SEARCH · Engineering Papers

Results for “Metadata management”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

SourceResearch connectionFrom yearThrough year

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 91 records · Page 5

All sources NASA NTRS DOE OSTI

DOE OSTI2020

Community-Driven Metadata Standards for Agricultural Microbiome Research

Accelerating the pace of microbiome science to enhance crop productivity and agroecosystem health will require transdisciplinary studies, comparisons among datasets, and synthetic analyses of research from diverse crop management contexts. However, despite the widespread availability of crop-associated microbiome data, variation in field sampling and laboratory processing methodologies, as well as metadata collection and reporting, significantly constrains the potential for integrative and comparative analyses. Here we discuss the need for agriculture-specific metadata standards for microbiome research, and propose a list of “required” and “desirable” metadata categories and ontologies essential to be included in a future minimum information metadata standards checklist for describing agricultural microbiome studies. We begin by briefly reviewing existing metadata standards relevant to agricultural microbiome research, and describe ongoing efforts to enhance the potential for integration of data across research studies. Our goal is not to delineate a fixed list of metadata requirements. Instead, we hope to advance the field by providing a starting point for discussion, and inspire researchers to adopt standardized procedures for collecting and reporting consistent and well-annotated metadata for agricultural microbiome research.

59 BASIC BIOLOGICAL SCIENCES↗

DOE OSTI2026

(U)Vendor Sanitization Requirements – SIEMENS Control

This document defines the software requirements for the sanitization of the IPC and or the controller. The system is intended to manage part programs and associated data (e.g. simulations, probing data) by capturing metadata, securely sanitizing files, logging operations, backing up data to a defined path, deploying a user defined program to a controller, and restoring part programs and associated data when required.

42 ENGINEERING↗

DOE OSTI2020

Introduction to Metadata and Ontologies: Everything You Always Wanted to Know About Metadata and Ontologies (But Were Afraid to Ask)

Metadata are contextual data about your experimental data. Metadata are the who, what, when, where, and why of these data. Metadata puts these data into context. In microbiome research, metadata includes information about the sample: when it was collected, where it was collected from, what kind of sample it is, and what were the properties of the environment or experimental condition from which the sample was taken. Information about sample processing is also metadata: methods used to extract and purify molecules (e.g., DNA) from the sample, type of DNA sequencing or other ’omics analyses done, and where the raw experimental data are located. This documentation includes information on the following topics: What are metadata? What kinds of metadata are there? Why are metadata important? Why should metadata be standardized? Why should metadata be machine readable? What is an ontology? Why are ontologies important? What are some examples of ontologies? The documentation also includes further resources on metadata guides and tools and tips for formatting metadata for maximum utility.

59 BASIC BIOLOGICAL SCIENCES↗

DOE OSTI2021

Waste Compliance and Tracking System (WCATS) Version 3 Requirements Document

This document describes the end-user requirements for the Waste Compliance and Tracking System (WCATS) project in accordance with the WCATS Software Quality Management Plan, EPC-WMP-WCATSPLAN-001. The WCATS application shall support the generation, characterization, processing, and shipment of LANL radioactive, hazardous, and industrial waste. Regulatory drivers include RCRA hazardous waste, DOT shipping, NNSA nuclear material control and accountability, DOE nuclear safety, TSDF permit, and transuranic waste certification requirements. The system will utilize a task-based architecture that supports the spectrum of treatment, storage, disposal, administrative, and characterization based unit operations necessary to manage waste from cradle to grave. The application design shall readily accommodate new facilities, processes, workflow, signature requirements, and so forth, via end-user established metadata. WCATS will provide support for representing waste storage and disposal facilities, buildings, rooms, and grid layouts (x, y, z) to support waste and radioactive material inventory management. Nuclear material at risk (MAR), DOE hazard rating (e.g., Category II facility) compliance per DOE-STD-1027, and permit inventory requirements will be configurable for any storage or disposal facility, or waste operation, and the system will automatically evaluate and enforce those requirements. In addition, the application will support the characterization and management of the entire range of hazardous and radioactive wastes (TRU, MTRU, LLW, MLLW, hazardous waste, etc.) that might be colocated or processed at a permitted facility. Some capabilities not found in traditional systems include user-defined tank systems for liquid waste, user-defined work paths (i.e., sequence of operations), and an equipment subsystem for tracking the calibration, maintenance, and inspection of tools used to process waste, such as torque wrenches, scales, pH probes, etc. The application incorporates a desktop and mobile user interface as shown in Figure 1. The mobile interface supports field operations, such as waste item characterization, intra-facility transfers, internal and external audits, and shipment preparation and receipt.

12 MANAGEMENT OF RADIOACTIVE AND NON-RADIOACTIVE W↗

DOE OSTI2020

Efficient Data Management in Neutron Scattering Data Reduction Workflows at ORNL

Oak Ridge National Laboratory (ORNL) experimental neutron science facilities produce 1.2 TB a day of raw event-based data that is stored using the standard metadata-rich NeXus schema built on top of the HDF5 file format. Performance of several data reduction workflows is largely determined by the amount of time spent on the loading and processing algorithms in Mantid, an open-source data analysis framework used across several neutron sciences facilities around the world. The present work introduces new data management algorithms to address identified input output (I/O) bottlenecks on Mantid. First, we introduce an in-memory binary-tree metadata index that resemble NeXus data access patterns to provide a scalable search and extraction mechanism. Second, data encapsulation in Mantid algorithms is optimally redesigned to reduce the total compute and memory runtime footprint associated with metadata I/O reconstruction tasks. Results from this work show speed ups in wall-clock time on ORNL data reduction workflows, ranging from 11% to 30% depending on the complexity of the targeted instrument-specific data. Nevertheless, we highlight the need for more research to address reduction challenges as experimental data volumes increase.

Godoy, William↗

DOE OSTI2023

RCSB Protein Data Bank: visualizing groups of experimentally determined PDB structures alongside computed structure models of proteins

Recent advances in Artificial Intelligence and Machine Learning (e.g., AlphaFold, RosettaFold, and ESMFold) enable prediction of three-dimensional (3D) protein structures from amino acid sequences alone at accuracies comparable to lower-resolution experimental methods. These tools have been employed to predict structures across entire proteomes and the results of large-scale metagenomic sequence studies, yielding an exponential increase in available biomolecular 3D structural information. Given the enormous volume of this newly computed biostructure data, there is an urgent need for robust tools to manage, search, cluster, and visualize large collections of structures. Equally important is the capability to efficiently summarize and visualize metadata, biological/biochemical annotations, and structural features, particularly when working with vast numbers of protein structures of both experimental origin from the Protein Data Bank (PDB) and computationally-predicted models. Moreover, researchers require advanced visualization techniques that support interactive exploration of multiple sequences and structural alignments. This paper introduces a suite of tools provided on the RCSB PDB research-focused web portal RCSB. org, tailor-made for efficient management, search, organization, and visualization of this burgeoning corpus of 3D macromolecular structure data.

3D visualization↗

DOE OSTI2023

EDX Spatial: Leveraging cloud and hybrid data management resources for spatial data

In the last few years, the National Energy Technology Laboratory has started leveraging cloud-hosted services for hosting spatial data collections published on the Energy Data eXchange (EDX). Using cloud-hosted storage and compute options offers multiple benefits for visualization and tool development through utilization of spatial data resources. The transition from on-premises services to cloud-hosted services has enabled a few key features: increased accessibility of large derivative datasets, dynamic integration of external authoritative data resources directly from outside entities through representational state transfer application programming interfaces (REST API), and the ability to produce complex mapping applications, dashboards, and online maps. As a result of the shift, NETL has launched EDX Spatial, a platform that leverages on-premises resources combined with cloud compute capabilities to enable enhanced online mapping interfaces and optimize data access. In addition, a unified workflow for handling the public release of spatial data products through EDX has been developed, including standardization of data hosting practices, metadata, symbology, and application elements. This poster reviews the opportunity of leveraging cloud-hosted and hybrid data management solutions for spatial data, and discusses the benefits and lessons learned while leveraging these services through the data repository EDX.

Morkner, Paige↗

DOE OSTI2022

gait

SAND2022-6991 O gait is an extension to zeek, an open-source network analyzer that adds metadata to the existing connection and SSL logs. Most of this metadata is already exposed by the zeek script API. Sandia National Laboratories is a multimission laboratory managed and operated by National Technology & Engineering Solutions of Sandia, LLC, a wholly owned subsidiary of Honeywell International Inc., for the U.S. Department of Energy’s National Nuclear Security Administration under contract DE-NA0003525.

Smutz, Charles↗

DOE OSTI2021

Metadata Schemas and Ontologies for Building Energy Applications: A Critical Review and Use Case Analysis

With the increasing digitalization of processes throughout the lifecycle of buildings, data exchanged between stakeholders and between building systems has grown significantly. However, a lack of semantic interoperability between data in different systems is still prevalent, hindering the development of applications that can be reused across buildings and limiting the scalability of innovative solutions. Semantics refers to the description of the meaning of the data in a way that can be consistently understood by applications. Recently, several competing initiatives have been developing metadata schemas and ontologies to express this semantic information for different applications in the building domain. This paper systematically reviews these schemas and conducts an analysis of five of them to evaluate their applicability to three high-value use cases for building operations: energy audits, automated fault detection and diagnostics and optimal control. The survey finds 40 schemas published in the last 10 years but but their actual use in industry is difficult to estimate. Among the five selected ontologies, several gaps are highlighted in relation to the three use cases. Recommendations for the future include better harmonization of these initiatives, more centralized repositories and search engines for these schemas as well as better industry engagement to facilitate their adoption.

Smart Building, Sematic, Metadata, Ontology, Data ↗

DOE OSTI2020

NREL HPC Eagle Jobs Data

3 months of Eagle jobs data including Slurm metadata and derived power use characterization.

96 KNOWLEDGE MANAGEMENT AND PRESERVATION↗

DOE OSTI2020

Transitioning CMS to Rucio Data Managment

Following a thorough review in 2018, the CMS experiment at the CERN LHC decided to adopt Rucio as its new data management system. Rucio is emerging as a community software project and will replace an aging CMSonly system before the start-up of LHC Run 3 in 2021. Rucio was chosen after an evaluation determined that Rucio could meet the technical and scale needs of CMS. The data management system for CMS needs to manage the current data sample of approximately 200 PB of data with 1 PB of transfers per day. The data management system must also have a development path suitable for LHC Run 4 (2026) data rates, which are expected to be 50 times larger.This contribution details the ongoing CMS adoption process as we replace our legacy system with Rucio, focusing on the challenges of integrating Rucio into an established set of experimental tools and procedures. This will include the migration of metadata, the construction of interfaces to the rest of the CMS computing tools, scale tests, operations, monitoring, and the plan to gradually turn over primary responsibility for the management of data to Rucio. A description of CMS user data management with Rucio will also be provided.

Vaandering, Eric↗

DOE OSTI2024

Spatialyze: A Geospatial Video Analytics System with Spatial-Aware Optimizations

Videos that are shot using commodity hardware such as phones and surveillance cameras record various metadata such as time and location. We encounter suchgeospatial videoson a daily basis and such videos have been growing in volume significantly. Yet, we do not have data management systems that allow users to interact with such data effectively. In this paper, we describe Spatialyze, a new framework for end-to-end querying of geospatial videos. Spatialyze comes with a domain-specific language where users can construct geospatial video analytic workflows using a 3-step, declarative,build-filter-observeparadigm. Internally, Spatialyze leverages the declarative nature of such workflows, the temporal-spatial metadata stored with videos, and physical behavior of real-world objects to optimize the execution of workflows. Our results using real-world videos and workflows show that Spatialyze can reduce execution time by up to 5.3×, while maintaining up to 97.1% accuracy compared to unoptimized execution.

Computer Science↗

DOE OSTI2026

infrastore [SWR-26-077]

Infrastore is time-series storage for energy-systems simulations, backed by HDF5 + SQLite, with Rust, Python, Julia, gRPC, and CLI bindings. It is a Rust library for managing time-series data in power-systems and energy simulations. Numerical arrays are persisted in HDF5, and the metadata associating each array with its owning component lives in SQLite. Identical arrays are stored once and shared through content addressing. It ships native Rust, Python (PyO3), and Julia (C ABI) interfaces, the infrastore command-line tool, and a read-only gRPC server with a Rust client. Documentation: https://natlabrockies.github.io/infrastore/latest/ — start with the Quick Start or the Architecture.

Thom, Daniel [National Laboratory of the Rockies (↗

DOE OSTI2025

Mic-hackathon 2024: hackathon on machine learning for electron and scanning probe microscopy

Microscopy is one of the primary sources of information on materials structure and functionality at the nanometer and atomic scales. The data generated through microscopy is often contained in well-structured datasets, enriched with extensive metadata and sample histories, although not always with the same level of detail or storage format. The broad incorporation of data management plans by major funding agencies ensures the preservation and accessibility of this data. However, deriving insights from these rich datasets remains challenging due to the lack of established code ecosystems, standardized benchmarks, and integration strategies. Correspondingly, the efficiency of data usage is very low, and time expenditures at the analysis stage are enormous. In addition to post-acquisition data analysis, the emergence of application programming interfaces by major microscope manufacturers now creates opportunities for real-time ML-based data analytics to enable automated decision making, and particularly ML-agent controlled real-time microscope operation. Despite these opportunities, there is a significant gap in integrating the ML community with the broader microscopy community, limiting the value that these methods bring to physics and materials discovery and materials optimization. Hackathons address these challenges by fostering collaboration between ML experts and microscopy professionals, encouraging the development of innovative solutions that leverage ML for microscopy and preparing the workforce of the future both for microscopy-intensive domains areas, instrument manufacturers, and ML scientists interested in real world applications for fundamental research, materials optimization, and manufacturing. The hackathon generated benchmark datasets and digital twins of microscopes that further contribute to the development of the field and establish data analysis ecosystems. All the codes can be found at GitHub(https://github.com/KalininGroup/Mic-hackathon-2024-codes-publication/tree/1.0.0.1) and Zenodo (https://zenodo.org/records/15579940).

97 MATHEMATICS AND COMPUTING↗

DOE OSTI2025

BULKI-Store v0.3.2

BULKI-Store is a distributed object storage system optimized for high-performance computing environments. Built with a Rust core and Python bindings, it efficiently manages scientific and machine learning datasets across HPC clusters. The system employs a client-server architecture with MPI integration, enabling seamless scaling on supercomputers like Perlmutter. BULKI-Store's object-oriented approach provides intuitive data organization with rich metadata support, contrasting with traditional file-based solutions. Key optimizations include selective checkpoint loading, unified checkpoint files, and object chunking for large data transfers. For machine learning workloads, BULKI-Store offers advantages through fine-grained access patterns, dynamic data sharing between training instances, and reduced memory pressure. Memory management features include strategic Python GC calls, minimized data copies, and batch processing capabilities. The system leverages Rayon's thread pool for asynchronous data prefetching and supports multiple CPU architectures (ARM64, x86, AMD, RISC-V). By combining performance optimizations with developer-friendly APIs, BULKI-Store addresses the complex data management challenges of modern HPC applications while maintaining compatibility across heterogeneous computing environments.

Zhang, Wei [Lawrence Berkeley National Laboratory ↗

DOE OSTI2025

mzPeak: Designing a Scalable, Interoperable, and Future-Ready Mass Spectrometry Data Format

Advances in mass spectrometry (MS) instrumentation, such as higher resolution, faster scan speeds, and improved sensitivity, have significantly increased the volume and complexity of data. The growing adoption of imaging and ion mobility further amplifies these challenges across MS-based omics fields, including proteomics, metabolomics, and lipidomics. While these technologies unlock new possibilities, they also present significant challenges in data management, storage, and accessibility. Existing open formats, such as the XML-based community standards mzML and imzML, struggle to meet the demands of modern MS workflows due to their large file sizes, slow data access, and limited metadata support. Vendor-specific formats, while optimized for proprietary instruments, lack interoperability, comprehensive metadata support and long-term archival reliability. This white paper lays the groundwork for mzPeak, a next-generation community data format designed to address these challenges and support high-throughput, multi-dimensional MS workflows. By adopting a hybrid model that combines efficient binary storage for numerical data and both human and machine-readable metadata storage, mzPeak will reduce file sizes, accelerate data access, and offer a scalable, adaptable solution for evolving MS technologies. For researchers, mzPeak will enable enhanced interoperability across platforms, seamless support for complex workflows including ion mobility and MS imaging, and faster data access compared to existing community formats such as mzML. Its design will ensure data is managed in compliance with regulatory standards, essential for applications such as precision medicine and chemical safety, where long-term data integrity and accessibility are critical. For vendors, mzPeak provides a streamlined, open alternative to proprietary formats, reducing the burden of regulatory compliance while aligning with the industry's push for transparency and standardization. By offering a high-performance, interoperable solution, mzPeak positions vendors to meet customer demands for sustainable data management tools which will be able to handle emerging and future data types and workflows. mzPeak aspires to become the cornerstone of MS data management, empowering researchers, vendors, and developers to innovate and collaborate more effectively.

data formats↗

DOE OSTI2021

The future low-temperature geochemical data-scape as envisioned by the U.S. geochemical community

Data sharing benefits the researcher, the scientific community, and the public by allowing the impact of data to be generalized beyond one project and by making science more transparent. However, many scientific communities have not developed protocols or standards for publishing, citing, and versioning datasets. One community that lags in data management is that of low-temperature geochemistry (LTG). This paper resulted from an initiative from 2018 through 2020 to convene LTG and data scientists in the U.S. to strategize future management of LTG data. Through webinars, a workshop, a preprint, a townhall, and a community survey, the group of U.S. scientists discussed the landscape of data management for LTG – the data-scape. Currently this data-scape includes a “street bazaar” of data repositories. This was deemed appropriate in the same way that LTG scientists publish articles in many journals. The variety of data repositories and journals reflect that LTG scientists target many different scientific questions, produce data with extremely different structures and volumes, and utilize copious and complex metadata. Nonetheless, the group agreed that publication of LTG science must be accompanied by sharing of data in publicly accessible repositories, and, for sample-based data, registration of samples with globally unique persistent identifiers. LTG scientists should use certified data repositories that are either highly structured databases designed for specialized types of data, or unstructured generalized data systems. Recognizing the need for tools to enable search and cross-referencing across the proliferating data repositories, the group proposed that the overall data informatics paradigm in LTG should shift from “build data repository, data will come” to “publish data online, cybertools will find”. Funding agencies could also provide portals for LTG scientists to register funded projects and datasets, and forge approaches that cross national boundaries. Finally, the needed transformation of the LTG data culture requires emphasis in student education on science and management of data.

58 GEOSCIENCES↗

DOE OSTI2023

A use-case-driven approach for demonstrating the added value of digitalisation in wind energy

Digitalisation is one of the key drivers for reducing the costs and risks of wind energy. When considering whether to embark on a digitalisation initiative, two key questions arise. The first is what business or operational opportunities might feasibly be addressed and the second is which of the many potential aspects of digitalisation are relevant to those opportunities. In this work, we show how these questions can be answered with a use-case-driven approach, based around a survey aiming to collect and collate the main "pain points" (or everyday challenges) of people in the wind energy sector. Although the relatively low number of participants of the survey (46) means that the results should only be used indicatively, it is still possible to make some general recommendations for priorities for digitalisation efforts in the wind energy sector. Firstly, digitalisation efforts should focus both on supporting people carrying out cross-lifecycle tasks, in particular sharing data, managing data, undertaking general data analyses and accessing data. Tools to do this should deal with varying data formats and naming conventions, make metadata more accessible, define data and metadata standards, make more data publicly available and improve the quality of data. Secondly, efforts should also focus on supporting people in the wind farm operational phase, in particular with failure detection, fault diagnosis, failure rate modelling and predictive maintenance. Solutions to do this should focus on accessible and validated tools for fault detection, cloud or other data pipeline solutions for SCADA data and tools for exhaustive data documentation. Finally, digitalisation efforts should focus on better communicating and helping people become aware of existing solutions and tools, as well as on helping people to exert a stronger influence on possible solutions.

17 WIND ENERGY↗