Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “Metadata management”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 19 records
All sourcesNASA NTRSDOE OSTI
DOE OSTI2023

Microbiome Metadata Management

Tremendous amounts of microbiome omics data have been generated in recent years, which holds great potential for large-scale studies and metaanalyses. The utility of these datasets is unfortunately limited by their lack of searchability and interoperability. Thorough, standardized metadata is essential for unlocking the scientific discovery potential of disparate datasets. Several national and international initiatives exist to gather and standardize microbiome omics data to improve accessibility and reusability. Continuing to improve consistency in metadata will allow advances in automation and reduction in manual mapping between databases.

59 BASIC BIOLOGICAL SCIENCES↗
DOE OSTI2021

MetaCat - metadata catalog for data management systems

Metadata management is one of three major areas and parts of functionality of scientific data management along with replica management and workflow management. Metadata is the information describing the data stored in a data item, a file or an object. It includes the data item provenance, recording conditions, format and other attributes. MetaCat is a metadata management database designed and developed for High Energy Physics experiments. As a component of a data management system, it’s main objectives are to provide efficient metadata storage and management and fast data items selection functionality. MetaCat is supposed to work on the scale of 100 million files (or objects) and beyond. The article will discuss the functionality of MetaCat and technological solutions used to implement the product.

Mandrichenko, Igor↗
DOE OSTI2024

Towards FAIR Workflows for Federated Experimental Sciences

A de-centralized, peer-to-peer AI metadata framework is demonstrated which can enable end-to-end metadata & lineage tracking for distributed Machine Learning pipelines spanning edge, High Performance Computing, and cloud environments. With a specific example of end-to-end microscopy algorithm and datasets, the proposed method shows how to enable reproducibility, audit trail, provenance of metadata artifacts. The emerging needs of automation in experimental sciences, ML-centric workflows, and FAIR metadata management across federated compute environments is addressed.

machine learning↗
DOE OSTI2022

Best Practices for Energy Management Information Systems Metadata Schemas

The Federal Energy Management Program (FEMP) promotes best practices for impactful utilization of Energy Management Information Systems (EMIS) at federal facilities. This best practice document is part of a series of fact sheets created to help accelerate the market adoption and use of EMIS in the federal sector. The use of standard naming conventions and a metadata schema, which may be referred to as 'data tags,' improves the ability of the EMIS to consistently analyze, visualize, and derive value from operational data. To help address these issues, this fact sheet highlights best practices for use of metadata in an EMIS stack, which can significantly reduce the initial time required to deploy an EMIS.

EMIS↗
DOE OSTI2024

Scaling Ensembles of Data-Intensive Quantum Chemical Calculations for Millions of Molecules

Deep learning models are efficient computational tools that can accelerate the inverse design of molecules with desired functional properties by generating predictions at a fraction of the time required by traditional quantum chemical approaches. To ensure that a model maintains accuracy and transferability across broad regions of the chemical space explored during the inverse design, it must be trained on massively large volumes of simulation data. This requires running large-scale ensemble quantum chemical calculations on high-performance computing (HPC) systems for data collection. However, the efficient execution of such large ensemble calculations and the management of large volumes of output data require tools that can judiciously utilize computational resources and manage metadata overhead on the file system. Therefore, we present a high-performance, scalable, ensemble management framework for performing data-intensive quantum chemical electronic structure calculations for organic molecules. This framework provides abstractions to plug different ab initio, first principles, and first principles-based semi-empirical methods and executes them efficiently at large scale on HPC systems. It dynamically distributes tasks to resources and uses tiered storage for managing large collections of files. We employed this framework to process over ten million organic molecules and generate open-source datasets that provide UV-vis absorption spectra by running time-dependent density-functional tight-binding calculations. It is the largest database containing molecular optical spectra that were simulated with quantum chemical methods in a consistent manner.

Mehta, Kshitij↗
DOE OSTI2024

Centralized Interactive Phenomics Resource: an integrated online phenomics knowledgebase for health data users

Development of clinical phenotypes from electronic health records (EHRs) can be resource intensive. Several phenotype libraries have been created to facilitate reuse of definitions. However, these platforms vary in target audience and utility. Here, we describe the development of the Centralized Interactive Phenomics Resource (CIPHER) knowledgebase, a comprehensive public-facing phenotype library, which aims to facilitate clinical and health services research. The platform was designed to collect and catalog EHR-based computable phenotype algorithms from any healthcare system, scale metadata management, facilitate phenotype discovery, and allow for integration of tools and user workflows. Phenomics experts were engaged in the development and testing of the site. The knowledgebase stores phenotype metadata using the CIPHER standard, and definitions are accessible through complex searching. Phenotypes are contributed to the knowledgebase via webform, allowing metadata validation. Data visualization tools linking to the knowledgebase enhance user interaction with content and accelerate phenotype development. The CIPHER knowledgebase was developed in the largest healthcare system in the United States and piloted with external partners. The design of the CIPHER website supports a variety of front-end tools and features to facilitate phenotype development and reuse. Health data users are encouraged to contribute their algorithms to the knowledgebase for wider dissemination to the research community, and to use the platform as a springboard for phenotyping. CIPHER is a public resource for all health data users available at https://phenomics.va.ornl.gov/ which facilitates phenotype reuse, development, and dissemination of phenotyping knowledge.

60 APPLIED LIFE SCIENCES↗
DOE OSTI2022

Position Papers for the ASCR Workshop on the Management and Storage of Scientific Data

The purpose of this workshop is to identify priority research directions in the area of data management for high-performance and scientific computing above and beyond HPC’s traditional "the parallel file system is the data-management system" model. Supporting the breadth of the DOE mission, including the explosion of AI uses and the growing needs of experimental and observational science, motivates revisiting our assumptions about data management. There are many facets of this topic to explore including: (1) Interfaces for accessing data that resides on traditional persistent storage as well as memory devices; (2) Storage-system architecture design that supports scientific workflows on varied hierarchical storage and networking devices; (3) Devising metadata management infrastructure to support FAIR principles (Findability, Accessibility, Interoperability, and Reusability); (4) Capturing provenance information about scientific data; (5) Utilizing AI to learn I/O patterns of emerging workloads for efficient data management; (6) Providing data management support for AI and complex workflows; and (7) Understanding the overlap between traditional storage systems and I/O (SSIO) efforts and data management. While the program committee has identified these topics as important areas for discussion, we welcome position papers from the community that propose additional topics of interest for discussion at the workshop. The workshop agenda will include breakout sessions for discussing these and selected topic areas to inform priority research directions for data management for high-performance and scientific computing.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2022

Report for the ASCR Workshop on the Management and Storage of Scientific Data

The purpose of this workshop is to identify priority research directions in the area of data management for high-performance and scientific computing above and beyond HPC’s traditional "the parallel file system is the data-management system" model. Supporting the breadth of the DOE mission, including the explosion of AI uses and the growing needs of experimental and observational science, motivates revisiting our assumptions about data management. There are many facets of this topic to explore including: (1) Interfaces for accessing data that resides on traditional persistent storage as well as memory devices; (2) Storage-system architecture design that supports scientific workflows on varied hierarchical storage and networking devices; (3) Devising metadata management infrastructure to support FAIR principles (Findability, Accessibility, Interoperability, and Reusability); (4) Capturing provenance information about scientific data; (5) Utilizing AI to learn I/O patterns of emerging workloads for efficient data management; (6) Providing data management support for AI and complex workflows; and (7) Understanding the overlap between traditional storage systems and I/O (SSIO) efforts and data management. While the program committee has identified these topics as important areas for discussion, we welcome position papers from the community that propose additional topics of interest for discussion at the workshop. The workshop agenda will include breakout sessions for discussing these and selected topic areas to inform priority research directions for data management for high-performance and scientific computing.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2020

Performance Improvements on SNS and HFIR Instrument Data Reduction Workflows Using Mantid

Performance of data reduction workflows at the High Flux Isotope Reactor (HFIR) and the Spallation Neutron Source (SNS) at Oak Ridge National Laboratory (ORNL) is mainly determined by the time spent loading raw measurement events stored in large and sparse datasets. This paper describes: (1) our long-term view to leverage SNS and HFIR data management needs with our experience at ORNL’s world-class high performance computing (HPC) facilities, and (2) our short-term efforts to speed up current workflows using Mantid, a data analysis and reduction community framework used across several neutron scattering facilities. We show that minimally invasive short-term improvements in metadata management have a moderate impact in speeding up current production workflows. We propose a more disruptive domain-specific solution: the No Cost Input Output (NCIO) framework, we provide an overview, the risks and challenges in NCIO’s adoption by HFIR and SNS stakeholders.

Godoy, William↗
DOE OSTI2022

Best Practices for Energy Management Information Systems Metadata Schemas

Fact sheet highlights best practices for metadata schemas and standard naming conventions, which improve the ability of an energy management information system (EMIS) to consistently analyze, visualize, and derive value from operational data. This best practice document is part of a series of fact sheets created to help accelerate the market adoption and use of EMIS in the federal sector. It provides an overview of best practices for using metadata in an EMIS stack, which can significantly reduce the initial time required to deploy an EMIS.

data tags↗
DOE OSTI2021

Sample Identifiers and Metadata to Support Data Management and Reuse in Multidisciplinary Ecosystem Sciences

Physical samples are foundational entities for research across biological, Earth, and environmental sciences. Data generated from sample-based analyses are not only the basis of individual studies, but can also be integrated with other data to answer new and broader-scale questions. Ecosystem studies increasingly rely on multidisciplinary team-science to study climate and environmental changes. While there are widely adopted conventions within certain domains to describe sample data, these have gaps when applied in a multidisciplinary context. In this study, we reviewed existing practices for identifying, characterizing, and linking related environmental samples. We then tested practicalities of assigning persistent identifiers to samples, with standardized metadata, in a pilot field test involving eight United States Department of Energy projects. Participants collected a variety of sample types, with analyses conducted across multiple facilities. We address terminology gaps for multidisciplinary research and make recommendations for assigning identifiers and metadata that supports sample tracking, integration, and reuse. Furthermore, our goal is to provide a practical approach to sample management, geared towards ecosystem scientists who contribute and reuse sample data.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2026

Metadata Standards for the NSE: Extended Field Standards

This standard presents a set of optional metadata fields for managed digital objects within the Nuclear Security Enterprise (NSE) and provides a deeper look at data representation in metadata by looking at the representation of 1) Records Management required metadata, and 2) common representations of technical/scientific data. Metadata standardization is a critical enabler for effectively sharing data, documents, and other digital objects between NSE sites, and for tracing the digital thread at the object level. Standardization is necessary for both schemas and vocabularies, meaning that both field standards and value standards must be specified. This document serves as a complementary field standard, recommending an optional set of fields that should be uniformly built for all managed digital objects within the NSE. This document specifically focuses on extending the shared discovery layer defined in the first white paper by introducing additional descriptive and data representation fields that improve cross-site search and interpretation.

96 KNOWLEDGE MANAGEMENT AND PRESERVATION↗
DOE OSTI2021

Enabling modern data discovery for atmospheric measurements

The Atmospheric Radiation Measurement (ARM) user facility is a US Department of Energy Office of Science user facility that is managed and operated through a collaborative effort led by nine US Department of Energy national laboratories. The ARM Data Center, located at Oak Ridge National Laboratory, is responsible for the timely collection, processing, and delivery of data products to the scientific community. The ARM Data Center holds more than 11,000 data products, including metadata collected from field campaigns, instruments, value-added products, and principal investigator–contributed data. These data sets are checked for successful transfer (for most data, this transfer is carried out automatically via the network; however, some of the largest data sets and some of the most remote sites require manual shipping of hard disks) and both the data and metadata are processed to a standard format, which is an ARM-standardized structure, via the Network Common Data Form. The Network Common Data Form is a self-describing binary format with many compatible software tools. Once processed, the data are cataloged, stored in the ARM Data Archive, and made discoverable through association with an array of metadata-characterizing information, such as location and measurement classification. These metadata enable powerful search capabilities through the ARM Data Center Data Discovery interface. This paper discusses the workflow of how the new discovery system has been redesigned from user requirements and how the data are distributed to the scientific community.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2020

Persistent Memory Object Storage and Indexing for Scientific Computing

This paper presents Mosiqs, a persistent memory object storage framework with metadata indexing and querying for scientific computing. We design Mosiqs based on the key idea that memory objects on shared PM pool can live beyond the application lifetime and can become the sharing currency for applications and scientists. Mosiqs provides an aggregate memory pool atop an array of persistent memory devices to store and access memory objects. Mosiqs uses a lightweight persistent memory key-value store to manage the metadata of memory objects such as persistent pointer mappings, which enables memory object sharing for effective scientific collaborations. Mosiqs is implemented atop PMDK. We evaluate the proposed approach on many-core server with an array of real PM devices. The preliminary evaluation confirms a 100% improvement for write and 30% in read performance against a PM-aware file system approach.

Khan, Awais↗
DOE OSTI2021

MOSIQS: Persistent Memory Object Storage With Metadata Indexing and Querying for Scientific Computing

Scientific applications often require high-bandwidth shared storage to perform joint simulations and collaborative data analytics. Shared memory pools provide a chance to satisfy such needs. Recently, a high-speed network such as Gen-Z utilizing persistent memory (PM) offers an opportunity to create a shared memory pool connected to compute nodes. However, there are several challenges to use scientific applications on the shared memory pool directly such as scalability, failure-atomicity, and lack of scientific metadata-based search and query. In this paper, we propose MOSIQS, a persistent memory object storage framework with metadata indexing and querying for scientific computing. We design MOSIQS based on the key idea that memory objects on PM pool can live beyond the application lifetime and can become the sharing currency for applications and scientists. MOSIQS provides an aggregate memory pool atop an array of persistent memory devices to store and access memory objects to accelerate scientific computing. MOSIQS uses a lightweight persistent memory key-value store to manage the metadata of memory objects, which enables memory object sharing. To facilitate metadata search and query over millions of memory objects resident on memory pool, we introduce Group Split and Merge (GSM), a novel persistent index data structure designed primarily for scientific datasets. GSM splits and merges dynamically to minimize the query search space and maintains low query processing time while overcoming the index storage overhead. MOSIQS is implemented on top of PMDK. We evaluate the proposed approach on many-core server with an array of real PM devices. Experimental results show that MOSIQS gains a 100% write performance improvement and executes multi-attribute queries efficiently with 2.7× less index storage overhead offering significant potential to speed up scientific computing applications.

97 MATHEMATICS AND COMPUTING↗