SEARCH · Engineering Papers

Results for “Metadata”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

SourceResearch connectionFrom yearThrough year

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 1,117 records · Page 62

All sources NASA NTRS DOE OSTI

NASA NTRS2021

Developing a Sustainable, User-Friendly Literature Database to Support the Microgravity Simulation Support Facility (MSSF) at NASA's Kennedy Space Center (KSC)

Established in 2017, the Microgravity Simulation Support Facility (MSSF) at NASA’s Kennedy Space Center is the only centralized, dedicated facility supporting ground microgravity research in the United States. The MSSF offers the research community the ability to conduct simulated microgravity research with experimental conditions functionally resembling those aboard the International Space Station (ISS) and in other flight-based experimental environments. Since its inception, the MSSF has supported numerous studies and has since collected an extensive library of relevant and pertinent literature. The goal of our research was to develop and implement a sustainable, user-friendly literature database to better house this literature at the MSSF. To achieve this, our team focused on sorting, optimizing, and analyzing preexisting literature libraries to determine a best suitable and sustainable platform for the MSSF. After establishing initial database platforms, the team worked to develop descriptive and structural metadata categories to best sort the literature, which was followed by rigorous testing and optimization of the database as it was implemented. The MSSF has now been outfitted with a reliable, accessible database that effectively houses literature and provides diverse analysis to the user. Our team is continuing to test and update our platform and parameters as we aim for the formal implementation, expansion, and evolution of our database to better sustain future research ventures at the MSSF and beyond.

Database Development↗

NASA NTRSDate not supplied

Stewardship Best Practices for Improved Discovery and Reuse of Heterogeneous and Cross-Disciplinary Earth System Data

Some of the Earth system data products such as those from NASA airborne and field investigations (a.k.a. campaigns), are highly heterogeneous and cross-disciplinary, making the data extremely challenging to manage. For example, airborne and field campaign measurements tend to be sporadic over a period of time, with large gaps. Data products generated are of various processing levels and utilized for a wide range of inter- and cross-disciplinary research and applications. Data and derived products have been historically stored in a variety of domain-specific standard (and some non-standard) formats and in various locations such as NASA Distributed Active Archive Centers (DAACs), NASA airborne science facilities, field archives, or even individual scientists’ computer hard drives. As a result, airborne and field campaign data products have often been managed and represented differently, making it onerous for data users to find, access, and utilize campaign data. Some difficulties in discovering and accessing the campaign data originate from the incomplete data product and contextual metadata that may contain details relevant to the campaign (e.g. campaign acronym and instrument deployment locations), but tend to lack other significant information needed to understand conditions surrounding the data. Such details can be burdensome to locate after the conclusion of a campaign. Utilizing consistent terminology, essential for improved discovery and reuse, is also challenging due to the variety of involved disciplines. To help address the aforementioned challenges faced by many repositories and data managers handling airborne and field data, this presentation will describe stewardship practices developed by the Airborne Data Management Group (ADMG) within the Interagency Implementation and Advanced Concepts Team (IMPACT) under the NASA’s Earth Science Data systems (ESDS) Program.

best practices↗

NASA NTRSDate not supplied

NASA Life Sciences Portal (NLSP): Supporting Scientific Transparency and Reproducibility

NASA’s Life Sciences Ports (NLSP) serves the scientific community by providing curated data from space life science experiment. The Human Research Program (HRP) with the help of NLSP is currently transforming their life sciences data archive systems and processes to improve compliance with the FAIR principles [1]. Some of these improvements will at the same time support the twin pillars of Open Science [2]: transparency of methods and reproducibility of results. Scientific transparency is marked by the easily intelligible communication of what has been investigated: what were the procedures for collecting sample and the characteristics of samples collected? what kinds of measurements were made, what were the environmental conditions of the measurements? What were the analysis techniques of the collected data? Reproducibility of the results and findings from the investigation requires a high level of transparency for all but the simplest investigations; the slightest deviation in communicating and replicating complex experimental procedures or data analyses can often yield quite different data and even findings, thwarting their validation. One of the ways the NLSP is aiming to improve the communication of scientific information is through the use of ontology-driven metadata. Ontologies are powerful, graph-based knowledge representation structures, which can be leveraged to increase data interoperability, the area of the FAIR principles in which many data systems most lack compliance. Over the past decade, there has been a concerted effort in the biomedical community to develop modular and narrowly focused domain and application-specific ontologies in a common, open-source framework, the Open Biological and Biomedical Ontology (OBO) Foundry [3]. The open sharing and modular nature of this effort promises huge increases in harmonized data sharing for systems that leverage these models. Which is in line with the FAIR Data Principles of Findability, Accessibility, Interoperability, and Reuse for scientific data management and stewardship. 1. Wilkinson, M.D., et al., The FAIR Guiding Principles for scientific data management and stewardship. Sci Data, 2016. 3: p. 160018. 2. National Academies of Sciences, E. and Medicine, Open Science by Design: Realizing a Vision for 21st Century Research. 2018, Washington, DC: The National Academies Press. 232. 3. Smith, B., et al., The OBO Foundry: coordinated evolution of ontologies to support biomedical data integration. Nat Biotechnol, 2007. 25(11): p. 1251-5.

Life Sciences data↗

NASA NTRSDate not supplied

NASA Life Sciences Portal (NLSP): Supporting Scientific Transparency and Reproducibility

Life Sciences data↗

DOE OSTI2015

DOE Geothermal Data Repository: Getting More Mileage Out of Your Data: Preprint

All data submitted to the U.S. Department of Energy's Geothermal Data Repository (GDR) is eventually made public. The metadata for these data submissions is searchable in multiple data catalogs, including the GDR catalog and the data catalog on OpenEI.org. Because it is a node on the National Geothermal Data System (NGDS), all data on the GDR are also discoverable through both the regular Identifier (DOI), and as a byproduct of this assignment, these submissions are automatically registered in the Office of Science and Technical Information (OSTI) DataCite catalog. From there, these data are federated to additional sites both domestic and international, including Science.gov and WorldWideScience.org. This paper will explore in detail the wide reach of data submitted to the GDR from and how this exposure can dramatically increase the utility of submitted data.

15 GEOTHERMAL ENERGY↗

DOE OSTI2017

HodDB: Design and Analysis of a Query Processor for Brick.

Brick is a recently proposed metadata schema and ontology for describing building components and the relationships between them. It represents buildings as directed labeled graphs using the RDF data model. Using the SPARQL query language, building-agnostic applications query a Brick graph to discover the set of resources and relationships they require to operate. Latency-sensitive applications, such as user interfaces, demand response and modelpredictive control, require fast queries — conventionally less than 100ms. We benchmark a set of popular open-source and commercial SPARQL databases against three real Brick models using seven application queries and find that none of them meet this performance target. This lack of performance can be attributed to design decisions that optimize for queries over large graphs consisting of billions of triples, but give poor spatial locality and join performance on the small dense graphs typical of Brick. We present the design and evaluation of HodDB, a RDF/SPARQL database for Brick built over a node-based index structure. HodDB performs Brick queries 3-700x faster than leading SPARQL databases and consistently meets the 100ms threshold, enabling the portability of important latency-sensitive building applications.

32 ENERGY CONSERVATION, CONSUMPTION, AND UTILIZATI↗

DOE OSTI2025

Lessons Learned from AskGDR: Usage and Impact Analysis of the Geothermal Data Repository's AI Research Assistant: Preprint

In October of 2024, the Department of Energy's (DOE) Geothermal Data Repository (GDR) team officially launched AskGDR, an AI research assistant resulting from the integration of a Large Language Model (LLM) with the metadata and supporting documents associated with GDR datasets. AskGDR allows GDR users to ask deeper questions about the origin of datasets, the methods used to collect them, and the findings they help support. Using Retrieval Augmented Generation (RAG), AskGDR can be used to summarize findings spread across dozens of papers and technical reports or to extract relevant information describing a single data field. However, generative AI is experimental. The National Renewable Energy Laboratory (NREL) has been collecting metrics on AskGDR and documenting lessons learned during its deployment. This paper will outline the efficacy and impact of AskGDR through analysis of its use, operating costs, number and types of questions asked, and the quality of answers provided.

15 GEOTHERMAL ENERGY↗

DOE OSTI2015

Load‐balanced and locality‐aware scheduling for data‐intensive workloads at extreme scales

Summary Data‐driven programming models such as many‐task computing (MTC) have been prevalent for running data‐intensive scientific applications. MTC applies over‐decomposition to enable distributed scheduling. To achieve extreme scalability, MTC proposes a fully distributed task scheduling architecture that employs as many schedulers as the compute nodes to make scheduling decisions. Achieving distributed load balancing and best exploiting data locality are two important goals for the best performance of distributed scheduling of data‐intensive applications. Our previous research proposed a data‐aware work‐stealing technique to optimize both load balancing and data locality by using both dedicated and shared task ready queues in each scheduler. Tasks were organized in queues based on the input data size and location. Distributed key‐value store was applied to manage task metadata. We implemented the technique in MATRIX, a distributed MTC task execution framework. In this work, we devise an analytical suboptimal upper bound of the proposed technique, compare MATRIX with other scheduling systems, and explore the scalability of the technique at extreme scales. Results show that the technique is not only scalable but can achieve performance within 15% of the suboptimal solution. Copyright © 2015 John Wiley & Sons, Ltd.

Wang, Ke↗

DOE OSTI2022

Evaluation of a scientific data search infrastructure

The ability to search over large scientific datasets has become crucial to next-generation scientific discoveries as data generated from scientific facilities grow dramatically. In previous work, we developed and deployed ScienceSearch, a search infrastructure for scientific data which uses machine learning to automate metadata creation. Our current deployment is deployed atop a container based platform at a HPC center. In this article, we present an evaluation and discuss our experiences with the ScienceSearch infrastructure. Specifically, we present a performance evaluation of ScienceSearch's infrastructure focusing on scalability trends. The obtained results show that ScienceSearch is able to serve up to 130 queries/min with latency under 3 s. We discuss our infrastructure setup and evaluation results to provide our experiences and a perspective on opportunities and challenges of our search infrastructure.

96 KNOWLEDGE MANAGEMENT AND PRESERVATION↗

DOE OSTI2018

FishGen.net: An Online Genetic Repository for Salmon and Steelhead Genetic Baselines

Abstract FishGen is a final repository for Pacific salmon Oncorhynchus spp. and steelhead O. mykiss genetic data generated as part of the genetic stock identification and parentage-based tagging projects in the Columbia River basin and throughout the Pacific Coast of North America. Resource Data, Inc., developed this web-based, GIS-interfaced software, which is freely available to the public, with funding from the Pacific Coastal Salmon Recovery Fund and Bonneville Power Administration. FishGen currently houses genetic stock identification baselines for both Chinook Salmon O. tshawytscha and steelhead in the Columbia and Snake river basins, as well as hatchery, parentage-based, tagging baselines for both species in the Snake River basin. Because it has a user-friendly interface and protocol for submitting and storing standardized genetic and sample metadata, it is an excellent tool for supporting genetic research and monitoring projects throughout the region.

McCane, Jesse↗

DOE OSTI2013

Instrumental temperature series in eastern and central China back to the nineteenth century

In this study, we bring together different source data sets and use quality control, interpolation, and homogeneity methods to construct a set of homogenized monthly mean surface air temperature (SAT) series for 18 stations in eastern and central China from the late nineteenth century. Missing values are statistically interpolated, and cross validation is used to assess the accuracy of the interpolation approaches. Results show that the errors of interpolation are small, and the interpolated values are statistically acceptable. Multiple homogeneity methods and all available metadata are used to assess the consistency of the time series and then to develop adjustments when necessary. Thirty-three homogeneity breakpoints are detected in the 18 stations, and the time series is adjusted to the latest segment of the data series. The adjusted annual mean SAT generally shows a range of trends of 1.0° to 4.2°C/100 years in northeastern and southeastern China and a range of trends of -0.3° to 1.0°C/100 years in central China near 30°N. Compared to the adjusted time series, the unadjusted time series underestimates the warming trend during the past 100 years. The regional and annual mean SAT over eastern and central China agrees well with estimates from a much denser station network over this region of China since 1951 and shows a warming trend of 1.52°C/100 years during 1909–2010.

54 ENVIRONMENTAL SCIENCES↗

DOE OSTI2026

The Zooplankton International Geospatial dataset: A global repository of spatiotemporal freshwater zooplankton community composition data from lakes and reservoirs to support ecological research

Zooplankton transfer substantial energy in aquatic food webs and are used as indicators of environmental change. Syntheses of zooplankton community dynamics globally require datasets that span a wide range of environmental gradients; however, these datasets are limited due to methodological differences across programs, taxonomic inconsistencies, and a lack of standardized metadata. To reconcile these challenges, we created the Zooplankton International Geospatial (ZIG) dataset, which includes original zooplankton, water physical and chemical variables, and lake morphometric data from 311 inland lakes and reservoirs. ZIG includes waterbodies ranging in size from 0.005 to 82,100 km2 and spanning broad latitudinal (−47.26 to 64.90) and longitudinal ranges (−165.04 to 176.53). Temporal coverage for individual waterbodies ranges between 1 and 60 yr with sampling frequency ranging from annually to weekly. With its extensive coverage and content, we consider ZIG to be a cornerstone for future investigations of global scale lake biodiversity change.

Figary, Stephanie [Cornell University, Ithaca, NY]↗

DOE OSTI2018

Advancing our predictive understanding of river corridor exchange

Despite decades of research, we lack an accurate framework to predict and manage hydrologic exchange in the river corridor and the associated ecosystem services and functions at the scales of stream reaches and entire networks. While many individual studies have been conducted to investigate specific mechanisms, they have not been synthesized to account for heterogeneity in space, nonstationarity, and multiscale feedbacks that typify the river corridor. As a result, contradictory predictions of exchange flux, geometry, and timescale are prevalent in the literature. We attribute these contradictions to (a) failure to account for multiscale feedbacks and (b) uncertainty in the information content of common measurement methods. Here, we apply the concept of interacting multiscale spatial and time‐variable domains to the river corridor to demonstrate why more complete multiscale characterization is necessary to achieve predictive understanding. Next, we highlight uncertainties and inconsistencies in measurement methods which may obfuscate our understanding of river corridor exchange. Finally, we briefly outline four necessary advances to achieve predictive understanding of river corridor exchange: standardization of metadata, critical evaluation of the information content and support volumes for measurement techniques, multiscale model‐data integration, and advancing theoretical models of river corridor exchange. This article is categorized under: Science of Water > Hydrological Processes Water and Life > Nature of Freshwater Ecosystems Science of Water > Methods

Ward, Adam S.↗

DOE OSTI2019

A Quantitative Approach to Architecting All-Flash Lustre File Systems

New experimental and AI-driven workloads are moving into the realm of extreme-scale HPC systems at the same time that high-performance flash is becoming cost-effective to deploy at scale. This confluence poses a number of new technical and economic challenges and opportunities in designing the next generation of HPC storage and I/O subsystems to achieve the right balance of bandwidth, latency, endurance, and cost. In this work, we present quantitative models that use workload data from existing, disk-based file systems to project the architectural requirements of all-flash Lustre file systems. Using data from NERSC’s Cori I/O subsystem, we then demonstrate the minimum required capacity for data, capacity for metadata and data-on-MDT, and SSD endurance for a future all-flash Lustre file system.

Lockwood, GK↗

DOE OSTI2020

Performance Improvements on SNS and HFIR Instrument Data Reduction Workflows Using Mantid

Performance of data reduction workflows at the High Flux Isotope Reactor (HFIR) and the Spallation Neutron Source (SNS) at Oak Ridge National Laboratory (ORNL) is mainly determined by the time spent loading raw measurement events stored in large and sparse datasets. This paper describes: (1) our long-term view to leverage SNS and HFIR data management needs with our experience at ORNL’s world-class high performance computing (HPC) facilities, and (2) our short-term efforts to speed up current workflows using Mantid, a data analysis and reduction community framework used across several neutron scattering facilities. We show that minimally invasive short-term improvements in metadata management have a moderate impact in speeding up current production workflows. We propose a more disruptive domain-specific solution: the No Cost Input Output (NCIO) framework, we provide an overview, the risks and challenges in NCIO’s adoption by HFIR and SNS stakeholders.

Godoy, William↗

DOE OSTI2023

Data Center Facility Monitoring with Physics Aware Approach

U.S. Department of Energy's National Renewable Energy Laboratory (NREL) hosts one of the world's most energy-efficient HPC data centers; this system uses component-level warm-water liquid cooling to efficiently remove heat from the data center and capture it for reuse in the building or rejection to the atmosphere. Given the complexity of this system, building data-driven tools for holistically monitoring and operating the entire data center is a priority for ensuring maximal efficiency and resiliency. In this advanced smart facility, over one million metrics are recorded per minute using state-of-the-art streaming data architecture and software to capture and process the state of the system in real time. Here we detail two efforts to effectively analyze, visualize, and interpret this large volume streaming data. We have developed a novel, flexible system for identifying and visualizing individual metric anomalies and component performance across the data center through automatic metadata extraction and physically-motivated visualization for quick interpretation. Additionally, to directly connect system maintenance to data stream processing we explore a physics informed multi-metric drift and anomaly detection application to detect scale-build up in heat exchangers.

anomaly detection↗

DOE OSTI2014

RADAR: Runtime Asymmetric Data-access Driven Scientific Data Replication

Efficient I/O on large-scale spatiotemporal scientific data requires scrutiny of both the logical layout of the data (e.g., row-major vs. column-major) and the physical layout (e.g., distribution on parallel filesystems). For increasingly complex datasets, hand optimization is a difficult matter prone to error and not scalable to the increasing heterogeneity of analysis workloads. Given these factors, we present a partial data replication system called RADAR. We capture datatype- and collective-aware I/O access patterns (indicating logical access) via MPI-IO tracing and use a combination of coarse-grained and fine-grained performance modeling to evaluate and select optimized physical data distributions for the task at hand. Unlike conventional methods, we store all replica data and metadata, along with the original untouched data, under a single file container using the object abstraction in parallel filesystems. Our system results in manyfold improvements in some commonly used subvolume decomposition access patterns.Moreover, the modeling approach can determine whether such optimizations should be undertaken in the first place.

access pattern↗

DOE OSTI2013

Analytics-Driven Lossless Data Compression for Rapid In-situ Indexing, Storing, and Querying

The analysis of scientific simulations is highly data-intensive and is becoming an increasingly important challenge. Peta-scale data sets require the use of light-weight query-driven analysis methods, as opposed to heavy-weight schemes that optimize for speed at the expense of size. This paper is an attempt in the direction of query processing over losslessly compressed scientific data. We propose a co-designed double-precision compression and indexing methodology for range queries by performing unique-value-based binning on the most significant bytes of double precision data (sign, exponent, and most significant mantissa bits), and inverting the resulting metadata to produce an inverted index over a reduced data representation. Without the inverted index, our method matches or improves compression ratios over both general-purpose and floating-point compression utilities. The inverted index is light-weight, and the overall storage requirement for both reduced column and index is less than 135%, whereas existing DBMS technologies can require 200-400%. As a proof-of-concept, we evaluate univariate range queries that additionally return column values, a critical component of data analytics, against state-of-the-art bitmap indexing technology, showing multi-fold query performance improvements.

Jenkins, John↗