SEARCH · Engineering Papers

Results for “metadata evaluation”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

SourceResearch connectionFrom yearThrough year

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 19 records

All sources NASA NTRS DOE OSTI

DOE OSTI2018

A large-scale evaluation of automated metadata inference approaches on sensors from air handling units

Building automation systems provide abundant sensor data to enable the potential of using data analytics to, among other things, improve the energy efficiency of the building. However, deployment of these applications for buildings, such as, fault detection and diagnosis (FDD) on multiple buildings remains a challenge due to the non-trivial efforts of organizing, managing and extracting metadata associated with sensors (e.g., information about their location, function, etc.), which is required by applications. One of the reasons leading to the problem is that varying conventions, acronyms, and standards are used to define this metadata. To better understand the nature of the problem, as well as the performance and scalability of existing solutions, in this report we implement and test 6 different time-series based metadata inference approaches on sensors from 614 air handling units (AHU) instrumented in 35 building sites accounting for more than 400 buildings distributed across United States of America. We infer 12 types of sensors and actuators in AHUs required by a rule-based FDD application: AHU performance and assessment rules (APAR). Our results show that: (1) the average performance of these approaches in terms of accuracy is similar across building sites, though there is significant variance; (2) the expected accuracy of classifying the type of points required by APAR for a new unseen building is, on average, 75%; (3) the performance of the model does not decrease as long as training data and testing data are extracted from adjacent months.

42 ENGINEERING↗

DOE OSTI2018

Advancing our predictive understanding of river corridor exchange

Despite decades of research, we lack an accurate framework to predict and manage hydrologic exchange in the river corridor and the associated ecosystem services and functions at the scales of stream reaches and entire networks. While many individual studies have been conducted to investigate specific mechanisms, they have not been synthesized to account for heterogeneity in space, nonstationarity, and multiscale feedbacks that typify the river corridor. As a result, contradictory predictions of exchange flux, geometry, and timescale are prevalent in the literature. We attribute these contradictions to (a) failure to account for multiscale feedbacks and (b) uncertainty in the information content of common measurement methods. Here, we apply the concept of interacting multiscale spatial and time‐variable domains to the river corridor to demonstrate why more complete multiscale characterization is necessary to achieve predictive understanding. Next, we highlight uncertainties and inconsistencies in measurement methods which may obfuscate our understanding of river corridor exchange. Finally, we briefly outline four necessary advances to achieve predictive understanding of river corridor exchange: standardization of metadata, critical evaluation of the information content and support volumes for measurement techniques, multiscale model‐data integration, and advancing theoretical models of river corridor exchange. This article is categorized under: Science of Water > Hydrological Processes Water and Life > Nature of Freshwater Ecosystems Science of Water > Methods

Ward, Adam S.↗

DOE OSTI2026

Envelope-driven comfort risk in residential demand response

Residential demand response (DR) is a valuable resource for grid reliability, but remains challenging because the highly heterogeneous residential building stock leads to widely varying and hard-to-predict load and comfort responses during DR events. Although prior research has estimated the technical potential of DR-capable technologies for achieving energy demand savings, little is known about how they affect thermal comfort. In particular, it remains unclear how indoor thermal conditions due to DR depend on the thermal envelope characteristics of the housing stock. To address this gap, this study provides a systematic, location-specific assessment of indoor thermal performance during DR-events across the US housing stock using both typical DR weather data and detailed building metadata. We evaluate how envelope characteristics influence indoor temperatures during realistic simulated summer and winter DR events across 37 US locations, applying both temperature threshold and rate of temperature change criteria to estimate region-level probabilities of discomfort. Additionally, we show the impact of distinct weather patterns that intensify or abate thermal stress on comfort outcomes. Results show a near-universal overheating risk in summer DR events, where comfort outcomes are strongly influenced by rapid risk of comfort violations. In contrast, overall winter DR discomfort risk is lower, risk escalation is more gradual and shows greater sensitivity to event duration. These findings offer a data-driven quantification of comfort risk across diverse climates and building envelopes, demonstrating the need for region-specific DR scheduling and discomfort mitigation strategies tailored to local weather patterns and the performance of existing residential buildings.

Demand response↗

DOE OSTI2019

Systems and methods for generating slice files from native CAD geometries

A system and method for improving the production of objects with fully defined CAD models by generating CAD slice files from the native CAD geometries for use by AM machines. An electronic processing element receives the CAD model having a native and/or neutral format with metadata, determines a desired resolution, slices the CAD model to create the slice files having a slice format (e.g., point, edge, surface, volume) and retaining the metadata, and evaluates the slice files to determine whether the resolution has been achieved. If so, the slice files are sent to the AM machine. If not, a new desired resolution is determined and the CAD model is re-sliced, which may include combining the slice files to regenerate the CAD model. A support structure for the object may be analyzed, and if distortion is predicted, a modified CAD model may be created and sliced to create modified slice files.

Vernon, Gregory J.↗

DOE OSTI2024

Quantifying Error in Photovoltaic Installation Metadata: Preprint

In this research, we quantify the level of metadata error for a fleet of 2860 photovoltaic (PV) systems, using metadata values provided by fleet owners. Using satellite imagery and time series analysis techniques available in open-source Python packages Panel-Segmentation and PVAnalytics, respectively, we evaluate the accuracy of PV system metadata such as location, azimuth, tilt, and mounting configuration (fixed tilt vs. tracking). We find that approximately 75% of provided latitude-longitude coordinates are within 190 meters of the actual solar installation. We were unable to link 7.8% of latitude-longitude coordinates to any solar installation via satellite imagery analysis. We evaluate the level of error in owner-provided mounting configuration (fixed tilt vs. single-axis tracking), finding only 8 systems with an incorrect mounting configuration. When evaluating azimuth and tilt parameters, we find that approximately 64% of the data is correct, with data for 860 systems (approximately 30%) not provided by system owners. To illustrate the importance of having correct solar metadata, we evaluate how incorrect metadata affects solar performance estimates by modeling system AC energy output at ground-truth vs. incorrect latitude-longitude coordinates, mounting configurations, and azimuth-tilt configurations. Energy output estimates can vary significantly if incorrect metadata parameters are used, with incorrect mounting configuration leading to the largest discrepancy with over 20% variation in expected energy output.

azimuth↗

DOE OSTI2013

Scalable PGAS Metadata Management on Extreme Scale Systems

Programming models intended to run on exascale systems have a number of challenges to overcome, specially the sheer size of the system as measured by the number of concurrent software entities created and managed by the underlying runtime. It is clear from the size of these systems that any state maintained by the programming model has to be strictly sub-linear in size, in order not to overwhelm memory usage with pure overhead. A principal feature of Partitioned Global Address Space (PGAS) models is providing easy access to global-view distributed data structures. In order to provide efficient access to these distributed data structures, PGAS models must keep track of metadata such as where array sections are located with respect to processes/threads running on the HPC system. As PGAS models and applications become ubiquitous on very large transpetascale systems, a key component to their performance and scalability will be efficient and judicious use of memory for model overhead (metadata) compared to application data. We present an evaluation of several strategies to manage PGAS metadata that exhibit different space/time tradeoffs. We use two real-world PGAS applications to capture metadata usage patterns and gain insight into their communication behavior.

Global Arrays↗

DOE OSTI2026

SetGo: Metadata Readiness for Scientific AI Datasets

Scientific datasets intended for AI use require both computational readiness for model training and metadata readiness for discovery, sharing, and reuse. The Readiness Engine for Data Integration (REDI) addresses computational readiness, but no corresponding tool evaluates whether a dataset’s metadata are sufficiently complete, governed, and standards-compliant for publication and agent-based consumption. Existing FAIR assessors operate only on published repository records, and no single system covers FAIR compliance, licensing, provenance, governance, reproducibility, and catalog readiness together. We present SetGo, an open-source Python toolkit that assesses and repairs metadata readiness across these six dimensions before a dataset is published or archived. Applied to four scientific corpora, SetGo surfaces deficiencies that general-purpose tools do not detect: ERA5 climate metadata scores 4% on ACDD 1.3 compliance; materials datasets fail OPTIMADE species-definition requirements; and PDB-derived proteomics data carries licensing terms incompatible with standard SPDX identifiers. Guided enrichment raises overall FAIR scores from 52–57% to 81–91%, and a single setgo publish command pushes to Hugging Face Hub, CKAN, or OpenMetadata with ML Commons Croissant 1.0 metadata sidecars. To support interactive and automated workflows, SetGo integrates with coding agents powered by large language models (LLMs) through a /setgo skill that enables natural-language execution of the full assess–enrich–publish loop, with user involvement limited to supplying missing metadata values.

Wilkinson, Sean [ORNL] (ORCID:0000000214437479)↗

DOE OSTI2024

Prompt Phrase Ordering Using Large Language Models in HPC: Evaluating Prompt Sensitivity

Large language models (LLMs) have demonstrated effective performance in domain-specific tasks, often requiring a well-designed prompt to guide their responses. However, optimizing the right prompt is challenging due to prompt sensitivity—the phenomenon where small changes in the prompt can lead to significant variations in performance. In this study, we evaluate prompt performance by examining all permutations of independent phrases to investigate prompt sensitivity and robustness. We used two datasets: the GSM8k dataset, which assesses mathematical reasoning, and a custom template prompt for summarizing database metadata. Our goal was to evaluate the performance across all permutations of a sequence of prompt phrases. The study was conducted using the llama3-instruct- 7B model hosted on Ollama, with computations parallelized in a high-performance computing environment. By comparing the average index of phrases in the best and worst-performing prompts, we found that the order of independent phrases within a prompt significantly impacts LLM performance. Additionally, we used Hamming distance to assess changes between phrase orderings, concluding that prompt modifications can dramatically affect scores, often by almost random chance. These findings support existing research on prompt sensitivity. We discuss the challenges of prompt optimization, noting that altering phrases in a successful prompt does not always result in another successful prompt.

97 MATHEMATICS AND COMPUTING↗

DOE OSTI2018

Managing Rich Metadata in High-Performance Computing Systems Using a Graph Model

High-performance computing (HPC) systems generate huge amounts of metadata about different entities such as jobs, users, and files. Existing systems can efficiently record and manage part of these metadata, mainly the POSIX metadata of data files (e.g., file size, name, and permissions mode). But another important set of metadata, referred to as "rich" metadata in this study, which record not only wider range of entities (e.g., running processes and jobs) but also more complex relationships between them, are mostly missing in current HPC systems. Yet such rich metadata are critical for supporting many advanced data management functions such as identifying data sources and parameters behind a given result; auditing data usage; or understanding details about how inputs are transformed into outputs. To uniformly and efficiently manage the rich metadata generated in HPC systems, We propose to utilize a graph model in this study. We identify the key challenges of implementing such a graph-based HPC rich metadata management system and present GraphMeta, a graph-based rich metadata management system designed and optimized for HPC platforms, to tackle these challenges. In conclusion, extensive evaluations on both synthetic and real HPC metadata workloads show its advantages in both performance and scalability compared with existing solutions.

97 MATHEMATICS AND COMPUTING↗

DOE OSTI2026

AI-Ready Data Pilot Project Report

The proliferation of artificial intelligence in scientific research has created an urgent need to define "AI-ready data" for researchers and, more importantly, provide resources to help them produce AI-ready data. At Pacific Northwest National Laboratory, we conducted a pilot study with three data scientists evaluating three CSV datasets from different scientific domains, followed by semi-structured interviews capturing assessment practices. Our findings reveal that AI-readiness evaluation is intuition-based, with practitioners asking "How fast can I go from raw data to my machine learning pipeline?" Data scientists consistently prioritized workflow efficiency, human interpretability, and quality stewardship signals. From these insights, we developed a practical evaluation framework comprising data requirements, metadata standards, and validation tests that provides actionable criteria for producing and curating AI-ready datasets, addressing the gap between theoretical understanding and practical implementation.

97 MATHEMATICS AND COMPUTING↗

DOE OSTI2021

MOFLUX Intensified Soil Moisture Extremes Decrease Soil Organic Carbon Decomposition: Modeling Archive

This Modeling Archive is in support the publication “Intensified Soil Moisture Extremes Decrease Soil Organic Carbon Decomposition: A Mechanistic Modeling Analysis” (Liang et al., 2021). Here we provide model code, inputs, outputs and evaluation datasets for the Microbial ENzyme Decomposition (MEND) model for the Missouri Ozarks AmeriFlux eddy covariance measurement site (MOFLUX) near Ashland, Missouri USA. The MEND model was developed with explicit representation of microbial and enzyme pools to mechanistically simulate the role of microbial organisms and extracellular enzymes in soil organic carbon (SOC) decomposition. Long-term SOC dynamics under intensified moisture extremes are studied using the MEND model that is parameterized with 11 years of measurements from the MOFLUX forest. The model explicitly represents microbial dormancy and resuscitation, different types of SOC-degrading enzymes, and how they vary with changes in soil moisture (Wang et al. 2015, 2019). A combination of two levels of frequency and severity of soil moisture, as well as a control with normal interannual variability, are used to simulate a range of moisture scenarios over 100 years. The code of Microbial-ENzyme Decomposition (MEND) as well as the input and output data are included in the archive. A user’s manual (MEND_Readme.pdf) is included with instructions for compiling and running the model to simulate soil organic carbon decomposition under various moisture scenarios. This dataset contains the modelling archive contained within a compressed (*.zip) file, a file-level metadata file in comma separate (*.csv) format, and two instructional files in PDF (*.pdf) format.

54 ENVIRONMENTAL SCIENCES↗

DOE OSTI2022

Machine Learning for Automated Metadata Assignment in Buildings: Cooperative Research and Development (Final Report, CRADA Number CRD-18-00767)

RealTerm Energy and NREL have identified a shared vision to evaluate opportunities to facilitate the organization and assignment of metadata to building control system (BCS) data via industry-informed machine learning (ML). Manual metadata assignment is labor intensive and costly, slowing down any Energy Management and Information System (EMIS) deployment in the building space. This project aims to develop methodologies to accurately assign this metadata and significantly decrease the level of effort associated with deploying EMIS. The objective of this project is to identify/design methodologies to assign metadata to HVAC control points automatically. The identified methodologies will be programmed in analytics algorithms so they can ingest a list of points and produce a detailed tagging following the Haystack classification nomenclature. To validate the efficacy of each methodology, tagging results will be compared utilizing a list of points extracted from RealTerm's building database-as well as data extracted from the NREL campus via the Intelligent Campus program-enabling testing against large datasets with real world challenges. The developed methodologies may leverage building manager/operator input on a limited basis to add context to the classifying algorithms. The partnership aims to advance global efforts in areas related to the DOE missions through improving operational performance of commercial buildings. It is well documented that buildings fall out of commission after they are occupied, wasting significant energy and incurring associated costs simply due to poor operational performance. Emerging EMIS technologies that perform continuous commissioning help to address this issue, yet integration of these systems can be labor intensive both for the technology vendor and the building owner/operator. This project will enable more efficient and cost-effective analytics for buildings, enabling improvement in building operations at lower cost points.

29 ENERGY PLANNING, POLICY, AND ECONOMY↗

DOE OSTI2020

MIQS v0.6

In this research, we propose a novel Metadata Indexing and Querying for Self-describing formats (MIQS), which removes the external DBMS and utilizes in-memory index to achieve efficient metadata searching. MIQS follows the self-contained data management paradigm and provides portable and schema-free metadata indexing and querying functionalities for self-describing file formats. We have evaluated MIQS with the state-of-the-art MongoDB-based metadata indexing solution. MIQS achieved up to 99% time reduction in index construction and up to 172,000X search performance improvement with up to 75% reduction in memory footprint.

Byna, Suren↗

DOE OSTI2020

MIQS v0.6

Byna, Suren↗

DOE OSTI2011

Object-Oriented Database for Managing Building Modeling Components and Metadata: Preprint

Building simulation enables users to explore and evaluate multiple building designs. When tools for optimization, parametrics, and uncertainty analysis are combined with analysis engines, the sheer number of discrete simulation datasets makes it difficult to keep track of the inputs. The integrity of the input data is critical to designers, engineers, and researchers for code compliance,validation, and building commissioning long after the simulations are finished. This paper discusses an application that stores inputs needed for building energy modeling in a searchable, indexable, flexible, and scalable database to help address the problem of managing simulation input data.

32 ENERGY CONSERVATION, CONSUMPTION, AND UTILIZATI↗

DOE OSTI2016

GraphMeta: Managing HPC Rich Metadata in Graphs

High-performance computing (HPC) systems face increasingly critical metadata management challenges, especially in the approaching exascale era. These challenges arise not only from exploding metadata volumes, but also from increasingly diverse metadata, which contains data provenance and arbitrary user-defined attributes in addition to traditional POSIX metadata. This ‘rich’ metadata is becoming critical to supporting advanced data management functionality such as data auditing and validation. In our prior work, we identified a graph-based model as a promising solution to uniformly manage HPC rich metadata due to its flexibility and generality. However, at the same time, graph-based HPC rich metadata anagement also introduces significant challenges to the underlying infrastructure. In this study, we first identify the challenges on the underlying infrastructure to support scalable, high-performance rich metadata management. Based on that, we introduce GraphMeta, a graphbased engine designed for this use case. It achieves performance scalability by introducing a new graph partitioning algorithm and a write-optimal storage engine. We evaluate GraphMeta under both synthetic and real HPC metadata workloads, compare it with other approaches, and demonstrate its advantages in terms of efficiency and usability for rich metadata management in HPC systems.

Dai, Dong↗

DOE OSTI2024

Analytical methods for online data quality assessment

This chapter provides a comprehensive overview of the main steps for algorithmic sensor signal quality assessment, which can enhance the decision-making process for water resource recovery facility (WRRF) operation and optimization. It introduces the concept of redundancy as the basis for data quality assessment. It also explains the typical data processing pipeline, which consists of preliminary analysis, data pre-processing, and specific algorithmic approaches. Each of these processes is presented and discussed in three separate sections. Importantly, this chapter introduces the main approaches for data quality assessment, provides guidelines for selecting the most suitable one and the key performance indicators to evaluate them and explains how to collect metadata through such an algorithmic approach.

Aguado, Daniel↗

DOE OSTI2022

PDFDataExtractor: A Tool for Reading Scientific Text and Interpreting Metadata from the Typeset Literature in the Portable Document Format

The layout of portable document format (PDF) files is constant to any screen, and the metadata therein are latent, compared to mark-up languages such as HTML and XML. No semantic tags are usually provided, and a PDF file is not designed to be edited or its data interpreted by software. However, data held in PDF files need to be extracted in order to comply with opensource data requirements that are now government-regulated. In the chemical domain, related chemical and property data also need to be found, and their correlations need to be exploited to enable data science in areas such as data-driven materials discovery. Such relationships may be realized using text-mining software such as the “chemistry-aware” natural-language-processing tool, ChemDataExtractor; however, this tool has limited data-extraction capabilities from PDF files. This study presents the PDFDataExtractor tool, which can act as a plug-in to ChemDataExtractor. It outperforms other PDF-extraction tools for the chemical literature by coupling its functionalities to the chemical-named entityrecognition capabilities of ChemDataExtractor. The intrinsic PDF-reading abilities of ChemDataExtractor are much improved. The system features a template-based architecture. This enables semantic information to be extracted from the PDF files of scientific articles in order to reconstruct the logical structure of articles. While other existing PDF-extracting tools focus on quantity mining, this template-based system is more focused on quality mining on different layouts. PDFDataExtractor outputs information in JSON and plain text, including the metadata of a PDF file, such as paper title, authors, affiliation, email, abstract, keywords, journal, year, document object identifier (DOI), reference, and issue number. With a self-created evaluation article set, PDFDataExtractor achieved promising precision for all key assessed metadata areas of the document text.

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗