SEARCH · Engineering Papers

Results for “evaluation protocols”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

SourceResearch connectionFrom yearThrough year

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 37 records · Page 2

All sources NASA NTRS DOE OSTI

DOE OSTI2022

Durability Evaluation of Advanced Fenestration Technologies

In the U.S., more than 40% of primary energy and 70% of electricity is consumed in residential and commercial buildings, resulting in annual energy costs of more than $430 billion. Approximately 35% of this consumption can be attributed to losses through the building envelope, of which windows are currently the weakest link. Multiple technologies are under development to improve this performance. This includes dynamic and photovoltaic glazing as well as emerging highly insulating technologies including vacuum insulating glass (VIG), aerogels and thin-glass based multi-pane configurations. While windows are specified based on expected performance as installed, the energy savings realized by high performance windows are delivered over time. This makes it critical to understand and maximize the durability of high-performance windows to ensure those projected energy savings are delivered. Present methods for evaluating durability are based on existing technologies. These methods may not apply or be adequate for newer emerging technologies which often present novel failure mechanisms that need to be understood and evaluated differently. In this presentation, we will discuss our efforts to define appropriate methods to improve the evaluation of many existing technologies as well as evaluate the durability of emerging window technologies. This work results from the combination of an extensive review of various international standards as well as existing scientific literature. In addition, input was gathered from multiple industry stakeholders regarding present practice as well as feedback on proposed improvements to existing methods. Here we will share these findings and present proposed improvements to developing and existing durability evaluation protocols.

building envelope↗

DOE OSTI2022

Model Assumptions and Data Characteristics: Impacts on Domain Adaptation in Building Segmentation

Studies on domain adaptation (DA) for remote sensing (RS) imagery analysis lack consistency in selection and description of evaluation scenarios. Without properly characterizing datasets, model assumptions, and evaluation scenarios, it is difficult to objectively compare DA methods and reach conclusions about their suitability across different applications. With this motivation, this work seeks to empirically assess to which extent the interaction between data characteristics and model assumptions influences the effectiveness of DA methods. Using the widely explored task of building footprint segmentation as a case study, we perform a large-scale study across over 200 DA scenarios that include variations across view angles, areas observed, and sensors used for data acquisition. Rather than adopting different model architectures or optimization criteria, we contrast the performances of two DA methods based on adversarial learning that differ only in their assumptions about source and target domains. Informed by metadata and data characteristics unveiled using traditional computer vision (CV) techniques as well as pretrained deep models, we provide a detailed meta-analysis of experiments highlighting the importance of accurately considering data assumptions for DA in RS segmentation tasks. As demonstrated by a “cherry-picking” exercise, different claims regarding which model is best could be made by selecting different subsets of evaluation scenarios. While well-calibrated assumptions can be beneficial, mismatching assumptions can lead to negative biases in DA applications. Furthermore, this study intends to motivate the community toward more consistent evaluation protocols while providing recommendations and insights toward creating novel benchmark datasets, documenting data characteristics, application-specific knowledge, and model assumptions.

42 ENGINEERING↗

DOE OSTI2026

Scaling open-weight large language models for hydropower regulatory information extraction: A systematic analysis

Information extraction from regulatory and technical documents using large language models (LLMs) involves practical trade-offs between extraction quality and computational cost. We evaluate eight open-weight LLMs spanning 0.6B–70B parameters on hydropower licensing documents and report deployment-oriented evidence under a unified extraction schema and evaluation protocol. Across the model set, we observe clear scale-dependent trends in both baseline extraction quality and the effectiveness of reflective reasoning (self-checking) under our fixed-prompt, no-augmentation setting. Mid-scale models often provide a favorable balance of accuracy and efficiency, whereas the smallest models show limited or inconsistent gains from the reasoning variants tested. Larger models achieve the highest overall F1 scores but incur substantially greater compute and infrastructure requirements. We further find that reliability failure modes can distort conventional metrics in this domain: in particular, high recall can coincide with systematic extraction errors when models fabricate values for fields that are absent from the source text, underscoring the importance of conservative null handling and evidence-grounded evaluation. Overall, our study provides a reproducible resource–performance comparison for open-weight LLM-based extraction in hydropower regulatory documentation and offers practical guidance for model selection under different deployment constraints.

Evaluation protocol↗

DOE OSTI2025

Exponential Backoff and Its Security Implications for Safety-Critical OT Protocols over TCP/IP Networks

The convergence of Operational Technology (OT) and Information Technology (IT) networks has become increasingly prevalent with the growth of Industrial Internet of Things (IIoT) applications. This shift, while enabling enhanced automation, remote monitoring, and data sharing, also introduces new challenges related to communication latency and cybersecurity. Oftentimes, legacy OT protocols were adapted to the TCP/IP stack without an extensive review of the ramifications to their robustness, performance, or safety objectives. To further accommodate the IT/OT convergence, protocol gateways were introduced to facilitate the migration from serial protocols to TCP/IP protocol stacks within modern IT/OT infrastructure. However, they often introduce additional vulnerabilities by exposing traditionally isolated protocols to external threats. This study investigates the security and reliability implications of migrating serial protocols to TCP/IP stacks and the impact of protocol gateways, utilizing two widely used OT protocols: Modbus TCP and DNP3. Our protocol analysis finds a significant safety-critical vulnerability resulting from this migration, and our subsequent tests clearly demonstrate its presence and impact. A multi-tiered testbed, consisting of both physical and emulated components, is used to evaluate protocol performance and the effects of device-specific implementation flaws. Through this analysis of specifications and behaviors during communication interruptions, we identify critical differences in fault handling and the impact on time-sensitive data delivery. The findings highlight how reliance on lower-level IT protocols can undermine OT system resilience, and they inform the development of mitigation strategies to enhance the robustness of industrial communication networks.

DNP3↗

DOE OSTI2025

An Evaluation of Representation Learning Methods in Particle Physics Foundation Models

We present a systematic evaluation of representation learning objectives for particle physics within a unified framework. Our study employs a shared transformer-based particle-cloud encoder with standardized preprocessing, matched sampling, and a consistent evaluation protocol on a jet classification dataset. We compare contrastive (supervised and self-supervised), masked particle modeling, and generative reconstruction objectives under a common training regimen. In addition, we introduce targeted supervised architectural modifications that achieve state-of-the-art performance on benchmark evaluations. This controlled comparison isolates the contributions of the learning objective, highlights their respective strengths and limitations, and provides reproducible baselines. We position this work as a reference point for the future development of foundation models in particle physics, enabling more transparent and robust progress across the community.

Chen, Michael [Caltech]↗

DOE OSTI2026

Taming the Wild West: Assessing Impacts-Relevant Climate Data Products (Abbreviated Report)

Impacts-relevant Earth system data refers to observational and ESM data that are downscaled, debiased, validated, and provisioned for use by decision-makers. Impacts-relevant Earth system data is essential for mitigation and adaptation planning across a variety of regions and sectors. A vast number of these data products have emerged in recent years, which has led to confusion among stakeholders and scientists as to the best product to use. With no standard evaluation protocol available for these products, the decision on which product to use was sometimes made because it was pragmatic rather than the best product to use. This project sought to develop foundational capabilities around impacts-relevant data products that would support more informed selection and application of these products. This work has been immensely successful, driving several academic publications and supported the development of a community of practice around impacts-relevant data products. Over the project’s three years we have addressed six tasks: First, the development of standard evaluation metrics for impacts-relevant climate data; second, the development of a novel suite of atmospheric river metrics; third, the development of novel metrics for precipitation feature analysis; fourth, the development of novel metrics for assessing co-variances between temperature and precipitation; fifth, the development of a dashboard for interactive examination of impacts-relevant climate data; and sixth, the establishment of a community of practice around impacts-relevant climate data that will continue beyond the conclusion of this project.

54 ENVIRONMENTAL SCIENCES↗

DOE OSTI2026

Application of artificial intelligence methods in the international roughness index prediction of rigid and composite pavements: a systematic review

The International Roughness Index (IRI) is a widely adopted metric for quantifying pavement roughness, directly influencing vehicle safety, ride comfort, and overall roadway performance. In recent years, the use of Machine Learning (ML) models for IRI prediction has gained momentum, with the goal of improving the allocation of maintenance and rehabilitation resources by enabling accurate assessments of pavement conditions. Most prior reviews, however, have concentrated on flexible pavements, leaving a notable gap regarding rigid and composite pavements. To address this gap, the present study conducts a systematic review of Artificial Intelligence (AI) methods applied to IRI prediction for rigid and composite pavements. Literature published between 2004 and 2025 is synthesized to highlight prevailing trends, methodological contributions, and directions for future research. Particular attention is given to the types of models employed, the datasets used for training and validation, and the role of input variables and data-processing strategies. Across the included studies, ensemble learning methods (especially gradient boosting variants such as XGBoost), artificial neural networks, and hybrid architectures frequently achieved high predictive skill, with several models reporting test-set coefficients of determination approaching 0.9–0.96, indicating strong potential for capturing the influence of traffic, pavement structure, and climatic factors. Since these results are obtained from heterogeneous datasets and evaluation protocols, they are interpreted qualitatively rather than as strict cross-study rankings. Analysis of input variables revealed that pavement age and initial IRI were included in 91% (21 of 23) and 78% (18 of 23) of studies, respectively. Climatic variables such as the freezing index appeared in 57% (13 of 23), while traffic-related factors were considered in 65% (15 of 23). The findings underscore the importance of standardized, high-quality datasets, such as those from the Long-Term Pavement Performance (LTPP) program, along with data consistency, model interpretability, computational efficiency, and replicability in enhancing IRI prediction. Future research should focus on incorporating input variable selection techniques to identify the most influential predictors, thereby improving accuracy and robustness. Integrating these approaches with advanced non-linear data-driven models, coupled with robust hyperparameter optimization, holds considerable promise for strengthening the reliability of IRI prediction and supporting resilient pavement management strategies.

42 ENGINEERING↗

DOE OSTI2022

Model Validation Database for Fires Involving Fuels at Liquefied Natural Gas Facilities

This document provides a description of the model evaluation protocol (MEP) database for fires involving liquefied natural gas (LNG) and processing fuels at LNG facilities. The purpose of the MEP is to provide procedures regarding the assessment of a model's suitability to predict thermal exclusion zones resulting from a fire. The database includes measurements from pool fire, jet fire, and fireball experiments which are provided in a spreadsheet. Users are to enter model results into the spreadsheet which automatically generates statistical performance measures and graphical comparisons with the experimental data. The intent of this document is to provide a description of the experiments and of the procedure required to carry out the validation portion of the MEP. In addition, the statistical performance measures, measurements for comparisons, and parameter variation are provided.

03 NATURAL GAS↗

DOE OSTI2026

Model Validation Database for Fires Involving Fuels at Liquefied Natural Gas Facilities (Version 2)

This document provides a description of the model evaluation protocol (MEP) database for fires involving liquefied natural gas (LNG) and processing fuels at LNG facilities. The purpose of the MEP is to provide procedures regarding the assessment of a model’s suitability to predict thermal exclusion zones resulting from a fire. The database includes measurements from pool fire, jet fire, and fireball experiments which are provided in a spreadsheet. Users are to enter model results into the spreadsheet which automatically generates statistical performance measures and graphical comparisons with the experimental data. The intent of this document is to provide a description of the experiments and of the procedure required to carry out the validation portion of the MEP. In addition, the statistical performance measures, measurements for comparisons, and parameter variation are provided.

03 NATURAL GAS↗

DOE OSTI2026

Taming the Wild West: Assessing Impacts-Relevant Climate Data Products (Final Report)

54 ENVIRONMENTAL SCIENCES↗

DOE OSTI2023

Evaluating energy justice metrics in early-stage science and technology research using the JUST-R metrics framework

Embedding principles of energy justice throughout all aspects of clean energy technology research and development (R&D) can facilitate a more just energy transition; yet gaps remain in our understanding of how to best integrate energy justice from the earliest R&D stages. The Justice Underpinning Science and Technology Research (JUST-R) metrics framework has been developed to enable early-stage energy researchers to assess and address justice considerations associated with their research, but the impacts of the framework, and others like it, have yet to be evaluated. This study seeks to evaluate the JUST-R metrics framework in terms of its effectiveness and appeal to researchers engaged in early-stage technical R&D using qualitative analyses of documents and workshop transcripts. We find that the metrics framework helps researchers identify problems and potential solutions surrounding the energy justice implications of their work and spurs a change in perspective for researchers, but, simultaneously, there is no evidence of solution follow-through within the evaluation timeframe. Greater institutional support, specialization to research areas, knowledge of energy justice fundamentals, and earlier incorporation of energy justice considerations in research projects arise as factors needed to aid continued use of the framework and pursuit of identified solutions. This evaluation protocol and these findings can serve as a guide for improving other frameworks with similar goals of encouraging sociotechnical engagement in early-stage energy R&D.

29 ENERGY PLANNING, POLICY, AND ECONOMY↗

DOE OSTI2023

Evaluation of preservation protocols for oxygen-sensitive minerals within laminated aquatic sediments

Laminated sediments can record seasonal changes in sedimentation of material from anoxic waters, including minerals of the redox-sensitive elements Fe, Mn, and S that form under varying oxygen levels, mineral saturation conditions, and from microbial metabolism. However, preserving the oxygen-sensitive minerals for identification is challenging when preservation of the spatial arrangement of laminae is also required. In this study, we compare methods for embedding sedimentary materials from anoxic waters and sediments from Brownie Lake, Minnesota, USA for analysis of the speciation for Fe, Mn, and S using synchrotron-based X-ray absorption near edge spectroscopy (XANES). We found that acetone dehydration and resin replacement in a 100% N 2 glovebox successfully preserved the speciation of Fe and Mn minerals within laminated sediments. However, acetone removed some sulfur species from sediments, and epoxies contained sulfur species, which challenged identification of native sulfur species. Results from this study will aid researchers who are interested in spatial analysis of oxygen sensitive sediments, soils, or microbial mats in choosing a preservation method.

58 GEOSCIENCES↗

DOE OSTI2024

MatFold: Cross-validation Protocols to Systematically Evaluate Generalization Errors in Materials Discovery Models

Explore the source record for details and available documents.

Schindler, Peter [Northeastern University]↗

DOE OSTI2020

dCache – Efficient Message Encoding For Inter-Service Communication in dCache: Evaluation of Existing Serialization Protocols as a Replacement for Java Object Serialization

As a well established, large-scale distributed storage system, dCache is used to manage and serve huge amounts of data collected by high energy physics, astrophysics and photon science experiments. Based on a microservices-like architecture, dCache is built as a modular distributed system, where each component provides a different core functionality. These services communicate by passing serialized messages to each other, a core behavior whose performance properties can consequently affect the entire system.This paper compares and evaluates different data serialization protocols in computer science with the objective of replacing and improving upon Java Object Serialization (JOS), which has increasingly presented itself as no longer being sufficiently performant for encoding messages. The criteria for choosing a new framework are collected, analyzed and formalized. The primary motivation for replacing Java serialization for encoding dCache messages is increasing the general speed of message-passing and thereby reducing the round-trip time for user requests. Emphasis is also placed on schema evolution capabilities and framework usability. Approaches for generalizing (de)serialization speed and size measurements based on data structure complexity are introduced, criteria for measuring documentation, learning curve, maintainability and introduction effort are defined. Finally, several selected serialization protocols are evaluated and compared accordingly, concluding with a recommendation for a suitable JOS replacement.

Morschel, Lea↗

DOE OSTI2021

Achieving Runtime State Verification Assurance in Critical Cyber-Physical Infrastructures

Industrial Cyber-Physical Systems (ICPS) are an essential backbone of national critical infrastructures. They help monitor and control crucial cyber-enabled services such as energy generation. Commonly ICPS monitors the physical process through Supervisory Control and Data Acquisition (SCADA) systems. The SCADA ecosystem takes critical real-time and future system operational decisions based on the runtime state behavior of field sensors. Traditional SCADA systems use legacy and insecure communication protocols such as the Modbus protocol that lack adequate security mechanisms to provide robust runtime state behavior assurance of constrained field sensors. Therefore, constrained field sensors are commonly vulnerable to standard semantic attacks that gradually change the behavior state of infected devices. This paper discusses process integrity assurance techniques necessary to enhance the security of behavior-based protocols such as the Modbus protocol. The Runtime State Verification (RSV) protocol proposed in this paper aims to address semantic attacks in the SCADA ecosystem by integrating behavior-based Mandatory Results Automata (MRA) and a Hyperledger Fabric (HLF) network. The RSV protocol provides high process integrity assurance through enhanced behavior-based MRA suitable for the constrained field devices. A proof of concept of the RSV protocol has been evaluated in an emulated water-tube boiler. Preliminary evaluations of the RSV protocol aimed to measure the efficiency of the proposed protocol by monitoring an Combustion Efficiency (CE) process necessary to preserve optimal combustion, thus minimizing costs and future maintenance of water-tube boilers. We analyze the overall network overhead and latency of the proposed RSV protocol by evaluating the HLF network performance and comparing the proposed RSV protocol with the state-ofart BloSPAI protocol. Through the preliminary evaluations of the proposed RSV protocol, this paper demonstrates that the proposed RSV protocol overcomes the shortcomings and network overhead of the BloSPAI protocol by integrating behavior-based authentication through novel MRAs and HLF networks.

Rivera, Abel Gomez↗

DOE OSTI2024

Experimental Characterization Test of a Grid-Forming Inverter for Microgrid Applications

Standardized experimental testing protocols for grid forming (GFM) inverters to ensure expected operation under both normal and contingency conditions do not exist. Such protocols increase the confidence of system owner/operators that an inverter deployed in a proposed system will engage in typical behaviors to ensure interoperability with other units and ancillary equipment (e.g. protection equipment). This paper presents systematic and comprehensive test protocols to evaluate the performance of GFM inverters under the following operational configurations: islanded operation, heterogeneous islanded operation (parallel with a synchronous generator), grid-connected operation, and transition operation. A commercial GFM inverter is used to verify the test protocols and to understand the inverter's performance and functionalities. In particular, required configuration and tuning of the inverter will be explained in the full paper to enrich the testing protocol.

black start↗

DOE OSTI2023

High throughput structure–property relationship for additively manufactured 316L/IN625 alloy mixtures leveraging 2-step Bayesian estimation

While the fabrication of graded materials by directed energy deposition (DED) has led to accelerated materials discovery, the ability to rapidly explore sufficiently large material composition spaces is limited due to the time-intensive nature of conventional materials characterization techniques. The present study investigates the viability of small punch test (SPT) protocols for rapidly evaluating DED-fabricated alloy mixtures of stainless steel 316L (316L) and Inconel 625 (IN625). The SPT protocols evaluated in this study include both the recently established two-step Bayesian estimation framework as well as the empirical relationships established in prior literature. It is shown that these protocols are capable of reliably and quantitatively tracking the changes in the mechanical properties of the alloy mixtures studied. Enhancement of mechanical properties was observed with the addition of IN625 to 316L, which is attributed to the austenite stabilization in the matrix and the formation of fine δ - Ni3Nb precipitates. It is shown that CALPHAD-based Scheil model simulations predicted the formation of different precipitate phases for each composition. The novel protocols presented in this paper open new avenues for high throughput material explorations for additive manufacturing.

2-step Bayesian estimation↗

DOE OSTI2021

Evaluation of DNA extraction protocols from liquid-based cytology specimens for studying cervical microbiota

Cervical microbiota (CM) are considered an important factor affecting the progression of cervical intraepithelial neoplasia (CIN) and are implicated in the persistence of human papillomavirus (HPV). Collection of liquid-based cytology (LBC) samples is routine for cervical cancer screening and HPV genotyping and can be used for long-term cytological biobanking. We sought to determine whether it is possible to access microbial DNA from LBC specimens, and compared the performance of four different extraction protocols: (ZymoBIOMICS DNA Miniprep Kit; QIAamp PowerFecal Pro DNA Kit; QIAamp DNA Mini Kit; and IndiSpin Pathogen Kit) and their ability to capture the diversity of CM from LBC specimens. LBC specimens from 20 patients (stored for 716 ± 105 days) with CIN values of 2 or 3 were each aliquoted for each of the four kits. Loss of microbial diversity due to long-term LBC storage could not be assessed due to lack of fresh LBC samples. Comparisons with other types of cervical sampling were not performed. We observed that all DNA extraction kits provided equivalent accessibility to the cervical microbial DNA within stored LBC samples. Approximately 80% microbial genera were shared among all DNA extraction protocols. Potential kit contaminants were observed as well. Variation between individuals was a significantly greater influence on the observed microbial composition than was the method of DNA extraction. We also observed that HPV16 was significantly associated with community types that were not dominated by Lactobacillus iners.

59 BASIC BIOLOGICAL SCIENCES↗