SEARCH · Engineering Papers

Results for “evaluation protocols”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

SourceResearch connectionFrom yearThrough year

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 37 records · Page 2

All sources NASA NTRS DOE OSTI

DOE OSTI2024

Toward Wireless Smart Grid Communications: An Evaluation of Protocol Latencies in an Open-Source 5G Testbed

Fifth-generation networks promise wide availability of wireless communication with inherent security features. The 5G standards also outline access for different applications requiring low latency, machine-to-machine communication, or mobile broadband. These networks can be advantageous to numerous applications that require widespread and diverse communications. One such application is found in smart grids. Smart grid networks, and Operational Technology (OT) networks in general, utilize a variety of communication protocols for low-latency control, data monitoring, and reporting at every level. Transitioning these network communications from wired Wide Area Networks (WANs) to wireless communication through 5G can provide additional benefits to their security and network configurability. However, introducing these wireless capabilities may also result in a degradation of network latency. In this paper, we propose utilizing 5G for smart grid communications, and we evaluate the latency impacts of encapsulating GOOSE, Modbus, and DNP3 for transmission over a 5G network. The OpenAirInterface open-source library is utilized to deploy an in-lab 5G Core Network and gNB for testing with off-the-shelf User Equipment (UE). This creates an effective 5G test platform for experimenting with different OT protocols such as GOOSE. The results are validated by measuring two different Intelligent Electronic Devices’ contact closure times for each network configuration. These tests are also conducted for varying packet sizes in order to isolate different sources of network latency. Our study outlines the latency impact of communication over 5G for time-critical and non-critical applications regarding their transition toward private 5G-based OT network implementations. The conducted experiments illustrate that in the case of GOOSE packets, simple encapsulation may exceed the protocol’s time-critical nature, and, therefore, additional measures must be taken to ensure a viable transition of GOOSE to 5G services. However, non-critical applications are shown to be viable for migration to 5G.

42 ENGINEERING↗

DOE OSTI2021

Analysis of Ion-Exchanged ZSM-5, BEA, and SSZ-13 Zeolite Trapping Materials under Realistic Exhaust Conditions

An industry-defined evaluation protocol was used to evaluate the hydrocarbon trapping (HCT) and passive NOx adsorption (PNA) potential for BEA, ZSM-5, and SSZ-13 zeolites with ion-exchanged Pd or Ag. All materials underwent 700 °C degreening prior to exposure to an industry-derived protocol gas stream, which included NOx, ethylene, toluene, and decane as measured trapping species as well as common exhaust gasses CO, H2O, O2, CO2, and H2. Evaluation showed that BEA and ZSM-5 zeolites were effective at trapping hydrocarbons (HCs), as saturation was not achieved after 30 min of exposure. SSZ-13 also stored HCs but was only able to adsorb 20–25% compared to BEA and ZSM-5. The presence of Ag or Pd did not impact the overall HC uptake, particularly in the first three minutes. Pd/zeolites had significantly lower THC release temperature, and it aided in the conversion of the released HCs; Ag only had a moderate effect in both areas. With respect to NOx adsorption, the level of uptake was much lower than HCs on all samples, and Ag or Pd was necessary with Pd being notably more effective. Additionally, only Pd/ZSM-5 and Pd/SSZ-13 continue to store a portion of the NOx above 200 °C, which is critical for downstream selective catalytic NOx reduction (SCR). Hydrothermal aging (800 °C for 50 h) of a subset of the samples were performed: BEA, Pd/BEA, ZSM-5, Pd/ZSM-5, and Pd/SSZ-13. There was a minimal effect on the HC storage, ~10% reduction in capacity with no effect on release temperature; however, only Pd/SSZ-13 showed significant NOx storage after aging.

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗

DOE OSTI2025

Modeling Protocol for Evaluation of Air Emissions for the Idaho National Laboratory Site-wide Permit to Construct and Facility Emission Cap (PTC/FEC)

This document outlines the methods to be used for calculating non-radiological air quality impacts from proposed emissions at the Idaho National Laboratory (INL) Site in support of applying for a renewal to a Permit to Construct with a Facility Emissions Cap (PTC/FEC). The INL Site is seeking to maintain a state enforceable limit on emissions to maintain synthetic minor status in accordance with Idaho Administrative Procedures Act (IDAPA) 58.01.01.175-181 and IDAPA 58.01.01.200-228.

54 ENVIRONMENTAL SCIENCES↗

DOE OSTI2024

Protocol for Evaluating Anion Exchange Membranes for Nonaqueous Redox Flow Batteries

Not provided.

Materials Science↗

NASA NTRS1992

Development of a test protocol for evaluating EVA glove performance

Testing gloved hand performance involves work from several disciplines. Evaluations performed in the course of reenabling a disabled hand, designing a robotic end effector or master controller, or hard-suit design have all yielded relevant information, and, in most cases, produced performance test methods. Most times, these test methods have been primarily oriented toward their parent discipline. For space operations, a comparative test which would provide a way to quantify pressure glove and end effector performance would be useful in dividing tasks between humans and robots. Such a test would have to rely heavily on sensored measurement, as opposed to questionnaires, to produce relevant data. However, at some point human preference would have to be taken into account. This paper presents a methodology for evaluating gloved hand performance which attempts to respond to these issues. Glove testing of a prototype glove design using this method is described.

Hinman, Elaine M.↗

DOE OSTI2022

Durability Evaluation of Advanced Fenestration Technologies

In the U.S., more than 40% of primary energy and 70% of electricity is consumed in residential and commercial buildings, resulting in annual energy costs of more than $430 billion. Approximately 35% of this consumption can be attributed to losses through the building envelope, of which windows are currently the weakest link. Multiple technologies are under development to improve this performance. This includes dynamic and photovoltaic glazing as well as emerging highly insulating technologies including vacuum insulating glass (VIG), aerogels and thin-glass based multi-pane configurations. While windows are specified based on expected performance as installed, the energy savings realized by high performance windows are delivered over time. This makes it critical to understand and maximize the durability of high-performance windows to ensure those projected energy savings are delivered. Present methods for evaluating durability are based on existing technologies. These methods may not apply or be adequate for newer emerging technologies which often present novel failure mechanisms that need to be understood and evaluated differently. In this presentation, we will discuss our efforts to define appropriate methods to improve the evaluation of many existing technologies as well as evaluate the durability of emerging window technologies. This work results from the combination of an extensive review of various international standards as well as existing scientific literature. In addition, input was gathered from multiple industry stakeholders regarding present practice as well as feedback on proposed improvements to existing methods. Here we will share these findings and present proposed improvements to developing and existing durability evaluation protocols.

building envelope↗

DOE OSTI2022

Model Assumptions and Data Characteristics: Impacts on Domain Adaptation in Building Segmentation

Studies on domain adaptation (DA) for remote sensing (RS) imagery analysis lack consistency in selection and description of evaluation scenarios. Without properly characterizing datasets, model assumptions, and evaluation scenarios, it is difficult to objectively compare DA methods and reach conclusions about their suitability across different applications. With this motivation, this work seeks to empirically assess to which extent the interaction between data characteristics and model assumptions influences the effectiveness of DA methods. Using the widely explored task of building footprint segmentation as a case study, we perform a large-scale study across over 200 DA scenarios that include variations across view angles, areas observed, and sensors used for data acquisition. Rather than adopting different model architectures or optimization criteria, we contrast the performances of two DA methods based on adversarial learning that differ only in their assumptions about source and target domains. Informed by metadata and data characteristics unveiled using traditional computer vision (CV) techniques as well as pretrained deep models, we provide a detailed meta-analysis of experiments highlighting the importance of accurately considering data assumptions for DA in RS segmentation tasks. As demonstrated by a “cherry-picking” exercise, different claims regarding which model is best could be made by selecting different subsets of evaluation scenarios. While well-calibrated assumptions can be beneficial, mismatching assumptions can lead to negative biases in DA applications. Furthermore, this study intends to motivate the community toward more consistent evaluation protocols while providing recommendations and insights toward creating novel benchmark datasets, documenting data characteristics, application-specific knowledge, and model assumptions.

42 ENGINEERING↗

DOE OSTI2026

Scaling open-weight large language models for hydropower regulatory information extraction: A systematic analysis

Information extraction from regulatory and technical documents using large language models (LLMs) involves practical trade-offs between extraction quality and computational cost. We evaluate eight open-weight LLMs spanning 0.6B–70B parameters on hydropower licensing documents and report deployment-oriented evidence under a unified extraction schema and evaluation protocol. Across the model set, we observe clear scale-dependent trends in both baseline extraction quality and the effectiveness of reflective reasoning (self-checking) under our fixed-prompt, no-augmentation setting. Mid-scale models often provide a favorable balance of accuracy and efficiency, whereas the smallest models show limited or inconsistent gains from the reasoning variants tested. Larger models achieve the highest overall F1 scores but incur substantially greater compute and infrastructure requirements. We further find that reliability failure modes can distort conventional metrics in this domain: in particular, high recall can coincide with systematic extraction errors when models fabricate values for fields that are absent from the source text, underscoring the importance of conservative null handling and evidence-grounded evaluation. Overall, our study provides a reproducible resource–performance comparison for open-weight LLM-based extraction in hydropower regulatory documentation and offers practical guidance for model selection under different deployment constraints.

Evaluation protocol↗

DOE OSTI2025

Exponential Backoff and Its Security Implications for Safety-Critical OT Protocols over TCP/IP Networks

The convergence of Operational Technology (OT) and Information Technology (IT) networks has become increasingly prevalent with the growth of Industrial Internet of Things (IIoT) applications. This shift, while enabling enhanced automation, remote monitoring, and data sharing, also introduces new challenges related to communication latency and cybersecurity. Oftentimes, legacy OT protocols were adapted to the TCP/IP stack without an extensive review of the ramifications to their robustness, performance, or safety objectives. To further accommodate the IT/OT convergence, protocol gateways were introduced to facilitate the migration from serial protocols to TCP/IP protocol stacks within modern IT/OT infrastructure. However, they often introduce additional vulnerabilities by exposing traditionally isolated protocols to external threats. This study investigates the security and reliability implications of migrating serial protocols to TCP/IP stacks and the impact of protocol gateways, utilizing two widely used OT protocols: Modbus TCP and DNP3. Our protocol analysis finds a significant safety-critical vulnerability resulting from this migration, and our subsequent tests clearly demonstrate its presence and impact. A multi-tiered testbed, consisting of both physical and emulated components, is used to evaluate protocol performance and the effects of device-specific implementation flaws. Through this analysis of specifications and behaviors during communication interruptions, we identify critical differences in fault handling and the impact on time-sensitive data delivery. The findings highlight how reliance on lower-level IT protocols can undermine OT system resilience, and they inform the development of mitigation strategies to enhance the robustness of industrial communication networks.

DNP3↗

DOE OSTI2025

An Evaluation of Representation Learning Methods in Particle Physics Foundation Models

We present a systematic evaluation of representation learning objectives for particle physics within a unified framework. Our study employs a shared transformer-based particle-cloud encoder with standardized preprocessing, matched sampling, and a consistent evaluation protocol on a jet classification dataset. We compare contrastive (supervised and self-supervised), masked particle modeling, and generative reconstruction objectives under a common training regimen. In addition, we introduce targeted supervised architectural modifications that achieve state-of-the-art performance on benchmark evaluations. This controlled comparison isolates the contributions of the learning objective, highlights their respective strengths and limitations, and provides reproducible baselines. We position this work as a reference point for the future development of foundation models in particle physics, enabling more transparent and robust progress across the community.

Chen, Michael [Caltech]↗

DOE OSTI2026

Taming the Wild West: Assessing Impacts-Relevant Climate Data Products (Abbreviated Report)

Impacts-relevant Earth system data refers to observational and ESM data that are downscaled, debiased, validated, and provisioned for use by decision-makers. Impacts-relevant Earth system data is essential for mitigation and adaptation planning across a variety of regions and sectors. A vast number of these data products have emerged in recent years, which has led to confusion among stakeholders and scientists as to the best product to use. With no standard evaluation protocol available for these products, the decision on which product to use was sometimes made because it was pragmatic rather than the best product to use. This project sought to develop foundational capabilities around impacts-relevant data products that would support more informed selection and application of these products. This work has been immensely successful, driving several academic publications and supported the development of a community of practice around impacts-relevant data products. Over the project’s three years we have addressed six tasks: First, the development of standard evaluation metrics for impacts-relevant climate data; second, the development of a novel suite of atmospheric river metrics; third, the development of novel metrics for precipitation feature analysis; fourth, the development of novel metrics for assessing co-variances between temperature and precipitation; fifth, the development of a dashboard for interactive examination of impacts-relevant climate data; and sixth, the establishment of a community of practice around impacts-relevant climate data that will continue beyond the conclusion of this project.

54 ENVIRONMENTAL SCIENCES↗

DOE OSTI2026

Application of artificial intelligence methods in the international roughness index prediction of rigid and composite pavements: a systematic review

The International Roughness Index (IRI) is a widely adopted metric for quantifying pavement roughness, directly influencing vehicle safety, ride comfort, and overall roadway performance. In recent years, the use of Machine Learning (ML) models for IRI prediction has gained momentum, with the goal of improving the allocation of maintenance and rehabilitation resources by enabling accurate assessments of pavement conditions. Most prior reviews, however, have concentrated on flexible pavements, leaving a notable gap regarding rigid and composite pavements. To address this gap, the present study conducts a systematic review of Artificial Intelligence (AI) methods applied to IRI prediction for rigid and composite pavements. Literature published between 2004 and 2025 is synthesized to highlight prevailing trends, methodological contributions, and directions for future research. Particular attention is given to the types of models employed, the datasets used for training and validation, and the role of input variables and data-processing strategies. Across the included studies, ensemble learning methods (especially gradient boosting variants such as XGBoost), artificial neural networks, and hybrid architectures frequently achieved high predictive skill, with several models reporting test-set coefficients of determination approaching 0.9–0.96, indicating strong potential for capturing the influence of traffic, pavement structure, and climatic factors. Since these results are obtained from heterogeneous datasets and evaluation protocols, they are interpreted qualitatively rather than as strict cross-study rankings. Analysis of input variables revealed that pavement age and initial IRI were included in 91% (21 of 23) and 78% (18 of 23) of studies, respectively. Climatic variables such as the freezing index appeared in 57% (13 of 23), while traffic-related factors were considered in 65% (15 of 23). The findings underscore the importance of standardized, high-quality datasets, such as those from the Long-Term Pavement Performance (LTPP) program, along with data consistency, model interpretability, computational efficiency, and replicability in enhancing IRI prediction. Future research should focus on incorporating input variable selection techniques to identify the most influential predictors, thereby improving accuracy and robustness. Integrating these approaches with advanced non-linear data-driven models, coupled with robust hyperparameter optimization, holds considerable promise for strengthening the reliability of IRI prediction and supporting resilient pavement management strategies.

42 ENGINEERING↗

DOE OSTI2022

Model Validation Database for Fires Involving Fuels at Liquefied Natural Gas Facilities

This document provides a description of the model evaluation protocol (MEP) database for fires involving liquefied natural gas (LNG) and processing fuels at LNG facilities. The purpose of the MEP is to provide procedures regarding the assessment of a model's suitability to predict thermal exclusion zones resulting from a fire. The database includes measurements from pool fire, jet fire, and fireball experiments which are provided in a spreadsheet. Users are to enter model results into the spreadsheet which automatically generates statistical performance measures and graphical comparisons with the experimental data. The intent of this document is to provide a description of the experiments and of the procedure required to carry out the validation portion of the MEP. In addition, the statistical performance measures, measurements for comparisons, and parameter variation are provided.

03 NATURAL GAS↗

DOE OSTI2026

Model Validation Database for Fires Involving Fuels at Liquefied Natural Gas Facilities (Version 2)

This document provides a description of the model evaluation protocol (MEP) database for fires involving liquefied natural gas (LNG) and processing fuels at LNG facilities. The purpose of the MEP is to provide procedures regarding the assessment of a model’s suitability to predict thermal exclusion zones resulting from a fire. The database includes measurements from pool fire, jet fire, and fireball experiments which are provided in a spreadsheet. Users are to enter model results into the spreadsheet which automatically generates statistical performance measures and graphical comparisons with the experimental data. The intent of this document is to provide a description of the experiments and of the procedure required to carry out the validation portion of the MEP. In addition, the statistical performance measures, measurements for comparisons, and parameter variation are provided.

03 NATURAL GAS↗

DOE OSTI2026

Taming the Wild West: Assessing Impacts-Relevant Climate Data Products (Final Report)

54 ENVIRONMENTAL SCIENCES↗

DOE OSTI2023

Evaluating energy justice metrics in early-stage science and technology research using the JUST-R metrics framework

Embedding principles of energy justice throughout all aspects of clean energy technology research and development (R&D) can facilitate a more just energy transition; yet gaps remain in our understanding of how to best integrate energy justice from the earliest R&D stages. The Justice Underpinning Science and Technology Research (JUST-R) metrics framework has been developed to enable early-stage energy researchers to assess and address justice considerations associated with their research, but the impacts of the framework, and others like it, have yet to be evaluated. This study seeks to evaluate the JUST-R metrics framework in terms of its effectiveness and appeal to researchers engaged in early-stage technical R&D using qualitative analyses of documents and workshop transcripts. We find that the metrics framework helps researchers identify problems and potential solutions surrounding the energy justice implications of their work and spurs a change in perspective for researchers, but, simultaneously, there is no evidence of solution follow-through within the evaluation timeframe. Greater institutional support, specialization to research areas, knowledge of energy justice fundamentals, and earlier incorporation of energy justice considerations in research projects arise as factors needed to aid continued use of the framework and pursuit of identified solutions. This evaluation protocol and these findings can serve as a guide for improving other frameworks with similar goals of encouraging sociotechnical engagement in early-stage energy R&D.

29 ENERGY PLANNING, POLICY, AND ECONOMY↗

NASA NTRS2000

Survey of Advanced Applications Over ACTS

The Advanced Communications Technology Satellite (ACTS) system provided a national testbed that enabled advanced applications to be tested and demonstrated over a live satellite link. Of the applications that used ACTS. some offered unique advantages over current methods, while others simply could not be accommodated by conventional systems. The initial technical and experiments results of the program were reported at the 1995 ACTS Results Conference. in Cleveland, Ohio. Since then, the Experiments Program has involved 45 new experiments comprising 30 application experiments and 15 technology related experiments that took advantage of the advanced technologies and unique capabilities offered by ACTS. The experiments are categorized and quantified to show the organizational mix of the experiments program and relative usage of the satellite. Since paper length guidelines preclude each experiment from being individually reported, the application experiments and significant demonstrations are surveyed to show the breadth of the activities that have been supported. Experiments in a similar application category are collectively discussed, such as. telemedicine. or networking and protocol evaluation. Where available. experiment conclusions and impact are presented and references of results and experiment information are provided. The quantity and diversity of the experiments program demonstrated a variety of service areas for the next generation of commercially available, advanced satellite communications.

Bauer, Robert↗

NASA NTRS2004

Advanced Communications Architecture Demonstration Made Significant Progress

Simulation for a ground station located at 44.5 deg latitude. The Advanced Communications Architecture Demonstration (ACAD) is a concept architecture to provide high-rate Ka-band (27-GHz) direct-to-ground delivery of payload data from the International Space Station. This new concept in delivering data from the space station targets scientific experiments that buffer data onboard. The concept design provides a method to augment the current downlink capability through the Tracking Data Relay Satellite System (TDRSS) Ku-band (15-GHz) communications system. The ACAD concept pushes the limits of technology in high-rate data communications for space-qualified systems. Research activities are ongoing in examining the various aspects of high-rate communications systems including: (1) link budget parametric analyses, (2) antenna configuration trade studies, (3) orbital simulations (see the preceding figure), (4) optimization of ground station contact time (see the following graph), (5) processor and storage architecture definition, and (6) protocol evaluations and dependencies.

Carek, David Andrew↗