Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “Evaluation Framework”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 127 records · Page 7
All sourcesNASA NTRSDOE OSTI
DOE OSTI2021

Opportunities for Treatment and Reuse of Agricultural Drainage in the United States

Irrigation accounts for 42% of the total freshwater withdrawals in the United States. Climate change, the pressure of a growing population, degrading water quality, and increased competition from other sectors could constrain continuous supply to meet future agricultural water demand. This study presents an evaluation framework to assess the potential reuse of agricultural drainage water for crop irrigation. Using a regional approach, we review the current state of agricultural drainage treatment and reuse and the institutional, economic, and other barriers that can influence the reuse decision. In the 31 eastern states, agricultural drainage contains valuable nutrients that can be reused for irrigation with minimal treatment, while the 17 western states struggle with large volumes of saline drainage that can contain constituents of concern (e.g., selenium), preventing reuse without treatment. Using a new decision-support tool called WaterTAP3, a potential treatment train for saline agricultural drainage was analyzed to identify treatment challenges, research needs, and the potential implementation at a larger scale. As demonstrated by our case study, desalination of agricultural drainage is costly and energy intensive and will require sizable investments to fully develop and optimize technologies as well as manage the generated waste and brine.

29 ENERGY PLANNING, POLICY, AND ECONOMY↗
DOE OSTI2026

AutoLabs: cognitive multi-agent systems with self-correction for autonomous chemical experimentation

The automation of chemical research through self-driving laboratories (SDLs) promises to accelerate scientific discovery, yet the reliability and granular performance of the underlying AI agents remain critical, under-examined challenges. In this work, we introduce AutoLabs, a self-correcting, multi-agent architecture designed to autonomously translate natural-language instructions into executable protocols for a high-throughput liquid handler. The system engages users in dialogue, decomposes experimental goals into discrete tasks for specialized agents, performs tool-assisted stoichiometric calculations, and iteratively self-corrects its output before generating a hardware-ready file. We present a comprehensive evaluation framework featuring five benchmark experiments of increasing complexity, from simple sample preparation to multi-plate timed syntheses. Through a systematic ablation study of 20 agent configurations, we assess the impact of reasoning capacity, architectural design (single- vs. multi-agent), tool use, and self-correction mechanisms. Our results demonstrate that agent reasoning capacity is the most critical factor for success, reducing quantitative errors in chemical amounts (nRMSE) by over 85% in complex tasks. When combined with a multi-agent architecture and iterative self-correction, AutoLabs approaches expert-authored reference procedures on the benchmark (F1-score > 0.89) on challenging multi-plate syntheses. These findings establish a clear blueprint for developing robust and trustworthy AI partners for autonomous laboratories, highlighting the synergistic effects of modular design, advanced reasoning, and self-correction to ensure both performance and reliability in high-stakes scientific applications. Code: https://github.com/pnnl/autolabs

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗
DOE OSTI2022

GPU-based optical simulation of the DARWIN detector

Abstract Understanding propagation of scintillation light is critical for maximizing the discovery potential of next-generation liquid xenon detectors that use dual-phase time projection chamber technology. This work describes a detailed optical simulation of the DARWIN detector implemented using Chroma, a GPU-based photon tracking framework. To evaluate the framework and to explore ways of maximizing efficiency and minimizing the time of light collection, we simulate several variations of the conventional detector design. Results of these selected studies are presented. More generally, we conclude that the approach used in this work allows one to investigate alternative designs faster and in more detail than using conventional Geant4 optical simulations, making it an attractive tool to guide the development of the ultimate liquid xenon observatory.

Instruments & Instrumentation↗
DOE OSTI2023

Evaluating the Risk of Enabling Energy Storage Systems to Provide Multiple Services

Battery energy storage systems (BESS) are emerging as a promising, versatile technology for modern power grids due to their inherent distributed nature, their ability to inject bidirectional power flows and to provide multiple grid services. With the proliferation of intermittent renewable energy resources, the BESS procurement is also expected to increase, thereby creating systematic and strategic opportunities for BESS to various revenue streams for providing multiple grid services. Utilizing a co-simulation architecture, this work presents evaluation framework for enabling Utility-scale BESS with simultaneous provision of multiple services. The proposed framework is implemented on a real-life feeder modeled in GridLAB-D, leveraging multi-service grid support. A Risk-Aware control strategy is also implemented in this work that that takes into account the uncertain nature of some of the stochastic services and ensures the BESS to deliver their service commitments. This promotes overall grid reliability and encourages BESS implementation through the revenue streams from providing multiple grid services.

Theisen, John↗
DOE OSTI2026

A System Level Comparative Study of SecondarySide Rectifiers for UAV Wireless Power Transfer in High-Altitude Airborne Energy Platforms

Wireless power transfer (WPT) systems for unmanned aerial vehicles (UAVs) must satisfy stringent design requirements to ensure reliable operation under constraints related to efficiency, thermal performance, altitude, weight, and reliability, particularly in airborne and high-altitude environments. Thermal management becomes increasingly challenging at higher altitudes due to reduced air density, while reliability is strongly affected by the number and complexity of active components. The weight and compactness of secondary-side rectifiers directly impact UAV flight endurance and system integration, influencing energy consumption and payload allocation. These factors collectively define the practical feasibility of UAV WPT systems and guide rectifier topology selection. This paper presents a comparative and experimentally validated evaluation framework for secondary-side rectifiers under UAV-specific constraints. Three representative rectifier architectures are analyzed using component-based assessment of weight, thermal behavior, and reliability, together with experimental efficiency measurements obtained from a 1 kW WPT prototype. The results show that the single-switch active rectifier achieves the highest peak efficiency of approximately 93.2% and an overall efficiency of 91%, while also offering advantages in weight and reliability. The findings demonstrate how constraint-oriented analysis, combined with experimental validation, enables informed selection of secondary-side rectifiers for UAV wireless power transfer applications, including emerging airborne and high-altitude platforms.

Asa, Erdem [ORNL] (ORCID:0000000190884812)↗
DOE OSTI2021

A Multi-Site Networked Hardware-in-Loop Platform for Evaluation of Interoperability and Distributed Intelligence at Grid-Edge

Electric power systems have experienced large increases in the number of intelligent, connected and controllable devices being deployed, leading to a high degree of distributed intelligence at the grid-edge. These devices, both utility-owned and consumer-owned, include but are not limited to: renewable generation sources, energy storage, remote switches, voltage regulators, and smart controllable loads such as electric vehicles. These new devices provide significant potential for increased operational flexibility that can be leveraged to achieve system reconfiguration, resiliency improvements, power quality improvements, and distribution system automation. However, there are two significant challenges that must be addressed before these assets can be leveraged for operations: interoperability and system level validation prior to deployment. Because of the complexity of distributed control systems, and their interactions with legacy centralized controls, a purely simulations-based approach for pre-deployment validation is not sufficient. It requires hardware-in-loop testing to emulate the operational hardware devices and evaluate their performance. Additionally, securely integrating multiple test facilities at utility operators and vendors might enable rapid scale-up of evaluation platforms, and remove the need for multiple expensive standalone installations. Presented in this paper, is the development of a multi-site evaluation platform that employs Advanced Distribution Management Systems (ADMS), distributed control devices, real-time hardware-in-loop assets, secure communication links, and protocol adapters. This platform uses standards-based approaches and open-source tools, and hence can serve as a template for other researchers and institutions to implement their multi-site evaluation frameworks for pre-deployment testing.

Essakiappan, Somasundaram↗
DOE OSTI2026

Detect the Unobservable: Abnormality Detection in mixed Autonomy for Lane Change Maneuver with Following Vehicles’ Trajectories Only

Highly Automated Vehicles (HAVs) and Advanced Driver-Assistance Systems (ADAS) are transforming modern transportation with enhanced mobility, safety, and efficiency. Despite their advantages, cybersecurity vulnerabilities in these systems can lead to abnormal behavior, posing significant risks to surrounding human-driven vehicles (HDVs) in mixed traffic environments. Here, this article addresses the challenge of detecting abnormal lateral movements of HAVs/ADAS vehicles using only trajectory profiles of following HDVs. Specifically, we propose a novel modeling approach that captures both normal and abnormal lateral behaviors through vehicle kinematics, integrated decision-making processes, vehicle control using symbolic regression for lane change vehicles. Additionally, we introduce an abnormality detection framework that relies on observable HDV data, even in occlusion scenarios. The framework evaluates the sensitivity of various car-following models to detect abnormal behaviors, providing insights into the interaction between HAVs/ADAS and HDVs in mixed autonomy systems.

Connected and Automated vehicles↗
DOE OSTI2023

pnnl/EXPERT2

This software includes the Jupyter notebooks, model pretraining and evaluation code for the EXPERT 2.0 Human-AI Reasoning Engine V0.1. It supports pre-training a Human-AI model for reasoning over multi-layer network representations. It includes prompt-based evaluation framework in a Jupyter notebook for AI reasoning and Jupyter widgets with AI-based techniques for evidence generation and uncertainty quantification to support human-AI reasoning.

Horawalavithana, Sameera↗
DOE OSTI2025

Using ARM Observations to Evaluate Process-Interactions in MCS Simulations Across Scales (Final Progress Report)

This project, funded by DOE Atmospheric System Research (DE-SC0020050), focused on improving the representation of mesoscale convective systems (MCSs) in numerical weather and climate models by leveraging high-resolution observations from the DOE Atmospheric Radiation Measurement (ARM) program. The research aimed to evaluate model sensitivities to grid spacing, microphysics, and planetary boundary layer (PBL) schemes, with a particular emphasis on improving convection parameterization for high-resolution modeling. Findings from this work highlight several key advancements. Model validation against ARM radar wind profiler data from the Southern Great Plains (SGP) and Manaus (MAO) sites revealed systematic biases in simulated convective mass flux profiles, leading to the development of an observationally constrained evaluation framework for diagnosing and improving model performance. Sensitivity analyses demonstrated that the representation of Amazonian MCSs was highly dependent on PBL scheme selection, while mid-latitude MCSs were more strongly influenced by microphysics parameterizations. A series of high-resolution WRF simulations, ranging from 4 km to 125 m grid spacing, provided insight into the behavior of convective drafts across scales. While updraft properties converged at sub-kilometer resolutions, biases in downdraft intensity persisted even at the finest resolution tested, emphasizing the need for further refinements in model physics. Additionally, comparisons of MCS vertical structures between mid-latitude and tropical environments revealed stronger updrafts and larger mass flux in mid-latitude MCSs, providing critical insights for improving climate model representations of storm-scale dynamics. The project’s findings have already contributed to advancing numerical modeling capabilities, particularly in WRF, MPAS, ICON, and DOE’s SCREAM model, by refining how convective processes are represented in high-resolution climate simulations. Results were disseminated through peer-reviewed publications, conference presentations, and ARM/ASR Research Highlights, engaging the broader scientific community. The project also provided valuable training opportunities for two postdoctoral researchers, who played central roles in model development, analysis, and dissemination of results. Their work contributed to several publications and conference presentations, helping prepare them for careers in atmospheric modeling. By improving the simulation of MCSs, this research directly supports the development of next-generation climate models capable of more accurately representing extreme precipitation and convective processes. The insights gained will inform future improvements in convective parameterization and guide the design of high-resolution weather and climate simulations, ultimately enhancing the reliability of climate projections and weather forecasts.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2025

Evaluation of LLM-Generated Kokkos Code Using Compile-Time and Run-Time Testing

Due to the growing use of large language models (LLMs) by developers and researchers, it has become essential to reliably evaluate their ability to generate code that uses specialized libraries. We explore the use of compile-time and run-time evaluation of LLM-generated Kokkos code through extending the methods used by OpenAI with the HumanEval dataset. Our evaluation framework is based on the first 40 prompts from the Kokkos138 dataset. We start by discussing two different forms of LLM prompting, using entirely plain English or providing pseudocode for added context. These two methods are used to generate Kokkos code with the Llama-3.1-8B-Instruct and CodeQwen1.5-7B-Chat models. We found that both forms of prompting led to high failure rates and difficulties with reliably parsing LLM-generated code, while prompts with pseudocode for context generally led to improved results on more complicated tests.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2026

Pull Force Evaluation of CCS Insulated End Caps

Insulated end caps installed on DC pins of Combined Charging System (CCS) inlets have been identified as potential debris sources within electric vehicle supply equipment (EVSE) connectors, increasing the risk of electric shock and fire hazards. Although standards such as IEC 62196-1:2022 specify mechanical pull-force requirements for these end caps, it remains unclear whether these requirements provide adequate robustness under real-world conditions. To address this concern, the National Charging Experience (ChargeX) Consortium's Hardware Task Force conducted evaluations of insulated end caps used in OEM CCS inlets. This study assesses the performance of insulated end caps used in OEM CCS inlets, specifically those installed on DC pins, using the procedures defined in IEC 62196-1:2022 (Section Sign) 26.7 and SAE J3400, supplemented by additional experimental conditioning. To ensure relevance to real-world conditions, NLR collaborated with charging network operators (CNOs) to guide the selection of charging inlet samples. Material analyses of the insulated end cap samples were conducted to confirm that the tested materials reflected the types and properties commonly observed in the field. The selected samples underwent temperature and humidity conditioning followed by pull-force testing to evaluate the end-cap integrity. Although SAE J3400 inlets were not directly tested due to limited diversity in available field data, the underlying end-cap principles between CCS and SAE J3400 are comparable, allowing the study's insights to be relevant to both technologies. Overall, this study provides a structured evaluation framework to inform potential refinements to mechanical pull-force requirements in charging standards.

33 ADVANCED PROPULSION SYSTEMS↗
DOE OSTI2026

AI-Ready Data Pilot Project Report

The proliferation of artificial intelligence in scientific research has created an urgent need to define "AI-ready data" for researchers and, more importantly, provide resources to help them produce AI-ready data. At Pacific Northwest National Laboratory, we conducted a pilot study with three data scientists evaluating three CSV datasets from different scientific domains, followed by semi-structured interviews capturing assessment practices. Our findings reveal that AI-readiness evaluation is intuition-based, with practitioners asking "How fast can I go from raw data to my machine learning pipeline?" Data scientists consistently prioritized workflow efficiency, human interpretability, and quality stewardship signals. From these insights, we developed a practical evaluation framework comprising data requirements, metadata standards, and validation tests that provides actionable criteria for producing and curating AI-ready datasets, addressing the gap between theoretical understanding and practical implementation.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2023

A Multi-Dimensional Benefit Assessment of Automated Mobility Platforms (AMP) for Large Facilities: Mobility, Energy, Equity, and Facility Management & Design

The goal of the automated mobility platforms (AMPs) initiative is to raise the bar of service regarding equity and sustainability for public mobility systems that are crucial to large facilities, and doing so using electrified, energy efficient technology. Using airports as an example, the rapid growth in air travel demand has led to facility expansions and congested terminals, which directly impacts equity (e.g., increased challenges for Passengers with Reduced Mobility [PRMs]) and sustainability - both of which are important metrics often overlooked during the engineering design process. Therefore, to evaluate systems and inform critical near- and long-term decisions more effectively, a holistic evaluation framework is proposed focused on four key areas: (1) mobility, with emphasis on travel time and accessibility within an airport, (2) environment, focused on energy consumption and greenhouse gas (GHG) emissions associated with intra-airport mobility, (3) equity, specifically to the PRM community, but with an eye to the whole of society, and (4) built environment, or the fundamental changes in building design enabled by different mobility systems for larger and more flexible, functional, and energy-efficient structures. Below, AMPs are defined, and each metric is discussed further, all with a focus on airport mobility.

ADVANCED PROPULSION SYSTEMS↗
DOE OSTI2026

Antarctic ice sheet model comparison with uncurated geological constraints shows that higher spatial resolution improves deglacial reconstructions

Accurately reconstructing past changes to the shape and volume of the Antarctic ice sheet relies on the use of physically based and thus internally consistent ice sheet modeling, benchmarked against spatially limited geologic data. The challenge in model benchmarking against geologic data is diagnosing whether model-data misfits are the result of an inadequate model, inherently noisy or biased geologic data, and/or incorrect association between modeled quantities and geologic observations. In this work we address this challenge by (i) the development and use of a new model-data evaluation framework applied to an uncurated data set of geologic constraints, and (ii) nested high-spatial-resolution modeling designed to test the hypothesis that model resolution is an important limitation in matching geologic data. While previous approaches to model benchmarking employed highly curated datasets, our approach applies an automated screening and quality control algorithm to an uncurated public dataset of geochronological observations (specifically, cosmogenic-nuclide exposure-age measurements from glacial deposits in ice-free areas). This optimizes data utilization by including more geological constraints, reduces potential interpretive bias, and allows unsupervised assimilation of new data as they are collected. We also incorporate a nested model framework in which high-resolution domains are downscaled from a continent-wide ice sheet model. We highlight the application of this framework by applying these methods to a small ensemble of deglacial ice-sheet model simulations, and demonstrate that the nested approach improves the ability of model simulations to match exposure age data collected from areas of complex topography and ice flow. We develop a range of diagnostic model-data comparison metrics to provide more insight into model performance than possible from a single-valued misfit statistic, showing that different metrics capture different aspects of ice sheet deflation.

Geosciences↗
DOE OSTI2021

Call to Action for Global Access to and Harmonization of Quality Information of Individual Earth Science Datasets

Knowledge about the quality of data and metadata is important to support informed decisions on the (re)use of individual datasets and is an essential part of the ecosystem that supports open science. Quality assessments reflect the reliability and usability of data. They need to be consistently curated, fully traceable, and adequately documented, as these are crucial for sound decision- and policy-making efforts that rely on data. Quality assessments also need to be consistently represented and readily integrated across systems and tools to allow for improved sharing of information on quality at the dataset level for individual quality attribute or dimension. Although the need for assessing the quality of data and associated information is well recognized, methodologies for an evaluation framework and presentation of resultant quality information to end users may not have been comprehensively addressed within and across disciplines. Global interdisciplinary domain experts have come together to systematically explore needs, challenges and impacts of consistently curating and representing quality information through the entire lifecycle of a dataset. This paper describes the findings of that effort, argues the importance of sharing dataset quality information, calls for community action to develop practical guidelines, and outlines community recommendations for developing such guidelines. Practical guidelines will allow for global access to and harmonization of quality information at the level of individual Earth science datasets, which in turn will support open science.

96 KNOWLEDGE MANAGEMENT AND PRESERVATION↗
DOE OSTI2020

Are You Resilient? Site Energy and Water Resilience Indicators

There is growing interest among operators of individual or co-located facilities to plan for utility disruptions caused by natural hazards and threats. Energy and water resilience at the site level can be characterized as the ability to withstand, adapt, and recover from an energy and/or water supply disruption. But how can a site’s energy and water resilience posture be measured? An evaluation framework with clear performance measures or indicators can help organizations establish what it means to be energy and water resilient and assess progress towards meeting resilience goals across sites and over time. Tracking a consistent set of indicators can help organizations identify problem areas that should be addressed and ascertain if energy and water resilience has improved as a result of investments or operational or policy changes. Resilience indicators also provide a means for organizations with geographically distributed sites to identify common problem areas that may require higher-level attention. This paper outlines an approach that organizations can use to measure site energy and water resilience. This approach was developed by Pacific Northwest National Laboratory for the U.S. Department of Energy’s Federal Energy Management Program as part of an effort to help federal agencies plan for energy and water resilience in a more systematic and rigorous way. The paper defines a general structure for defining energy and water resilience indicators, and provides example indicators and potential rating methods.

resilience, metrics for resilience, site resilienc↗
DOE OSTI2022

Evaluation of WRF-Solar Cloud Forecast Using the NSRDB: Preprint

Cloud forecast is a crucial component in predicting solar irradiance from numerical weather prediction (NWP) models. Assessing cloud properties from the NWP models requires significant work due to the need for high-quality data, spatial analysis covering model extent, and detailed analysis of model performance for different types of clouds. This study presents an evaluation of the WRF-Solar cloud forecast using the National Solar Radiation Database (NSRDB). We propose an evaluation framework applied to a single model prediction as well as ensemble-based forecasts. Various cloud detection metrics are calculated when comparing with the satellite-derived dataset. The mismatched clouds from the WRF-Solar model are quantified using nine cloud types classified by cloud top height and cloud optical depth. The results based on the WRF-Solar forecasts covering the entire U.S. for the full year of 2018 shows mismatched cloud frequency in the range of 8% - 46% for thick and high-level (deep convective) to thin and low-level (cumulus) clouds.

cloud mask forecast↗
DOE OSTI2024

A Multi-Dimensional Benefit Assessment of Automated Mobility Platforms (AMP) for Large Facilities

Automated Mobility Platform (AMP) is a general term for intelligently managed passenger mobility systems that leverage automation and electrification to improve system performance and sustainability through the integration of real-time data streams with advanced decision-making processes tailored for use in large facilities. Using airports as an example, the rapid growth in air travel demand has led to facility expansions and congested terminals, which directly impacts equity (e.g., increased challenges for passengers with reduced mobility [PRMs]) and sustainability - both of which are important metrics often neglected during the engineering design process. Therefore, to evaluate systems and inform critical decisions more effectively, a holistic evaluation framework is proposed that includes equity, sustainability, and infrastructure perspectives - referred to as a multidimensional benefit assessment. More specifically, the following analysis focuses on: (1) mobility, with emphasis on travel time and accessibility within an airport, contrasting existing systems with the proposed AMP system; (2) equity analysis, specifically to the PRM community, but with an eye to the whole of society; (3) energy consumption and greenhouse gas (GHG) emissions associated with intra-airport mobility, comparing and contrasting modes as well as impacts on overall airport operations efficiency; and (4) fundamental changes in building design enabled by AMPs for larger and more flexible, functional, and energy-efficient structures.

ADVANCED PROPULSION SYSTEMS↗