Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “Evaluation Framework”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 91 records · Page 5
All sourcesNASA NTRSDOE OSTI
NASA NTRSDate not supplied

A Framework for Evaluating Climate Model Performance Metrics

The CMIP5 archive contains future climate projections from over 50 models provided by dozens of modeling centers from around the world. Individual model projections, however, are subject to biases created by structural model uncertainties. As a result, ensemble averaging of multiple models is often used to add value to model projections: consensus projections have been shown to consistently outperform individual models. Previous reports for the IPCC establish climate change projections based on an equal-weighted average of all model projections. However, certain models reproduce climate processes better than other models. Should models be weighted based on performance? Unequal ensemble averages have previously been constructed using a variety of mean state metrics. What metrics are most relevant for constraining future climate projections? This project develops a framework for systematically testing metrics in models to identify optimal metrics for unequal weighting multi-model ensembles. A unique aspect of this project is the construction and testing of climate process-based model evaluation metrics. A climate process-based metric is defined as a metric based on the relationship between two physically related climate variables?e.g., outgoing longwave radiation and surface temperature. Metrics are constructed using high-quality Earth radiation budget data from NASA's Clouds and Earth's Radiant Energy System (CERES) instrument and surface temperature data sets. It is found that regional values of tested quantities can vary significantly when comparing weighted and unweighted model ensembles. For example, one tested metric weights the ensemble by how well models reproduce the time-series probability distribution of the cloud forcing component of reflected shortwave radiation. The weighted ensemble for this metric indicates lower simulated precipitation (up to .7 mm/day) in tropical regions than the unweighted ensemble: since CMIP5 models have been shown to overproduce precipitation, this result could indicate that the metric is effective in identifying models which simulate more realistic precipitation. Ultimately, the goal of the framework is to identify performance metrics for advising better methods for ensemble averaging models and create better climate predictions.

Noel C Baker↗
DOE OSTI2022

Building a better framework for evaluating human well-being impacts in global change analysis: The example of energy security

Human well-being can be greatly impacted by the global environmental and socio-economic change captured in Integrated Models of Global Change (IMGCs). Though most IMGCs address some aspects of well-being, their underlying modeling approach and ‘philosophy’ differ widely, and some key elements – like energy security – are omitted. In this report, we describe a project in which we set out to a create a framework through which the well-being dimensions of the household are connected to key drivers of socio-economic and environmental change – and how the needed metrics, data and modeling methods can be brought to bear. We focus on the well-being dimensions of energy, and lay out the necessary elements to capturing household energy security – using household energy burden as the relevant metric. We begin by showing the conceptual linkage of energy burden to environmental drivers like temperature change, using a simple and straightforward conceptual framework. We then go further to use the example of GCAM-USA to show how some key analytical features of the model can provide insight into how energy security across different groups can change along alternative pathways to sustainability. We compare our preliminary assessment of household energy burden to existing data and suggest further steps to improve and refine this analysis in future research.

29 ENERGY PLANNING, POLICY, AND ECONOMY↗
DOE OSTI2022

gdess: A framework for evaluating simulated atmospheric CO 2 in Earth System Models

Atmospheric carbon dioxide (CO 2 ) plays a key role in the global carbon cycle and global warming. Climate-carbon feedbacks are often studied and estimated using Earth System Models (ESMs), which couple together multiple model components—including the atmosphere, ocean, terrestrial biosphere, and cryosphere—to jointly simulate mass and energy exchanges within and between these components. Despite tremendous advances, model intercomparisons and benchmarking are aspects of ESMs that warrant further improvement (Fer et al., 2021; Smith et al., 2014). Such benchmarking is critical because comparing the value of state variables in these simulations against observed values provides evidence for appropriately refining model components; moreover, researchers can learn much about Earth system dynamics in the process (Randall et al., 2019). We introduce `gdess` (a.k.a., Greenhouse gas Diagnostics for Earth System Simulations), which parses observational datasets and ESM simulation output, combines them to be in a consistent structure, computes statistical metrics, and generates diagnostic visualizations. In its current incarnation, `gdess` facilitates evaluating a model's ability to reproduce observed temporal and spatial variations of atmospheric CO 2 . The diagnostics implemented modularly in `gdess` support more rapid assessment and improvement of model-simulated global CO 2 sources and sinks associated with land and ocean ecosystem processes. We intend for this set of automated diagnostics to form an extensible, open source framework for future comparisons of simulated and observed concentrations of various greenhouse gases across Earth system models.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2020

Dynamic Microgrids in Resilient Distribution Systems With Reconfigurable Cyber-Physical Networks

Modern distribution systems energized by inverter-interfaced distributed generators (DGs) operate as coupled cyber-physical networks (C/P-networks), where the controllable components in the physical network (P-network) are coordinated through the cyber network (C-network). The concept of dynamic microgrids (MGs) operation has been adopted to enable distribution system autonomous operation with varying electric boundaries. To further enhance system operation resiliency and flexibility, dynamic MGs operation with reconfigurable C/P-networks is discussed in this work. An evaluation framework is proposed to assess the operational feasibility of distribution feeders with multiple inverter-based dynamic MGs and come out with possible restoration solutions in the context of cross-layer C/P-network reconfiguration. Furthermore, distributed controllers are developed for components with different operating characteristics to realize seamless system topology variations and provide coordinated secondary regulation in various operation modes. Furthermore, the proposed evaluation framework along with the developed distributed controller has been validated using a Hardware-in-the-Loop (HIL) real-time CPS testbed.

24 POWER TRANSMISSION AND DISTRIBUTION↗
DOE OSTI2025

ReSpike: A Co-Design Framework for Evaluating SNNs on ReRAM-Based Neuromorphic Processors

With Moore’s law approaching its end, traditional von Neumann architectures are struggling to keep up with the exceeding performance and memory requirements of artificial intelligence and machine learning algorithms. Unconventional computing approaches such as neuromorphic computing that leverage spiking neural networks (SNNs) to perform computation are gaining traction and seek the paradigm shift necessary to sustain the increasing demands of modern applications. Novel memory technologies, such as resistive RAM (ReRAM), employ a crossbar architecture that possesses the inherent capability of efficiently computing vector-matrix multiplication—a dominant operation in SNNs. The prospect of naturally mapping SNNs to the crossbar structures provides a unique opportunity for achieving a high-performance, power-efficient neuromorphic system. In this work, we present ReSpike, which is a new framework, behavioral simulator, and architectural design based on ReRAM crossbar architectures, enabling modeling and co-design to achieve efficient execution of SNNs. We drive this co-design forward by quantifying the impact that ReRAM cell nonidealities have on the corresponding accuracy of an SNN application.

Asifuzzaman, Kazi [ORNL] (ORCID:0000000240044791)↗
DOE OSTI2025

A cost–benefit framework to evaluate capacity upgrade options in overhead line transmission planning

This paper presents the methodology behind the new Reconductoring Economic and Financial Analysis (REFA) tool, an open-access software, used by transmission utilities to evaluate transmission capacity enhancement options. The proposed methodology is intended to be used in a new planning stage, after the capacity expansion and prior to the individual transmission project engineering, allowing capacity upgrade options (reconductoring, rebuild or voltage upgrade), and respective conductor selection, to be compared under the same economic basis. Furthermore, the REFA tool implements a methodology to rank project options and conductor types based on economic criteria, considering an approximation of the ampacity and sag constraints. Results, using 5 real transmission lines in the US, show that least-cost combinations of project and conductor types can be very diverse, which emphasizes the need for the proposed methodology and tool.

Advanced conductors↗
DOE OSTI2025

A hybrid numerical and machine learning framework for evaluating the performance of a 780 cm 2 aqueous organic redox flow battery

Aqueous organic redox flow battery (AORFB) is a promising cost-competitive technology for large-scale energy storage. Among existing work, the dihydroxyphenazine (DHP)-based AORFB has demonstrated high energy density and low capacity degradation in 10 cm2 cells during lab tests. However, its commercial-scale performance in more complex environments remains unknown, posing a barrier for commercialization. To address this gap, this work presents a comprehensive performance evaluation of a 780 cm 2 DHP-based AORFB by combining physics-based numerical model, machine learning (ML)-based surrogate models, and ML-derived sensitivity quantification. Specifically, we first select 12 key battery parameters that include 10 physicochemical quantities and 2 operation quantities, then select 6 performance metrics that include energy efficiency (EE), discharging capacity, charging energy, and power losses due to concentration, activation, and ohmic over-potentials. With such selection, 12800 combinations of the 12 parameters are subsequently generated using the Latin Hypercube Sampling method. These combinations, together with 38 pre-defined State of Charge, are then integrated to a validated AORFB model developed in COMSOL to compute the performance metrics. With both input parameters and performance metrics, 60 deep neural network (DNN) surrogate models are then trained to approximate the relationship between the 10 physicochemical quantities and 6 performance metrics at each flow rate and current density. Sensitivity scores are then calculated based on the DNN models. Two additional sensitivity analysis tools, i.e., MARS, and SHAP, are also used to cross-validate the sensitivity scores from the DNN. The results demonstrate that 1) the standard potential ranks the first in controlling EE and charging energy, 2) the membrane conductivity is most critical for power loss and EE, and 3) specific area and reaction rate control activation power loss.

25 ENERGY STORAGE↗
DOE OSTI2023

A Statistical Framework for Evaluating Rain Microphysics in Model Simulations and Disdrometer Observations

Abstract Statistical analyses of a large disdrometer data set and a diverse set of model simulations for convection using the Regional Atmospheric Modeling System were conducted, with the mutual goal of providing insights into precipitation formation and microphysical processes. We demonstrate that a two‐moment bulk microphysical model successfully captures the dominant observed modes of variability in rainfall related to rainfall intensity and raindrop size distributions. The model reproduced the general distribution of observed precipitation groups (PGs) derived from Principal Component Analysis. The multi‐variable analysis also uncovered some shortcomings in the model as well as limitations of the disdrometer data. The model solutions were constrained in their predicted drop size distributions (DSDs) due to the fixed DSD parameters assumed in a two‐moment microphysics scheme. A case study from the Mid‐latitude Continental Clouds and Convection Experiment field project demonstrated how model results can be used to contextualize the disdrometer observations which are limited in sample size, spatial coherence, and detection of small drops and low drop concentrations. The case study also showed that the spatial patterns of the statistically derived PGs revealed by the model are consistent with the hypothesized microphysical processes that determine surface rain DSDs. This work demonstrates how leveraging the strengths of observations and models together can improve our understanding and representation of rain microphysical processes.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2025

Author Correction: A framework to evaluate machine learning crystal stability predictions

In the version of this article initially published, Figs. 1–3, Table 1 and the Supplementary Information presented more models than were present in the accepted version of the article, and which were not discussed in the text. The Supplementary Information has been revised and the figures and table are now updated in the HTML and PDF versions of the article.

Riebesell, Janosh↗
DOE OSTI2024

A Novel Framework to Evaluate the Costs and Potential of Bioenergy in Decarbonization of the U.S. Economy

The long-term strategy of the United States targets reaching economy-wide net-zero emissions by 2050 and a carbon-neutral electricity grid by 2035 (U.S. Department of State and U.S. Executive Office of the President, 2021). Meeting these targets would require considerable changes to the energy system. Some key characteristics of illustrative net-zero energy systems include increased penetration of renewable energy and carbon sources, use of CO2 capture and storage (CCS) in hard-to-abate sectors, and a greater role for energy carriers such as electricity and hydrogen (Davis et al, 2018). Another common feature of such energy systems is the need for carbon dioxide removal (CDR) approaches (Horowitz et al, 2022). Across all these characteristics of net-zero energy systems, bioenergy and biomass feedstock is anticipated to play an important role. Biomass feedstock serves as a renewable carbon source. This can enable conversion of such feedstock into fuels and energy carriers for hard-to-abate sectors such as aviation. Indeed, the U.S. Government has a target to meet all jet fuel demand by 2050 from sustainable aviation fuel (SAF), where biofuel pathways are likely to have an important role (EERE, 2020). Bioenergy is also highly versatile with the possibility to convert feedstock into electricity, hydrogen, liquid fuels, heat or high-value products, based on biomass type, demand and technology availability (Clarke et al, 2022). Combination of bioenergy with CCS can also nominally deliver CDR (Fuhrman et al, 2023). As such, the share of bioenergy is expected to grow by at least five time across scenarios studied for the long-term strategy of the U.S. between 2020 and 2050 (Horowitz et al, 2022). Notwithstanding the role of bioenergy in the energy systems, its deployment, costs and scalability are influenced by a number of factors. Some of these factors pertain to policy interventions such as imposition of a binding decarbonization target either at an economy-wide level or the sectoral level. Resource availability and type of biomass feedstock also varies considerably across regions. From a technological perspective, the readiness of bioenergy conversion pathways is subject to high variability. This influences the costs of deployment. Moreover, the sourcing of feedstock, grid carbon intensity, and co-product handling approaches all affect the life cycle efficacy of bioenergy. The latter, in turn, is particularly important in determining the extent to which bioenergy with CCS or BECCS can effectively deliver CDR (Fajardy and Mac Dowell, 2017).

air emission↗
DOE OSTI2021

A Cybersecurity Event Simulation Tool and Platform

Digital Instrumentation and Control Systems (ICSs) have replaced analog control systems in nuclear power plants raising cybersecurity concerns. To study and understand the cybersecurity risks of nuclear power plants both high fidelity models of the plant physics and controllers must be created, and a framework to test and evaluate cyber security events must be established. A testing and evaluation framework of cybersecurity events consists of a method of interfering with control systems, a simulation of the plant network, and a network packet capture and recording tool. Sandia National Labs (SNL) in collaboration with the University of New Mexico’s Institute for Space and Nuclear Power Studies (UNM-ISNPS) is developing such a cybersecurity testing framework.

21 SPECIFIC NUCLEAR REACTORS AND ASSOCIATED PLANTS↗
DOE OSTI2025

Applying a Multisector Scenario Framework to Evaluate Past and Future Public Surface Water Supply Infrastructure Strategies in Texas

Datasets supporting the index model and scenario analysis used in evaluating surface water supply strategies across different water system types in Texas. These data underpin the scenario development and application of five key indicators: Water Availability Index (WAI), Water Quality Index (WQI), Energy Requirement Index (ERI), Water Treatment Cost (WTC), and Water Infrastructure Cost (WIC). The datasets are organized by system type—stream reaches (flowlines), waterbodies, and reservoirs—and include both raw and standardized index values. The integrated datasets also provide scenario classifications (original and adjusted) based on infrastructure and planning priorities, enabling comparison across Shared Socioeconomic Pathways (SSPs). Additional strategy-level data are included to support evaluation of state-level new reservoir projects in relation to cost and availability tradeoffs. Please refer to the README file provided in Files for more details. Descriptions of the datasets are provided below. Dataset(s) Descriptions Folder: Index_model_database.zip Subfolder: Stream_reach.zip Fl_wf.csv, Fl_wq.csv, Fl_er.csv, Fl_wf_wtcUV.csv, Fl_wf_wtcnoUV.csv, Fl_allfac_wic1.csv, Fl_allfac_wic2.csvDatasets for computing WAI, WQI, ERI, WTC, and WIC for surface water systems classified as stream reaches (flowlines). Subfolder: Waterbody.zip Wb_wf.csv, Wb_wq.csv, Wb_er.csv, Wb_wf_wtcUV.csv, Wb_wf_wtcnoUV.csv, Wb_allfac_wic1.csv, Wb_allfac_wic2.csvEquivalent index model datasets for waterbodies, reflecting hydrologic and infrastructure attributes specific to impounded natural systems. Subfolder: Reservoir.zip Rs_wf.csv, Rs_wq.csv, Rs_er.csv, Rs_wf_wtcUV.csv, Rs_wf_wtcnoUV.csv, Rs_allfac_wic1.csv, Rs_allfac_wic2.csvIndex model datasets specific to regulated reservoir systems, incorporating both resource indicators and cost parameters. Folder: Integrated data.zip combined_merged_data.csv, combined_merged_data_scenario.csvDatasets integrating index model indicators (both raw and scaled) with scenario classifications, including adjustments reflecting SSP-aligned transitions and planning shifts. Folder: Additional data.zip wai_supplystrat_wic_merged.csvCurated dataset capturing proposed major reservoir-based municipal water supply strategies in Texas. Integrates site-level planning data with estimated capital infrastructure costs and water availability scores for comparative assessment.

geospatial↗