Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “evaluation framework”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 55 records · Page 3
All sourcesNASA NTRSDOE OSTI
DOE OSTI2024

​​Integrity Enhancing Protocols (IEP) Evaluation Framework​

​Protocols play an essential role in Advance Reactor systems. A diverse set of protocols are available to these reactors. Advanced Reactors benefit from technologies that can minimize their resource utilization and costs. Evaluation frameworks are often used when assessing protocols and processes related to cryptographic security systems. The following report discusses the various characteristics associated with these protocol evaluation frameworks, and derives a novel evaluative framework.

22 GENERAL STUDIES OF NUCLEAR REACTORS↗
DOE OSTI2024

A Pathway Analysis Framework for Evaluating the Economic and Environmental Viability of Biomass-Based Plastic Production

Plastic production from fossil feedstocks (e.g., naphtha, coal, and natural gas) is not sustainable and causes known environmental impacts such as global warming. A possible solution is to shift production pathways to use biomass, which is a sustainable feedstock that can sequester atmospheric carbon dioxide. This study presents an optimization-based pathway analysis framework for evaluating the carbon footprints of the production of mainstream plastics from biomass and fossil feedstocks. We use the modeling framework to quickly navigate complex interdependencies that exist between the production pathways of different plastics and to determine pathways of minimum production cost under a range of carbon pricing scenarios. The framework interprets carbon prices as an exogenous taxation scheme or an endogenous negative value perceived by producers. The proposed approach reveals the biomass feedstock quantities needed to displace fossil counterparts and the plastics and technologies that should be prioritized. The framework can also be used for evaluating system-wide trade-offs between production costs and carbon footprints that arise from pathway interdependencies. We also evaluate hidden environmental impacts associated with the large-scale use of biomass as a feedstock, such as land use and water eutrophication that results from a significant increase in fertilizer use. Therefore, it is important to highlight that there are trade-offs between decarbonization and other environmental issues. Here, the proposed framework provides an integrative platform for basic techno-economic and life-cycle data that can be used for analyzing diverse scenarios and determining necessary technology targets (e.g., yields, footprints, and costs) to achieve required levels of decarbonization.

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗
DOE OSTI2023

Generalized Bayesian Framework for Evaluation of Integral Benchmark Experiments

A recently published generalized Bayesian optimization framework has provided a way to retract any or all of the three common assumptions underlying the conventional Generalized Linear Least Squares (GLLS) optimization method based on the concepts introduced in Ref. [2]. These assumptions are: 1. Perfection: The model used for data evaluation and the prior probability distribution function (PDF) of generalized data are perfect. 2. Normality: The prior and posterior PDF are normal. 3. Linearity: The model is linear. In this work we outline how the framework in [1] could be directly adopted for improved evaluation of nuclear criticality integral benchmark experiments (IBEs) by: 1. Removing the first assumption alone by utilizing the concept of imperfections introduced in [1] to enable evaluation in the presence of discrepancies between the data and model or of missing covariance information by a GLLS method that will be seen as a generalization of the conventional GLLS method employed by the TSURFER code, and by 2. Removing the remaining two assumptions by implementing a Markov Chain Monte Carlo method for computation of the posterior PDF in the SAMPLER code, where TSURFER and SAMPLER are the uncertainty quantification (UQ) codes for IBEs in the SCALE code system based on the GLLS and the stochastic method, respectively. The graphic in Figure 1 categorizes the methods discussed in terms of the assumptions that they employ to determine posterior PDFs.

73 NUCLEAR PHYSICS AND RADIATION PHYSICS↗
DOE OSTI2021

Generalized Bayesian Framework for Evaluation of Integral Benchmark Experiments

A recently published generalized Bayesian optimization framework has provided a way to retract any or all of the three common assumptions underlying the conventional Generalized Linear Least Squares (GLLS) optimization method based on the concepts introduced in reference two. These assumptions are: 1. Perfection: The model used for data evaluation and the prior probability distribution function (PDF) of generalized* data are perfect; 2. Normality: The prior and posterior PDF are normal; and 3. Linearity: The model is linear. In this work we outline how the framework in 1 could be directly adopted for improved evaluation of nuclear criticality integral benchmark experiments (IBEs) by: 1. Removing the first assumption alone by utilizing the concept of imperfections introduced in 1 to enable evaluation in the presence of discrepancies between the data and model or of missing covariance information by a GLLS method that will be seen as a generalization of the conventional GLLS method employed by the TSURFER code, and by 2. Removing the remaining two assumptions by implement- ing a Markov Chain Monte Carlo method for computation of the posterior PDF in the SAMPLER code, where TSURFER and SAMPLER are the uncertainty quantification (UQ) codes for IBEs in the SCALE code system based on the GLLS, and the stochastic method, respectively.

73 NUCLEAR PHYSICS AND RADIATION PHYSICS↗
DOE OSTI2020

A Framework for Evaluating the Resilience Contribution of Solar PV and Battery Storage on the Grid

Motivated by decreased cost and climate change concerns, penetration of solar photovoltaic (PV) and battery energy storage has been continually increasing. The variability in solar generation assets has led to many challenges for utilities and researchers. Therefore, there is increasing interest in the area of resilience of the grid and contribution from its assets. This paper describes a proposed framework for evaluating the resilience contribution of solar generation and battery storage. The metric provides a quantifiable adaptive capacity measure with uncertainty for the contribution assets proved to the bulk grid. A case study using long and short-term solar generation, with and without battery storage, demonstrates the framework and provides useful insight to the resilience solar and battery storage assets contribute.

14 SOLAR ENERGY↗
DOE OSTI2026

An AI-driven framework for evaluating local and state authorities’ permitting processes

The demand for new energy infrastructure is increasing across the United States, but heterogenous permitting processes and embedded requirements across different local jurisdictions can cause project delays, increase “soft costs,” and hinder developer expansion. This study analyzes the variability in local permitting requirements across the U.S. and develops a quantitative approach to describe their clarity and effectiveness in enabling infrastructure project development. By using an Energy Language Model (ELM), a large language model (LLM) for energy technologies, we systematically gathered permitting information from nearly 300 state-, county-, and city-level documents, creating a structured dataset of requirements and procedures on an unprecedented scale and speed. Our analysis revealed that local (city and county) permitting requirement documents are underrepresented compared to state-level guidance documents, which can impede timely and cost-effective installation of new electric infrastructure. Our validation process showed that the final database has an accuracy of approximately 95%. We, further, created a new quantitative method to score permitting requirements for clarity and efficiency, with electric vehicle supply equipment as an initial use case. The average local permitting document scored a 1.8 out of 5, which we interpret as meaning that half of the requirements developers face when installing electric infrastructure are ambiguous, increasing both cost and time. We also created a “Generalized Permit Process”, highlighting common procedural steps and identifying specific opportunities for municipalities to improve their documentation. This research establishes a systematic and scalable framework for evaluating the complexities of local infrastructure permitting processes by combining LLM-powered data collection and quantitative scoring. The framework enables policymakers and developers to identify and mitigate procedural bottlenecks, with the expectation that these improvements can accelerate application review and approval, reduce project costs, and expedite connection to utility distribution grids. As a foundational approach for streamlining local project development processes, this study’s methods are intended to be extended to a wide range of energy applications.

29 ENERGY PLANNING, POLICY, AND ECONOMY↗
DOE OSTI2023

A Framework to Evaluate the Grid Impacts of EV Fleet Charging Solutions

Growing Electric vehicle (EV) adoption in residential and commercial applications is driving the need for the charging infrastructures required to fulfill the charging needs. The resulting increase in demand for electric power will add a significant load to the electric grid, which could negatively impact the grid operation. EVs are power electronic loads and draw harmonic currents which can lead to a host of power quality issues. Therefore, it is crucial to assess the grid impacts and management of EV charging loads to ensure the reliable operation of the electric grid. In this paper, we develop a framework to help determine power-quality impacts given an EV charging schedule on the system. We compare the performance of unmanaged and managed charging solutions on factors like voltage drop, flicker, harmonic distortion and the peak site load - based on the EV depot in Hazelwood school district. The results allow insights into operation and site design of EV depots.

ADVANCED PROPULSION SYSTEMS,POWER TRANSMISSION AND↗
DOE OSTI2020

Development of a Comparison Framework for Evaluating Environmental Contours of Extreme Sea States

Environmental contours of extreme sea states are often utilized for the purposes of reliability-based offshore design. Many methods have been proposed to estimate environmental contours of extreme sea states, including, but not limited to, the traditional inverse first-order reliability method (I-FORM) and subsequent modifications, copula methods, and Monte Carlo methods. These methods differ in terms of both the methodology selected for defining the joint distribution of sea state parameters and in the method used to construct the environmental contour from the joint distribution. It is often difficult to compare the results of proposed methods to determine which method should be used for a particular application or geographical region. The comparison of the predictions from various contour methods at a single site and across many sites is important to making environmental contours of extreme sea states useful in practice. The goal of this paper is to develop a comparison framework for evaluating methods for developing environmental contours of extreme sea states. This paper develops generalized metrics for comparing the performance of contour methods to one another across a collection of study sites, and applies these metrics and methods to develop conclusions about trends in the wave resource across geographic locations, as demonstrated for a pilot dataset. These proposed metrics and methods are intended to judge the environmental contours themselves relative to other contour methods, and are thus agnostic to a specific device, structure, or field of application. The metrics developed and applied in this paper include measures of predictive accuracy, physical validity, and aggregated temporal performance that can be used to both assess contour methods and provide recommendations for the use of certain methods in various geographical regions. The application and aggregation of the metrics proposed in this paper outline a comparison framework for environmental contour methods that can be applied to support design analysis workflows for offshore structures. This comparison framework could be extended in future work to include additional metrics of interest, potentially including those to address issues pertinent to a specific application area or analysis discipline, such as metrics related to structural response across contour methods or additional physics-based metrics based on wave dynamics.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2025

The Blending ToolKit: A simulation framework for evaluation of galaxy detection and deblending

We present an open source Python library for simulating overlapping (i.e., blended) images of galaxies and performing self-consistent comparisons of detection and deblending algorithms based on a suite of metrics. The package, named Blending Toolkit (BTK), serves as a modular, flexible, easy-to-install, and simple-to-use interface for exploring and analyzing systematic effects related to blended galaxies in cosmological surveys such as the Vera Rubin Observatory Legacy Survey of Space and Time (LSST). BTK has three main components: (1) a set of modules that perform fast image simulations of blended galaxies, using the open source image simulation package GalSim; (2) a module that standardizes the inputs and outputs of existing deblending algorithms; (3) a library of deblending metrics commonly defined in the galaxy deblending literature. In combination, these modules allow researchers to explore the impacts of galaxy blending in cosmological surveys. Additionally, BTK provides researchers who are developing a new deblending algorithm a framework to evaluate algorithm performance and make principled comparisons with existing deblenders. BTK includes a suite of tutorials and comprehensive documentation. The source code is publicly available on GitHub at https://github.com/LSSTDESC/BlendingToolKit.

79 ASTRONOMY AND ASTROPHYSICS↗
DOE OSTI2024

Preventive Power Outage Estimation Based on a Novel Scenario Clustering Strategy

The increasing occurrence of extreme weather events is challenging power grid operation. For extreme weather events, the system operator is responsible for estimating the power outages and scheduling the restoration resources. This paper proposes an outage evaluation framework to identify the possible unserved load profiles, vulnerable areas, and mobile energy adequacy. The outputs of an outage prediction model tool are used to generate numerous faulted line scenarios. Next, each scenario's nodal unserved load profile is obtained by solving a three-phase restoration model that considers repair crews and mobile energy resources (MERs). Then, a novel scenario clustering strategy is developed to cluster the unserved load profiles into multiple representative profiles which the system operator can focus on. Finally, case studies on a distribution system evaluate the damage caused by an extreme weather event and verify the effectiveness of the proposed scenario clustering strategy.

MATHEMATICS AND COMPUTING,POWER TRANSMISSION AND D↗
DOE OSTI2021

A Meta-Level Framework for Evaluating Resilience in Net-Zero Carbon Power Systems with Extreme Weather Events in the United States

Important changes are underway in the U.S. power industry in the way that electricity is sourced, transported, and utilized. Disruption from extreme weather events and cybersecurity events is bringing new scrutiny to power-system resilience. Recognizing the complex social and technical aspects that are involved, this article provides a meta-level framework for coherently evaluating and making decisions about power-system resilience. It does so by examining net-zero carbon strategies with quantitative, qualitative, and integrative dimensions across discrete location-specific systems and timescales. The generalizable framework is designed with a flexibility and logic that allows for refinement to accompany stakeholder review processes and highly localized decision-making. To highlight the framework’s applicability across multiple timescales, processes, and types of knowledge, power system outages are reviewed for extreme weather events, including 2021 and 2011 winter storms that impacted Texas, the 2017 Hurricane Maria that affected Puerto Rico, and a heatwave/wildfire event in California in August 2020. By design, the meta-level framework enables utility decision-makers, regulators, insurers, and communities to analyze and track levels of resilience safeguards for a given system. Future directions to advance an integrated science of resilience in net-zero power systems and the use of this framework are also discussed.

29 ENERGY PLANNING, POLICY, AND ECONOMY↗
DOE OSTI2026

A simulation framework for evaluating electronic order workflows in integrated health records

Electronic health record (EHR) systems are critical to modern healthcare delivery, yet the dynamic workflows that govern electronic order processing remain underexplored. Inefficiencies in these digital pathways can cause delays in care, repetitive workloads, and even patient harm. This study presents a discrete-event simulation framework used to reconstruct and evaluate EHR-based order workflows in a large integrated healthcare system. Using real-world data extracted from the Veterans Health Administration’s Corporate Data Warehouse, the authors mapped order events to standardized state transitions and modeled their progression across different facilities of varying complexity levels. After being calibrated with empirical distributions of transition times and validated against observed time-in-system metrics, the simulation demonstrates close alignment with historical performance. Scenario analyses reveal that resource capacity constraints significantly amplify the impact of electronic order surges, which are reflected in the disproportionate growth in backlogs and processing delays. Adjustments in transition probabilities further increased recirculation and extended workflow paths. Network-based analysis identified Reserved, InProgress, and Completed as structurally critical states that function as hubs within the process network but the transitions in-between also act as major bottlenecks. These results showcased the effectiveness of simulation-based approaches in monitoring EHR order processing performance and evaluating consequences of workflow changes on healthcare network resources planning. The proposed simulation framework provides a scalable data-driven tool to support operational decision-making and improve the efficiency of electronic order management in complex healthcare environments.

Engineering↗
DOE OSTI2021

Geant4Reweight: a framework for evaluating and propagating hadronic interaction uncertainties in Geant4

Geant4Reweight is an open-source C++ framework that allows users to weight tracks produced by the Geant4 particle transport Monte Carlo simulation according to hadron interaction cross section variations and estimate uncertainties in Geant4 interaction models by comparing the simulation’s hadron interaction cross section predictions to data. The ability to weight hadron transport as simulated by Geant4 is crucial to the propagation of systematic uncertainties related to secondary hadronic interactions in current and upcoming neutrino oscillation experiments, including MicroBooNE, NOvA, and DUNE, aswell as hadron test beam experiments such as ProtoDUNE. Here, we provide motivation for weighting hadron tracks in Geant4 in the context of systematic uncertainty propagation, a description of Geant4’s transport simulation technique, and a description of our weighting technique and fitting framework in the momentum range 0–10 GeV/c, which is typical for the hadrons produced by neutrino interactions in these experiments.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2020

A metric framework for evaluating the resilience contribution of hydropower to the grid

This paper will describe a proposed framework for expressing the resilience of hydropower generation and provide initial case studies for three classes of hydropower, run-of-river hydropower, hydropower with reservoirs, and pumped storage hydropower. Hydropower has great flexibility to provide support during and after natural and man-made events that can disrupt critical infrastructure functionality. The concept of the framework provides for consideration of policy and rules, constraints of the water shed and other allocations of water, storage and plant capabilities to produce real and reactive power, and the strength of the delivery network. The paper details a resilience response metric that has inputs of state of storage and plant level constraints on real and reactive power production. Using the definition of resilience, based on maintaining a minimally normal operations, we provide a qualitative assessment of hydropower’s ability to address the various time scales comprising the “R”s of resilience.

13 HYDRO ENERGY↗
DOE OSTI2021

Modeling framework for evaluating the impacts of hydrodynamic pressure on hydrologic exchange fluxes and residence time for a large-scale river section over a long-term period

Quantifying hydrologic exchange fluxes (HEF) at the river and subsurface interface and their residence times (RT) in subsurface are important for managing the water quality and ecosystem health in dynamic river corridor systems. In this study, a modeling framework is developed for coupling the three-dimensional (3D) multi-phase surface, subsurface flow transport, and numerical tracer model for RT in a large-scale river section over a long period. The framework is utilized to evaluate the impacts of hydrodynamic pressure on HEFs and RT for a 30 km section of the Columbia River in Washington State over a three-year period. Based on comparisons between model simulations with and without considering hydrodynamic pressure, we found that hydrodynamic pressure increases the net HEFs by 7% with river gaining water from the subsurface domain, and leads to slight reduction of RT.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2022

Developing a heterogeneous ensemble learning framework to evaluate Alkali-silica reaction damage in concrete using acoustic emission signals

The monitoring and evaluation of Alkali-silica reaction (ASR) damage in concrete structures are required to ensure the serviceability and integrity of concrete infrastructures such as bridges and dams. The innovation of this paper lies in the development of an automatic ASR monitoring and evaluation approach by leveraging acoustic emission (AE) and a heterogeneous ensemble learning framework. Here, in this paper, ASR was monitored by AE sensors attached to a concrete specimen, which was placed in a chamber with high humidity and temperature. The recorded AE signals were filtered and divided by four ASR phases according to signal strength, crack width and expansion strain. A heterogeneous ensemble network including convolutional neural networks (CNN) and random forest models was employed to learn different features from AE signals and classify the AE signals into their corresponding phases. The results suggest that the proposed model has a high performance and classifies the signals into the assigned phases with high accuracy.

42 ENGINEERING↗
DOE OSTI2024

Integrity Enhancing Protocols: Performance and Recommendations for Nuclear Systems

In today’s communication landscape there are multiple technologies and protocols used for communication between end devices. Within security paradigms for these protocols, integrity management is a common goal of system designers. Communication protocols focused on maintaining message integrity can provide assurance that some received data has not been altered or tampered with. While integrity is often coupled with confidentiality in protocol design, this analysis focuses on an evaluation of only integrity protocols. This report outlines various ways message integrity may be preserved with respect to high performance operational technology (OT) systems. It describes a series of experiments and an evaluation framework used to evaluate the performance of the identified integrity approaches regarding common system design goals. Finally, it addresses the testing environment utilized and closes the report with a summary of experimental results.

22 GENERAL STUDIES OF NUCLEAR REACTORS↗