Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “Evaluation Framework”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 73 records · Page 4
All sourcesNASA NTRSDOE OSTI
DOE OSTI2025

The Blending ToolKit: A simulation framework for evaluation of galaxy detection and deblending

We present an open source Python library for simulating overlapping (i.e., blended) images of galaxies and performing self-consistent comparisons of detection and deblending algorithms based on a suite of metrics. The package, named Blending Toolkit (BTK), serves as a modular, flexible, easy-to-install, and simple-to-use interface for exploring and analyzing systematic effects related to blended galaxies in cosmological surveys such as the Vera Rubin Observatory Legacy Survey of Space and Time (LSST). BTK has three main components: (1) a set of modules that perform fast image simulations of blended galaxies, using the open source image simulation package GalSim; (2) a module that standardizes the inputs and outputs of existing deblending algorithms; (3) a library of deblending metrics commonly defined in the galaxy deblending literature. In combination, these modules allow researchers to explore the impacts of galaxy blending in cosmological surveys. Additionally, BTK provides researchers who are developing a new deblending algorithm a framework to evaluate algorithm performance and make principled comparisons with existing deblenders. BTK includes a suite of tutorials and comprehensive documentation. The source code is publicly available on GitHub at https://github.com/LSSTDESC/BlendingToolKit.

79 ASTRONOMY AND ASTROPHYSICS↗
DOE OSTI2024

Preventive Power Outage Estimation Based on a Novel Scenario Clustering Strategy

The increasing occurrence of extreme weather events is challenging power grid operation. For extreme weather events, the system operator is responsible for estimating the power outages and scheduling the restoration resources. This paper proposes an outage evaluation framework to identify the possible unserved load profiles, vulnerable areas, and mobile energy adequacy. The outputs of an outage prediction model tool are used to generate numerous faulted line scenarios. Next, each scenario's nodal unserved load profile is obtained by solving a three-phase restoration model that considers repair crews and mobile energy resources (MERs). Then, a novel scenario clustering strategy is developed to cluster the unserved load profiles into multiple representative profiles which the system operator can focus on. Finally, case studies on a distribution system evaluate the damage caused by an extreme weather event and verify the effectiveness of the proposed scenario clustering strategy.

MATHEMATICS AND COMPUTING,POWER TRANSMISSION AND D↗
NASA NTRS2009

Towards a Framework for Evaluating and Comparing Diagnosis Algorithms

Diagnostic inference involves the detection of anomalous system behavior and the identification of its cause, possibly down to a failed unit or to a parameter of a failed unit. Traditional approaches to solving this problem include expert/rule-based, model-based, and data-driven methods. Each approach (and various techniques within each approach) use different representations of the knowledge required to perform the diagnosis. The sensor data is expected to be combined with these internal representations to produce the diagnosis result. In spite of the availability of various diagnosis technologies, there have been only minimal efforts to develop a standardized software framework to run, evaluate, and compare different diagnosis technologies on the same system. This paper presents a framework that defines a standardized representation of the system knowledge, the sensor data, and the form of the diagnosis results and provides a run-time architecture that can execute diagnosis algorithms, send sensor data to the algorithms at appropriate time steps from a variety of sources (including the actual physical system), and collect resulting diagnoses. We also define a set of metrics that can be used to evaluate and compare the performance of the algorithms, and provide software to calculate the metrics.

Kurtoglu, Tolga↗
DOE OSTI2021

A Meta-Level Framework for Evaluating Resilience in Net-Zero Carbon Power Systems with Extreme Weather Events in the United States

Important changes are underway in the U.S. power industry in the way that electricity is sourced, transported, and utilized. Disruption from extreme weather events and cybersecurity events is bringing new scrutiny to power-system resilience. Recognizing the complex social and technical aspects that are involved, this article provides a meta-level framework for coherently evaluating and making decisions about power-system resilience. It does so by examining net-zero carbon strategies with quantitative, qualitative, and integrative dimensions across discrete location-specific systems and timescales. The generalizable framework is designed with a flexibility and logic that allows for refinement to accompany stakeholder review processes and highly localized decision-making. To highlight the framework’s applicability across multiple timescales, processes, and types of knowledge, power system outages are reviewed for extreme weather events, including 2021 and 2011 winter storms that impacted Texas, the 2017 Hurricane Maria that affected Puerto Rico, and a heatwave/wildfire event in California in August 2020. By design, the meta-level framework enables utility decision-makers, regulators, insurers, and communities to analyze and track levels of resilience safeguards for a given system. Future directions to advance an integrated science of resilience in net-zero power systems and the use of this framework are also discussed.

29 ENERGY PLANNING, POLICY, AND ECONOMY↗
DOE OSTI2026

A simulation framework for evaluating electronic order workflows in integrated health records

Electronic health record (EHR) systems are critical to modern healthcare delivery, yet the dynamic workflows that govern electronic order processing remain underexplored. Inefficiencies in these digital pathways can cause delays in care, repetitive workloads, and even patient harm. This study presents a discrete-event simulation framework used to reconstruct and evaluate EHR-based order workflows in a large integrated healthcare system. Using real-world data extracted from the Veterans Health Administration’s Corporate Data Warehouse, the authors mapped order events to standardized state transitions and modeled their progression across different facilities of varying complexity levels. After being calibrated with empirical distributions of transition times and validated against observed time-in-system metrics, the simulation demonstrates close alignment with historical performance. Scenario analyses reveal that resource capacity constraints significantly amplify the impact of electronic order surges, which are reflected in the disproportionate growth in backlogs and processing delays. Adjustments in transition probabilities further increased recirculation and extended workflow paths. Network-based analysis identified Reserved, InProgress, and Completed as structurally critical states that function as hubs within the process network but the transitions in-between also act as major bottlenecks. These results showcased the effectiveness of simulation-based approaches in monitoring EHR order processing performance and evaluating consequences of workflow changes on healthcare network resources planning. The proposed simulation framework provides a scalable data-driven tool to support operational decision-making and improve the efficiency of electronic order management in complex healthcare environments.

Engineering↗
DOE OSTI2021

Geant4Reweight: a framework for evaluating and propagating hadronic interaction uncertainties in Geant4

Geant4Reweight is an open-source C++ framework that allows users to weight tracks produced by the Geant4 particle transport Monte Carlo simulation according to hadron interaction cross section variations and estimate uncertainties in Geant4 interaction models by comparing the simulation’s hadron interaction cross section predictions to data. The ability to weight hadron transport as simulated by Geant4 is crucial to the propagation of systematic uncertainties related to secondary hadronic interactions in current and upcoming neutrino oscillation experiments, including MicroBooNE, NOvA, and DUNE, aswell as hadron test beam experiments such as ProtoDUNE. Here, we provide motivation for weighting hadron tracks in Geant4 in the context of systematic uncertainty propagation, a description of Geant4’s transport simulation technique, and a description of our weighting technique and fitting framework in the momentum range 0–10 GeV/c, which is typical for the hadrons produced by neutrino interactions in these experiments.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2020

A metric framework for evaluating the resilience contribution of hydropower to the grid

This paper will describe a proposed framework for expressing the resilience of hydropower generation and provide initial case studies for three classes of hydropower, run-of-river hydropower, hydropower with reservoirs, and pumped storage hydropower. Hydropower has great flexibility to provide support during and after natural and man-made events that can disrupt critical infrastructure functionality. The concept of the framework provides for consideration of policy and rules, constraints of the water shed and other allocations of water, storage and plant capabilities to produce real and reactive power, and the strength of the delivery network. The paper details a resilience response metric that has inputs of state of storage and plant level constraints on real and reactive power production. Using the definition of resilience, based on maintaining a minimally normal operations, we provide a qualitative assessment of hydropower’s ability to address the various time scales comprising the “R”s of resilience.

13 HYDRO ENERGY↗
DOE OSTI2021

Modeling framework for evaluating the impacts of hydrodynamic pressure on hydrologic exchange fluxes and residence time for a large-scale river section over a long-term period

Quantifying hydrologic exchange fluxes (HEF) at the river and subsurface interface and their residence times (RT) in subsurface are important for managing the water quality and ecosystem health in dynamic river corridor systems. In this study, a modeling framework is developed for coupling the three-dimensional (3D) multi-phase surface, subsurface flow transport, and numerical tracer model for RT in a large-scale river section over a long period. The framework is utilized to evaluate the impacts of hydrodynamic pressure on HEFs and RT for a 30 km section of the Columbia River in Washington State over a three-year period. Based on comparisons between model simulations with and without considering hydrodynamic pressure, we found that hydrodynamic pressure increases the net HEFs by 7% with river gaining water from the subsurface domain, and leads to slight reduction of RT.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2022

Developing a heterogeneous ensemble learning framework to evaluate Alkali-silica reaction damage in concrete using acoustic emission signals

The monitoring and evaluation of Alkali-silica reaction (ASR) damage in concrete structures are required to ensure the serviceability and integrity of concrete infrastructures such as bridges and dams. The innovation of this paper lies in the development of an automatic ASR monitoring and evaluation approach by leveraging acoustic emission (AE) and a heterogeneous ensemble learning framework. Here, in this paper, ASR was monitored by AE sensors attached to a concrete specimen, which was placed in a chamber with high humidity and temperature. The recorded AE signals were filtered and divided by four ASR phases according to signal strength, crack width and expansion strain. A heterogeneous ensemble network including convolutional neural networks (CNN) and random forest models was employed to learn different features from AE signals and classify the AE signals into their corresponding phases. The results suggest that the proposed model has a high performance and classifies the signals into the assigned phases with high accuracy.

42 ENGINEERING↗
DOE OSTI2024

Integrity Enhancing Protocols: Performance and Recommendations for Nuclear Systems

In today’s communication landscape there are multiple technologies and protocols used for communication between end devices. Within security paradigms for these protocols, integrity management is a common goal of system designers. Communication protocols focused on maintaining message integrity can provide assurance that some received data has not been altered or tampered with. While integrity is often coupled with confidentiality in protocol design, this analysis focuses on an evaluation of only integrity protocols. This report outlines various ways message integrity may be preserved with respect to high performance operational technology (OT) systems. It describes a series of experiments and an evaluation framework used to evaluate the performance of the identified integrity approaches regarding common system design goals. Finally, it addresses the testing environment utilized and closes the report with a summary of experimental results.

22 GENERAL STUDIES OF NUCLEAR REACTORS↗
DOE OSTI2023

Advancing the Theory of Nuclear Data Evaluations [Abstract]

We present recent advances in the R-matrix formalism as well as the Bayesian evaluation framework for improved nuclear data evaluations. The advances in the R matrix formalism include: 1) direct processes, 2) doorway, as well as multistep, processes, and 3) various forms of the Reich-Moore approximation for eliminated capture channels. Furthermore, to address unreasonably small posterior uncertainties often encountered in nuclear data evaluations of large data sets using the conventional form of the Bayes’ theorem, we introduce imperfections (of the data or the model) as a formal evaluation tool for taming the evaluated uncertainties in harmony with Bayes’ theorem. These theoretical advances were motivated by the nuclear data evaluations of differential resolved resonance cross section data using the code SAMMY, as well as the integral benchmark experiments using the SCALE code system, being performed at Oak Ridge National Laboratory for the Nuclear Criticality Safety Program. Some pedagogical applications of the new formalism, as well as a snapshot of the SAMMY modernization efforts, will be presented.

73 NUCLEAR PHYSICS AND RADIATION PHYSICS↗
DOE OSTI2026

A coupled hydrologic-agroeconomic modeling framework to evaluate adaptive irrigation strategies under groundwater withdrawal restrictions

Growing groundwater scarcity requires integrated tools to capture interactions among hydrology, agricultural production, markets, and land use. This study presents an iterative modeling framework that couples hydrologic, crop-yield, and economic models to capture two-way feedback among water availability, agricultural production, and market responses under groundwater constraints. The primary goal of this paper is to describe the methodological development of the coupled framework and demonstrate the significance of iterative model interaction. Applied to the western United States, we evaluated adaptive responses to restricting groundwater use beyond recharge levels, represented through changes in irrigation management and expansion or shrinkage of crop markets through land reallocation. Results demonstrate that the iterative coupling converges to stable equilibrium responses within 10 iterations. At equilibrium, deficit irrigation emerges as the dominant adaptation strategy in California, with irrigation levels stabilizing at approximately 70% of full irrigation demand, while Arizona and New Mexico experience stronger yield sensitivities. Early iterations produce commodity price increases of up to 10% for fruit and vegetable crops; however, these responses moderate as land allocation and production patterns adjust across regions. Deficit irrigation and spatial reallocation of irrigated land partially offset production losses, with variability observed across different states: California maintains yields primarily via deficit irrigation, whereas Arizona and New Mexico will rely mainly on reducing irrigated area to absorb the shock. By capturing feedback between biophysical and economic processes, this approach highlights how irrigation strategies and land-use decisions evolve under water stress and provides a transferable platform for evaluating water management policies.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2023

Decision-Making Framework to Evaluate Opportunities for Recovery of Rare Earth Elements and Critical Minerals in Produced Water Networks

Oil and gas development activities require a significant amount of water. Given the environmental impacts associated with high consumption and subsequent disposal of this water, there is a need for strategies to support effective reuse of, treatment of, and/or resource recovery from this produced water. These streams can contain appreciable concentrations of rare earth elements (REE) and critical minerals (CM) (e.g., Lithium), which are critical for many important applications in electronics, technology, manufacturing, energy, and medicine [1]. Along with REE/CM present in produced water, other waste streams such as fly ash from coal-fired power plants and acid mine discharge present other potential sources for recovery of REE/CM [2-4]. And with increases in demand for REE/CM, the need for recovery from wastewater streams is becoming even more important. There is a need to develop effective decision-making tools to evaluate the economic and environmental potential for REE/CM recovery from produced water networks while supporting the needs of drilling and hydraulic fracturing activities. In this work, we present our progress in developing a decision-making framework to efficiently design and operate produced water networks for REE/CM recovery. Our approach considers REE/CM recovery from produced water streams alongside other potential wastewater sources. The models consider the location and capacity of storage and decentralized treatment processes as well as the design and operation of pipeline and transportation networks to connect different sources and effectively schedule inventories, reuse, and recovery opportunities. We focus on the evaluation of opportunities for REE/CM recovery with treatment constraints on composition and flow. We demonstrate this framework with an illustrative case study that features diverse production sites, storage facilities, transportation, and treatment sites. This proposed framework will be deployed in an open-source software package that is compatible with the PARETO framework to deliver analysis and decision-making tools that allow stakeholders to quantitatively evaluate potential opportunities for REE/CM recovery in produced water networks.

Pulsipher, Joshua↗
NASA NTRS2004

Integration of Propulsion-Airframe-Aeroacoustic Technologies and Design Concepts for a Quiet Blended-Wing-Body Transport

This paper summarizes the results of studies undertaken to investigate revolutionary propulsion-airframe configurations that have the potential to achieve significant noise reductions over present-day commercial transport aircraft. Using a 300 passenger Blended-Wing-Body (BWB) as a baseline, several alternative low-noise propulsion-airframe-aeroacoustic (PAA) technologies and design concepts were investigated both for their potential to reduce the overall BWB noise levels, and for their impact on the weight, performance, and cost of the vehicle. Two evaluation frameworks were implemented for the assessments. The first was a Multi-Attribute Decision Making (MADM) process that used a Pugh Evaluation Matrix coupled with the Technique for Order Preference by Similarity to Ideal Solution (TOPSIS). This process provided a qualitative evaluation of the PAA technologies and design concepts and ranked them based on how well they satisfied chosen design requirements. From the results of the evaluation, it was observed that almost all of the PAA concepts gave the BWB a noise benefit, but degraded its performance. The second evaluation framework involved both deterministic and probabilistic systems analyses that were performed on a down-selected number of BWB propulsion configurations incorporating the PAA technologies and design concepts. These configurations included embedded engines with Boundary Layer Ingesting Inlets, Distributed Exhaust Nozzles installed on podded engines, a High Aspect Ratio Rectangular Nozzle, Distributed Propulsion, and a fixed and retractable aft airframe extension. The systems analyses focused on the BWB performance impacts of each concept using the mission range as a measure of merit. Noise effects were also investigated when enough information was available for a tractable analysis. Some tentative conclusions were drawn from the results. One was that the Boundary Layer Ingesting Inlets provided improvements to the BWB's mission range, by increasing the propulsive efficiency at cruise, and therefore offered a means to offset performance penalties imposed by some of the advanced PAA configurations. It was also found that the podded Distributed Exhaust Nozzle configuration imposed high penalties on the mission range and the need for substantial synergistic performance enhancements from an advanced integration scheme was identified. The High Aspect Ratio Nozzle showed inconclusive noise results and posed significant integration difficulties. Distributed Propulsion, in general, imposed performance penalties but may offer some promise for noise reduction from jet-to-jet shielding effects. Finally, a retractable aft airframe extension provided excellent noise reduction for a modest decrease in range.

Hill, G. A.↗
DOE OSTI2026

From Bricks to Clicks: Mapping the White Space in Building Innovation

It is a critical national imperative to transform the buildings sector, yet innovation is impeded by deployment failures that leave promising technologies stranded. Conventional market reports and techno-economic analysis provide an insufficient understanding of markets and resource allocation for emerging building technologies. They omit crucial commercialization factors such as ecosystem maturity and adoption friction, where the coordinated participation of a network of suppliers, contractors, financiers, regulators, and integrators is required to scale solutions. This study addresses these gaps by introducing an evaluation framework grounded in front-line data from six years of the DOE's IMPEL incubator, comprising experience from 300 building-sector innovators and the adjacent, complex ecosystem. Our methodology synthesizes top-down market analysis with bottom-up, practitioner-level data across five megatrends: (M1) Affordable materials and industrialized construction; (M2) Healthy and efficient mechanical systems; (M3) Intelligent building operations; (M4) Buildings as grid assets; and (M5) High-density power and cooling for data centers and therein identify twelve "white space" technology opportunities. Next, we develop a multi-criteria scoring rubric to rank these opportunities based on parameters, i.e., Affordability, Quality of Life, Reliability, and Security, yielding composite ‘Demand’ and ‘Maturity’ indices. Our results indicate that the most significant white spaces may not be incremental products but a new class of ‘Ecosystem Enablers’, such as logistics platforms, orchestration layers, and automated compliance software that solve structural deployment gaps. This paper summarizes this transparent, evidence-based, practitioner-informed evaluation framework for policymakers and investors to re-evaluate policy and resource allocation and unlock scalable market transformation.

Singh, Reshma↗
NASA NTRS2014

"Efficiency Space" - A Framework for Evaluating Joint Evaporation and Runoff Behavior

At the land surface, higher soil moisture levels generally lead to both increased evaporation for a given amount of incoming radiation (increased evaporation efficiency) and increased runoff for a given amount of precipitation (increased runoff efficiency). Evaporation efficiency and runoff efficiency can thus be said to vary with each other, motivating the development of a unique hydroclimatic analysis framework. Using a simple water balance model fitted, in different experiments, with a wide variety of functional forms for evaporation and runoff efficiency, we transform net radiation and precipitation fields into fields of streamflow that can be directly evaluated against observations. The optimal combination of the functional forms the combination that produces the most skillful stream-flow simulations provides an indication for how evaporation and runoff efficiencies vary with each other in nature, a relationship that can be said to define the overall character of land surface hydrological processes, at least to first order. The inferred optimal relationship is represented herein as a curve in efficiency space and should be valuable for the evaluation and development of GCM-based land surface models, which by this measure are often found to be suboptimal.

GCM↗
DOE OSTI2022

A framework for evaluating power electronic systems for grid integration

As power electronic systems (PESs) grow increasingly complex, integrating and testing these systems in the development stages is becoming more crucial and more challenging. New modes and resource integration strategies are in development and must be tested before pilot demonstrations and mass production. This paper proposes a framework for testing of PES technologies including early validation with integration of simulated resources. The framework is demonstrated to work in two example configurations as proof of principle.

Dean, Ben↗