Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “evaluation framework”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 55 records · Page 3
All sourcesNASA NTRSDOE OSTI
DOE OSTI2025

Evaluation of Seismic Artificial Intelligence with Uncertainty

Artificial intelligence has transformed the seismic community with deep learning models (DLMs) that are trained to complete specific tasks within workflows. However, there is still a lack of robust evaluation frameworks for evaluating and comparing DLMs. Here, we address this gap by designing an evaluation framework that jointly incorporates two crucial aspects: performance uncertainty and learning efficiency. To target these aspects, we meticulously construct the training, validation, and test splits using a clustering method tailored to seismic data and enact an expansive training design to segregate performance uncertainty arising from stochastic training processes and random data sampling. The framework’s ability to guard against misleading declarations of model superiority is demonstrated through the evaluation of PhaseNet (Zhu and Beroza, 2018), a popular seismic phase picking DLM, under three training approaches. Our framework helps practitioners choose the best model for their problem and set performance expectations by explicitly analyzing model performance with uncertainty at varying budgets of training data.

58 GEOSCIENCES↗
NASA NTRS2023

A Framework for Evaluating Distributed Electric Propulsion on the SUSAN Electrofan Aircraft

This work presents a framework for evaluating models and algorithms for Distributed Electric Propulsion (DEP) on the SUSAN Electrofan Aircraft. Throughout the development of the SUSAN aircraft, the performance of various configurations of the aircraft will need to be analyzed. However, the static behavior alone is not sufficient to describe the performance of these configurations. Therefore, simulation with fully integrated subsystem models is required. The proposed framework considers the vehicle aerodynamic, propulsion, and control subsystems. The presented framework automatically generates control laws for any vehicle configuration in response to changes in these subsystems. To compare these different vehicle configurations, various time and frequency domain performance metrics are compared. Three different system modifications are used as cases to evaluate this framework. The first modification integrates the propulsion control system with the flight controller to enable differential thrust without stalling the main engine. This evaluation case is used to validate the framework for aircraft configurations with coupled subsystems. The second modification compares the effect of the vertical tail size on open and closed loop performance. This evaluation case is used to validate the framework for controlling different configurations and tuning towards comparable closed loop performance despite changes to the aircraft's aerodynamic model. The third modification implements two different control allocation schemes. This evaluation case demonstrates the framework's ability to evaluate allocation modifications needed to take advantage of DEP. The first evaluation case is used to show that controller integration enables differential thrust, improving realized wingfan bandwidth by up to 40\% in simulation. The second evaluation case demonstrates that the framework can stabilize the reduced tail size aircraft with closed loop control. The third evaluation case demonstrates that a pseudoinverse control allocation scheme improves lateral velocity settling time by approximately 17~seconds over a symmetric-thrust allocation. These cases show that the framework is useful for evaluating the performance of integrated system designs, enabling analyses of new models and algorithms for the SUSAN distributed electric propulsion vehicle.

Nicholas C Ogden↗
DOE OSTI2023

Internal consistency and diversity scenario development: A comparative framework to evaluate energy model scenarios

Energy modeling frameworks and scenario analysis help us explore the potential impact of our actions and uncertainties in future energy systems. Despite their importance, there is no systematic procedure for evaluating the scenario development process. In a literature review, we identify two core elements of the scenario development process: internal consistency and diversity which are oftentimes missing from scenarios. Here, to address this gap, we create the Internal consistency and Diversity Scenario Development (IDSD) comparative framework which aims to assess the feasibility and diversity of scenarios for a given energy model. With this framework, we review commonly used energy models and demonstrate our framework on their scenarios. The IDSD comparative framework can serve several purposes absent from previous scenario development work by aiding energy modelers and report writers in crafting high-quality scenarios. First, the IDSD is a reflective tool which can improve the quality of the scenario development process, enabling a comparative assessment of energy models and scenarios. Second, the IDSD can provide guidance to modeling frameworks with existing scenarios and those still in development; this feedback will enable modelers to improve the development and the communication of the limitations of their scenarios. Third, this study has highlighted areas for improvement in the scenario development of some commonly used energy model frameworks. Finally, there is a complete lack of explanation regarding the stakeholder selection process. Addressing these identified items could increase opportunities for advanced energy technology uptake and improve our options for achieving a more resilient energy system.

29 ENERGY PLANNING, POLICY, AND ECONOMY↗
DOE OSTI2023

Generalized Bayesian Framework for Evaluation of Integral Benchmark Experiments [Slides]

This presentation is on generalized Bayesian framework for evaluation of integral benchmark experiments. This presentation starts off with assumptions and approximations used with Bayes Theorem. then an overview of approximations used by ORNL codes, and Generalized Bayesian Monto Carlo (GBMC). The presentation then details out a precise framework. This presentation then concludes with considerations.

97 MATHEMATICS AND COMPUTING↗
NASA NTRS1998

Evaluation of Frameworks for HSCT Design Optimization

This report is an evaluation of engineering frameworks that could be used to augment, supplement, or replace the existing FIDO 3.5 (Framework for Interdisciplinary Design and Optimization Version 3.5) framework. The report begins with the motivation for this effort, followed by a description of an "ideal" multidisciplinary design and optimization (MDO) framework. The discussion then turns to how each candidate framework stacks up against this ideal. This report ends with recommendations as to the "best" frameworks that should be down-selected for detailed review.

Krishnan, Ramki↗
DOE OSTI2026

A Benchmarking Framework for Evaluating Large Language Model Capabilities in Nuclear Reactor Safety Applications

Large language models (LLMs) are increasingly capable of answering technical questions, synthesizing domain knowledge, and supporting engineering workflows. For nuclear science and engineering, these capabilities require careful, domain-specific evaluation before they can be credibly incorporated into safety-related activities, regulatory review, or technical decision support. This paper presents preliminary results from benchmarking framework for evaluating LLM capabilities in nuclear contexts. The framework is organized into three evaluation categories: nuclear fundamentals, general dual-use knowledge, and plant specific knowledge. These categories are intended to distinguish general nuclear engineering competence from broader technical reasoning and more context-dependent nuclear knowledge. Initial evaluations focus on nuclear fundamentals using questions representative of the knowledge expected of a nuclear professional engineer. Results indicate that contemporary frontier models perform at a high level and substantially exceed the performance of older model generations, with some models approaching saturation of the current benchmark. These findings suggest both the rapid improvement of LLM capabilities in specialized technical domains and the need for more discriminating evaluation methods. The paper presents the benchmark structure, preliminary model-comparison results, and ongoing work. This work supports development of verifiable, responsible, and safety-conscious methods for assessing AI systems in nuclear engineering applications.

21 - SPECIFIC NUCLEAR REACTORS AND ASSOCIATED PLAN↗
DOE OSTI2024

​​Integrity Enhancing Protocols (IEP) Evaluation Framework​

​Protocols play an essential role in Advance Reactor systems. A diverse set of protocols are available to these reactors. Advanced Reactors benefit from technologies that can minimize their resource utilization and costs. Evaluation frameworks are often used when assessing protocols and processes related to cryptographic security systems. The following report discusses the various characteristics associated with these protocol evaluation frameworks, and derives a novel evaluative framework.

22 GENERAL STUDIES OF NUCLEAR REACTORS↗
DOE OSTI2024

A Pathway Analysis Framework for Evaluating the Economic and Environmental Viability of Biomass-Based Plastic Production

Plastic production from fossil feedstocks (e.g., naphtha, coal, and natural gas) is not sustainable and causes known environmental impacts such as global warming. A possible solution is to shift production pathways to use biomass, which is a sustainable feedstock that can sequester atmospheric carbon dioxide. This study presents an optimization-based pathway analysis framework for evaluating the carbon footprints of the production of mainstream plastics from biomass and fossil feedstocks. We use the modeling framework to quickly navigate complex interdependencies that exist between the production pathways of different plastics and to determine pathways of minimum production cost under a range of carbon pricing scenarios. The framework interprets carbon prices as an exogenous taxation scheme or an endogenous negative value perceived by producers. The proposed approach reveals the biomass feedstock quantities needed to displace fossil counterparts and the plastics and technologies that should be prioritized. The framework can also be used for evaluating system-wide trade-offs between production costs and carbon footprints that arise from pathway interdependencies. We also evaluate hidden environmental impacts associated with the large-scale use of biomass as a feedstock, such as land use and water eutrophication that results from a significant increase in fertilizer use. Therefore, it is important to highlight that there are trade-offs between decarbonization and other environmental issues. Here, the proposed framework provides an integrative platform for basic techno-economic and life-cycle data that can be used for analyzing diverse scenarios and determining necessary technology targets (e.g., yields, footprints, and costs) to achieve required levels of decarbonization.

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗
NASA NTRS1993

Sensor fusion display evaluation using information integration models in enhanced/synthetic vision applications

Based on existing integration models in the psychological literature, an evaluation framework is developed to assess sensor fusion displays as might be implemented in an enhanced/synthetic vision system. The proposed evaluation framework for evaluating the operator's ability to use such systems is a normative approach: The pilot's performance with the sensor fusion image is compared to models' predictions based on the pilot's performance when viewing the original component sensor images prior to fusion. This allows for the determination as to when a sensor fusion system leads to: poorer performance than one of the original sensor displays, clearly an undesirable system in which the fused sensor system causes some distortion or interference; better performance than with either single sensor system alone, but at a sub-optimal level compared to model predictions; optimal performance compared to model predictions; or, super-optimal performance, which may occur if the operator were able to use some highly diagnostic 'emergent features' in the sensor fusion display, which were unavailable in the original sensor displays.

Foyle, David C.↗
DOE OSTI2023

Generalized Bayesian Framework for Evaluation of Integral Benchmark Experiments

A recently published generalized Bayesian optimization framework has provided a way to retract any or all of the three common assumptions underlying the conventional Generalized Linear Least Squares (GLLS) optimization method based on the concepts introduced in Ref. [2]. These assumptions are: 1. Perfection: The model used for data evaluation and the prior probability distribution function (PDF) of generalized data are perfect. 2. Normality: The prior and posterior PDF are normal. 3. Linearity: The model is linear. In this work we outline how the framework in [1] could be directly adopted for improved evaluation of nuclear criticality integral benchmark experiments (IBEs) by: 1. Removing the first assumption alone by utilizing the concept of imperfections introduced in [1] to enable evaluation in the presence of discrepancies between the data and model or of missing covariance information by a GLLS method that will be seen as a generalization of the conventional GLLS method employed by the TSURFER code, and by 2. Removing the remaining two assumptions by implementing a Markov Chain Monte Carlo method for computation of the posterior PDF in the SAMPLER code, where TSURFER and SAMPLER are the uncertainty quantification (UQ) codes for IBEs in the SCALE code system based on the GLLS and the stochastic method, respectively. The graphic in Figure 1 categorizes the methods discussed in terms of the assumptions that they employ to determine posterior PDFs.

73 NUCLEAR PHYSICS AND RADIATION PHYSICS↗
DOE OSTI2021

Generalized Bayesian Framework for Evaluation of Integral Benchmark Experiments

A recently published generalized Bayesian optimization framework has provided a way to retract any or all of the three common assumptions underlying the conventional Generalized Linear Least Squares (GLLS) optimization method based on the concepts introduced in reference two. These assumptions are: 1. Perfection: The model used for data evaluation and the prior probability distribution function (PDF) of generalized* data are perfect; 2. Normality: The prior and posterior PDF are normal; and 3. Linearity: The model is linear. In this work we outline how the framework in 1 could be directly adopted for improved evaluation of nuclear criticality integral benchmark experiments (IBEs) by: 1. Removing the first assumption alone by utilizing the concept of imperfections introduced in 1 to enable evaluation in the presence of discrepancies between the data and model or of missing covariance information by a GLLS method that will be seen as a generalization of the conventional GLLS method employed by the TSURFER code, and by 2. Removing the remaining two assumptions by implement- ing a Markov Chain Monte Carlo method for computation of the posterior PDF in the SAMPLER code, where TSURFER and SAMPLER are the uncertainty quantification (UQ) codes for IBEs in the SCALE code system based on the GLLS, and the stochastic method, respectively.

73 NUCLEAR PHYSICS AND RADIATION PHYSICS↗
DOE OSTI2020

A Framework for Evaluating the Resilience Contribution of Solar PV and Battery Storage on the Grid

Motivated by decreased cost and climate change concerns, penetration of solar photovoltaic (PV) and battery energy storage has been continually increasing. The variability in solar generation assets has led to many challenges for utilities and researchers. Therefore, there is increasing interest in the area of resilience of the grid and contribution from its assets. This paper describes a proposed framework for evaluating the resilience contribution of solar generation and battery storage. The metric provides a quantifiable adaptive capacity measure with uncertainty for the contribution assets proved to the bulk grid. A case study using long and short-term solar generation, with and without battery storage, demonstrates the framework and provides useful insight to the resilience solar and battery storage assets contribute.

14 SOLAR ENERGY↗
NASA NTRS2017

SpaceCubeX: A Framework for Evaluating Hybrid Multi-Core CPU FPGA DSP Architectures

The SpaceCubeX project is motivated by the need for high performance, modular, and scalable on-board processing to help scientists answer critical 21st century questions about global climate change, air quality, ocean health, and ecosystem dynamics, while adding new capabilities such as low-latency data products for extreme event warnings. These goals translate into on-board processing throughput requirements that are on the order of 100-1,000 more than those of previous Earth Science missions for standard processing, compression, storage, and downlink operations. To study possible future architectures to achieve these performance requirements, the SpaceCubeX project provides an evolvable testbed and framework that enables a focused design space exploration of candidate hybrid CPU/FPGA/DSP processing architectures. The framework includes ArchGen, an architecture generator tool populated with candidate architecture components, performance models, and IP cores, that allows an end user to specify the type, number, and connectivity of a hybrid architecture. The framework requires minimal extensions to integrate new processors, such as the anticipated High Performance Spaceflight Computer (HPSC), reducing time to initiate benchmarking by months. To evaluate the framework, we leverage a wide suite of high performance embedded computing benchmarks and Earth science scenarios to ensure robust architecture characterization. We report on our projects Year 1 efforts and demonstrate the capabilities across four simulation testbed models, a baseline SpaceCube 2.0 system, a dual ARM A9 processor system, a hybrid quad ARM A53 and FPGA system, and a hybrid quad ARM A53 and DSP system.

Hybrid Flight Architectures↗
DOE OSTI2026

An AI-driven framework for evaluating local and state authorities’ permitting processes

The demand for new energy infrastructure is increasing across the United States, but heterogenous permitting processes and embedded requirements across different local jurisdictions can cause project delays, increase “soft costs,” and hinder developer expansion. This study analyzes the variability in local permitting requirements across the U.S. and develops a quantitative approach to describe their clarity and effectiveness in enabling infrastructure project development. By using an Energy Language Model (ELM), a large language model (LLM) for energy technologies, we systematically gathered permitting information from nearly 300 state-, county-, and city-level documents, creating a structured dataset of requirements and procedures on an unprecedented scale and speed. Our analysis revealed that local (city and county) permitting requirement documents are underrepresented compared to state-level guidance documents, which can impede timely and cost-effective installation of new electric infrastructure. Our validation process showed that the final database has an accuracy of approximately 95%. We, further, created a new quantitative method to score permitting requirements for clarity and efficiency, with electric vehicle supply equipment as an initial use case. The average local permitting document scored a 1.8 out of 5, which we interpret as meaning that half of the requirements developers face when installing electric infrastructure are ambiguous, increasing both cost and time. We also created a “Generalized Permit Process”, highlighting common procedural steps and identifying specific opportunities for municipalities to improve their documentation. This research establishes a systematic and scalable framework for evaluating the complexities of local infrastructure permitting processes by combining LLM-powered data collection and quantitative scoring. The framework enables policymakers and developers to identify and mitigate procedural bottlenecks, with the expectation that these improvements can accelerate application review and approval, reduce project costs, and expedite connection to utility distribution grids. As a foundational approach for streamlining local project development processes, this study’s methods are intended to be extended to a wide range of energy applications.

29 ENERGY PLANNING, POLICY, AND ECONOMY↗
DOE OSTI2023

A Framework to Evaluate the Grid Impacts of EV Fleet Charging Solutions

Growing Electric vehicle (EV) adoption in residential and commercial applications is driving the need for the charging infrastructures required to fulfill the charging needs. The resulting increase in demand for electric power will add a significant load to the electric grid, which could negatively impact the grid operation. EVs are power electronic loads and draw harmonic currents which can lead to a host of power quality issues. Therefore, it is crucial to assess the grid impacts and management of EV charging loads to ensure the reliable operation of the electric grid. In this paper, we develop a framework to help determine power-quality impacts given an EV charging schedule on the system. We compare the performance of unmanaged and managed charging solutions on factors like voltage drop, flicker, harmonic distortion and the peak site load - based on the EV depot in Hazelwood school district. The results allow insights into operation and site design of EV depots.

ADVANCED PROPULSION SYSTEMS,POWER TRANSMISSION AND↗
NASA NTRS2015

The AgMIP Framework to Evaluate Agricultural Pathways

This talk will describe the community and research framework that AgMIP has built to enable evidence-based adaptation investment. We provide expertise on the ground and connect various disciplines in order to allow specific adaptations to be evaluated for their biophysical and socio-economic ramifications.

agriculture↗
DOE OSTI2020

Development of a Comparison Framework for Evaluating Environmental Contours of Extreme Sea States

Environmental contours of extreme sea states are often utilized for the purposes of reliability-based offshore design. Many methods have been proposed to estimate environmental contours of extreme sea states, including, but not limited to, the traditional inverse first-order reliability method (I-FORM) and subsequent modifications, copula methods, and Monte Carlo methods. These methods differ in terms of both the methodology selected for defining the joint distribution of sea state parameters and in the method used to construct the environmental contour from the joint distribution. It is often difficult to compare the results of proposed methods to determine which method should be used for a particular application or geographical region. The comparison of the predictions from various contour methods at a single site and across many sites is important to making environmental contours of extreme sea states useful in practice. The goal of this paper is to develop a comparison framework for evaluating methods for developing environmental contours of extreme sea states. This paper develops generalized metrics for comparing the performance of contour methods to one another across a collection of study sites, and applies these metrics and methods to develop conclusions about trends in the wave resource across geographic locations, as demonstrated for a pilot dataset. These proposed metrics and methods are intended to judge the environmental contours themselves relative to other contour methods, and are thus agnostic to a specific device, structure, or field of application. The metrics developed and applied in this paper include measures of predictive accuracy, physical validity, and aggregated temporal performance that can be used to both assess contour methods and provide recommendations for the use of certain methods in various geographical regions. The application and aggregation of the metrics proposed in this paper outline a comparison framework for environmental contour methods that can be applied to support design analysis workflows for offshore structures. This comparison framework could be extended in future work to include additional metrics of interest, potentially including those to address issues pertinent to a specific application area or analysis discipline, such as metrics related to structural response across contour methods or additional physics-based metrics based on wave dynamics.

54 ENVIRONMENTAL SCIENCES↗