Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “Evaluation Framework”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 37 records · Page 2
All sourcesNASA NTRSDOE OSTI
DOE OSTI2022

Power System Resilience Evaluation Framework and Metric Review

Power system resilience has been an emerging hot topic in recent years to investigate the increasing threats of extreme events, such as natural disasters, severe weather, and cyberattacks. Although much research has been done to define, model, and quantify resilience from different aspects, the lack of universally accepted evaluation methods and resilience metrics makes it difficult to assess and compare resilience across different power systems, such as what is typically done in power system reliability studies. In this paper, first, we review the definitions of resilience, and we summarize two core concepts shared by most of the literature. Then, we develop a new framework to assess power system resilience from two perspectives - i.e., pre-event estimation and post-event evaluation - to capture system resilience performance in both general and specific fashions. We conduct a thorough review of existing resilience metrics and categorize them using the proposed framework, where recommendations are also proposed to capture core concepts of resilience.

power system resilience↗
DOE OSTI2023

Wildfire Risk Evaluation Framework for Grid Operations and Planning

The United States (US) environmental protection agency's (EPA's) climate change indicators for wildfires show a long-term trend of increased annual wildfire activity, larger wildfire size, and more variable dynamics in wildfire behavior. This has caused more frequent preemptive public safety power shutoff (PSPS) events in the regions with recognized high wildfire risk. These preemptive power shutoffs attempt to prevent the ignition of a wildfire but it nonetheless renders the transmission line non-operational, which often sheds load of downstream communities. As wildfires and PSPS events become more frequent, it is crucial to find the communities most at risk of load shedding. To that end, this paper proposes the wildfire risk evaluation of the system (WiRES) framework, which is a performance-based framework that translates extreme weather-related and PSPS event probabilities into a cumulative probability of a non-operational transmission line. This study also provides a geospatial visualization tool that breaks down the entire western electrical coordinating council (WECC) region into 50 km grid cells which can be used to 1) filter out transmission lines with higher than a threshold outage probability, and 2) graphically discover the affected regions and their biophysical and socioeconomic metrics. %such as the social vulnerability index, population density, gross domestic product, etc. Lastly, an impact assessment study is conducted which connects the results of the proposed framework to python and powerworld-based contingency analysis to highlight the applicability of the framework.

Wildfire, Resilience, Extreme Events, Grid Reslien↗
DOE OSTI2022

Power System Resilience Evaluation Framework and Metric Review: Preprint

Power system resilience is an emerging hot topic in recent years to study the increasing threats of extreme events such as natural disasters, severe weather, and cyberattacks. Although many research works have been done to define, model, and quantify resilience from different aspects, the lack of universally accepted resilience metrics and evaluation methods makes it difficult to assess and compare resilience across different power systems like what is typically done in power system reliability studies. In this paper, we first review the definitions of resilience and summarized two core concepts shared by most literature. On top of that, we conduct a thorough review of resilience metrics and develop a new framework to assess power system resilience from two perspectives, i.e., pre-event estimation and post-event evaluation, to capture system resilience performance in both general and specific fashions. Existing resilience metrics are summarized and categorized using the proposed framework, where recommendations are also proposed to capture core concepts of resilience.

power system resilience↗
DOE OSTI2022

Nanoscale defect evaluation framework combining real-time transmission electron microscopy and integrated machine learning-particle filter estimation

Observation of dynamic processes by transmission electron microscopy (TEM) is an attractive technique to experimentally analyze materials’ nanoscale phenomena and understand the microstructure-properties relationships in nanoscale. Even if spatial and temporal resolutions of real-time TEM increase significantly, it is still difficult to say that the researchers quantitatively evaluate the dynamic behavior of defects. Images in TEM video are a two-dimensional projection of three-dimensional space phenomena, thus missing information must be existed that makes image’s uniquely accurate interpretation challenging. Therefore, even though they are still a clustering high-dimensional data and can be compressed to two-dimensional, conventional statistical methods for analyzing images may not be powerful enough to track nanoscale behavior by removing various artifacts associated with experiment; and automated and unbiased processing tools for such big-data are becoming mission-critical to discover knowledge about unforeseen behavior. We have developed a method to quantitative image analysis framework to resolve these problems, in which machine learning and particle filter estimation are uniquely combined. The quantitative and automated measurement of the dislocation velocity in an Fe-31Mn-3Al-3Si autunitic steel subjected to the tensile deformation was performed to validate the framework, and an intermittent motion of the dislocations was quantitatively analyzed. The framework is successfully classifying, identifying and tracking nanoscale objects; these are not able to be accurately implemented by the conventional mean-path based analysis.

36 MATERIALS SCIENCE↗
DOE OSTI2023

Hydropower Evaluation Framework for Wildfire Resilient Microgrids

This paper presents a framework for characterizing hydropower plants based on plant and site attributes, assessing their feasibility to function within microgrids during periods of wildfire-induced outages. A set of nine metrics is developed to assign scores to hydropower plants, considering their suitability for forming wildfire-resilient microgrids, their capability to operate within such microgrids, and their simulated performance during wildfire events. A comprehensive database is constructed, comparing hydropower plants according to their suitability and capability metrics. To evaluate the performance score of a specific hydropower plant, namely the Hills Creek plant situated in a wildfire-prone region of Oregon, a case study is conducted, simulating a wildfire scenario. The results of steady-state and dynamic simulations demonstrate that the Hills Creek hydropower plant can effectively provide crucial microgrid services and power nearby communities during a prolonged wildfire-related outages.

29 ENERGY PLANNING, POLICY, AND ECONOMY↗
DOE OSTI2023

Qualification Framework Evaluations, Status Quo, and Recommendations

The Advanced Materials and Manufacturing Technology (AMMT) program develops cross-cutting technologies in support of a broad range of nuclear reactor technologies and maintains U.S. leadership in materials and manufacturing technologies for nuclear energy applications. The overarching vision of AMMT is to accelerate the development, qualification, demonstration, and deployment of advanced materials and manufacturing technologies to enable reliable and economical nuclear energy. The acceleration of qualification processes is one of the key aspects and qualification processes for the nuclear industry is not necessarily equivalent or benchmarked against other industries.

36 MATERIALS SCIENCE↗
DOE OSTI2024

A Framework for Evaluating the Implementation Cost of Attacks on Large Language Models

Large Language Models (LLMs) have been increasingly proposed as a method to enhance productivity in tasks that involve language and code. However, these models are large, complex, and their capabilities are not easily understood and controlled, meaning that their adoption opens many possibilities for new cyberattacks and misuse. Numerous attacks on LLMs have been reported and summarized in literature reviews, but we found existing reviews lacking in understanding the implementation cost of the attacks - i.e., how much effort would an attacker need in terms of coding, expertise, and resources to adopt attacks presented in the literature. Therefore, we divide existing attacks on LLMs into a taxonomy, and define a cost evaluation framework to determine the cost of the attack. An attack’s cost can be 1) estimated from reading the publication about the attack or 2) determined by implementing the attack from that publication. We provide an example evaluation of a couple jailbreaking frameworks based on experiments, and then apply the more lightweight cost estimate to a representative selection of attacks across the taxonomy we define. We discuss the relative difficulty of the attacks and also highlight defenses that have attempted to mitigate these attacks and assess their effectiveness.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2025

A framework to evaluate machine learning crystal stability predictions

The rapid adoption of machine learning in various scientific domains calls for the development of best practices and community agreed-upon benchmarking tasks and metrics. We present Matbench Discovery as an example evaluation framework for machine learning energy models, here applied as pre-filters to first-principles computed data in a high-throughput search for stable inorganic crystals. We address the disconnect between (1) thermodynamic stability and formation energy and (2) retrospective and prospective benchmarking for materials discovery. Alongside this paper, we publish a Python package to aid with future model submissions and a growing online leaderboard with adaptive user-defined weighting of various performance metrics allowing researchers to prioritize the metrics they value most. To answer the question of which machine learning methodology performs best at materials discovery, our initial release includes random forests, graph neural networks, one-shot predictors, iterative Bayesian optimizers and universal interatomic potentials. We highlight a misalignment between commonly used regression metrics and more task-relevant classification metrics for materials discovery. Accurate regressors are susceptible to unexpectedly high false-positive rates if those accurate predictions lie close to the decision boundary at 0 eV per atom above the convex hull. The benchmark results demonstrate that universal interatomic potentials have advanced sufficiently to effectively and cheaply pre-screen thermodynamic stable hypothetical materials in future expansions of high-throughput materials databases.

Riebesell, Janosh↗
DOE OSTI2020

A framework to evaluate IMEX schemes for atmospheric models

We present a new evaluation framework for implicit and explicit (IMEX) Runge–Kutta time-stepping schemes. The new framework uses a linearized nonhydrostatic system of normal modes. We utilize the framework to investigate the stability of IMEX methods and their dispersion and dissipation of gravity, Rossby, and acoustic waves. We test the new framework on a variety of IMEX schemes and use it to develop and analyze a set of second-order low-storage IMEX Runge–Kutta methods with a high Courant–Friedrichs–Lewy (CFL) number. We show that the new framework is more selective than the 2-D acoustic system previously used in the literature. Schemes that are stable for the 2-D acoustic system are not stable for the system of normal modes.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2021

Modeling Framework for Evaluating Grid Disturbances

Recent research in the literature proposed parameter-based functional forms to quantify the impact of disturbances on electric power grids. This modeling was based on a concept of generalized grid disturbances that harmonized reliability and resilience. Three main stages—avoid, react, and recover—were identified, and to support practicality, were intended to be independently planned. An objective of the grid disturbance model was to fairly analyze the potential impacts of novel mitigations like transactive energy systems (TES) on grid reliability and resilience. Quantitative evaluation of a common performance metric (e.g., area under the curve of percentage customers online before, during, and after an event) over time was shown as an effective tool for such analysis. Such quantification may also enable system planners to assess the impact of appropriate mitigative actions for improving grid performance against potential disturbances. This is the focus of this report, i.e., presenting an evaluation modeling framework that can be utilized to evaluate grid performance against disturbances, under a variety of existing and possible new mitigative actions.

24 POWER TRANSMISSION AND DISTRIBUTION↗
DOE OSTI2022

Role of Zeolite Structural Properties toward Iodine Capture: A Head-to-head Evaluation of Framework Type and Chemical Composition

This study evaluated zeolite-based sorbents for iodine gas [I2(g)] capture. Based on the framework structures and porosities, five zeolites, including two faujasite (FAU), one ZSM-5 (MFI), one mesoMFI, one ZSM-22 (TON), as well as two mesoporous materials, were evaluated for I2(g) capture at room temperature and 150 °C in an iodine-saturated environment. From these preliminary studies, the three best-performing zeolites were ion-exchanged with Ag+ and evaluated for I2(g) capture under similar conditions. Energy-dispersive X-ray spectroscopy data suggest that Ag-FAU frameworks were the materials with the highest capacity for I2(g) in this study, showing ~3× higher adsorption compared to Ag-mordenite (Ag-MOR) at room temperature, but X-ray diffraction measurements show that the faujasite structure collapsed during the adsorption studies because of dealumination. The Ag-MFI zeolites are decent sorbents in real-life applications, showing both good sorption capacities and higher stability. In-depth analyses and characterizations, including synchrotron X-ray absorption spectroscopy, revealed the influence of structural and chemical properties of zeolites on the performance for iodine adsorption from the gas phase.

aluminosilicate zeolites, iodine, porosity, acidit↗
DOE OSTI2025

Evaluation of Seismic Artificial Intelligence with Uncertainty

Artificial intelligence has transformed the seismic community with deep learning models (DLMs) that are trained to complete specific tasks within workflows. However, there is still a lack of robust evaluation frameworks for evaluating and comparing DLMs. Here, we address this gap by designing an evaluation framework that jointly incorporates two crucial aspects: performance uncertainty and learning efficiency. To target these aspects, we meticulously construct the training, validation, and test splits using a clustering method tailored to seismic data and enact an expansive training design to segregate performance uncertainty arising from stochastic training processes and random data sampling. The framework’s ability to guard against misleading declarations of model superiority is demonstrated through the evaluation of PhaseNet (Zhu and Beroza, 2018), a popular seismic phase picking DLM, under three training approaches. Our framework helps practitioners choose the best model for their problem and set performance expectations by explicitly analyzing model performance with uncertainty at varying budgets of training data.

58 GEOSCIENCES↗
DOE OSTI2023

Internal consistency and diversity scenario development: A comparative framework to evaluate energy model scenarios

Energy modeling frameworks and scenario analysis help us explore the potential impact of our actions and uncertainties in future energy systems. Despite their importance, there is no systematic procedure for evaluating the scenario development process. In a literature review, we identify two core elements of the scenario development process: internal consistency and diversity which are oftentimes missing from scenarios. Here, to address this gap, we create the Internal consistency and Diversity Scenario Development (IDSD) comparative framework which aims to assess the feasibility and diversity of scenarios for a given energy model. With this framework, we review commonly used energy models and demonstrate our framework on their scenarios. The IDSD comparative framework can serve several purposes absent from previous scenario development work by aiding energy modelers and report writers in crafting high-quality scenarios. First, the IDSD is a reflective tool which can improve the quality of the scenario development process, enabling a comparative assessment of energy models and scenarios. Second, the IDSD can provide guidance to modeling frameworks with existing scenarios and those still in development; this feedback will enable modelers to improve the development and the communication of the limitations of their scenarios. Third, this study has highlighted areas for improvement in the scenario development of some commonly used energy model frameworks. Finally, there is a complete lack of explanation regarding the stakeholder selection process. Addressing these identified items could increase opportunities for advanced energy technology uptake and improve our options for achieving a more resilient energy system.

29 ENERGY PLANNING, POLICY, AND ECONOMY↗
DOE OSTI2023

Generalized Bayesian Framework for Evaluation of Integral Benchmark Experiments [Slides]

This presentation is on generalized Bayesian framework for evaluation of integral benchmark experiments. This presentation starts off with assumptions and approximations used with Bayes Theorem. then an overview of approximations used by ORNL codes, and Generalized Bayesian Monto Carlo (GBMC). The presentation then details out a precise framework. This presentation then concludes with considerations.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2026

A Benchmarking Framework for Evaluating Large Language Model Capabilities in Nuclear Reactor Safety Applications

Large language models (LLMs) are increasingly capable of answering technical questions, synthesizing domain knowledge, and supporting engineering workflows. For nuclear science and engineering, these capabilities require careful, domain-specific evaluation before they can be credibly incorporated into safety-related activities, regulatory review, or technical decision support. This paper presents preliminary results from benchmarking framework for evaluating LLM capabilities in nuclear contexts. The framework is organized into three evaluation categories: nuclear fundamentals, general dual-use knowledge, and plant specific knowledge. These categories are intended to distinguish general nuclear engineering competence from broader technical reasoning and more context-dependent nuclear knowledge. Initial evaluations focus on nuclear fundamentals using questions representative of the knowledge expected of a nuclear professional engineer. Results indicate that contemporary frontier models perform at a high level and substantially exceed the performance of older model generations, with some models approaching saturation of the current benchmark. These findings suggest both the rapid improvement of LLM capabilities in specialized technical domains and the need for more discriminating evaluation methods. The paper presents the benchmark structure, preliminary model-comparison results, and ongoing work. This work supports development of verifiable, responsible, and safety-conscious methods for assessing AI systems in nuclear engineering applications.

21 - SPECIFIC NUCLEAR REACTORS AND ASSOCIATED PLAN↗