Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “Evaluation Framework”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 37 records · Page 2

Bayesian Monte Carlo Evaluation Framework for Imperfect Data and Models [Abstract]

Nuclear data evaluation methods conventionally make the following assumptions: prior and posterior probability distribution functions (PDFs) of all model parameters and data are normal (Gaussian); the linear approximation is sufficiently accurate for minimization of a cost function (even for non-linear models); and that both the model (of, e.g., neutron cross section) and experimental data (including their covariance data) are perfect. These assumptions are inherent to the well-known generalized linear least squares (GLLS) minimization method commonly used for evaluations of resolved resonance region (RRR) neutron cross sections. However, these assumptions are often not justified due to the presence of non-normal PDFs, non-linear models (e.g. R -matrix formalism), and inherent imperfections in data and models (e.g. discrepant data sets, discrepancies between the previous evaluation and newly measured data, or imperfect covariance data). We remove the said assumptions in a mathematical framework of Bayes’ theorem, and implement it using the Metropolis-Hastings Monte Carlo method. Parameters of a new kind are introduced to parameterize inherent imperfections, e.g. , any discrepancies between the theoretical model and measured data. These new parameters enable evaluators to quantify their expert judgement about any discrepancies or imperfections in a reproducible manner. We demonstrate the framework with an ongoing evaluation of 233 U in the eV region using the ENDF-B/VIII library and transmission data measured by Guber, et al. , and compare the posterior parameters to those obtained by conventional evaluation methods.

73 NUCLEAR PHYSICS AND RADIATION PHYSICS↗

Bayesian Monte-Carlo Evaluation Framework for Imperfect Nuclear Data [Slides]

BMC evaluation is a tool used to address imperfect data and models, non-linear models, and non-normal PDFs. ENDF-6 format does not allow non-normal parameter PDFs. Storing posterior sets allows for variance, covariance, skewness, etc. To better predict criticality, we should document non-normal parameter PDFs (i.e. asymmetric uncertainty) and consider non-linear sensitivity of $k_{\text{eff}}$ to resonance parameters.

73 NUCLEAR PHYSICS AND RADIATION PHYSICS↗

Long-Range Biometric Identification in Real World Scenarios: A Comprehensive Evaluation Framework Based on Missions

The considerable body of data available for evaluating biometric recognition systems in Research and Development (R&D) environments has contributed to the increasingly common problem of target performance mismatch. Biometric algorithms are frequently tested against data that may not reflect the real world applications they target. From a Testing and Evaluation (T&E) standpoint, this domain mismatch causes difficulty assessing when improvements in State-of-the-Art (SOTA) research actually translate to improved applied outcomes. This problem can be addressed with thoughtful preparation of data and experimental methods to reflect specific use-cases and scenarios.To that end, this paper evaluates research solutions for identifying individuals at ranges and altitudes, which could support various application areas such as counterterrorism, protection of critical infrastructure facilities, military force protection, and border security. We address challenges including image quality issues and reliance on face recognition as the sole biometric modality. By fusing face and body features, we propose developing robust biometric systems for effective long-range identification from both the ground and steep pitch angles. Preliminary results show promising progress in whole-body recognition. This paper presents these early findings and discusses potential future directions for advancing long-range biometric identification systems based on mission-driven metrics.

Aykac, Deniz↗

PRIME: An evaluation framework for protein representation inference and generalization in viral mutation space

Background Protein language models (PLMs) have revolutionized protein fitness prediction, yet their application to rapidly evolving viral pathogens is often confounded by extreme sequence homology. This homology leads to “data leakage” in standard random validation splits, yielding inflated performance metrics that fail to translate into real-world biosurveillance utility. Results We present Protein Representation Inference for Mutation Evaluation (PRIME), a framework that integrates domain-specific fine-tuning with a rigorous position-stratified validation protocol to evaluate viral threats. Using a dataset of 347,432 SARS-CoV-2 receptor binding domain (RBD) sequences, we demonstrate that while random training data split yields deceptive R 2 values (> 0.90), they fail to generalize to novel mutational sites. By benchmarking models up to 650 M parameters, we show that domain-specific fine-tuning of the ESM-C 600 M model with correctly stratified data provides an initial demonstration of predictive signal for binding affinity and expression at unseen mutational sites of binding affinity and expression on unseen sites (R 2 ~0.23), a significant advancement over base foundation models which exhibit no predictive power (R 2 <0). PRIME’s embedding-based clustering identified 3.03% of bat coronavirus sequences as candidates for further experimental prioritization based on their functional similarity to human-infective strains in embedding space, offering a perspective complementary to traditional phylogenetic methods. Conclusion PRIME establishes a new benchmark for the application of PLMs in pathogen surveillance. Our findings demonstrate that state-of-the-art models and fine-tuning, when paired with stratified validation, provide biologically meaningful insights into pathogen evolution and zoonotic risk.

59 BASIC BIOLOGICAL SCIENCES↗

Power System Resilience Evaluation Framework and Metric Review

Power system resilience has been an emerging hot topic in recent years to investigate the increasing threats of extreme events, such as natural disasters, severe weather, and cyberattacks. Although much research has been done to define, model, and quantify resilience from different aspects, the lack of universally accepted evaluation methods and resilience metrics makes it difficult to assess and compare resilience across different power systems, such as what is typically done in power system reliability studies. In this paper, first, we review the definitions of resilience, and we summarize two core concepts shared by most of the literature. Then, we develop a new framework to assess power system resilience from two perspectives - i.e., pre-event estimation and post-event evaluation - to capture system resilience performance in both general and specific fashions. We conduct a thorough review of existing resilience metrics and categorize them using the proposed framework, where recommendations are also proposed to capture core concepts of resilience.

power system resilience↗

Wildfire Risk Evaluation Framework for Grid Operations and Planning

The United States (US) environmental protection agency's (EPA's) climate change indicators for wildfires show a long-term trend of increased annual wildfire activity, larger wildfire size, and more variable dynamics in wildfire behavior. This has caused more frequent preemptive public safety power shutoff (PSPS) events in the regions with recognized high wildfire risk. These preemptive power shutoffs attempt to prevent the ignition of a wildfire but it nonetheless renders the transmission line non-operational, which often sheds load of downstream communities. As wildfires and PSPS events become more frequent, it is crucial to find the communities most at risk of load shedding. To that end, this paper proposes the wildfire risk evaluation of the system (WiRES) framework, which is a performance-based framework that translates extreme weather-related and PSPS event probabilities into a cumulative probability of a non-operational transmission line. This study also provides a geospatial visualization tool that breaks down the entire western electrical coordinating council (WECC) region into 50 km grid cells which can be used to 1) filter out transmission lines with higher than a threshold outage probability, and 2) graphically discover the affected regions and their biophysical and socioeconomic metrics. %such as the social vulnerability index, population density, gross domestic product, etc. Lastly, an impact assessment study is conducted which connects the results of the proposed framework to python and powerworld-based contingency analysis to highlight the applicability of the framework.

Wildfire, Resilience, Extreme Events, Grid Reslien↗

Power System Resilience Evaluation Framework and Metric Review: Preprint

Power system resilience is an emerging hot topic in recent years to study the increasing threats of extreme events such as natural disasters, severe weather, and cyberattacks. Although many research works have been done to define, model, and quantify resilience from different aspects, the lack of universally accepted resilience metrics and evaluation methods makes it difficult to assess and compare resilience across different power systems like what is typically done in power system reliability studies. In this paper, we first review the definitions of resilience and summarized two core concepts shared by most literature. On top of that, we conduct a thorough review of resilience metrics and develop a new framework to assess power system resilience from two perspectives, i.e., pre-event estimation and post-event evaluation, to capture system resilience performance in both general and specific fashions. Existing resilience metrics are summarized and categorized using the proposed framework, where recommendations are also proposed to capture core concepts of resilience.

power system resilience↗

Nanoscale defect evaluation framework combining real-time transmission electron microscopy and integrated machine learning-particle filter estimation

Observation of dynamic processes by transmission electron microscopy (TEM) is an attractive technique to experimentally analyze materials’ nanoscale phenomena and understand the microstructure-properties relationships in nanoscale. Even if spatial and temporal resolutions of real-time TEM increase significantly, it is still difficult to say that the researchers quantitatively evaluate the dynamic behavior of defects. Images in TEM video are a two-dimensional projection of three-dimensional space phenomena, thus missing information must be existed that makes image’s uniquely accurate interpretation challenging. Therefore, even though they are still a clustering high-dimensional data and can be compressed to two-dimensional, conventional statistical methods for analyzing images may not be powerful enough to track nanoscale behavior by removing various artifacts associated with experiment; and automated and unbiased processing tools for such big-data are becoming mission-critical to discover knowledge about unforeseen behavior. We have developed a method to quantitative image analysis framework to resolve these problems, in which machine learning and particle filter estimation are uniquely combined. The quantitative and automated measurement of the dislocation velocity in an Fe-31Mn-3Al-3Si autunitic steel subjected to the tensile deformation was performed to validate the framework, and an intermittent motion of the dislocations was quantitatively analyzed. The framework is successfully classifying, identifying and tracking nanoscale objects; these are not able to be accurately implemented by the conventional mean-path based analysis.

36 MATERIALS SCIENCE↗

Hydropower Evaluation Framework for Wildfire Resilient Microgrids

This paper presents a framework for characterizing hydropower plants based on plant and site attributes, assessing their feasibility to function within microgrids during periods of wildfire-induced outages. A set of nine metrics is developed to assign scores to hydropower plants, considering their suitability for forming wildfire-resilient microgrids, their capability to operate within such microgrids, and their simulated performance during wildfire events. A comprehensive database is constructed, comparing hydropower plants according to their suitability and capability metrics. To evaluate the performance score of a specific hydropower plant, namely the Hills Creek plant situated in a wildfire-prone region of Oregon, a case study is conducted, simulating a wildfire scenario. The results of steady-state and dynamic simulations demonstrate that the Hills Creek hydropower plant can effectively provide crucial microgrid services and power nearby communities during a prolonged wildfire-related outages.

29 ENERGY PLANNING, POLICY, AND ECONOMY↗

Qualification Framework Evaluations, Status Quo, and Recommendations

The Advanced Materials and Manufacturing Technology (AMMT) program develops cross-cutting technologies in support of a broad range of nuclear reactor technologies and maintains U.S. leadership in materials and manufacturing technologies for nuclear energy applications. The overarching vision of AMMT is to accelerate the development, qualification, demonstration, and deployment of advanced materials and manufacturing technologies to enable reliable and economical nuclear energy. The acceleration of qualification processes is one of the key aspects and qualification processes for the nuclear industry is not necessarily equivalent or benchmarked against other industries.

36 MATERIALS SCIENCE↗

A Framework for Evaluating the Implementation Cost of Attacks on Large Language Models

Large Language Models (LLMs) have been increasingly proposed as a method to enhance productivity in tasks that involve language and code. However, these models are large, complex, and their capabilities are not easily understood and controlled, meaning that their adoption opens many possibilities for new cyberattacks and misuse. Numerous attacks on LLMs have been reported and summarized in literature reviews, but we found existing reviews lacking in understanding the implementation cost of the attacks - i.e., how much effort would an attacker need in terms of coding, expertise, and resources to adopt attacks presented in the literature. Therefore, we divide existing attacks on LLMs into a taxonomy, and define a cost evaluation framework to determine the cost of the attack. An attack’s cost can be 1) estimated from reading the publication about the attack or 2) determined by implementing the attack from that publication. We provide an example evaluation of a couple jailbreaking frameworks based on experiments, and then apply the more lightweight cost estimate to a representative selection of attacks across the taxonomy we define. We discuss the relative difficulty of the attacks and also highlight defenses that have attempted to mitigate these attacks and assess their effectiveness.

97 MATHEMATICS AND COMPUTING↗

A framework to evaluate machine learning crystal stability predictions

The rapid adoption of machine learning in various scientific domains calls for the development of best practices and community agreed-upon benchmarking tasks and metrics. We present Matbench Discovery as an example evaluation framework for machine learning energy models, here applied as pre-filters to first-principles computed data in a high-throughput search for stable inorganic crystals. We address the disconnect between (1) thermodynamic stability and formation energy and (2) retrospective and prospective benchmarking for materials discovery. Alongside this paper, we publish a Python package to aid with future model submissions and a growing online leaderboard with adaptive user-defined weighting of various performance metrics allowing researchers to prioritize the metrics they value most. To answer the question of which machine learning methodology performs best at materials discovery, our initial release includes random forests, graph neural networks, one-shot predictors, iterative Bayesian optimizers and universal interatomic potentials. We highlight a misalignment between commonly used regression metrics and more task-relevant classification metrics for materials discovery. Accurate regressors are susceptible to unexpectedly high false-positive rates if those accurate predictions lie close to the decision boundary at 0 eV per atom above the convex hull. The benchmark results demonstrate that universal interatomic potentials have advanced sufficiently to effectively and cheaply pre-screen thermodynamic stable hypothetical materials in future expansions of high-throughput materials databases.

Riebesell, Janosh↗

A framework to evaluate IMEX schemes for atmospheric models

We present a new evaluation framework for implicit and explicit (IMEX) Runge–Kutta time-stepping schemes. The new framework uses a linearized nonhydrostatic system of normal modes. We utilize the framework to investigate the stability of IMEX methods and their dispersion and dissipation of gravity, Rossby, and acoustic waves. We test the new framework on a variety of IMEX schemes and use it to develop and analyze a set of second-order low-storage IMEX Runge–Kutta methods with a high Courant–Friedrichs–Lewy (CFL) number. We show that the new framework is more selective than the 2-D acoustic system previously used in the literature. Schemes that are stable for the 2-D acoustic system are not stable for the system of normal modes.

54 ENVIRONMENTAL SCIENCES↗

Modeling Framework for Evaluating Grid Disturbances

Recent research in the literature proposed parameter-based functional forms to quantify the impact of disturbances on electric power grids. This modeling was based on a concept of generalized grid disturbances that harmonized reliability and resilience. Three main stages—avoid, react, and recover—were identified, and to support practicality, were intended to be independently planned. An objective of the grid disturbance model was to fairly analyze the potential impacts of novel mitigations like transactive energy systems (TES) on grid reliability and resilience. Quantitative evaluation of a common performance metric (e.g., area under the curve of percentage customers online before, during, and after an event) over time was shown as an effective tool for such analysis. Such quantification may also enable system planners to assess the impact of appropriate mitigative actions for improving grid performance against potential disturbances. This is the focus of this report, i.e., presenting an evaluation modeling framework that can be utilized to evaluate grid performance against disturbances, under a variety of existing and possible new mitigative actions.

24 POWER TRANSMISSION AND DISTRIBUTION↗

Role of Zeolite Structural Properties toward Iodine Capture: A Head-to-head Evaluation of Framework Type and Chemical Composition

This study evaluated zeolite-based sorbents for iodine gas [I2(g)] capture. Based on the framework structures and porosities, five zeolites, including two faujasite (FAU), one ZSM-5 (MFI), one mesoMFI, one ZSM-22 (TON), as well as two mesoporous materials, were evaluated for I2(g) capture at room temperature and 150 °C in an iodine-saturated environment. From these preliminary studies, the three best-performing zeolites were ion-exchanged with Ag+ and evaluated for I2(g) capture under similar conditions. Energy-dispersive X-ray spectroscopy data suggest that Ag-FAU frameworks were the materials with the highest capacity for I2(g) in this study, showing ~3× higher adsorption compared to Ag-mordenite (Ag-MOR) at room temperature, but X-ray diffraction measurements show that the faujasite structure collapsed during the adsorption studies because of dealumination. The Ag-MFI zeolites are decent sorbents in real-life applications, showing both good sorption capacities and higher stability. In-depth analyses and characterizations, including synchrotron X-ray absorption spectroscopy, revealed the influence of structural and chemical properties of zeolites on the performance for iodine adsorption from the gas phase.

aluminosilicate zeolites, iodine, porosity, acidit↗