Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “evaluations”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 415 records · Page 23
All sourcesNASA NTRSDOE OSTI
DOE OSTI2022

The DOE E3SM Model Version 2: Overview of the Physical Model and Initial Model Evaluation

Abstract This work documents version two of the Department of Energy's Energy Exascale Earth System Model (E3SM). E3SMv2 is a significant evolution from its predecessor E3SMv1, resulting in a model that is nearly twice as fast and with a simulated climate that is improved in many metrics. We describe the physical climate model in its lower horizontal resolution configuration consisting of 110 km atmosphere, 165 km land, 0.5° river routing model, and an ocean and sea ice with mesh spacing varying between 60 km in the mid‐latitudes and 30 km at the equator and poles. The model performance is evaluated with Coupled Model Intercomparison Project Phase 6 Diagnosis, Evaluation, and Characterization of Klima simulations augmented with historical simulations as well as simulations to evaluate impacts of different forcing agents. The simulated climate has many realistic features of the climate system, with notable improvements in clouds and precipitation compared to E3SMv1. E3SMv1 suffered from an excessively high equilibrium climate sensitivity (ECS) of 5.3 K. In E3SMv2, ECS is reduced to 4.0 K which is now within the plausible range based on a recent World Climate Research Program assessment. However, a number of important biases remain including a weak Atlantic Meridional Overturning Circulation, deficiencies in the characteristics and spectral distribution of tropical atmospheric variability, and a significant underestimation of the observed warming in the second half of the historical period. An analysis of single‐forcing simulations indicates that correcting the historical temperature bias would require a substantial reduction in the magnitude of the aerosol‐related forcing.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2023

Evaluating Shallow Convection Parameterization Assumptions With a q t – w Quadrant Analysis

Abstract Uncertainties associated with the response of shallow clouds to global warming remain a great challenge for climate projection. Due to the small size of these clouds, parameterizations are required to represent them in both current‐ and next‐generation climate models. We present a quantitative evaluation of several important assumptions used in both mass‐flux and Assumed‐PDF Higher‐Order Closure parameterizations of shallow convection. We use large‐eddy simulations (LESs) of four different shallow convection regimes as benchmarks and apply a q t (total moisture)– w (vertical velocity) quadrant analysis technique to identify the “coherent structures” of moist and dry up/down‐drafts. The statistics of these coherent structures and the environment are then used to evaluate commonly used assumptions in mass‐flux parameterizations related to (a) the convective downdrafts and (b) the subplume variability of convective updrafts. For quantitative evaluation of the PDF closure used in AP‐HOCs, we perform offline calculations with the PDF closure used in the Cloud Layers Unified By Bi‐normals and the Simplified Higher‐Order Closure schemes by supplying the closure with statistical quantities directly calculated from the LESs. Then, the impact of parameterization assumptions embedded in the PDF closure in the same two categories is analyzed and understood through comparisons of the output from the closure with the statistics of the moist and dry up/down‐drafts from the LESs.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2024

Evaluation of a New Approach for Entrainment and Detrainment Rate Estimation

Entrainment and detrainment rates (ε and δ) constitute the most critical free parameters in mass flux schemes commonly employed for cumulus parameterizations. Recently, Zhu et al. (2021) introduced a new approach that utilizes aircraft observations to simultaneously estimate ε and δ for cumulus clouds, overcoming the limitation of other observation-based approaches that solely yield ε without offering insights into δ. This study aims to comprehensively evaluate the reliability of this new approach. First, evaluation using an Explicit Mixing Parcel Model demonstrates the capability of the new approach to back-calculate predetermined ε and δ based on the physical properties before and after the entrainment mixing. Second, evaluation using large-eddy simulations illustrates that the new approach yields consistent ε and δ profiles compared to the traditional approach. Sensitivity tests indicate a weak sensitivity of the estimated δ with the new approach to the entrained air source. A decrease in the proportion of cloudy air in the assumed detrained air leads to a reduction in the estimated δ, while ε remains unaffected. Finally, the most appropriate assumptions for entrained and detrained air are discussed. Estimating ε for cumulus parameterizations involves acquiring ambient air more than 500 m away from the cloud edge as entrained air. Due to implicit mean field approximations in the traditional approach, determining the optimal assumption for detrained air properties proves challenging. Finally, this study confirms the reliability of the new approach in estimating ε and δ, providing confidence in its application to extensive observational data and advancement in parameterization.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2024

Using an Isotope Enabled Mass Balance to Evaluate Existing Land Surface Models

Abstract Land surface models (LSMs) play a crucial role in elucidating water and carbon cycles by simulating processes such as plant transpiration and evaporation from bare soil, yet calibration often relies on comparing LSM outputs of landscape total evapotranspiration ( ET ) and discharge with measured bulk fluxes. Discrepancies in partitioning into component fluxes predicted by various LSMs have been noted, prompting the need for improved evaluation methods. Stable water isotopes serve as effective tracers of component hydrologic fluxes, but data and model integration challenges have hindered their widespread application. Leveraging National Ecological Observation Network measurements of water isotope ratios at 16 US sites over 3 years combined with LSM‐modeled fluxes, we employed an isotope‐enabled mass balance framework to simulate ET isotope values ( δET ) within three operational LSMs (Mosaic, Noah, and VIC) to evaluate their partitioning. Models simulating δET values consistent with observations were deemed more reflective of water cycling in these ecosystems. Mosaic exhibited the best overall performance (Kling‐Gupta Efficiency of 0.28). For both Mosaic and Noah there were robust correlations between bare soil evaporation fraction and error (negative) as well as transpiration fraction and error (positive). We found the point at which errors are smallest ( x ‐intercept of the multi‐site regression) is at a higher transpiration fraction than is currently specified in the models. Which means that transpiration fraction is underestimated on average. Stable isotope tracers offer an additional tool for model evaluation and identifying areas for improvement, potentially enhancing LSM simulations and our understanding of land‐surface hydrologic processes.

58 GEOSCIENCES↗
DOE OSTI2020

A single-component water-lean post-combustion CO 2 capture solvent with exceptionally low operational heat and total costs of capture – comprehensive experimental and theoretical evaluation

A comprehensive evaluation of a recently developed water-lean amine-based solvent, namely N-(2-ethoxyethyl)-3-morpholinopropan-1-amine (2-EEMPA), has been performed to analyze its post-combustion CO 2 capture performance. This evaluation comprises (1) fundamental characterization of the solvent–CO 2 interaction using vapor–liquid equilibria, kinetics and viscosity measurements; (2) process characterization of the CO 2 capture performance as measured in a laboratory scale continuous flow system and via Aspen Plus® simulation using a flue gas simulant; as well as (3) a full techno economic analysis of the capture process at industrial scale with corresponding projections of critical metrics. This paper summarizes the many parts of this comprehensive evaluation and shows how the various parts come together to empower validated conclusions about its process performance. Notably, it is projected that this solvent can operate at a regeneration heat rate of 2.0 GJ per tonne CO 2 for post-combustion capture, and at a total cost of capture of $50.6/tonne CO 2 . Overall, with further process optimization significant reductions in the capture cost are predicted.

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗
DOE OSTI2023

Database work for the new cross section standards evaluation

An effort is now underway to produce a new evaluation of the neutron standards. It is important to maintain experimental programs to increase the quality and extend the database for the neutron cross section standards in order to improve evaluations of them that will be used to convert cross section measurements made relative to those standards. Measurements have been made for most of the standard cross sections since the last evaluation of the standards. The improved database includes the cross sections for the H(n,n), 6Li(n,t), 10B(n,αγ), 10B(n,α), C(n,n), Au(n,γ), 235U(n,f) and 238U(n,f) standard reactions and ratios among them. The database also includes the 238U(n,γ) and 239Pu(n,f) cross sections in addition to the standard cross sections. Those data were included since there are many ratio measurements of those cross sections with the standards and absolute data are available for them.

73 NUCLEAR PHYSICS AND RADIATION PHYSICS↗
DOE OSTI2025

Fission Product Yield Modeling and Evaluation

Although independent and cumulative fission product yields have been a part of evaluated libraries for decades, there have been few updates over the years. The fission product yield sub-library in the ENDF/B-VIII.0 library is still largely based on the evaluation of England and Rider from the mid-90’s, with only more recent updates to the energy dependence of 239 Pu below 2 MeV and fixes to isomeric states and missing fission products. Over the past several years, there have been a wealth of new measurements of independent and cumulative fission product yields, particularly those with short half-lives, and there have been significant improvements in the modeling of prompt and delayed fission observables. Here, we describe recent progress in the improvement of fission product yield calculations, using the BeoH code and the underlying Hauser Fesh-bach Fission Fragment Decay (HF 3 D) model, developed at Los Alamos National Laboratory. We will describe our recent calculations for consistent prompt and delayed fission observables for major and minor actinides, including new work investigating isomeric ratios. We will detail the ongoing evaluation process for energy-dependent fission product yields from thermal up to 20 MeV incident neutron energy and some validation work that has been performed for these new fission product yield calculations. Additionally, we will discuss future perspectives of this work, highlighting the need for additional data.

38 RADIATION CHEMISTRY, RADIOCHEMISTRY, AND NUCLEA↗
DOE OSTI2021

Stochastic evaluation of fourth-order many-body perturbation energies

A scalable, stochastic algorithm evaluating the fourth-order many-body perturbation (MP4) correction to energy is proposed. Three hundred Goldstone diagrams representing the MP4 correction are computer generated and then converted into algebraic formulas expressed in terms of Green’s functions in real space and imaginary time. They are evaluated by the direct (i.e., non-Markov, non-Metropolis) Monte Carlo (MC) integration accelerated by the redundant-walker and control-variate algorithms. The resulting MC-MP4 method is efficiently parallelized and is shown to display O(n 5.3 ) size-dependence of cost, which is nearly two ranks lower than the O(n 7 ) dependence of the deterministic MP4 algorithm. Furthermore, it evaluates the MP4/aug-cc-pVDZ energy for benzene, naphthalene, phenanthrene, and corannulene with the statistical uncertainty of 10 mE h (1.1% of the total basis-set correlation energy), 38 mE h (2.6%), 110 mE h (5.5%), and 280 mE h (9.0%), respectively, after about 10 9 MC steps.

74 ATOMIC AND MOLECULAR PHYSICS↗
DOE OSTI2026

Expert evaluation of LLM world models: A high-T c superconductivity case study

Large Language Models (LLMs) show great promise as a powerful tool for scientific literature exploration. However, their effectiveness in providing scientifically accurate and comprehensive answers to complex questions within specialized domains remains an active area of research. Using the field of high-temperature cuprates as an exemplar, we evaluate the ability of LLM systems to understand the literature at the level of an expert. We construct an expert-curated database of 1,726 scientific papers that covers the history of the field, and a set of 67 expert-formulated questions that probe deep understanding of the literature. We then evaluate six different LLM-based systems for answering these questions, including both commercially available closed models and a custom retrieval-augmented generation (RAG) system capable of retrieving images alongside text. Experts then evaluate the answers of these systems against a rubric that assesses balanced perspectives, factual comprehensiveness, succinctness, and evidentiary support. Among the six systems, two using RAG on curated literature outperformed existing closed models across key metrics, particularly in providing comprehensive and well-supported answers. We discuss promising aspects of LLM performances as well as critical short-comings of all the models. The set of expert-formulated questions and the rubric will be valuable for assessing expert level performance of LLM based reasoning systems.

36 MATERIALS SCIENCE↗
DOE OSTI2020

Dynamic PRA Methods to Evaluate the Impact on Accident Progression of Accident Tolerant Fuels

Accident tolerant fuels (ATFs) are new nuclear fuels developed in response to the accident at the Fukushima power station in March 2011. The goal of ATFs is to withstand accident scenarios through better performance compared to currently employed fuels (e.g., small-scale hydrogen generation). This paper targets a method for evaluating and comparing ATF performance from a probabilistic risk assessment (PRA) perspective by employing a newly developed combination of event trees and dynamic PRA methods. Compared to classical PRA methods based on event trees and fault trees, dynamic PRA can evaluate with higher resolution the safety impacts of physics dynamics and the timing/sequencing of events on the accident progression without the need to introduce overly conservative modeling assumptions and success criteria. In this paper, we analyze the impact on the accident progression of three different cladding configurations for two initiating events [a large break loss-of-coolant accident (LB-LOCA) and a station blackout (SBO)] by employing dynamic PRA methods. The goal is to compare the safety performance of ATFs (FeCrAl and Cr-coated cladding) and the currently employed Zr-based clad fuel. We employ two different strategies. The first focuses on the identification of success criteria discrepancies between the accident sequences generated by the classical PRA model and the set of simulation runs generated by dynamic PRA using ATF. The second one, on the other hand, directly uses dynamic PRA to evaluate the impact of timing of events (e.g., recovery actions) on accident progression. In conclusion, by applying these methods to the LB-LOCA and SBO initiating events, we show how dynamic PRA methods can provide analysts with detailed and quantitative information on the safety impact of ATFs.

97 - MATHEMATICS AND COMPUTING↗
DOE OSTI2025

Preliminary Evaluation of Joint Electricity-Hydrogen Concept of Operations

An initial thermal power dispatch (TPD) concept of operations was evaluated that couples a nuclear power plant to a nearby hydrogen production plant. GSE Systems’ generic pressurized water reactor full-scope simulator was modified with a TPD model comprised of a thermal power extraction and delivery system. A prototype human-system interface (HSI) was developed to interact with the TPD model and allow participants to execute the basic operating scenarios for normal operations. Four retired operators performed the evaluation, and due to COVID-19 travel restrictions, the original in-person experimental design was restructured to support a remote participator evaluation using a web meeting platform. Data from operator feedback, observations from the research team, and quantitative survey responses revealed that the initial TPD concept of operations is feasible. The operators were comfortable with the engineered system and HSI and could manage it without adverse impacts to reactor power, plant safety, or equipment. Findings are discussed in terms of both the TPD system design and HSI performance.

human factors↗
DOE OSTI2021

Design and evaluation of an open-source, conformable skin-cooling system for body magnetic resonance guided focused ultrasound treatments

Magnetic resonance guided focused ultrasound (MRgFUS) treatment of tumors uses inter-sonication delays to allow heat to dissipate from the skin and other near-field tissues. Despite inter-sonication delays, treatment of tumors close to the skin risks skin burns. This work has designed and evaluated an open-source, conformable, skin-cooling system for body MRgFUS treatments to reduce skin burns and enable ablation closer to the skin. A MR-compatible skin cooling system is described that features a conformable skin-cooling pad assembly with feedback control allowing continuous flow and pressure maintenance during the procedure. System performance was evaluated with hydrophone, phantom and in vivo porcine studies. Sonications were performed 10 and 5 mm from the skin surface under both control and forced convective skin-cooling conditions. 3D MR temperature imaging was acquired in real time and the accumulated thermal dose volume was measured. Gross analysis of the skin post-sonication was further performed. Device conformability was demonstrated at several body locations. Hydrophone studies demonstrated no beam aberration, but a 5–12% reduction of the peak pressure due to the presence of the skin-cooling pad assembly in the acoustic near field. Phantom evaluation demonstrated there is no MR temperature imaging precision reduction or any other artifacts present due to the coolant flow during MRgFUS sonication. The porcine studies demonstrated skin burns were reduced in size or eliminated when compared to the control condition. An open-source design of an MRgFUS active skin cooling system demonstrates device conformability with a reduction of skin burns while ablating superficial tissues.

60 APPLIED LIFE SCIENCES↗
DOE OSTI2020

An Evaluation of Advanced Tools for Distributed Wind Turbine Performance Estimation

We evaluate various classes of distributed wind turbine performance tools across two sites in the United States. The class of tools ranges from the simple mass conservation model to the coupled Reynolds-averaged Navier-Stokes model, all initiated by the WIND Toolkit data set. The resource estimation at the site is evaluated against measured data at the mast location. Taking a sample 100-kW wind turbine and constant losses, we evaluate Openwind, Continuum, and WindNinja tools and document annual energy production (AEP) and time-series statistics associated with the performance estimation of the wind turbine. Using a methodology that is consistent and unbiased across the three options currently available in the industry, we elaborate results at the two sample locations and discuss the probable sources of discrepancy in the AEP estimates. Two main sources of the discrepancy come from the input WIND Toolkit data and the spatial modeling techniques of the tools to capture atmospheric physics. The discussion includes additional values that these tools may bring into the energy assessment process to enhance the owners' confidence over the distributed wind power systems.

17 WIND ENERGY↗
DOE OSTI2025

Evaluating algorithmic bias on biomarker classification of breast cancer pathology reports

Objectives: This work evaluated algorithmic bias in biomarkers classification using electronic pathology reports from female breast cancer cases. Bias was assessed across 5 subgroups: cancer registry, race, Hispanic ethnicity, age at diagnosis, and socioeconomic status. Materials and Methods: We utilized 594 875 electronic pathology reports from 178 121 tumors diagnosed in Kentucky, Louisiana, New Jersey, New Mexico, Seattle, and Utah to train 2 deep-learning algorithms to classify breast cancer patients using their biomarkers test results. We used balanced error rate (BER), demographic parity (DP), equalized odds (EOD), and equal opportunity (EOP) to assess bias. Results: We found differences in predictive accuracy between registries, with the highest accuracy in the registry that contributed the most data (Seattle Registry, BER ratios for all registries >1.25). BER showed no significant algorithmic bias in extracting biomarkers (estrogen receptor, progesterone receptor, human epidermal growth factor receptor 2) for race, Hispanic ethnicity, age at diagnosis, or socioeconomic subgroups (BER ratio <1.25). DP, EOD, and EOP all showed insignificant results. Discussion: We observed significant differences in BER by registry, but no significant bias using the DP, EOD, and EOP metrics for socio-demographic or racial categories. This highlights the importance of employing a diverse set of metrics for a comprehensive evaluation of model fairness. Conclusion: A thorough evaluation of algorithmic biases that may affect equality in clinical care is a critical step before deploying algorithms in the real world. We found little evidence of algorithmic bias in our biomarker classification tool. Artificial intelligence tools to expedite information extraction from clinical records could accelerate clinical trial matching and improve care.

60 APPLIED LIFE SCIENCES↗
DOE OSTI2026

Empirical evaluation of all unique Cas9 protospacers in E. coli reveal widespread functionality and rules for gRNA design

The Cas9 nuclease has become central to modern methods and technologies in synthetic biology, largely due to the ease with which it can be targeted to specific DNA loci via guide RNAs (gRNAs). Reports vary widely on the actual specificity of this targeting, with some studies observing 60% of gRNAs possessing no activity against the genome, yet an assumption persists within the E. coli community that inactive gRNAs are rare. To resolve these contradictions, we evaluated the activity of 463 000 unique gRNAs in the E. coli K12 MG1655 genome. We show that the overwhelming majority (at least 93%) of unique gRNAs are functional while only 0.3% are nonfunctional. These nonfunctional gRNAs exhibit strong spacer self-interaction, which can either be excluded using a simple design rule or “repaired” during library design. Finally, this work provides the greater microbial synthetic biology community both a set of nearly half a million empirically evaluated E. coli gRNAs as well as a thoroughly evaluated experimental procedure, complete with appropriate controls for Cas9 activity, for conducting Cas9 assays in E. coli specifically and bacteria more generally. Lastly, we have produced a webapp to allow users to easily browse and extract gRNA sequences from the E. coli genome, which can be accessed at https://grna.ornl.gov.

Kammerdiener, Elise K. [Oak Ridge National Laborat↗
DOE OSTI2026

Fast Fourier transform evaluation of the Fresnel integral for gravitational-wave lensing

Gravitational waves (GWs) exhibit wave-optics effects when their wavelength is comparable to the scale of the gravitational lens. This may occur in lensing from galactic subhalos in GWs emitted by binary black-hole mergers and is gaining interest as a novel probe of dark matter. Predictions for observables in these cases ultimately rely on evaluating a Fresnel integral that quantifies the effect of lensing on the amplitude of a GW at a given frequency. However, numerical evaluation of this Fresnel integral is tricky, and several algorithms and publicly available codes that implement it have been developed. Here, we show that the dependence of this integral on the lens position can be written as a two-dimensional Fourier transform. Modern FFT techniques then enable rapid evaluation at all-sky positions simultaneously for general lenses without symmetry. Vectorization of FFT routines allows for derivatives with respect to model parameters to be obtained with only incremental additional computational cost. If the lens is axisymmetric, further speedups can be achieved with recently developed techniques for nonuniform fast Hankel transforms. To demonstrate, we make available Fresnel Integral Optimization with Nonuniform Transforms (fiona), an efficient and accurate code that is significantly faster than current methods for dense source grids, reaching 2 orders of magnitude speedups for ∼10 6 GW-emitting points. As part of FIONA , we developed code that provides vectorized nonuniform fast Hankel transforms that may have other uses (e.g., calculation of cosmological two-point correlation functions) beyond those considered here.

dark matter↗
DOE OSTI2026

Feature Engineering and Ensemble Methods for Imbalanced ICS Intrusion Detection: Pipeline Audit and Constrained Evaluation

Industries are becoming increasingly connected and are more vulnerable to cyberattacks due to the widened attack surface. Industrial Control Systems (ICS) are among the most critical sectors that malicious actors can target, as such attacks can cause significant operational disruption and physical damage. It is imperative to detect such attacks as early as possible. This paper evaluates constraint-conditioned optimistic performance estimates for traditional ML models in ICS intrusion detection (i.e., estimates obtained under contiguous, non-shuffled temporal evaluation without test-set alteration, but with pre-split feature engineering that may introduce temporal leakage, due to dataset constraints). Our findings are threefold. First, we quantify how iterative feature engineering affects tree-based ensemble performance and examine how pipeline decisions (split strategy, sampling scope, and cleaning policy) can inflate or reduce reported IDS results under constraint-bound evaluation. Second, we compare intrinsic class-imbalance handling across ensemble models. Third, under our current pipeline constraints (including pre-split feature engineering), CatBoost achieves the best performance on Water Storage Tank (accuracy: 0.9831, class-1 F1: 0.9682), while Light- GBM achieves the best performance on Gas Pipeline (accuracy: 0.9618, class-1 F1: 0.9086).

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2023

AI ATAC 1: An Evaluation of Prominent Commercial Malware Detectors

This work presents an evaluation of six prominent commercial endpoint malware detectors, a network malware detector, and a file-conviction algorithm from a cyber technology vendor. The evaluation was administered as the first of the Artificial I ntelligence Applications t o Autonomous Cybersecurity (AI ATAC) prize challenges, funded by / completed in service of the US Navy. The experiment employed 100K files (50/50% benign/malicious) with a stratified distribution of file types, including ~1K zero-day program executables (increasing experiment size two orders of magnitude over previous work). We present an evaluation process of delivering a file to a fresh virtual machine donning the detection technology, waiting 90s to allow static detection, then executing the file and waiting another period for dynamic detection; this allows greater fidelity in the observational data than previous experiments, in particular, resource and time-to-detection statistics. To execute all 800K trials (100K files × 8 tools), a software framework is designed to choreograph the experiment into an automated, time-synced, and reproducible workflow with substantial parallelization. Software with base classes for this framework are provided. A cost-benefit model was configured to integrate the tools’ detection statistics into a comparable quantity by simulating costs of use. This provides a ranking methodology for cyber competitions and a lens for reasoning about the varied statistical results. The results provide insights on state of commercial malware detection.

Bridges, Robert↗