SEARCH · Engineering Papers

Results for “test metrics”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

SourceResearch connectionFrom yearThrough year

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 37 records · Page 2

All sources NASA NTRS DOE OSTI

DOE OSTI2025

The Triple Catalytic Action of Tertiary Nitrogen Catalysts in Recyclable Epoxy-Anhydride Thermosets

The thermosetting polymer matrix in fiber reinforced composites is an important component for energy related applications, such as the lightweighting of vehicles or their use in wind and waterpower turbine blades, due to their ability to provide superior adhesion, stiffness, and applicability to a wide range of manufacturing processes. Despite these benefits, today's thermosets are widely considered to be unrecyclable; thus, there is a large interest in redesigning these materials to be inherently recyclable so that energy intensive production of fibers and monomers can be circumvented, bolstering composite manufacture supply chains. Polyester covalent adaptable networks (PECANs) are one such promising alternative to the incumbent, nonrecyclable epoxy-amine thermosets. PECANs can be formed from the ring-opening co-polymerization (ROCOP) of epoxy-anhydride monomer mixtures and subsequent curing at mild temperatures to exhibit similar performance to conventional epoxies while also possessing unique dynamic chemistries along the ester-hydroxyl backbone that are capable of transesterification and thus reprocessability. While significant advancements have been made in formulating these materials for improved mechanical properties or optimizing solvolysis and reprocessing strategies, less attention has been placed on the impact of the residing amine catalyst used to generate the polyester network. In this work, we evaluated the triple-catalytic efficacy of 12 tertiary amines that act as a curing (bulk ROCOP), a transesterification (internal bond exchange), and a deconstruction (methanolysis) catalyst for PECAN thermosets. Specifically, we first distinguish between chain-growth and step-growth polymerization mechanisms for epoxy-amine and epoxy-anhydride mechanisms. We also utilized density functional theory (DFT) to estimate the basicity (pKb) of each catalyst. Of the tested catalysts, the ROCOP of the studied PECAN network can be completed between 95 and 247 min (at 80 degrees C), with variable gelation phenomena. Additionally, the stress relaxation (transesterification metric) efficiency of the tested PECAN networks with alternative embedded catalysts ranged from 95% to 15% reduction in stress after 5 h at 200 degrees C, and the depolymerization efficacy ranged from 2.5% to 9.8% deconstruction after 36 h at 130 degrees C. Overall, the nitrogen-based moieties were demonstrated to influence polymerization kinetics, catalyze the dynamic transesterification exchange mechanism, and aid in the solvolysis of the thermosets at end-of-life.

36 MATERIALS SCIENCE↗

DOE OSTI2024

Prioritizing Off-gas Metrics

Off-gas research within the Material Recovery and Waste Form Development (MRWFD) team is dispersed among four national laboratories and multiple universities and small businesses. Ensuring research is conducted consistently and in a manner conducive to direct comparison is a nontrivial task. The team assembled to realign goals and expectations by identifying distinct R&D phases, prioritizing crucial metrics, and outlining standardized tests specifically associated with Kr, Xe, and I capture.

11 - NUCLEAR FUEL CYCLE AND FUEL MATERIALS↗

DOE OSTI2025

A building performance-based approach to determining energy resilience for grocery stores in the event of a power outage

Evaluating progress toward a built environment that is best equipped to serve communities during a regional power outage will require metrics that capture the energy resilience of the unique buildings and businesses most crucial to the well-being of those nearby. We focused on grocery stores as key buildings for which access, and thus energy resilience, is critical during a disaster when power is unavailable. We evaluated the energy resilience of these buildings by offering and testing building-scale metrics that assess business continuity potential during a power outage. Metrics proposed in this study are calculated based on the unique power loads characteristic to grocery stores, primarily refrigeration and maintaining safe indoor environmental conditions. Building simulations based on varying levels of backup power were carried out against occupant safety and comfort parameters to apply these metrics, with additional criteria imposed on grocery stores to capture the inventory and sales loss from food spoilage resulting from a lack of refrigeration power. Findings from this study demonstrate the feasibility of our proposed metrics and methodology to serve as a low-data burden means for stakeholders to evaluate the energy resilience of grocery stores, with greater implications in helping to understand the impact on community-scale energy resilience.

Siegel, Lino Sanchez↗

DOE OSTI2024

Optimal Transport for e/$\pi^0$ Particle Classification in LArTPC Neutrino Experiments

Separation of electron signals from $\pi^0$ backgrounds is crucial for neutrino oscillation measurements and searches for Beyond Standard Model (BSM) physics in current and future Liquid Argon Time Projection Chamber (LArTPC) experiments. e/$\pi^0$ separation has been a reconstruction challenge since both e and $\pi^0$ present as electromagnetic showers, and often only one out of the two showers produced by $\pi^0$ is reconstructed correctly. This research aims to improve the performance of e/$\pi^0$ separation using optimal transport (OT), by leveraging on the topological differences in the showers produced by the two particles. OT is a method which compares two distributions by finding the most efficient way to transform, or “move” from one to the other. This work uses the MicroBooNE open samples public dataset to test the e/$\pi^0$ separation performance of the method on events which incorporate realistic modeling of LArTPC detector response. Reconstructed 3D energy deposits are projected onto a plane perpendicular to the primary shower, allowing OT to better detect the topological differences between the two types of particles without the need to separately reconstruct all the showers in the events. Different distance metrics for OT are tested and preliminary results on e/$\pi^0$ separation are presented.

43 PARTICLE ACCELERATORS↗

DOE OSTI2025

Exxon Mobil-NETL Testing of DAC Materials

This joint initiative is aimed at extending our understanding of real DAC testing conditions at the NETL DAC testbed facility. Of particular focus will be: • Small Scale testing of powdered and formulated materials (supplied by ExxonMobil) to evaluate various performance metrics under DAC process cycles • Pilot scale testing of formulated materials at larger scales ExxonMobil will work with NETL to shake down equipment, validate testing methods, and define best practices for data analysis. Three tasks are proposed: 1. Validation of multi-cycle test data on powdered and formulated materials to benchmark various performance metrics of these materials. 2. Steam regeneration of powdered samples at small scale to define baseline performance under commercially relevant conditions. 3. Pilot scale testing of larger formulated materials to test commercially relevant samples under actual process cycles.

36 MATERIALS SCIENCE↗

DOE OSTI2024

Proof-of-concept chemometric approach for environmental forensic sourcing of crude oil samples using SPME-GC-MS

Environmental exposure to crude oil through seepage and spillage poses risks to the immediate environment and the broader ecosystem as areas along the oil distribution path are affected by the influx of crude petroleum as well as the environmental, economic, and civil unrest that accompanies it. There is a large financial burden associated with the lost resources, including the cost of rehabilitation, and the affected sources of revenue for communities affected by oil spills. As such, it is crucial to determine the responsible parties. This work outlines an environmental forensics approach to determining the source of an un-weathered crude oil sample. The researchers employed solid phase microextraction coupled with gas chromatography mass spectrometry (SPME-GC-MS) to capture and analyze the gaseous components emitted by crude oil samples sourced from five locations. Samples were analyzed using Spearman's rank correlation and 3D covariance analysis. Both chemometric approaches yielded optimal performance results with no misclassifications, true positive rate (TPR) = 100 % and false positive rate (FPR) = 0 %. The similarity metrics calculated by each test noted clear delineations between the values of same-source and differently sourced samples. The Spearman's rank correlation test and 3D covariance calculations both demonstrated the ability to correctly identify sample source origin in this dataset. Finally, the authors outline an approach to the future application of these tests and suggest their joint use in future crude oil sourcing endeavors.

3D covariance mapping↗

DOE OSTI2026

Design and Prototyping of the 162.5 MHz PIP-II RF Reference-Line Station

The Proton Improvement Plan II (PIP-II) Reference Line distributes phase-stable radio-frequency signals so accelerating cavities transfer energy to the proton beam at the correct point in each RF cycle. The full system contains 162.5, 325, and 650 MHz sections; this project focuses on the 162.5 MHz station. Commercial filters, amplifiers, splitters, mixers, and hardware were assembled into a compact, serviceable heat-plate layout and tested to verify performance metrics at 162.5 MHz.

Subedi, Harsheet [Unlisted, US]↗

DOE OSTI2024

2024 Photovoltaic Inverter Reliability Workshop Summary Report & Proceedings

The National Renewable Energy Laboratory (NREL) organized the 2024 Photovoltaic Inverter Reliability Workshop on April 11-12, 2024, hosted at NREL's South Table Mountain campus in Golden, Colorado. The workshop was organized around seven key topics, including the present state of inverter reliability; solutions for reliability challenges; life cycle cost and ownership issues; testing, standards, performance, and reliability metrics; data reporting, analytics, and sharing; and the future of PV inverter reliability research. Participants included inverter manufacturers, national laboratory researchers, academics, independent testing laboratories, and more. Over the course of the two-day workshop, attendees arrived at several key priorities and conclusions. This report summarizes these conclusions and then collects presentations from the workshop into a record of the workshop's proceedings.

14 SOLAR ENERGY↗

DOE OSTI2025

Performance Evaluation of a Single-Phase Grid-Forming Inverter Through Hardware Experiments

This study conducts hardware experiments to assess the performance of a commercial single-phase grid-forming (GFM) inverter using a purely hardware-based approach. We adhere to a testing protocol for the GFM inverter and enhance it by exploring the transient performance of GFM inverters across various grid dynamic events. Quantifiable performance metrics are established for each testing scenario to gain insights into the performance of the single-phase GFM inverters. Based on the comprehensive tests, the following observations are summarized: 1) The performance of the single-phase GFM inverter is satisfactory, and it is capable of being the islanding master for residence homes when the main grid is gone. 2) For frequency response, the inverter is able to stay connected but cannot inject the needed active power to support the grid. This may be improved by the manufacturer by using droop control. 3) The GFM inverter exhibits harmful transients in voltage and frequency during islanding operation, which can be enhanced by maintaining the same operating points before and after the breaker is open.

black start↗

DOE OSTI2024

Performance Evaluation of a Single-Phase Grid-Forming Inverter Through Hardware Experiments

dynamic performance↗

DOE OSTI2025

Optimizing cost-effective and benchmarked industry standards to quantify nutrient bioextraction by seaweed

Interest in the utility of seaweed farms to mitigate coastal eutrophication, or nutrient loading, has grown commensurate with the recent rise of the farmed seaweed industry in the U.S. But economic valuation of this ecosystem service remains elusive in part because of challenges in quantifying this spatiotemporally variable biological process with reproducible and comparable metrics. Regulatory bodies that permit wastewater discharge or lease area for aquaculture farms require water quality testing and reporting of dissolved total nitrogen (N) in nearshore marine environments. These metrics must meet EPA standards for testing and reporting (e.g., Total Kjeldahl Nitrogen - TKN). However, these metrics are inherently highly variable over space and time in dynamic nearshore systems, and expensive to evaluate with sufficient breadth to constrain this variance, creating a critical bottleneck to direct quantification of farmed seaweed net uptake rates in situ.

59 BASIC BIOLOGICAL SCIENCES↗

DOE OSTI2026

A multi‐variable framework for selecting WRF physics configurations at convection‐permitting scales: An Amazon wet‐season case study

Tropical convection over rainforests modulates atmospheric circulation and the energy and hydrological cycles across multiple scales. However, the scarcity of observations still limits our understanding of these processes. Although numerical models are utilized to investigate atmospheric physical processes, their performance depends on the choice of parameterizations and grid resolution. Here, in this work, we introduce and apply a multi‐variable, multi‐physics framework to evaluate and rank the Advanced Research Weather Research and Forecasting (WRF–ARW) configurations at 1‐km resolution over the central Amazon during the wet season. A 48‐member ensemble combines three land‐surface models (LSMs), four planetary boundary layer (PBL), and four microphysics (MP) schemes. Model performance for seven convection‐related near‐surface and boundary‐layer variables is assessed using Taylor diagrams and the Taylor skill score (TSS), analysis of variance (ANOVA)‐based sensitivity metrics, and non‐parametric rank tests. We then construct a combined, weighted TSS to identify configurations that are comparatively robust across variables. Results showed that most configurations reproduce near‐surface temperature, sensible and latent heat fluxes, and boundary‐layer height reasonably well, whereas humidity and rainfall remain challenging. LSM choice has the strongest impact on the surface fluxes and a secondary influence on near‐surface temperature and humidity, PBL schemes dominate boundary‐layer height, and MP schemes exert the largest control on rainfall. No single configuration is optimal for all variables, but the combination of Noah (LSM), Yonsei University (PBL), and Morrison (MP) emerges as the most robust configuration for this case, with the WRF single‐moment six‐class scheme (WSM6) providing a competitive, computationally cheaper MP alternative. The framework is general and can be applied to other regions, seasons, and convective regimes.

Amazon rainforest↗

DOE OSTI2024

Smoothing and flattening the universe through slow contraction versus inflation

In a systematic study, we use an equivalent pair of improved numerical relativity codes based on a tetrad-formulation of the classical Einstein-scalar field equations to examine whether slow contraction or inflation (or both) can resolve the homogeneity, isotropy and flatness problems. Our finding, based on a set of gauge/frame invariant diagnostics and the models considered, is that slow contraction robustly and rapidly smooths and flattens spacetime beginning from initial conditions that are outside the perturbative regime of the flat Friedmann-Robertson-Walker metric, whereas inflation fails these tests. We present new numerical evidence supporting the conjecture that the combination of ultralocal evolution and an effective equation-of-state with pressure much greater than energy density is the key to having robust and rapid smoothing. The opposite of ultralocality occurs in expanding spacetimes, which is the leading obstruction to smoothing following a big bang.

79 ASTRONOMY AND ASTROPHYSICS↗

DOE OSTI2024

ADRIANO2 Calorimeter Performance from 2022 Test Beams

A novel high-granularity dual-readout calorimetric technique was developed as part of the T1604 collaboration. The ADRIANO2 Calorimeter Prototype consists of a pair of optically isolated, small sized tiles made of scintillating plastic and lead glass. Čerenkov light from the lead glass are exploited to for high resolution timing measurements, while high granularity from scintillating plastic can be used to probe the spatial component of the particle shower. This setup works for excellent energy resolution and particle detection for REDTOP as it is crucial for a calorimeter to detect the decay products of eta/eta-prime mesons. Measurements were collected on ADRIANO2 between February to December 2022 to evaluate the detector performance at Fermilab’s Test Beam Facility. The key metrics extracted from my analysis are the detector’s efficiency for various tile configuration and light-yield which will then be used as parameters for an upgraded REDTOP monte-carlo simulation campaign. An in-depth analysis of ADRIANO2 performance are detailed in this presentation.

43 PARTICLE ACCELERATORS↗

DOE OSTI2024

Automating the Analysis of Large Language Models Responses through Zero-Shot Question Answering

Recent advancements in Large Language Models (LLMs) have shown significant potential in various applications, yet their evaluation, particularly in zero-shot question answering scenarios, remains a challenging task. In this study, our objective was to explore precision metrics for Large Language Models (LLM) and design and implement a software pipeline to automatically evaluate LLMs' outputs under zero-shot question answering. Zero-shot question answering involves a model providing answers to questions about topics it hasn't seen during training. It leverages the principles of zero-shot learning by relying on semantic understanding and generalization from related knowledge. The data used was metadata from medical databases on congenital heart disease. We explored eleven LLM metrics and selected three for our evaluation: BLEU, BERTScore, and MoverScore. BLEU calculates a score based on the overlap of n-grams (contiguous sequences of n items, typically words) between the machine-generated translation and the reference translations. Higher BLEU scores indicate better correspondence between the machine-generated and human-generated translations. BERTScore is a metric used to evaluate the quality of machine-generated text by measuring the similarity of token embeddings produced by BERT (Bidirectional Encoder Representations from Transformers) between the generated text and reference text. MoverScore is a metric that quantifies the dissimilarity between the distributions of word embeddings from machine-generated text and reference text, emphasizing semantic similarity over exact token overlap. We also introduced HBKI, a composite metric summarizing these approaches. We tested five models —GPT-3, Llama-2, Gemini 1.5 Pro, Solar 10.7B, and Mixtral-8x7b. Our software pipeline, designed and implemented using Object-Oriented Programming principles, allows users to customize the selection and extraction of features for topics of interest in their own research. Our results show that MoverScore delivered the most precise evaluation of the LLM's outputs, while Mixtral-8x7b achieved the best overall performance in extracting metadata from the databases.

97 MATHEMATICS AND COMPUTING↗

DOE OSTI2025

MS25: Materials Science-Focused Benchmark Data Set for Machine Learning Interatomic Potentials

Here, we present MS25, a benchmark data set for evaluating machine learning interatomic potentials (MLIPs) across diverse materials-relevant systems including MgO surfaces, liquid water, zeolites, a catalytic Pt surface reaction, high-entropy alloys (HEAs), and disordered Zr-oxides. Five MLIP architectures (MACE, NequIP, Allegro, MTP, and Torch-ANI) are trained and tested, focusing not only on traditional metrics (energies, forces, and stresses) but also explicitly validating derived physical observables such as lattice constants, volumes, and reaction barriers. We find that most models reach comparable accuracy on standard error metrics across the simple systems, although equivariant MLIPs offer 1.5–2× improvements over nonequivariant MLIPs in energy and force error for structurally complex or compositionally disordered environments such as HEAs and Zr–O systems. Our analysis highlights that low errors in energy and force predictions do not guarantee reliable observables, emphasizing the necessity of explicit validation. We demonstrate limitations in cross-framework transferability, as models trained on one zeolite framework (CHA) fail to reliably generalize to predictions of structurally distinct frameworks (e.g., MFI). Size-extensive tests show some dependence on system size for MgO, resulting from forced periodicity. The HEA and Zr–O data sets are identified as challenging tests for future benchmarks and MLIP model architecture developments as they show significant differentiation in error between MLIP architectures and are still relatively difficult at 1000 training images. Moving forward, we recommend that benchmarking efforts shift their focus from marginal accuracy improvements in energy and force errors toward identifying and understanding model failure modes, rigorously assessing transferability, and evaluating how their errors affect observable predictions. For researchers looking to choose an MLIP architecture, we suggest selecting equivariant MLIP architectures if the complexity of the system is a challenge. For simple materials problems, auxiliary features such as integration with molecular dynamics engines, trade-offs between computational data set generation cost vs MLIP inference speed, and framework integration may play a more important decision factor than small differences in error metrics that are unlikely to matter for production-level research.

chemical structure↗

DOE OSTI2025

Review of Grey Box/Black Box Data Contamination Metrics on Open and Commercial Models

Dataset contamination is a problem where benchmarks and tasks used to evaluate the capabilities of Large Language Models (LLMs) have been incorporated into the training dataset of the models. This gives a false sense of performance that can overestimate how these models will function on truly unseen data. This problem becomes worse with commercial LLMs with larger and non-accessible training data, so techniques have been developed to try to measure the degree to which a model is contaminated with a benchmark’s data. To understand the effectiveness of these techniques, particularly when evaluating contamination on coding tasks, we review trends and categorize techniques by the degree of access to the model that is required. The research literature on this topic has reported mixed effectiveness of these techniques, so we select a set of black box (text access only) and grey box (access to model loss/probabilities required) techniques and apply them to both commercial and non-commercial models. We implement these metrics as part of a framework to test the contamination of Python code in LLMs to see to what extent we can replicate the effectiveness (or ineffectiveness) of these contamination detection techniques. Though we find mixed results in the capabilities of these metrics to identify contamination, we do observe evidence that they can identify contamination (broadly) in fine-tuned models when both a baseline and fine-tuned model is present. Additionally, similarity metrics were able to identify between contaminated and uncontaminated data even in situations where the data is distributionally similar (e.g., drawn from the same set of code projects).

97 MATHEMATICS AND COMPUTING↗

DOE OSTI2025

Validation of Pronghorn for Natural-Circulation Molten Salt Loops

This paper presents the development and validation of a high-fidelity thermal-hydraulic model of a molten salt natural circulation flow loop, designed for integration within a digital twin framework. The study evaluates the performance of Idaho National Laboratory’s Pronghorn against experimental data from Texas A&M University Molten Salt Flow Loop (MSFL) four Hitec-salt test benchmark data. Natural circulation of high-Prandtl-number fluids exhibits complex, counter-intuitive flow patterns that make pointwise thermocouple readings unreliable. Experimental work at TAMU’s MSFL provides benchmark data, including flow visualization at a test-section and centerline steady-state temperature measurements along the loop. Validation includes four single-phase natural circulation test cases with Hitec salt. Key metrics include flow profile agreement and steady-state temperature accuracy. Pronghorn results for two-dimensional single-phase agree qualitatively with the experimental flow profile. This paper illustrates the importance of Computational Fluid Dynamics (CFD) in elucidating the behavior of high-Prandtl-number thermal-hydraulics, along with how misleading centerline temperature measurements can be. Pronghorn reproduces the axial and radial stratification that makes single thermocouple readings unreliable. Future research will focus on reduced-order modeling techniques to enable rapid simulation suitable for real-time digital twin applications. The validated cases provide a basis for developing reduced-order surrogates aimed at real-time digital-twin applications.

22 - GENERAL STUDIES OF NUCLEAR REACTORS↗