Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “METRICS”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 793 records · Page 44
All sourcesNASA NTRSDOE OSTI
DOE OSTI2021

Evaluation of Oxygen Consumption as a Sensitive Measure of Electrical Cable Polymer Insulation Degradation

Polymers used as dielectric materials in nuclear grade electrical cables, such as cross-linked polyethylene (XLPE) and ethylene-propylene rubber (EPR), are long-lived and durable. Estimating the decades-long useful service life of polymers experimentally in a feasible timeframe of weeks or months requires greatly accelerated aging in the laboratory at elevated temperatures and/or gamma dose rates. Many limitations of accelerated aging, such as due to artefacts of the process including diffusion limited oxidation and dose rate effects, have been identified that can make accurate lifetime prediction at service conditions challenging. This can be due to differences in degradation mechanisms between rapid and extended aging or to the presence of induction periods during which early aging is undetectable using common metrics of polymer insulation characterization such as tensile elongation at break. Conceptually, an ideal measure of aging would track clearly and predictably with material changes throughout the lifecycle of the evolving material state from early aging through end of useful life. With such a metric accelerated aging could be performed at much milder conditions, conditions closer to those found in long term service, and end of life calculated from the early data. Oxygen consumption has previously been proposed as a sensitive measure of insulation polymer oxidation that might meet such desirable characteristics. In this work, nuclear cable insulation samples of the two most common types, XLPE and EPR, from two of the most sourced manufacturers, RSCC and The Okonite Company, were subjected to gamma irradiation at dose rates of 100, 200, and 300 Gy/hr for up to 42 days at 26°C. The aged samples were used to evaluate oxygen consumption as a sensitive measure of aging for these materials and these conditions. For comparison and validation, the aged samples were also characterized using the more conventional tests of mass change, density, carbonyl index, and yellowness index. For both materials, oxygen consumption during irradiation was found to track consistently with applied dose—oxygen consumption rate was linear with time. Mass change and carbonyl index were also clearly proportional to exposure dose. Yellowness index, a surface sensitive measure, was observed to increase linearly with dose at low exposure levels before plateauing. Sample density was not observed to change under the conditions explored. Oxygen consumption was found to be a particularly sensitive metric to measure the degradation of polymeric cable insulation exposed to radiation. Oxygen consumption was compared to Arrhenius and time-temperature-superposition methods for activation energy calculation using literature date and found to be a useful and viable alternative for lifetime prediction. The techniques developed in this work are anticipated to be valuable in the estimation of insulation aging rates under mild conditions approaching service conditions. Direction comparison of useful life prediction by oxygen consumption with prediction via Arrhenius methodology for thermal aging and existing methods for radiation and combined thermal/radiation aging would serve to further validate this promising approach and refine our understanding of material-specific aging rates.

36 MATERIALS SCIENCE↗
DOE OSTI2023

Energy Resilience for Mission Assurance, Agile Co-simulation for Cyber Energy System Security (ACCESS) Model Advancements for Resilience Analysis (Version 3, July 2023)

Agile Co-simulation for Cyber Energy System Security (ACCESS) is a co-simulation platform developed by Lawrence Livermore National Laboratory (LLNL). The primary high-level use-case for ACCESS is to study existing or new cyber-physical critical infrastructure systems, with a heavy emphasis on 1) systems that utilize communication networks, and 2) studies that seek to understand cyber-related system impacts. ACCESS is currently used for several energy system resilience projects at LLNL. In the Energy Resilience for Mission Assurance (ERMA) project, ACCESS is used in the Mod eling for Metric Calculation task (specifically, subtask 4.3, Communications and Cyber Modeling) to model and simulate the cyber and communication system aspects of Defense Critical Electric Infrastructure (DCEI) systems, with a focus on computing specific communication system metrics that can impact system resilience and mission performance. Simulated communication system per formance will be fed back to other ERMA system components so that mission performance can be evaluated holistically. This report describes several enhancements to the ACCESS platform that were implemented during the execution of the ERMA project in support of reslience analysis. This includes the addition of new models and subsystems, enhancements to existing models, and integration with external systems. The remainder of this report is structured as follows. In Section 2, a brief background description of the ACCESS platform is provided, including an outline of ACCESS components, example use cases, and a set of communication network resilience metrics that can be computed with ACCESS. Section 3 describes the ACCESS model enhancements for ERMA in detail. Finally, Section 4 briefly outlines future integration opportunities between ACCESS and project participant capabilities identified during the progression of the project.

29 ENERGY PLANNING, POLICY, AND ECONOMY↗
DOE OSTI2021

Task 12 Sustainability - Methodological Guidelines on Net Energy Analysis of Photovoltaic Electricity (2nd Edition)

Net Energy Analysis (NEA) is a structured, comprehensive method of quantifying the extent to which a given energy source is able to provide a net energy gain (i.e., an energy surplus) to the end user, after accounting for all the energy losses occurring along the chain of processes that are required to exploit it (i.e., for its extraction, processing and transformation into a usable energy carrier, and delivery to the end user), as well as for all the additional energy 'investments' that are required in order to carry out the same chain of processes. However, this general framework leaves the individual practitioner with a range of choices that can affect the results and thus, the conclusions of a NEA study. The current IEA PVPS guidelines were developed to provide guidance on assuring consistency, balance, and quality to enhance the credibility and reliability of the results from photovoltaic (PV) NEAs. The guidelines represent a consensus among the authors - PV NEA experts in North America and Europe - for assumptions made on PV performance, process inputs and outputs, methods of analysis, and reporting of the results. Guidance is given on photovoltaic-specific parameters used as inputs in NEA and on choices and assumptions in inventory data analysis and on implementation of modelling approaches. A consistent approach towards system modelling, the functional unit, the system boundaries and allocation aspects enhance the credibility of PV electricity NEA studies and enables balanced NEA-based comparisons. Specifically, "apples-to-oranges" comparisons of different energy carriers (e.g., fuels vs. electricity) are not methodologically sound and are to be avoided in all cases; also, any comparison across renewable and non-renewable electricity generation technologies must clearly point out the intrinsically short-term nature of the NEA viewpoint, which does not capture the long-term sustainability implications of renewable vs. non-renewable primary energy harvesting and use: non-renewable primary energy resources are depleted and finally exhausted (irrespective of the size of the EROI), while renewable primary energy resources are not. This document provides an in-depth discussion of a common metric of NEA, namely the energy return on investment (EROI), and how this is to be interpreted vis-a-vis the deceptively similar-sounding metrics in the field of Life Cycle Assessment (LCA): cumulative energy demand (CED) and non-renewable cumulative energy demand (nr-CED) per unit output. Specifically, a number of key differences are highlighted between these metrics as applied to electricity production systems, which are listed in Table S-1.

14 SOLAR ENERGY↗
DOE OSTI2024

Cambium 2023 Scenario Descriptions and Documentation

The National Renewable Energy Laboratory's (NREL's) Cambium data sets are annually released sets of simulated hourly emission, cost, and operational data for a range of modeled futures of the U.S. electric sector with metrics designed to be useful for long-term decision- making. The 2023 Cambium data set is the fourth annual release. The data sets are a companion product to NREL's Standard Scenarios, which are likewise released annually and are a set of projections of how the U.S. electric sector could evolve across a suite of different potential futures, but covering more scenarios with less temporal granularity (Gagnon et al. 2024). Information about Cambium and related publications can be found at https://www.nrel.gov/analysis/cambium.html, and the Cambium data sets can be viewed and downloaded at https://scenarioviewer.nrel.gov/. In this documentation, we describe Cambium 2023's scenarios, define the metrics, and document the Cambium-specific methods for calculating those metrics.

24 POWER TRANSMISSION AND DISTRIBUTION↗
DOE OSTI2024

A Review of Resilience and Long-Term Planning in Power and Water Systems in the United States

There is recognition among power and water utilities that the frequency and magnitude of high consequence and low probability events could increase as a result of climate change. The interconnected nature of energy-water systems raises the possibility of cascading failures, increasing complexity and risks. Resilience and long-term planning are important ways of weathering the effects of climate change. First, to understand more about resilience, we reviewed existing literature on resilience definitions, metrics, and modeling, focusing on integrated water-power systems. Second, to understand how resilience and planning are being applied in practice, we interviewed utilities and organized, curated, and synthesized the interview data to arrive at several key findings, which are presented here. We found that there is not a consistent definition for resilience, yet it is something that utilities regularly plan for, often with different names and varying methods/measures. However, there is a tangible shift in the industry towards defining and determining measurable resilience metrics. While the exact metrics are a work in progress, utilities are taking steps forward by (1) putting people and culture at the center of resilience, (2) recognizing their own interdependencies, and (3) pursuing better cross-sector collaboration.

29 ENERGY PLANNING, POLICY, AND ECONOMY↗
DOE OSTI2024

HERO CarbonSAFE Phase 2 Project in the Columbia River Basalt Group

The Hermiston, Oregon Basalt CarbonSAFE Phase II project (HERO CarbonSAFE) seeks to accelerate the deployment of commercial carbon dioxide (CO2) storage projects in basaltic rocks. Basalt CO2 storage has several advantages to conventional saline storage reservoirs including 1. The potential for rapid mineralization of CO2, 2. Associated decreases in pressure and CO2 migration risks, 3. Reduced long-term monitoring requirements with respect to plume tracking, 4. Widespread geographic distribution and, 5. Large storage potential due to thickness, porosity, and CO2 interactions with basalt. And for locations such as the Pacific Northwest, Hawaii, Iceland, India and Japan, basalts may offer the only economically feasible option for local CO2 storage. However, there are limited field-scale assessments of CO2 storage in basalt, and current carbon capture utilization and storage (CCUS) permitting and regulatory frameworks were developed for conventional saline reservoirs. HERO CarbonSAFE is designed to address research gaps and uncertainties associated with basalt storage. Specifically, the project will assess the feasibility of CO2 injection in the deep layered basalts, long-term storage (mineralization), practical approaches for large-scale implementation (50+ million metric tons of CO2 over 30 years), lithology-specific risks, and the technoeconomic potential for CO2 storage in basalts. The HERO CarbonSAFE project will assess feasibility of developing a commercial-scale (50+ million metric tons of CO2) geological storage complex within the Columbia River Basalt Group (CRBG), a layered continental flood basalt complex that underlies Calpine’s natural gas-fired Hermiston Power Project (HPP) in Hermiston, OR (Figure 1). Under this 2-year CarbonSAFE Phase II project, the HERO team will conduct a data acquisition campaign that includes drilling a stratigraphic well to a total depth of ~1,500 m into the thick layered basalts proximal to HPP. A comprehensive well logging and hydrologic testing program will be augmented with new core collected from flow zones and sealing units, and comprehensive laboratory testing to help refine the kinetic rates of mineralization. The newly acquired information will be integrated with existing data from regional wells to correlate basalt injection zone properties to develop storage hub/commercial-scale models. Using these models, the project team will evaluate injection scenarios to define the technical and economic potential for storing a minimum of 50 million metric tons of CO2 over a 30-year period, along with a robust sensitivity analysis on key parameters governing reservoir viability for sustainable injection over a commercial project lifetime. Specific technical objectives of HERO are: (1) assessing the reservoir response of a series of stacked layered reservoir flowtop sequences occurring in this area of the CRBG to commercial-scale injection volumes; (2) extending prior efforts by the project team to characterize the deep layered basalts encountered in regional studies, to leverage prior investments by U.S. Department of Energy’s (DOE) Carbon Storage program; (3) leveraging DOE’s mineralization characterization efforts to advance model parametrization for commercial scale injection of CO2 in basalts; (4) conducting risk assessments associated with scaling up to commercial storage hub injection goals, while validating DOE’s National Risk Assessment Partnership (NRAP) tools, to identify potential constraints that would prevent the CRBG from serving as a commercial-scale storage complex; (5) developing mitigation plans to address identified risks; (6) developing a commercial-scale injection and monitoring, verification and accounting (MVA) strategy; (7) utilizing computational models to define and minimize, if possible, the Area of Review (AoR) under Class VI regulations; and (8) developing a robust CO2 management strategy for CRBG that also considers a regional source/sink approach that is responsive to stakeholder needs and industrial demand. Specific institutional objectives are: (1) identifying and developing plans to mitigate the nontechnical challenges associated with the build-out of a commercial-scale storage complex within the CRBG with integrated CO2 sources; (2) implementing the community outreach plan; (3) conducting regulatory research, including a survey of issues related to pore space ownership, MVA and long-term assurance of mineralization-based storage, to support an eventual application for a UIC Class VI permit; (4) advancing the project’s plan for CO2 liability management; and (5) continuing to refine and update the project’s economic model. The final objective is the preparation of a comprehensive Site Characterization Plan that draws upon the technical and institutional feasibility assessments to prepare the project for future commercialization efforts.

58 GEOSCIENCES↗
DOE OSTI2025

Open-Source Offshore Airfoil Summary (V.1.0)

The Open-Source Offshore (OSO) airfoils have been developed for research purposes for offshore wind turbines, offering a set of airfoils that align with modern turbine design requirements and industry design practices without proprietary constraints on research use. The eventual airfoil family will target the IEA 22 MW reference wind turbine, which was originally developed with the FFA airfoils. The two airfoils summarized in Table 1 (OSO-21-WT1 and OSO-30-WT1) started development as part of a family of airfoils being designed to target the IEA 22 MW wind turbine. The criteria used to design these airfoils are summarized in Table 1, which aim to encapsulate requirements of modern airfoils for offshore wind turbine applications, and were developed with feedback from industry and research experts. The airfoils were designed using XFOIL and candidate airfoils were then analyzed in RFOIL, which is considered more accurate than XFoil for high lift predictions of thicker airfoils. The design process for a preliminary family of airfoils is available, including a more detailed explanation of the design requirements and metrics similar to those used for these airfoils. Most of the design criteria are met for these two airfoils, with two exceptions. For both airfoils, the L/D Roughness Loss metric is exceeded (42% > 40% goal) and the desired lift coefficient margin over the design value (“CL_Margin”) was moderately exceeded (0.43 > 0.3) while smooth-stall characteristics (computed) were achieved. Note that all of the metrics were computed using RFOIL, and like other new airfoils, these will need to be experimentally validated at a range of Reynolds numbers. The airfoil coordinates will be shared publicly on Sadia National Laboratories’ public Github repository:

17 WIND ENERGY↗
DOE OSTI2025

Optimizing cost-effective and benchmarked industry standards to quantify nutrient bioextraction by seaweed

Interest in the utility of seaweed farms to mitigate coastal eutrophication, or nutrient loading, has grown commensurate with the recent rise of the farmed seaweed industry in the U.S. But economic valuation of this ecosystem service remains elusive in part because of challenges in quantifying this spatiotemporally variable biological process with reproducible and comparable metrics. Regulatory bodies that permit wastewater discharge or lease area for aquaculture farms require water quality testing and reporting of dissolved total nitrogen (N) in nearshore marine environments. These metrics must meet EPA standards for testing and reporting (e.g., Total Kjeldahl Nitrogen - TKN). However, these metrics are inherently highly variable over space and time in dynamic nearshore systems, and expensive to evaluate with sufficient breadth to constrain this variance, creating a critical bottleneck to direct quantification of farmed seaweed net uptake rates in situ.

59 BASIC BIOLOGICAL SCIENCES↗
DOE OSTI2025

Cambium 2024 Scenario Descriptions and Documentation

The National Renewable Energy Laboratory's (NREL's) Cambium data sets are annually released sets of simulated hourly data for a range of modeled futures of the U.S. electric sector with metrics designed to be useful for long-term decision- making. The 2024 Cambium data set is the fifth annual release. The data sets are a companion product to NREL's Standard Scenarios, which are likewise released annually and are a set of projections of how the U.S. electric sector could evolve across a suite of different potential futures, but covering more scenarios with less temporal granularity. Information about Cambium and related publications can be found at https://www.nrel.gov/analysis/cambium.html, and the Cambium data sets can be viewed and downloaded at https://scenarioviewer.nrel.gov/. In this documentation, we describe Cambium 2024's scenarios, define the metrics, and document the Cambium-specific methods for calculating those metrics.

24 POWER TRANSMISSION AND DISTRIBUTION↗
DOE OSTI2025

Templates for Risk Informed Assurance with Curvature Embeddings (TRACE)

We investigate recovery of geometric structure from networks embedded in manifolds with spatially varying curvature, extending the constant-curvature framework of Lubold et al. (2023). Our work supports cascade risk assessment in critical infrastructure through the Templates for Risk-informed Assurance with Curvature Embeddings (TRACE) framework. Simulations on a bi-modal Gaussian surface show that constant-curvature methods yield weighted averages shaped by clique patterns, while hierarchical clustering identifies distinct regimes. Localized estimation, however, reveals boundary contamination in transitional regions. To address heterogeneity, we develop distance metrics for graphs with edge and node features, proving their metric validity, and validate them via deterministic graph generation from canonical tilings. We further propose a diffusion-based anomaly detection approach that treats networks as glued manifolds, using curvature discontinuities to detect structural anomalies. Employing the carré-du-champ operator and scalar curvature, we achieve robust anomaly discrimination, demonstrated on the Singapore Water Treatment (SWaT) dataset with joint network-traffic and sensor features. Integration with TRACE reveals how curvature shapes cascade dynamics: positive curvature impedes, while negative curvature accelerates propagation. This geometric perspective provides interpretable risk metrics and visualization tools for critical infrastructure managers. While full validation remains ongoing, our contributions establish a rigorous foundation for geometric analysis of network resilience and cascade vulnerability.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2025

Remote Sensing Approach for Monitoring Tree Health Adjacent to Transmission Corridors

This study presents an initial proof-of-concept for a satellite-based remote sensing approach to identify and monitor potential areas of poor tree health across the entire BPA service territory on an annual basis. We tested three variants of “delta peak NDVI” ( ΔPN ) change detection metrics that express interannual variation in primary productivity relative to a baseline by comparing ΔPN values for known insect/disease disturbances and nearby reference locations. All three metrics showed promise for detecting poor tree health in the year during disturbance, but the metric based on the difference from the long-term (2016-2024) median ( Δ Med PN ) was preferred due to its responsiveness to change in the years during and after disturbance, resilience to interannual variation, and ease of interpretation as being above or below normal. Comparison of Δ Med PN grouped by relative severity of disturbance indicated it was not sensitive enough to detect “low” severity disturbances, as mapped by USGS’s LANDFIRE program, but could distinguish “moderate” and “high” severity disturbances from reference locations. These findings informed selection of a threshold for Δ Med PN , which was combined with areas exhibiting negative NDVI to map potential areas of concern. Visual inspection of before/after high-resolution imagery and NDVI time series showed that many areas of concern aligned with visible signs of defoliation and die-off as well as other types of disturbance (e.g., landslides, logging, road grading, flooding). Some areas of concern are thought to be false detections caused by persistent shadow, and some could not be explained with visual inspection due to spatiotemporal limitations of before/after imagery. In summary, our approach shows promise for large-scale monitoring of tree health adjacent to BPA transmission lines, but additional work is recommended to improve model sophistication and remove noise.

24 POWER TRANSMISSION AND DISTRIBUTION↗
DOE OSTI2025

Evaluating Tools and Technologies for Monitoring Baleen Whales During Offshore Wind Foundation Installation

Impacts to baleen whales are of particular concern in the context of offshore wind development because of their conservation status, the overlap of their habitat with current and planned offshore wind energy facilities, and their expected sensitivity to low-frequency sounds, such as those produced during pile driving. Therefore, this report focuses on monitoring technologies capable of detecting this species group. The scope of this report is to assess the current state of technologies and methods for monitoring baleen whales in low- and no-light conditions in conjunction with offshore wind foundation installation. This work considers the performance of various sensing technologies that can be used to observe the presence of baleen whales near offshore construction activities. The scope of work includes: 1. Technology Performance Metrics – Identify metrics that can be used to assess technology performance. 2. Technology Evaluation – Synthesize the past performance of technologies with respect to the performance metrics based on available literature. 3. Technology Characterization Framework – Describe a potential framework to characterize technology performance with a standardized approach. 4. Technology Research and Development Needs – Identify future needs and opportunities for research, development, and deployment to improve technologies.

17 WIND ENERGY↗
DOE OSTI2025

Automated Segmentation of Twin Boundaries in TRISO Silicon Carbide Using Deep Neural Networks

Coated particle fuels, such as the tristructural isotropic (TRISO) fuel particle, are essential for high-temperature gas reactor (HTGR) applications due to their efficiency and stability under normal and off-normal conditions. However, widespread commercialization and deployment of this technology for next-generation nuclear applications require robust quality assurance and quality control (QA/QC) methods linking fabrication, properties, and performance. Of the many important metrics for TRISO QA/QC, quantification of the silicon carbide (SiC) microstructure is critical because it correlates with fission product retention during irradiation. Previous work has shown extensive twinning of the SiC microstructure, which strongly affects microstructural metrics; however, twin grain boundaries are not expected play a significant role in fission product diffusion. This report summarizes the initial development, training, and testing of a machine learning image processing algorithm to detect twin grain boundaries in a backscattered electron image, which can be removed so that microstructural metrics can be recalculated for legacy data. Further development and deployment of this model will provide automated, scalable improvement of potential QA/QC methods for the SiC layer of TRISO particles.

11 NUCLEAR FUEL CYCLE AND FUEL MATERIALS↗
DOE OSTI2023

Assessing the Current State of U.S. Energy Equity Regulation and Legislation [Slides]

This report examined state energy equity actions from January 2020 to July 2022. Lawrence Berkeley National Laboratory and Pacific Northwest National Laboratory partnered with E9 Insight to screen executive, legislative, and regulatory actions focusing on energy equity. A total of 95 actions were identified across 22 states and Washington, DC. Energy equity was most associated with resource planning, decarbonization, and energy efficiency regulatory focus areas. States tended to focus on distributive and procedural justice tenets over recognition and restorative justice tenets. The review also suggested that energy equity metrics are in nascent stages; most outcomes identified in equity actions did not have corresponding metrics. Affordability and energy burden were identified as metrics in multiple cases and appeared in several different forms.

29 ENERGY PLANNING, POLICY, AND ECONOMY↗
DOE OSTI2025

Ultrasonic Characterization of Lithium Ion Thermal Runaway Conditions for Real-time Ultrasonic Enabled BMS Integration

The demand for energy storage is growing, and lithium-ion batteries are a promising technology to meet this need due to their high power/energy density, high round-trip efficiency, rapid response time, and portability. However, recent catastrophic events caused by thermal runaway have slowed their adoption, highlighting the need for an early warning system for battery failure. In this work, ultrasound is used to detect physical changes in 950 mAh batteries by identifying material property changes independent of voltage and current. [Ultrasound signal features (e.g., time of flight, maximum frequency component) were extracted as the batteries were cycled and subjected to both constant current and constant voltage overcharge and were used to develop two metrics identifying failure: a warning to detect the start of overcharge and an emergency stop (E-stop) to immediately take the battery out of service. The identification method involved locating magnitude differences of several ultrasound features compared to baseline operation considering different currents and temperatures, and the warning/stopping metrics were consistent across all experiments. For an average overcharge time of 140 minutes, the average warning was issued 124 minutes before the failure and the average E-stop was triggered 94 minutes before failure. As a test of using ultrasound for early warning detection, a battery was forced into overcharge and returned to normal cycling conditions based on the previously determined warning metric. Both the voltage profile and the ultrasound measurements returned to their baseline behavior, indicating that ultrasonic detection can not only identify battery failure before a catastrophic event, but can also provide early enough warnings such that overcharges can be detected and corrected quickly enough so the battery does not need to be decommissioned.]

25 ENERGY STORAGE↗
DOE OSTI2026

Measuring Flexibility through Reduction Potential

As electrical loads, electric vehicles (EVs) are distinguished in many cases by their substantial flexibility. Harnessing this flexibility, however, requires a precise characterization of its timing and degree. Current approaches to flexibility characterization tend to be limited either by the complexity of their representation or their inability to capture aggregate flexibility across multiple vehicles. In this paper, we present the reduction potential matrix, a novel metric capturing fleet-level EV load flexibility while being both straightforward to calculate and intuitive to interpret. We use the metric to quantify flexibility for two operationally distinct commercial groups: freight vehicles and transit buses. While we find that both vehicle groups have substantial flexibility, its timing, magnitude, and duration varies across the groups. We describe how this variability translates into variation in the each group's role as a grid resource, and conclude with a discussion of how system planners, fleet operators, and other stakeholders can use the novel metric to cultivate symbiosis between vehicles and the grid.

32 ENERGY CONSERVATION, CONSUMPTION, AND UTILIZATI↗
DOE OSTI2025

AI Model Benchmarking for Nonproliferation Applications: Steel Thread Benchmarking Task Force Technical Report (Rev. 2)

Steel Thread is a NA-22 venture that seeks to build trustworthy, reliable AI models that can be used in a wide variety of nonproliferation tasks. A key aspect of building these models is developing appropriate benchmarks and evaluation methods, which will enable the venture to identify and adapt models to provide the most value in the nonproliferation domain. Benchmarks must be relevant to key tasks in this domain, such as question answering, information retrieval, document summarization and classification, consensus analysis, and image and data analysis. This report 1) provides an overview of benchmark design, evaluation, and challenges; 2) reviews a variety of open benchmarks, with a focus on language models and tasks; and 3) identifies benchmarks that are most relevant to Steel Thread. This report is intended to serve as a basis for further efforts to classify and evaluate benchmarks and their correlation with success on nonproliferation-specific tasks. The Steel Thread venture has defined benchmarks to be a particular combination of a dataset (or datasets) and a metric (or metrics) conceptualized as representing one or more specific tasks or sets of abilities for a specific modality. It is adopted by a research community as a shared framework for comparing methods.1 It includes 1) Data: Labeled (a designated subset not used for training, which could be all the data), 2) Metric: A way to quantify performance, 3) Task/Ability: What the benchmark is testing, 4) Protocol: A structured and repeatable evaluation process, 5) Baseline/Reference Model: For comparison; could be statistical, rule-based, SME-derived, or another model, and 6) Maintenance Plan: to update with new information over time; important for long-term utility. For further clarity, the definition includes what a benchmark, in this context, is not. It is not a corpus of training data, specific to a model (it is intended to apply to a range of models), a universal evaluation of performance, a guarantee that the ‘top’ model on the leaderboard will be the best fit for every specific use case, an all-encompassing proof of a model’s universal quality, nor is it a one-size-fits-all measure of success. It does not cover every real-world constraint (like operational, ethical, or cost considerations), a systems integration test, or a unit test. This definition was inspired by and resulted from discussions within the Steel Thread Benchmarking Task Force. This group was formed to define what we would mean as a benchmark within Steel Thread but persisted as the need to develop a thorough understanding of the large and expanding existing benchmarking space. This technical report is a result of the group’s divide and conquer approach to exploring this space. The release of benchmarks might not be progressing as quickly as model development, but it is moving very fast, as many benchmarks quickly become saturated, when state-of-the-art models score so close to the benchmark’s ceiling that their results are virtually indistinguishable. At that point, the test no longer differentiates between new systems, so researchers usually stop reporting scores as the benchmark no longer informs about improvements from the next generation of models. In the OpenAI announcement of GPT-5, they reported results on six flagship public benchmarks (AIME 2025, SWE-bench Verified, Aider Polyglot, MMMU, HealthBench Hard, GPQA) but the full system-card covers roughly thirty-five separate evaluations, comprising hundreds of test task items in total. There have been some efforts to summarize benchmarks in specific fields, like for text-to-image generation, but these surveys have had a narrow methodology scope. Therefore, a comprehensive survey of all benchmarks or even all benchmarks that could be relevant to Steel Thread is outside of the scope of this report. We chose some specific benchmarks to investigate in detail.

97 MATHEMATICS AND COMPUTING↗