Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “needs assessment”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 325 records · Page 18

DER Cybersecurity Standards: Assessment and Gap Analysis

The purpose of this report is to share the comprehensive gap analysis of existing cybersecurity standards applicable to Distributed Energy Resources (DERs) within the electric power sector. This analysis aims to identify critical deficiencies in current standards, assess their alignment with industry needs, and provide actionable recommendations for enhancing cybersecurity measures. The scope encompasses various DER technologies, including solar, wind, energy storage, and hydrogen fuel cells, and emphasizes the significance of establishing robust cybersecurity frameworks and standards to safeguard these increasingly integrated systems. The report provides valuable insights for stakeholders in the DER ecosystem, including manufacturers, utilities, and regulators. It underscores the importance of continued development and refinement of cybersecurity standards to keep up with the technical advances in DERs and associated cybersecurity challenges. The analysis evaluated IEC, IEEE, ISA, ISO, and UL standards relevant to DER cybersecurity. Standards were assessed on their coverage of key requirements including data availability, integrity, confidentiality, access control, authentication, encryption, and system hardening. For each standard, the analysis assessed its alignment with current industry practices, regulatory compliance, effectiveness in addressing known risks, coverage of emerging risks, and how it promotes interoperability. The evaluation also considered potential integration challenges and barriers to adoption.

97 MATHEMATICS AND COMPUTING↗

Cybersecurity and Infrastructure Security Agency (CISA) Infrastructure Security Division (ISD) Assessment Prioritization Project

The Cybersecurity and Infrastructure Security Agency’s Infrastructure Security Division (CISA ISD) manages a diverse portfolio of assessments across the nation’s critical infrastructure sectors. These assessments—ranging in type, scope, and complexity—are essential for identifying vulnerabilities and strengthening national security. However, the wide variation in assessment offerings and the increasing demand for limited resources have highlighted the need for a transparent, structured approach to prioritizing assessment activities. To address this challenge, CISA ISD partnered with Lawrence Livermore National Laboratory (LLNL) to review current assessment methodologies, analyze existing prioritization practices, and develop a comprehensive, national-security-focused prioritization framework. This report summarizes the project’s approach, key findings, and actionable recommendations for enhancing ISD’s assessment program.

45 MILITARY TECHNOLOGY, WEAPONRY, AND NATIONAL DEF↗

Rapid Assessment of Sulfate Resistance in Mortar and Concrete

Extensive research has been conducted on the sulfate attack of concrete structures; however, the need to adopt the use of more sustainable materials is driving a need for a quicker test method to assess sulfate resistance. This work presents accelerated methods that can reduce the time required for assessing the sulfate resistance of mixtures by 70%. Class F fly ash has historically been used in concrete mixtures to improve sulfate resistance. However, environmental considerations and the evolving energy industry have decreased its availability, requiring the identification of economically viable and environmentally friendly alternatives to fly ash. Another challenge in addressing sulfate attack durability issues in concrete is that the standard sulfate attack test (ASTM C1012) is time-consuming and designed for only standard mortars (not concrete mixtures). To expedite the testing process, accelerated testing methods for both mortar and concrete mixtures were adopted from previous work to further the development of the accelerated tests and to assess the feasibility of testing the sulfate resistance of mortar and concrete mixtures rapidly. This study also established criteria for interpreting sulfate resistance for each of the test methods used in this work. A total of 14 mortar mixtures and four concrete mixtures using two types of Portland cement (Type I and Type I/II) and various supplementary cementitious materials (SCMs) were evaluated in this study. The accelerated testing methods significantly reduced the evaluation time from 12 months to 21 days for mortar mixtures and from 6 months to 56 days for concrete mixtures. The proposed interpretation method for mortar accelerated test results showed acceptable consistency with the ACI 318-19 interpretations for ASTM C1012 results. The interpretation methods proposed for the two concrete sulfate attack tests demonstrated excellent consistency with the ASTM C1012 results from mortar mixtures with the same cementitious materials combinations. Metakaolin was shown to improve sulfate resistance for both mortar and concrete mixtures, while silica fume and natural pozzolan had a limited impact. Using 15% metakaolin in mortar or concrete mixtures with Type I/II cement provided the best sulfate resistance.

Chemistry↗

Report on the AAPM grand challenge on deep generative modeling for learning medical image statistics

Abstract Background The findings of the 2023 AAPM Grand Challenge on Deep Generative Modeling for Learning Medical Image Statistics are reported in this Special Report. Purpose The goal of this challenge was to promote the development of deep generative models for medical imaging and to emphasize the need for their domain‐relevant assessments via the analysis of relevant image statistics. Methods As part of this Grand Challenge, a common training dataset and an evaluation procedure was developed for benchmarking deep generative models for medical image synthesis. To create the training dataset, an established 3D virtual breast phantom was adapted. The resulting dataset comprised about 108 000 images of size 512 512. For the evaluation of submissions to the Challenge, an ensemble of 10 000 DGM‐generated images from each submission was employed. The evaluation procedure consisted of two stages. In the first stage, a preliminary check for memorization and image quality (via the Fréchet Inception Distance [FID]) was performed. Submissions that passed the first stage were then evaluated for the reproducibility of image statistics corresponding to several feature families including texture, morphology, image moments, fractal statistics, and skeleton statistics. A summary measure in this feature space was employed to rank the submissions. Additional analyses of submissions was performed to assess DGM performance specific to individual feature families, the four classes in the training data, and also to identify various artifacts. Results Fifty‐eight submissions from 12 unique users were received for this Challenge. Out of these 12 submissions, 9 submissions passed the first stage of evaluation and were eligible for ranking. The top‐ranked submission employed a conditional latent diffusion model, whereas the joint runners‐up employed a generative adversarial network, followed by another network for image superresolution. In general, we observed that the overall ranking of the top 9 submissions according to our evaluation method (i) did not match the FID‐based ranking, and (ii) differed with respect to individual feature families. Another important finding from our additional analyses was that different DGMs demonstrated similar kinds of artifacts. Conclusions This Grand Challenge highlighted the need for domain‐specific evaluation to further DGM design as well as deployment. It also demonstrated that the specification of a DGM may differ depending on its intended use.

Radiology, Nuclear Medicine & Medical Imaging↗

Component level modeling of materials degradation for insights into operational flexibility of Existing Coal Power Plants

Increasingly, coal-fired power plants are required to balance power grids by compensating for the variable electricity supply from renewable energy sources. Fossil-fueled power plants, originally designed to be base loaded, will increasingly need to operate on a load following or cyclic basis. This demanding requirement for operational flexibility needs insights into accelerated material degradation arising due to the harsh operating conditions (e.g., fatigue, early oxide exfoliation due to stresses) along with current damage mechanisms (fireside corrosion, creep and erosion) observed in service. Our research objective is to develop component level modeling toolkit for materials-based degradation for two key mechanisms that can accelerate with cyclic operations. In more detail, this includes the fireside corrosion/steam oxidation/erosion/creep/fatigue of superheaters/reheaters and steam pipework and also the water droplet erosion/ fatigue of last stage steam turbine blades degradation mechanisms, that demand routine and sometimes unplanned maintenance and repair. The innovation is in developing a computational fluid dynamics/finite element (CFD/FE) modeling toolkit for the component level models of the boilers and low-pressure steam turbines in coal power plants that can tackle multidisciplinary failure mechanisms occurring concurrently for extreme environment materials. Lifetime assessment in such environments also needs to account for the unit-specific analyses, operational history and fuel feedstock; this can only be obtained by destructive analysis of components. This, in turn, enables validation of the model toolkits utilizing service feedback data, improving the probability of time/temperature dependent life prediction.

20 FOSSIL-FUELED POWER PLANTS↗

Challenges and Opportunities in Turning Large U.S. Archives of Analog Seismograms into a Modern Usable Resource

Abstract Quantitative seismology is based firmly on the analysis of actual ground motions, and the transition to digital recording in the 1980s enabled sophisticated new capabilities to extract useful results from waveforms. With some effort, these tools can also be applied to analog records. Focusing on assets available within U.S. institutions, we review the necessary steps and the challenges in enabling “data rescue”—that is, preserving the scientific information latent in large analog seismogram archives and making it usable. They include: determining what assets are available (the analog seismogram archives held by various institutions, with associated metadata on instrument responses, station locations, and timing information); developing a consensus on the top level of a triage process (which analog records most definitely should be rescued?); deciding the level of quality needed in copying original seismograms to media suitable for digitizing; assessing the relative merits of scanning and digitizing; and, the need for a community service in distributing scans and digital records, as they accumulate. The necessary level of effort can benefit from practical experience. For example, specific studies have used digitized versions of analog recordings to model earthquake sources and assess seismic hazard. Other studies have used them to gain experience with nuclear explosion signals recorded at regional distances, noting that regional signals enable explosions to be monitored down to levels much lower than those attainable teleseismically. The opportunities presented by large archives of analog seismograms include the insights they present to current and future seismologists studying earthquakes and explosions, into the practical areas of assessing seismic hazard, monitoring for test ban compliance down to low explosion yields—and prompt characterization of actual explosions should they occur, as well the traditional academic pursuit of a better understanding of earthquake physics.

Geochemistry & Geophysics↗

Industry Facing PV Degradation Prediction Tool and Database to Enable a 50-Year Life Module

The Photovoltaic (PV) industry constantly aims for lower costs, higher-efficiency cells, and improved module designs. These trends lead to using new materials, designs, and manufacturing processes, resulting in a continually changing technological landscape. These changes can potentially introduce new, unknown degradation mechanisms and failure modes that are difficult to diagnose, analyze, test, and model. This introduces uncertainty into the expected lifetime of PV modules of 25 years. Furthermore, research efforts aim for up to 50 years of service life while keeping performance degradation at a minimum for decades of outdoor weathering - putting additional pressure on improving the accuracy of long-term durability and reliability assessments. Here there is a need to organize the existing degradation data into an accessible format and to provide industry relevant tools for extrapolation from laboratory to field conditions. Because the core of this type of analysis involves calculations that are complicated but ubiquitous for many degradation processes, an enhanced predictive modeling framework will facilitate the analysis to help researchers keep up with the rapid pace of technological changes. In this work, we present an online tool that can be used to search for and analyze degradation information and extrapolate PV module performance and durability to field exposure. A graphical user interface will aid in the understanding of the results. The prediction tool will be built modular and published as open source, enabling users to expand on the existing framework. We use an integration pipeline approach that allows us to leverage weather data from the National Solar Radiation Database to perform geospatial degradation analysis in the US and worldwide. Our repository will contain various degradation models and material parameters suitable for the reliability and durability assessment of materials and components deployed outdoors. We hope to become a repository that can be used for weathering and degradation analysis for various applications beyond the PV industry.

degradation↗

A Methodology to Evaluate the Grid Reliability Impact of Oscillations Induced by Large Loads

The rapid growth of hyperscale AI data centers is bringing renewed attention to the reliability risk that sustained forced oscillations pose to bulk power systems, with cyclic computational workloads emerging as a new forcing source. Unlike the broadband, stochastic disturbances from traditional industrial loads such as arc furnaces, AI training and inference facilities can inject large active power swings concentrated at specific frequencies over extended durations - characteristics that existing grid planning practices do not account for. While the North American Electric Reliability Corporation (NERC) has recognized this gap and called for system-level studies of large load interconnections, no standardized methodology exists to screen, simulate, and quantify these risks at the planning stage. This report presents the Risk Assessment Tool for Large Load-induced Events (RATLLE), a Python-based, publicly available script suite developed at the Pacific Northwest National Laboratory to evaluate bulk power system reliability risks from data center-induced oscillations. RATLLE implements a three-module workflow: a screening module that identifies vulnerable interconnection locations and excitable system modes; a simulation module that models cyclic data center load behavior using a commercial positive sequence simulation platform; and an analysis module that computes risk metrics and generates interactive visualization dashboards. The risk metrics, formulated around simulation observables, map oscillation impacts to a three-stage severity scale spanning latent equipment fatigue through imminent cascading failure. The methodology is demonstrated on two Western Electricity Coordinating Council (WECC) system models: a publicly available 240-bus reduced representation and a detailed 2031 Heavy Winter planning case. Case studies illustrate that even modest 50 MW forced oscillations at resonant frequencies can produce wide-area power swings, N-1 security constraint violations, and cascading generator trips through protection actions - outcomes that would not occur under normal operating conditions without oscillations present. The results underscore the need for standardized oscillation impact assessment in large load interconnection studies and provide a reproducible, extensible framework for utilities to adopt or customize within their existing planning workflows.

Biswas, Shuchismita↗

Important Human Actions for Advanced Reactors: Implications for Human Factors

As advanced reactor platforms continue to develop and gain traction in the energy sector there is a need for risk-informed, scalable regulations that match that progress. This is a core component of the U.S. Nuclear Regulatory Commission’s proposed Part 53 Rule Making; the Accelerating Deployment of Versatile, Advanced Nuclear for Clean Energy (ADVANCE) Act; and other efforts that seek to update nuclear power regulations. This paper covers one key aspect of that regulatory evolution: Important Human Actions (IHA). In this paper, we discuss how the understanding and definitions of IHAs have changed and what that means for human factors engagement through the process of developing these technologies. Instead of a narrow focus on control actions that led to an increase in core damage risk, the new focus is on IHAs is “wherever they occur.” What this means is that having a highly automated or passive safety system does not eliminate IHAs. Rather, it shifts the focus point to all the actions that enable these systems. Everything from maintenance, to design, to training can be considered an IHA and that dramatically shifts the efforts and level of engagement necessary for human factors to enable these technologies. We discuss the notions of risk-informed human factors that underpin these efforts, give several examples, and briefly describe the risk assessment methodologies that will be needed. In the past, IHAs were identified and then became a focus point of human factors engineering (HFE) activities to ensure a robust evaluation of the task was completed. The future is less clear. HFE for nuclear energy will need to evolve and become more integrated in technology development than ever before.

22 - GENERAL STUDIES OF NUCLEAR REACTORS↗

Biomass increases attributed to both faster tree growth and altered allometric relationships under long‐term carbon dioxide enrichment at a temperate forest

Abstract Increases in atmospheric carbon dioxide (CO 2 ) concentrations are expected to lead to increases in the rate of tree biomass accumulation, at least temporarily. On the one hand, trees may simply grow faster under higher CO 2 concentrations, preserving the allometric relations that prevailed under lower CO 2 concentrations. Alternatively, the allometric relations themselves may change. In this study, the effects of elevated CO 2 (eCO 2 ) on tree biomass and allometric relations were jointly assessed. Over 100 trees, grown at Duke Forest, NC, USA, were harvested from eight plots. Half of the plots had been subjected to CO 2 enrichment from 1996 to 2010. Several subplots had also been subjected to nitrogen fertilization from 2005 to 2010. Allometric equations were developed to predict tree height, stem volume, and aboveground biomass components for loblolly pine ( Pinus taeda L.), the dominant tree species, and broad‐leaved species. Using the same diameter‐based allometric equations for biomass, it was estimated that plots with eCO 2 contained 21% more aboveground biomass, consistent with previous studies. However, eCO 2 significantly affected allometry, and these changes had an additional effect on biomass. In particular, P. taeda trees at a given diameter were observed to be taller under eCO 2 than under ambient CO 2 due to changes in both the allometric scaling exponent and intercept. Accounting for allometric change increased the treatment effect of eCO 2 on aboveground biomass from a 21% to a 27% increase. No allometric changes for the nondominant broad‐leaved species were identified, nor were allometric changes associated with nitrogen fertilization. For P. taeda , it is concluded that eCO 2 affects allometries, and that knowledge of allometry changes is necessary to accurately compute biomass under eCO 2 . Further observations are needed to determine whether this assessment holds for other taxa.

Kim, Dohyoung↗

Making a Water Data System Responsive to Information Needs of Decision Makers

Evidence-based environmental management requires data that are sufficient, accessible, useful and used. A mismatch between data, data systems, and data needs for decision making can result in inefficient and inequitable capital investments, resource allocations, environmental protection, hazard mitigation, and quality of life. In this paper, we examine the relationship between data and decision making in environmental management, with a focus on water management. We focus on the concept of decision-driven data systems —data systems that incorporate an assessment of decision-makers' data needs into their design. The aim of the research was to examine the process of translating data into effective decision making by engaging stakeholders in the development of a water data system. Using California's legislative mandate for state agencies to integrate existing water and other environmental data as a case study, we developed and applied a participatory approach to inform data-system design and identify unmet data needs. Using workshops and focused stakeholder meetings, we developed 20 diverse use cases to assess data sources, availability, characteristics, gaps, and other attributes of data used for representative decisions. Federal and state agencies made up about 90% of the data sources, and could readily adapt to a federated data system, our recommended model for the state. The remaining 10% of more-specialized data, central to important decisions across multiple use cases, would require additional investment or incentives to achieve data consistency, interoperability, and compatibility with a federated system. Based on this assessment, we propose a typology of different types of data limitations and gaps described by stakeholders. We also propose technical, governance, and stakeholder engagement evaluation criteria to guide planning and building environmental data systems. Data-system governance involving both producers and users of data was seen as essential to achieving workable standards, stable funding, convenient data availability, resilience to institutional change, and long-term buy-in by stakeholders. Our work provides a replicable lesson for using decision-maker and stakeholder engagement to shape the design of an environmental data system, and inform a technical design that addresses both user and producer needs.

Cantor, Alida↗

Short-Run Marginal Emission Factors Neglect Impactful Phenomena and are Unsuitable for Assessing the Power Sector Emissions Impacts of Hydrogen Electrolysis

This comment reacts to Ruhnau and Schiele's (2023) assessment of the cost and emissions impacts of electrolytic hydrogen production operating under different green hydrogen certification requirements in the EU. We critique the paper's use of short-run marginal emissions rates to estimate emissions impacts, a methodology which the literature has shown to be inadequate for assessing the full lifecycle emissions impacts of electricity sector interventions. We hope that our response clarifies the need to consider induced structural change when assessing the greenhouse gas emissions impacts of electricity sector decisions at all scales.

electricity↗

Environmental DNA as a tool for hydropower impact assessments: current status, special considerations, and future integration

Globally there is an urgent need to find sustainable solutions to balance energy production with the protection of vulnerable species and conservation of biodiversity. This is particularly critical for freshwater ecosystems, habitats, and species that may be impacted by hydropower development and operations needed to meet energy grid demands. Reliable and accurate environmental impact assessments (EIAs) that identify the biological, physical, or social impacts of hydropower are key to ensure biodiversity, ecosystem, and societal sustainability. The analysis of environmental DNA (eDNA) has the potential to transform hydropower EIAs, management and mitigation planning, and decision-making procedures. Further, the incorporation of eDNA surveys into EIAs during both hydropower planning and continued operations may streamline regulatory processes by improving our understanding of potentially impacted biota and habitats and evaluating environmental impacts mitigation. Here, we: (i) highlight current understanding and use of eDNA in freshwater environments; (ii) examine critical considerations for eDNA integration into hydropower EIAs and biological monitoring; (iii) identify knowledge gaps in eDNA analysis and applications unique to hydropower-regulated systems; and (iv) discuss future opportunities to bolster the incorporation of eDNA into hydropower research including regulatory acceptance and public engagement. While we acknowledge that there are several factors that may complicate the broad adoption of eDNA as a tool for assessing the impacts of hydropower, we anticipate that growing confidence in eDNA through hydropower-specific protocols, calibrations, and validations will overcome these inherent uncertainties.

aquatic biodiversity↗

Interactions between U.S. biofuels policy and the Endangered Species Act

The Renewable Fuel Standard (RFS) is the primary policy governing the use of biofuels in the United States. Following the 2007 enactment of the RFS, there has been substantial expansion in the use of cropland to produce biofuel feedstocks, including corn and soybeans. Both the conversion of land and the ongoing cultivation of these crops can, depending on management practices, lead to negative environmental outcomes including degraded water quality, increased water use, increased greenhouse gas emissions, and loss or degradation of wildlife habitat. These outcomes have the potential to affect threatened and endangered species protected under the U.S. Endangered Species Act (ESA). The ESA requires federal agencies to consult with the U.S. Fish and Wildlife Service or the National Marine Fisheries Service to ensure that an agency's actions are not likely to jeopardize the existence of imperiled species, nor result in the destruction or adverse modification of their habitat. However, formal consultation has not yet been completed between the U.S. Environmental Protection Agency—the agency charged with implementing the RFS—and the wildlife services, thereby hindering mitigation or remediation of potential impacts. This analysis explores the likely effects of the RFS on land use changes and critical habitat, illustrates example pathways by which threatened and endangered species may be affected, and proposes solutions to help redress this harm. The findings suggest a need for further consultation and assessment of the interactions between the RFS and ESA as well as additional research to support the species-conservation and policymaking processes.

09 BIOMASS FUELS↗

High-performance windows improve thermal survivability of occupants during cold snaps

Exposure to low indoor air temperature is a major contributor to temperature-related mortality during extreme cold events, especially when power outages disrupt operation of space heating systems. This study explores the impact of high-performance windows on the thermal resilience of residential buildings during extreme cold weather and grid power outages, as well as their long-term benefits through energy efficiency and reduced risk of property damage. Building performance simulations were conducted for reference residential buildings in three construction vintages and two major U.S. cities located in cold climate zones, considering two types of extreme cold events: short and severe, and long and milder. Our research found that even houses compliant with current energy codes struggle to maintain safe indoor temperatures for more than a few hours during power outages, necessitating rapid evacuations. High-performance windows can extend the thermal survivability time by up to 3.8 days within a 7-day cold snap and significantly reduce risk of bursting frozen water pipes, depending on the building’s insulation and infiltration level, cold event severity, and occupant vulnerability. This extended thermal safety time is crucial in scenarios where reduced mobility complicates emergency responses in senior housing. In addition to boosting thermal resilience, upgrading older homes with high-performance windows can reduce heating energy consumption by over 18% and cooling energy by 15%. Our findings highlight the need to incorporate thermal resilience assessments into new designs or major retrofits, including the use of typical and extreme weather scenarios and advanced technologies like high-performance windows.

Krelling, Amanda F↗

Evaluating direct vessel injection accident-event progression of AP1000 and key figures of merit to support the design and development of water-cooled small modular reactors

The passive safety systems (PSSs) within water-cooled reactors are meticulously engineered to function autonomously, requiring no external power source or manual intervention. They depend exclusively on inherent natural forces and the fundamental principles of reactor physics, such as gravity, natural convection, and phase changes, to manage, alleviate, and avert the release of radioactive materials into the environment during accident scenarios like a loss-of-coolant accident (LOCA). PSSs are already integrated into such operating commercial reactors as the Advanced Pressurized Reactor-1000 MWe (AP1000) and the Water-Water Energetic Reactor-1200 MWe (WWER-1200) are adopted in most of the upcoming small modular reactor (SMR) designs. Examples of water-cooled SMR PSSs are the passive emergency core-cooling system (ECCS), passive containment cooling system (PCCS), and passive decay-heat removal system, the designs of which vary based on reactor system-design requirements. However, understanding the accident-event progression and phases of a LOCA is pivotal for adopting a specific PSS for a new SMR design. This study covers the accident-event progression for direct vessel injection (DVI) small-break loss-of-coolant accident (SB-LOCA), associated physics phenomena, knowledge gaps, and important figures of merit (FOMs) that may need to be evaluated and assessed to validate thermal-hydraulics models with an available experimental dataset to support new SMR design and development.

21 SPECIFIC NUCLEAR REACTORS AND ASSOCIATED PLANTS↗

Consequence analyses of sabotage-induced radiological releases in sodium-cooled fast microreactors

Analysis of three sodium-cooled fast microreactors (SFMs) with thermal powers of 10, 30, and 50 MWt showed that smaller reactors result in lower radiological consequences during a postulated sabotage-induced event because of their reduced core inventory. All SFMs used U-10Zr metal fuel enriched to 15 wt% high-assay low-enriched uranium and operated until their respective effective multiplication factor (k eff ) reduced to less than 1 or until the end of their operational lifespan. Sabotage scenarios were simulated at this point, when the fuel inventory within the core contains the highest-level of radioactivity. Radionuclide core inventories were calculated using the SCALE code at shutdown and 3 days post-shutdown. Dose consequence analyses were performed for three sabotage scenarios using the RASCAL tool. As microreactor developers plan for minimal on-site or complete off-site emergency response, it remains essential to evaluate their physical protection needs and potential hazards, including assessing postulated sabotage-induced events that could become more relevant. SFM licensees should identify a credible worst-case, major accident, estimate release source terms, and perform dose consequence analyses to evaluate site-specific physical protection measures. In conclusion, this recommendation supports a risk-informed, performance-based approach, aligning with applicable regulatory requirements, i.e., 10 CFR Parts 100 and 53 rulemaking in the United States.

98 NUCLEAR DISARMAMENT, SAFEGUARDS, AND PHYSICAL P↗

Systematic Benchmarking of Climate Models: Methodologies, Applications, and New Directions

As climate models become increasingly complex, there is a growing need to comprehensively and systematically assess model performance with respect to observations. Given the increasing number and diversity of climate model simulations in use, the community has moved beyond simple model intercomparison and toward developing methods capable of benchmarking a large number of simulations against a suite of climate metrics. Here, we present a detailed review of evaluation and benchmarking methods and approaches developed in the last decade, focusing primarily on scientific implications for Coupled Model Intercomparison Project (CMIP) simulations and CMIP6 results that contributed to the Intergovernmental Panel on Climate Change (IPCC) Sixth Assessment Report (AR6). Based on this review, we explain the resulting contemporary philosophy of model benchmarking, and provide clear distinctions and definitions of the terms model verification, process validation, evaluation, and benchmarking. While significant progress has been made in model development based on systematic evaluation and benchmarking efforts, some climate system biases still remain. The development of open‐source community software packages has played a fundamental role in identifying areas of significant model improvement and bias reduction. We review the key features of several software packages that have been commonly used over the past decade to evaluate and benchmark global and regional climate models. Additionally, we discuss best practices for the selection of evaluation and benchmarking metrics and for interpreting the obtained results, the importance of selecting suitable sources of reference data and accurate uncertainty quantification.

Environmental sciences↗