SEARCH · Engineering Papers

Results for “Judgment”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

SourceResearch connectionFrom yearThrough year

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 163 records · Page 9

All sources NASA NTRS DOE OSTI

DOE OSTI2018

NSRD-15:Computational Capability to Substantiate DOE-HDBK-3010 Data

97 MATHEMATICS AND COMPUTING↗

DOE OSTI2016

Deep Borehole Disposal Concept: Development of Universal Canister Concept of Operations

This report documents key elements of the conceptual design for deep borehole disposal of radioactive waste to support the development of a universal canister concept of operations. A universal canister is a canister that is designed to be able to store, transport, and dispose of radioactive waste without the canister having to be reopened to treat or repackage the waste. This report focuses on the conceptual design for disposal of radioactive waste contained in a universal canister in a deep borehole. The general deep borehole disposal concept consists of drilling a borehole into crystalline basement rock to a depth of about 5 km, emplacing WPs in the lower 2 km of the borehole, and sealing and plugging the upper 3 km. Research and development programs for deep borehole disposal have been ongoing for several years in the United States and the United Kingdom; these studies have shown that deep borehole disposal of radioactive waste could be safe, cost effective, and technically feasible. The design concepts described in this report are workable solutions based on expert judgment, and are intended to guide follow-on design activities. Both preclosure and postclosure safety were considered in the development of the reference design concept. The requirements and assumptions that form the basis for the deep borehole disposal concept include WP performance requirements, radiological protection requirements, surface handling and transport requirements, and emplacement requirements. The key features of the reference disposal concept include borehole drilling and construction concepts, WP designs, and waste handling and emplacement concepts. These features are supported by engineering analyses.

12 MANAGEMENT OF RADIOACTIVE AND NON-RADIOACTIVE W↗

DOE OSTI2017

Risk-Informed Safety Margins Characterization (RISMC) Pathway Technical Program Plan

Safety is central to the design, licensing, operation, and economics of nuclear power plants (NPPs). As the current Light Water Reactor (LWR) NPPs age beyond 60 years, there are possibilities for increased frequency of systems, structures, and components (SSCs) degradations or failures that initiate safety-significant events, reduce existing accident mitigation capabilities, or create new failure modes. Plant designers commonly “over-design” portions of NPPs and provide robustness in the form of redundant and diverse engineered safety features to ensure that, even in the case of well-beyond design basis scenarios, public health and safety will be protected with a very high degree of assurance. This form of defense-in-depth is a reasoned response to uncertainties and is often referred to generically as “safety margin.” Historically, specific safety margin provisions have been formulated, primarily based on “engineering judgment.” The ability to better characterize (i.e., describe and quantify) safety margin is important to improved decision making about LWR design and long-term operation. A systematic approach to characterizing safety margins and the subsequent risk informed margins management options represents a vital input to the licensee and regulatory analysis and decision making that will be involved. In addition, as research and development (R&D) in the LWRS Program and other collaborative efforts yield new data and improved scientific understanding of physical processes that govern the aging and degradation of plant SSCs (and concurrently support technological advances in nuclear reactor fuels and plant instrumentation and control systems), needs and opportunities to better optimize plant safety, economics, and performance will become known. The purpose of the Risk-Informed Safety Margins Characterization (RISMC) Pathway R&D is to support plant decisions for risk-informed margins management with the aim to improve economics, reliability, and sustain safety of current NPPs over periods of extended plant operations. The goals of the RISMC Pathway are twofold: (1) develop and demonstrate a risk-assessment method that is coupled to safety margin quantification that can be used by NPP decision makers as part of risk-informed margin management strategies; (2) create an advanced RISMC Toolkit that enables more accurate representation of NPP safety margins. The methods and tools provided by RISMC are essential to a comprehensive and integrated risk-informed margin management approach that supports effective preservation of margin for both active and passive SSCs.

97 MATHEMATICS AND COMPUTING↗

DOE OSTI2018

Failure Rate Estimates for Passive Mechanical Components

Fusion design projects that reach the stage of design reviews typically require reliability, availability, maintainability and inspectability (RAMI) assessments. While many kinds of reliability and maintainability data are needed for a wide variety of fusion components to quantify a RAMI assessment, one of the types of components requiring data is the passive mechanical component. Passive mechanical components can be items such as metal plates, rods, shafts, brackets, bolts, and other items. Generally these passive components tend to have low failure rates since they are simple designs, have no or few moving parts, and do not need control signals to perform their design function. However, there are no widely recognized datasets for such components, so analysts must use multiple datasets and judgment for failure rates. This report gives recommended failure rates for various failure modes of these components, and gives suggested failure rate multipliers to apply to these failure rates if they reside in harsh environments. If the component resides far from the tokamak then typical industrial failure rates would suffice. Components near the tokamak find conditions can be demanding; similar to those in a fission reactor containment building. Some component failure rates can be modified if the components reside inside the tokamak.

70 PLASMA PHYSICS AND FUSION TECHNOLOGY↗

DOE OSTI2014

Expanded Materials Degradation Analysis (EMDA) Volume 3: Aging of Reactor Pressure Vessels

In NUREG/CR-6923 ORNL/TM-2013/532, "Expert Panel Report on Proactive Materials Degradation Assessment," referred to as the PMDA report, NRC conducted a comprehensive evaluation of potential aging-related degradation modes for core internal components, as well as primary, secondary, and some tertiary piping systems, considering operation up to 40 years. This document has been a very valuable resource, supporting NRC staff evaluations of licensees' aging management programs and allowing for prioritization of research needs.This report describes an expanded materials degradation assessment (EMDA), which significantly broadens the scope of the PMDA report. The analytical timeframe is expanded to 80 years to encompass a potential second 20-year license-renewal operating-period, beyond the initial 40-year licensing term and a first 20-year license renewal. Further, a broader range of structures, systems, and components (SSCs) was evaluated, including core internals, piping systems, the reactor pressure vessel (RPV), electrical cables, and concrete and civil structures. The EMDA uses the approach of the phenomena identification and ranking table (PIRT), wherein an expert panel is convened to rank potential degradation scenarios according to their judgment of susceptibility and current state of knowledge.

22 GENERAL STUDIES OF NUCLEAR REACTORS↗

DOE OSTI2020

An Overview of the State-of-the-Art Reactor Consequence Uncertainty Assessment Accident Progression Insights

The U.S. Nuclear Regulatory Commission (NRC) with Sandia National Laboratories (Sandia) have completed three uncertainty analyses (UAs) as part of the State-of-the-Art Reactor Consequence Analyses (SOARCA) program. The SOARCA UAs included an integrated evaluation of uncertainty in accident progression, radiological release, and offsite health consequence projections. The UA for Peach Bottom, a boiling-water reactor (BWR) with a Mark I containment located in the State of Pennsylvania, analyzed the unmitigated long-term station blackout SOARCA scenario. The UA for Sequoyah, a 4-loop Westinghouse pressurized-water reactor (PWR) located in the State of Tennessee, analyzed the unmitigated short-term station blackout SOARCA scenario, with a focus on issues unique to the ice condenser containment and the potential for early containment failure due to hydrogen deflagration. The UA for Surry, a 3-loop Westinghouse PWR with a sub-atmospheric large dry containment located in the State of Virginia, analyzed the unmitigated short-term station blackout SOARCA scenario including the potential for thermally-induced steam-generator tube rupture. These three UAs are currently documented in three NUREG/CR reports. This report provides input to planned NRC documentation on the insights and findings from the SOARCA UA program. The purpose of the summary report is to provide a useful reference for regulatory applications that require the evaluation of offsite consequence risk from beyond design basis event severe accidents. This report focuses on the accident progression and source term insights developed from the MELCOR analyses. MELCOR is the NRC's best-estimate, severe accident computer code used in the SOARCA UAs. In anticipation of the SOARCA UA insights work, NRC and Sandia benchmarked the response of the Peach Bottom model to selected reference calculations from the Peach Bottom SOARCA UA. Peach Bottom was the first SOARCA UA performed and was completed in 2015 using the MELCOR 1.8.6 code. The PWR SOARCA UAs evolved the original methodology and utilized the updated MELCOR 2.2 computer code. The Peach Bottom model has been systematically updated for other NRC research efforts and has been updated to MELCOR 2.2. computer code. The findings from the new reference calculations using the updated model with the MELCOR 2.2 code are also integrated into the report. A second objective is an assessment of the applicability of the results to the other nuclear reactors in the U.S. As the key findings are reviewed, judgments are presented on the applicability of the results to other U.S. nuclear power plants. An important objective of the SOARCA program relied on high- fidelity plant-specific modeling. However, the nature of the insights and conclusions allowed judgements to be made on the applicability of the various insights to the same general classification of plant (i.e., BWR or PWR) or the entire fleet of plants. Finally, the results from the SOARCA UA accident progression calculations contain a wealth of information not previously documented in the NUREG/CRs. This report includes new but related information that can be used to benchmark past or support future regulatory decisions related to severe accidents. The new work includes a benchmark of the NUREG-1465 licensing source term definitions, the variability of key accident progression events and timing to radionuclide release, and an improved understanding of the timing and source terms from consequential steam generator tube ruptures. iii ACKNOWLEDGEMENTS The Sandia authors gratefully acknowledge the significant technical and programmatic contributions from the NRC SOARCA team which are reflected throughout the report. Dr. Tina Ghosh has been involved throughout the SOARCA UAs, providing the primary managerial and technical oversight. The long lists of NRC and Sandia contributors from the SOARCA UAs are cited in the three NUREG/CRs and are also gratefully acknowledged by the small team of authors compiling the results of their efforts. Significant technical contributions, advice, and reviews were provided by Dr. Hossein Esmaili, Dr. Alfred Hathaway, and Dr. Edward Fuller (retired) of the NRC. Dr. Randal Gauntt (retired), Mr. Patrick Mattie, Mr. Joseph Jones (retired), and Dr. Doug Osborn from Sandia are recognized as the SOARCA UA managers guiding the past efforts. There is a comparable list of project managers at the NRC including Ms. Patricia Santiago, Dr. Salman Haq, and Mr. Jon Barr. Sadly, we have lost Mr. Charlie Tinkler and Mr. Robert Prato, who were important contributors to the original SOARCA project. Finally, Mr. Kyle Ross and Mr. Mark Leonard have also retired but were significant technical contributors. Mr. Kyle Ross was the technical lead on all three SOARCA UAs and the original pressurized water reactor SOARCA study. Mr. Leonard was the technical lead on the original boiling water reactor SOARCA study and a key contributor to the first Peach Bottom SOARCA UA. iv

21 SPECIFIC NUCLEAR REACTORS AND ASSOCIATED PLANTS↗

DOE OSTI2020

Group Structure Machine Learning Proposal

Nuclear data is the linchpin underwriting several fundamental capabilities and mission needs at LANL. New techniques such as machine learning can be brought to bear to solve old problems such as multigroup cross-section accuracy. In neutron transport, generating multigroup cross sections is a complex and arcane task, but a crucial one, as accurate solutions require appropriate cross sections. There are two key challenges when generating multigroup cross sections: (1) choosing an accurate weight function, and (2) choosing appropriate energy boundaries. Often, energy boundaries are chosen using “expert judgment” that is not documented and is difficult to replicate. The long-standing Los Alamos 30-group structure has been in use since at least 1969 and is still in use today. Simplistic attempts over the years since to improve on the 30-group structure have been met with limited success. Machine learning algorithms would enable the selection of appropriate, problem-dependent group boundaries without an inordinate investment of scientist time. We will develop workflows and tools to enable these improved group boundary choices, which will reduce uncertainty and increase predictive capability of neutron-transport applications at LANL.

73 NUCLEAR PHYSICS AND RADIATION PHYSICS↗

DOE OSTI2020

Applying Compression-Based Metrics to Seismic Data in Support of Global Nuclear Explosion Monitoring

The analysis of seismic data for evidence of possible nuclear explosion testing is a critical global security mission that relies heavily on human expertise to identify and mark seismic signals embedded in background noise. To assist analysts in making these determinations, we adapted two compression distance metrics for use with seismic data. First, we demonstrated that the Normalized Compression Distance (NCD) metric can be adapted for use with waveform data and can identify the arrival times of seismic signals. Then we tested an approximation for the NCD called Sliding Information Distance (SLID), which can be computed much faster than NCD. We assessed the accuracy of the SLID output by comparing it to both the Akaike Information Criterion (AIC) and the judgments of expert seismic analysts. Our results indicate that SLID effectively identifies arrival times and provides analysts with useful information that can aid their analysis process.

45 MILITARY TECHNOLOGY, WEAPONRY, AND NATIONAL DEF↗

DOE OSTI2020

Comparison Study of Upper Subcritical Limits Derived Using Sensitivity/Uncertainty Tools Case Studies of Benchmarks and Applications

Neutron transport methods used to establish subcriticality require validation by comparison to critical experiments considered to be benchmarks. Whisper is a sensitivity/uncertainty analysis tool developed to assist with the task of validation in nuclear criticality safety. Details on the Whisper methodology can be found in References 1-3 on the MCNP ® reference collection website at https://mcnp.lanl.gov. Whisper-1.0 was originally developed in 2014 and used to assist with nuclear criticality safety validation at Los Alamos National Laboratory. Whisper was upgraded in 2016 to Whisper-1.1 and prepared for release with MCNP6.2. Whisper contains a library of over 1100 critical experiment benchmarks and quantifies neutronic similarity of an application to benchmarks in the library. Using highest similarity benchmarks, Whisper computes a calculational margin (CM) encompassing of the worst-case bias and bias uncertainty at a 99% confidence level for each application. In addition, portions of the margin of subcriticality (MOS) for nuclear data uncertainty and potential code errors are computed. The baseline upper subcritical limit (USL) computed by Whisper is comprised of the CM, MOS nuclear data , and MOS code errors . The Whisper baseline USL is absent a portion of the MOS due to the area of application, which is applied based upon judgment by the criticality safety analyst. An objective of this paper is to present the baseline USL, CM and portions of the MOS as computed by Whisper for comparison with similar sensitivity/uncertainty tools, such as those used by IRSN and ORNL. The initial comparison involves four critical experiment benchmarks: HEU-MET-FAST-013-001, HEU-SOLTHERM-001-008, PU-MET-FAST-022-001, AND PU-SOL-THERM-001-001, which have been: 1. modeled independently by LANL, IRSN, and ORNL based upon information provided in the ICSBEP Handbook, 2. are common in S/U libraries for LANL, IRSN, and ORNL, 3. span a range of energy spectrum and fissile material, and 4. taken as applications for the purposes of this study and therefore excluded from use as a benchmark for calculating the upper subcritical limit. Results presented in this paper have been computed using covariance data for all isotopes in ENDF/BVII.0 using a 44-group energy structure. Benchmarks in the Whisper library were run in MCNP6.2 using 100,000 neutrons per cycle, skipping 100 cycles for 500 active cycles. Reference 10 also compares the results for baseline USL with an order of magnitude greater neutrons, using the same total number of cycles with 1,000,000 neutrons per cycle. Subsequent to the results presented in Reference 10 changes were made to the benchmark library, as discussed in Reference 11. Newer results using the revised benchmark library are presented herein.

73 NUCLEAR PHYSICS AND RADIATION PHYSICS↗

DOE OSTI2019

NSRD-16: Computational Capability to Substantiate DOE-HDBk-3010 Data

Safety basis analysts throughout the U.S. Department of Energy (DOE) complex rely heavily on the information provided in the DOE Handbook, DOE-HDBK-3010, Airborne Release Fractions/Rates and Respirable Fractions for Nonreactor Nuclear Facilities, to determine radionuclide source terms from postulated accident scenarios. In calculating source terms, analysts tend to use the DOE Handbook's bounding values on airborne release fractions (ARFs) and respirable fractions (RFs) for various categories of insults (representing potential accident release categories). This is typically due to both time constraints and the avoidance of regulatory critique. Unfortunately, these bounding ARFs/RFs represent extremely conservative values. Moreover, they were derived from very limited small-scale bench/laboratory experiments and/or from engineered judgment. Thus, the basis for the data may not be representative of the actual unique accident conditions and configurations being evaluated. The goal of this research is to develop a more accurate and defensible method to determine bounding values for the DOE Handbook using state-of-art multi-physics-based computer codes. This enables us to better understand the fundamental physics and phenomena associated with the types of accidents in the handbook. In this fourth year, we improved existing computational capabilities to better model fragmentation situations to capture small fragments during an impact accident. In addition, we have provided additional new information for various sections of Chapters 4 and 5 of the Handbook on free fall powders and impacts of solids, and have provided the damage ratio simulations for containers (7A drum and standard waste box) for various drops and impact scenarios. Thus, this work provides a low-cost method to establish physics-justified safety bounds by considering specific geometries and conditions that may not have been previously measured and/or are too costly to perform during an experiment.

61 RADIATION PROTECTION AND DOSIMETRY↗

DOE OSTI2021

Miscalculation, Misperception and Risk Reduction

In the current security environment, the most likely wars for which the US and its allies must be prepared are wars against nuclear-armed states. Such wars bring unique risks of both vertical and horizontal escalation. Some are “accepted risks;” for example, the US may choose to escalate a conflict to clarify its commitment to an issue and defend its interest. Other are “unacceptable risks;” especially, the risk of miscalculation by the U.S. or an adversary. The focus here is on this latter category. Miscalculation is an error of judgment—the failure to accurately assess a situation. Miscalculation can result from hidden, incomplete, or ambiguous information, or from inaccurate processing of available information because of bias, overconfidence, denial, delusion, etc. Miscalculation is related to misperception. For some, they are synonymous. This paper characterizes the risks of miscalculation and introduces a discussion of a risk mitigation strategy.

45 MILITARY TECHNOLOGY, WEAPONRY, AND NATIONAL DEF↗

DOE OSTI2021

Selection of 3013 Containers for Field Surveillance (FY 2021 Update)

This Update is the ninth in a series of reports that document the binning and sample selection of 3013 containers for the Field Surveillance program as part of the Integrated Surveillance Program. The last Update was in 2016. This Update documents changes to the binning and changes to the random and engineering judgment samples since 2016. This Update also documents field surveillance activities since 2016 and describes plans to complete random sampling in 2025. In addition, this Update describes a new effort to collect surveillance data as part of down blending operations and updates assumptions about Advanced Recovery and Integrated Extraction System (ARIES) 3013 containers.

11 NUCLEAR FUEL CYCLE AND FUEL MATERIALS↗

DOE OSTI2021

Updated Oregon Floating Offshore Wind Cost Modeling [Slides]

The study provides heat maps showing updated estimates of the levelized cost of energy (LCOE) for floating offshore wind energy off the coast of Oregon. This project builds on a 2019 National Renewable Energy Laboratory (NREL) floating offshore wind power cost study in Oregon (Musial et al. 2019) and a recent NREL California cost analysis (Beiter et al. 2020). Floating wind power cost data, modeling methodology, and resource data are updated and LCOE is estimated through 2032 using 2019 as a reference year. Comparisons are made to the previous offshore wind energy cost studies (Musial et al. 2019; Beiter et al. 2020). The study does not prioritize specific sites or make judgments about marine spatial planning viability.

17 WIND ENERGY↗

DOE OSTI2022

Titanium Material Comparison for Second Target Station Moderators

Two titanium alloys, Ti 15-3-3-3 and Grade 5 Ti, have been analyzed for their neutronic differences after being exposed to neutron fluxes from a 5-year operational period in the Second Target Station (STS)moderators. There are several differences in the original composition of the two alloys where Ti 15-3-3-3contains Cr and Sn and Grade 5 Ti does not. The differences in original composition propagate through the analysis to yield discrepancies in decay power density and decay gamma intensity density of more than a factor of 3.5 at some times during operation. Immediately after the 5-year operational period, Ti 15-3-3-3has a factor of 3.59 higher intensity of high-energy decay gammas than Grade 5 Ti, and Ti 15-3-3-3 also has a factor of 3.53 higher decay power than Grade 5 Ti. After 1 year of shutdown, Ti 15-3-3-3 has a 48% higher intensity of high-energy decay gammas and a 86% greater decay power density when compared with Grade 5 Ti. The quantities-of-interest in this report are per unit volume and the overall volume of titanium should be considered when making engineering judgment on final titanium alloy selection.

36 MATERIALS SCIENCE↗

DOE OSTI2021

Comparison Study of Upper Subcritical Limits Derived Using Sensitivity/Uncertainty Tools: Case Studies of U233-SOL-THERM-001-001, MIX-COMP-THERM-001-001, IEU-MET-FAST-002-001, LEU-COMP-THERM-001-001, LEU-SOL-THERM-004-001

Neutron transport methods used to establish subcriticality require validation by comparison to critical experiments considered to be benchmarks. Whisper is a sensitivity/uncertainty analysis tool developed to assist with the task of validation in nuclear criticality safety. Details on the Whisper methodology can be found in References 1-3 on the MCNP® reference collection website at https://mcnp.lanl.gov. Whisper-1.0 was originally developed in 2014 and used to assist with nuclear criticality safety validation at Los Alamos National Laboratory. Whisper was upgraded in 2016 to Whisper-1.1 and prepared for release with MCNP6.2 [References 3-5]. Whisper contains a library of over 1100 critical experiment benchmarks and quantifies neutronic similarity of an application to benchmarks in the library. Using highest similarity benchmarks, Whisper computes a calculational margin (CM) encompassing of the worst-case bias and bias uncertainty at a 99% confidence level for each application. In addition, portions of the margin of subcriticality (MOS) for nuclear data uncertainty and potential code errors are computed. The baseline upper subcritical limit (USL) computed by Whisper is comprised of the CM, MOS nuclear data , and MOS code errors . The Whisper baseline USL is absent a portion of the MOS due to the area of application, which is applied based upon judgment by the criticality safety analyst. An objective of this paper is to present the baseline USL, CM and portions of the MOS as computed by Whisper for comparison with similar sensitivity/uncertainty tools, such as those used by IRSN and ORNL. An initial comparison involved four critical experiment benchmarks: HEU-MET-FAST-013-001, HEU-SOLTHERM-001-008, PU-MET-FAST-022-001, AND PU-SOL-THERM-001-001, which have been documented in References 10-13. This study extends the comparison to include cases U233-SOL-THERM-001-001, MIX-COMP-THERM-001-001, IEU-MET-FAST-002-001, LEU-COMP-THERM-001-001, LEU-SOL-THERM- 004-001.

73 NUCLEAR PHYSICS AND RADIATION PHYSICS↗

DOE OSTI2023

Software Bill of Materials (SBOM) Sharing Lifecycle Report

As Software Bill of Materials (SBOM) adoption efforts mature, SBOM sharing continues to occur, but no single solution or set of solutions have become ubiquitous. The purpose of this report is to enumerate and describe the different parties and phases of the SBOM sharing lifecycle and assist readers in choosing suitable SBOM sharing solutions based on the amount of time, resources, subject-matter expertise, effort, and access to tooling that is available to the reader to implement a phase of the SBOM sharing lifecycle. The SBOM sharing lifecycle consists of the Discovery, Access, and Transport of an SBOM and this report details these individual phases and how an SBOM goes from author to the consumer. This report also details how potential enrichment activities may be performed on an SBOM to create a new product before or after it has been shared. The concept of a sophistication classification for SBOM sharing solutions is concurrently introduced with a focus on the inclusion or lack of certain features and effort associated with their implementation. Examples of low, medium, and high-sophistication solutions are provided; however, these examples and associated categorizations should not be seen as a qualitative judgment meant to push the reader towards a particular adoption strategy since sharing solutions are chosen based on the unique needs of the user. This report does recommend the SBOM community consider how to make current and future sharing solutions interoperable with each other as well as more automated methods to facilitate sharing and broader SBOM adoption. This report also highlights an SBOM sharing survey results obtained from interviews with stakeholders to understand the current SBOM sharing landscape. The categorized results of the survey suggest that SBOMs are currently transported directly to the receiver through email or similar informal communication mechanisms or alternatively the SBOM resides on a repository available to consumers. In addition to these transport methods, this report captures industry efforts to create private sharing solutions and services that can store and transport enrichment data and may use higher sophistication features that are cloud-based or using distributed ledger technologies.

97 MATHEMATICS AND COMPUTING↗

DOE OSTI2023

Reliability Estimation for One-Shot Devices (Rev. 1)

We present an engineering-oriented summary of statistical methods for estimation of the reliability (or equivalently, failure probability) of one-shot devices such as explosive detonators and other weapon components. Estimates may be given as single points or intervals, based on pass/fail tests, margin analysis, computational models, expert judgment, or a combination of these. We focus on highly reliable devices for which few or no failures are expected to occur in testing.

42 ENGINEERING↗

DOE OSTI2022

Method for Generating Expert Derived Confidence Scores

We executed a pilot demonstration of a methodology for developing a new confidence metric to help operators calibrate their trust in ML event classifiers. This confidence metric was derived from domain expert judgment and was accompanied with a qualitative description describing the reason for each confidence rating. After learning the boundaries of an ML’s performance by studying a subset of events an SME rated his confidence in the ML’s ability to classify similar events and provided an explanation for his ratings. To demonstrate this methodology, we developed our expert driven confidence scores for the ML event classifier within the ESAMS. Next, we assessed the accuracy of the human expert confidence scores relative to the ML’s uncertainty quantification scores. This report includes a description of our methodology, summary of our findings and future directions.

97 MATHEMATICS AND COMPUTING↗