SEARCH · Engineering Papers

Results for “Expert Judgment”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

SourceResearch connectionFrom yearThrough year

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 19 records

All sources NASA NTRS DOE OSTI

NASA NTRSDate not supplied

A Modified Delphi Method to Accelerate Consensus Building in Expert Judgment Elicitation

The 2017 Earth Science Decadal Survey recommends the implementation of a novel Earth Observing mission to study Aerosols, Clouds, Convection, and Precipitation. The assessment of the candidate architectures under consideration requires the use of Expert Judgment Elicitation. Some of the assessment scores are obtained through consensus among the Science Leadership Team. A modified Delphi method was developed to accelerate the consensus building process and reduce the number of cycles required to converge. This paper discusses which elements of the traditional method were modified, how the method was applied, and the impact of the modifications on generating consensus.

Expert Judgment↗

NASA NTRSDate not supplied

A Modified Delphi Method to Accelerate Consensus Building in Expert Judgment Elicitation

Expert Judgement↗

DOE OSTI2026

Criteria for Retention of 3013 S1 Containers Based on Relative Risk and Expert Judgment

An evaluation was performed to assess the suitability of thirty-three 3013 containers proposed for retention. These containers have moisture levels greater than 0.08 wt.% – the S1 population. The remainder of the S1 population stored at SRS will be down blended and disposed of by the end of 2028. Based on field surveillance and shelf-life data available to date as well as informed technical judgment, no container is currently expected to fail in its 50-year storage period. However, corrosion risk varies across the S1 population. Relative risks were evaluated using predicted Consensus Scores and their 95% Upper Prediction Limits (UPLs). The predicted values are based on a statistical model of Consensus Score as a function of moisture, chloride, and whether the packaged material was electrorefining scrap packaged at Hanford. Consensus Score has been shown to be a useful indicator of corrosion potential, and the UPL captures uncertainty in the model predictions, providing a conservative indicator of corrosion potential. Using UPLs to determine relative risks, together with expert review, three containers were identified as not suitable for retention, and the remainder were determined to be suitable.

11 NUCLEAR FUEL CYCLE AND FUEL MATERIALS↗

NASA NTRS2004

Development of an Expert Judgement Elicitation and Calibration Methodology for Risk Analysis in Conceptual Vehicle Design

A comprehensive expert-judgment elicitation methodology to quantify input parameter uncertainty and analysis tool uncertainty in a conceptual launch vehicle design analysis has been developed. The ten-phase methodology seeks to obtain expert judgment opinion for quantifying uncertainties as a probability distribution so that multidisciplinary risk analysis studies can be performed. The calibration and aggregation techniques presented as part of the methodology are aimed at improving individual expert estimates, and provide an approach to aggregate multiple expert judgments into a single probability distribution. The purpose of this report is to document the methodology development and its validation through application to a reference aerospace vehicle. A detailed summary of the application exercise, including calibration and aggregation results is presented. A discussion of possible future steps in this research area is given.

Unal, Resit↗

NASA NTRS2012

Safety Risk Knowledge Elicitation in Support of Aeronautical R and D Portfolio Management: A Case Study

Aviation is a problem domain characterized by a high level of system complexity and uncertainty. Safety risk analysis in such a domain is especially challenging given the multitude of operations and diverse stakeholders. The Federal Aviation Administration (FAA) projects that by 2025 air traffic will increase by more than 50 percent with 1.1 billion passengers a year and more than 85,000 flights every 24 hours contributing to further delays and congestion in the sky (Circelli, 2011). This increased system complexity necessitates the application of structured safety risk analysis methods to understand and eliminate where possible, reduce, and/or mitigate risk factors. The use of expert judgments for probabilistic safety analysis in such a complex domain is necessary especially when evaluating the projected impact of future technologies, capabilities, and procedures for which current operational data may be scarce. Management of an R&D product portfolio in such a dynamic domain needs a systematic process to elicit these expert judgments, process modeling results, perform sensitivity analyses, and efficiently communicate the modeling results to decision makers. In this paper a case study focusing on the application of an R&D portfolio of aeronautical products intended to mitigate aircraft Loss of Control (LOC) accidents is presented. In particular, the knowledge elicitation process with three subject matter experts who contributed to the safety risk model is emphasized. The application and refinement of a verbal-numerical scale for conditional probability elicitation in a Bayesian Belief Network (BBN) is discussed. The preliminary findings from this initial step of a three-part elicitation are important to project management practitioners as they illustrate the vital contribution of systematic knowledge elicitation in complex domains.

Shih, Ann T.↗

DOE OSTI2025

A mathematical approach to using the forgetting curve to evaluate experience and training factors in human reliability analysis

Traditional human reliability analysis (HRA) methods have difficulty dealing with the dynamic nature of factors such as time and rely on static and expert-judgment-based assessments of performance-shaping factors (PSFs) across limited levels. In this study, we introduce a mathematical approach for dynamically evaluating the experience and training PSF. Our proposed method integrates the psychological concept of the “forgetting curve” to evaluate how PSFs are impacted by the number of trainings and the time elapsed since training. To confirm the validity of the model, we provide experimental data fitted by identifying the quantitative relationship between training and human performance. This research enables dynamic and objective assessments, thus reducing reliance on subjective expert judgment and improving the accuracy of HRA.

99 - GENERAL AND MISCELLANEOUS↗

NASA NTRS2013

A Step-Wise Approach to Elicit Triangular Distributions

Adapt/combine known methods to demonstrate an expert judgment elicitation process that: 1.Models expert's inputs as a triangular distribution, 2.Incorporates techniques to account for expert bias and 3.Is structured in a way to help justify expert's inputs. This paper will show one way of "extracting" expert opinion for estimating purposes. Nevertheless, as with most subjective methods, there are many ways to do this.

Greenberg, Marc W.↗

DOE OSTI2022

Method for Generating Expert Derived Confidence Scores

We executed a pilot demonstration of a methodology for developing a new confidence metric to help operators calibrate their trust in ML event classifiers. This confidence metric was derived from domain expert judgment and was accompanied with a qualitative description describing the reason for each confidence rating. After learning the boundaries of an ML’s performance by studying a subset of events an SME rated his confidence in the ML’s ability to classify similar events and provided an explanation for his ratings. To demonstrate this methodology, we developed our expert driven confidence scores for the ML event classifier within the ESAMS. Next, we assessed the accuracy of the human expert confidence scores relative to the ML’s uncertainty quantification scores. This report includes a description of our methodology, summary of our findings and future directions.

97 MATHEMATICS AND COMPUTING↗

DOE OSTI2026

Examining Cloud Feedback Components in the Simple Cloud-Resolving E3SM Atmosphere Model (SCREAM)

Cloud feedback remains the main source of uncertainty in climate sensitivity estimated by global climate models (GCMs), largely because subgrid cloud responses are parameterized in GCMs due to their coarse resolution. Here, this study examines cloud feedback in the global 3.25-km Simple Cloud-Resolving Energy Exascale Earth System Model (E3SM) Atmosphere Model (SCREAM 3 km) through a pair of 1-yr atmosphere-only simulations with control and +4-K sea surface temperature perturbations. SCREAM 3 km produces a positive cloud feedback that falls within but at the upper end of the range of Coupled Model Intercomparison Project phase 5 (CMIP5) and CMIP phase 6 (CMIP6) models and expert judgment. The positive cloud feedback arises from positive contributions from both high- and low-level clouds, with increases in high-cloud altitude and decreases in low-cloud amount and optical depth playing key roles. The stronger-than-CMIP-average feedback is mainly attributable to the high-cloud altitude feedback, owing to cloud tops rising nearly isothermally in SCREAM 3 km. The positive low-cloud amount feedback is weaker in SCREAM than in GCMs because estimated inversion strength (EIS) increases more dramatically with warming. A coarser 12-km resolution version of SCREAM exhibits a weaker positive cloud feedback than SCREAM 3 km, mainly because its low-cloud-radiative flux is more sensitive to EIS, leading to a stronger negative low-cloud amount feedback. With this process-level assessment of cloud feedback, this study reveals where SCREAM aligns with and diverges from conventional GCMs and expert assessment, providing insights to inform further model improvement and future expert assessment.

Cloud radiative effects↗

DOE OSTI2026

Virtual refrigerant charge sensor for variable-speed heat pumps based on feature selection

The refrigerant charge level in heat pump systems significantly impacts their energy efficiency. Virtual refrigerant charge (VRC) sensing technology has been comprehensively investigated and well-established due to its lower cost compared to physical sensors. However, the previous VRC research often relied on expert judgment and physical reasoning for their variable selection, which can potentially select redundant (or highly correlated) or insignificant features, and it is also primarily focused on single-speed systems. To address these challenges, this study proposes a VRC algorithm for variable-speed heat pumps that selects features through a rigorous feature selection method in combination with physical insights. We also propose a piecewise linear model structure segmented by subcooling temperature to accurately predict charge levels, particularly when subcooling temperatures are substantially low. The proposed algorithm was evaluated using experimental data of a residential R410A heat pump, and the performance was compared with two baseline VRC algorithms. The results are: (1) The proposed algorithm outperforms for the case with subcooling temperature less than 1 °C. (2) The proposed algorithm achieves a tested mean absolute percentage error (MAPE) of 4.23%, and improves the overall accuracy for cooling conditions by approximately 60%, compared with the two baseline algorithms. (3) The proposed algorithm uses two fewer features and improves the accuracy for undercharge cooling conditions by 68.0%, compared with baseline algorithm 2. These improvements enhance prediction accuracy and prevent overfitting, providing a more reliable refrigerant charge level prediction and helping improve the heat pump energy efficiency.

Liang, Chenjiyu↗

DOE OSTI2025

Impact of representative ground motion level on seismic PSA with the boundary between overestimation and underestimation

One commonly used approach in seismic probabilistic safety assessment (PSA) is the discrete method. This method follows the standard PSA framework and can be applied to various models, such as multi-unit models, while reducing computational costs using standard software. However, due to the inability to subdivide intervals infinitely, the discrete method approximates with a finite number of subintervals. In practice, different numbers of subintervals are applied, and the representative ground motion level is selected based on expert judgment. When employing a smaller number of subintervals, it is important to take caution to prevent underestimation. This study analyzes the impact of the representative ground motion level on seismic risk. It confirms that underestimation can occur with a small number of subintervals depending on the representative ground motion level. This study also proposes a method for determining the boundary of underestimation and overestimation. The method is demonstrated through examples, providing a mathematical foundation for selecting appropriate representative ground motion levels. By avoiding underestimation, this research helps prevent the oversight of significant risk contributors and enhances the understanding of seismic risk.

99 - GENERAL AND MISCELLANEOUS↗

DOE OSTI2023

Dynamic probabilistic risk assessment for electric grid cybersecurity

Electric grid cybersecurity risk has become a significant concern of industries and governments. This paper proposes a dynamic probabilistic risk assessment method for electric grid cybersecurity risk analysis. The proposed method helps reduce the reliance on expert judgment, capture a broad range of components and system dynamics, and model the interactions between various contributing entities (e.g., attacker, operator). In addition, the scenarios with multiple events, such as the occurrence of both cyberattacks and failures of physical components, the occurrence of both cyberattacks and operators’ (in)correct reactions, are considered and analyzed. Further, for each cyberattack scenario, Monte Carlo simulations are used to obtain possible sequences of the system's evolution under study and then derive risk estimates. As an application of the proposed method, the risk assessment method serves as the basis of risk-informed defense resource allocation to improve electric grid cybersecurity. The proposed method is verified using the IEEE 14-bus system by evaluating different security resource allocations for selected cyberattack scenarios.

24 POWER TRANSMISSION AND DISTRIBUTION↗

DOE OSTI2023

WPEC SG50: Developing an Automatically Readable, Comprehensive and Curated Experimental Nuclear Reaction Database

The Organisation for Economic Co-operation and Development (OECD) Nuclear Energy Agency (NEA) Working Party on International Nuclear Data Evaluation Co-operation (WPEC) subgroup (SG) 50 was formed in 2020 to develop an automatically readable, comprehensive and curated experimental nuclear reaction database. This database is called MEDUSAL (Machine-readable Experimental Data User Application & Library), and will draw from EXFOR. The EXFOR database preserves experimental nuclear reaction data true to its original documentation and information from the authors of the data. MEDUSAL will deviate from EXFOR by storing additional information from users of the data for their fields of work (evaluation, model development, validation, etc.). This includes expert judgment on the data sets, identification of data points as outliers, renormalization of the data to the newest monitor reactions, and estimations of missing uncertainty sources. The format for MEDUSAL is being developed to enable easy automatic parsing of large amounts of data. Here, we will summarize the use cases, high-level requirements and first steps towards developing the database MEDUSAL and the API to access it.

73 NUCLEAR PHYSICS AND RADIATION PHYSICS↗

DOE OSTI2023

Reliability Estimation for One-Shot Devices (Rev. 1)

We present an engineering-oriented summary of statistical methods for estimation of the reliability (or equivalently, failure probability) of one-shot devices such as explosive detonators and other weapon components. Estimates may be given as single points or intervals, based on pass/fail tests, margin analysis, computational models, expert judgment, or a combination of these. We focus on highly reliable devices for which few or no failures are expected to occur in testing.

42 ENGINEERING↗

DOE OSTI2024

Understanding Decision-Relevant Regional Data Products: Workshop Report

A broad community of climate adaptation practitioners, stakeholders and policymakers rely on historical reconstructions and future projections of local to regional climate. To be of value to these users, climate data must be credible, salient, and authoritative (Cash et al. 2002). Namely, data must be consistent with our physical understanding of the global Earth system, must be relevant for informing the decision-making process, and must be backed by expert judgment. As more and more data products have become available, multiple challenges have emerged around the production, evaluation, selection, and use of these data products. Consequently, to ensure crucial decisions leverage the best possible historical and future physical climate data, there is a pressing need to develop a coordinated national climate data strategy that is inclusive of all relevant communities of practice.

54 ENVIRONMENTAL SCIENCES↗

DOE OSTI2026

Assessing and Enabling Trustworthy Predictions for High-Consequence Decisions

Predictions from physics-based computational models provide critical information to inform high consequence decisions, e.g., engineering design decisions. The ability to assess the reliability of such predictions is therefore critical. However, to date, reliability assessment rely heavily on expert judgment and qualitative arguments. This report details the efforts of LDRD 233072 to develop quantitative methods to assess reliability of model predictions, especially in the context of simplifying assumptions that can impact their reliability.

42 ENGINEERING↗

DOE OSTI2026

Battery Life Prediction Using Reduced-Order Physics Models and Machine Learning (CRADA Final Report)

Phase 1 (Original CRADA, plus no-cost extension modifications #1-3, 6/1/2017 to 3/13/2021): The Australian Department of Defence (AUDoD) is performing accelerated aging tests of Li-ion batteries to benchmark their reliability and degradation characteristics. Using its previously developed battery lifetime predictive model framework, the National Laboratory of the Rockies (NLR) will develop analytical models based the AUDoD data to predict lifetime of the multiple Li-ion battery chemistries under real-world use scenarios of interest to AUDoD. The NLR model is based on physical degradation mechanisms encountered by Li-ion batteries and has been previously validated. Phase 2 (CRADA modification #4, plus no-cost extension modification #5, 2/22/2021 to 3/30/2025): Train and support Australian Department of Defence personnel to use NLR software for model-based estimation of Li-ion battery lifetime using accelerated battery aging data collected by the Australian Department of Defence. Under separate DOE funding from 2019 to 2021, NLR enhanced its battery life-prediction software using machine learning algorithms to automate portions of the model-fitting process, requiring significantly less labor and expert judgment and also adding uncertainty quantification, increasing statistical rigor. Under Phase 2, NLR will customize NLR Software and provide it to AuDoD. NLR will enhance its NLR Model to capture aging modes of AuDoD's multi-cell modules, including cell-balancing effects. NLR will develop example single-cell and multi-cell models based on one AuDoD battery aging dataset. NLR will train AuDoD personnel on NLR Software. By the conclusion of the project, NLR will have provided AuDoD the training materials, a user manual and software needed to perform their own analysis of additional and/or future battery aging datasets.

33 ADVANCED PROPULSION SYSTEMS↗

DOE OSTI2025

Automated shaker placement and regularized input estimation for MIMO testing.

Multi-input, multi-output (MIMO) testing is used in component qualification to reproduce operational responses in the laboratory. It is often preferred to single-input and base-shake testing because of the potential for equivalent or better tests using smaller actuators and shorter test suites. Given a target response, two key steps in MIMO test design are selecting actuator locations and solving for input loads. Actuator locations are often manually selected using expert judgment. If an automatic method is used, locations are usually determined by simulating the vibration control problem and minimizing a combination of the input energy and control residuals. To select a configuration, the relative importance of input energy and residuals must be specified. Specifying relative weights is, in general, a manual and subjective process. This paper develops an objective function that compares actuator configurations based on control accuracy and required input energy without any manual parameter tuning. The objective function uses an optimally selected tradeoff parameter for each candidate configuration. To choose actuator locations using the new objective function, a pivoting algorithm for integer programming problems is developed. Starting with an initial configuration (such as the one generated by a greedy algorithm), the pivoting algorithm guarantees an objective function decrease in each iteration until convergence is reached. In a simulation featuring a structure excited by a diffuse acoustic field, electrodynamic shaker locations and regularized inputs are solved for without any analyst-specified parameters. Simulations are performed in MIMO configurations where the number of target responses is less than, equal to, and greater than the number of actuators.

Multi-input multi-output↗