Eliciting Experts' Judgments about Uncertain Probabilities: A Brief Primer to Using Expert Elicitation Well.
Abstract not provided.
SEARCH · Engineering Papers
Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.
Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.
Abstract not provided.
The vast majority of remediation and waste management decisions are made without quantitative consideration of economic and socio-political factors. They are made instead based on quantitative metrics of human health risk. Sustainable decisions need to be made based on all three 'pillars of sustainability' (economics, environment and social), and require understanding and characterization of the costs and values associated with each pillar. In addition, such decisions need to conform to regulatory or other legal requirements, which often constrains the decision space of interest. Although efforts are often made to include factors across all three pillars (for example, for decisions made under CERCLA), these efforts are usually qualitative, and hence difficult to defend. They lack technical defensibility, transparency and traceability. The purpose of GiSdT (Guided Interactive Statistical Decision Tools) is to provide a quantitative framework whereby all aspects of a decision problem, such as remediation and waste management decisions, can be addressed quantitatively, and hence, defensibly, transparently and traceably. Most decision-makers do not currently have access to useful or usable methods and approaches when are presented with choices that have significant impacts across all three pillars of sustainability. The goal of GiSdT is to provide that access by identifying or developing effective and user-friendly decision methods and approaches that empower decision-makers to explicitly and routinely incorporate all aspects of sustainability into their decision-making. To identify and develop these methods and approaches, GiSdT provides the tools needed for decision-makers and stakeholders to understand and characterize their knowledge of their current decision-making processes, and what methods and approaches they need to pro-actively and quantitatively address all aspects of sustainability in their decisions. GiSdT is an open source interactive framework program that provides the tools needed to address the relevant components of a sustainability-based decision problem, including describing the decision landscape, translating that landscape into goals and objectives, valuing the measurable attributes that describe each objective, identifying decision options and addressing the uncertainty in the attributes (e.g., human health risk, institutional controls) through appropriate probabilistic models. In addition to addressing these quantitative technical components, the GiSdT program is aimed at facilitating stakeholder involvement so that the perspectives, values and objectives are all relevant stakeholders are addressed explicitly in the decision making process. The technical underpinnings of our software program, called Guided Interactive Statistical Decision Tools (GiSdT), are described. (authors)
We introduce a new methodology for inferring the accuracy of computational simulations through the practice of solution verification. We demonstrate this methodology on examples from computational heat transfer, fluid dynamics and radiation transport. Our methodology is suited to both well- and ill-behaved sequences of simulations. Our approach to the analysis of these sequences of simulations incorporates expert judgment into the process directly via a flexible optimization framework, and the application of robust statistics. The expert judgment is systematically applied as constraints to the analysis, and together with the robust statistics guards against over-emphasis on anomalous analysis results. We have named our methodology Robust Verification. Our methodology is based on utilizing multiple constrained optimization problems to solve the verification model in a manner that varies the analysis' underlying assumptions. Constraints applied in the analysis can include expert judgment regarding convergence rates (bounds and expectations) as well as bounding values for physical quantities (e.g., positivity of energy or density). This approach then produces a number of error models, which are then analyzed through robust statistical techniques (median instead of mean statistics). This provides self-contained, data and expert informed error estimation including uncertainties for both the solution itself and order of convergence. Our method produces high quality results for the well-behaved cases relatively consistent with existing practice. The methodology can also produce reliable results for ill-behaved circumstances predicated on appropriate expert judgment. We demonstrate the method and compare the results with standard approaches used for both code and solution verification on well-behaved and ill-behaved simulations.
Two decades after the U.S. Senate declined in 1999 to give its consent to ratification of the Comprehensive Nuclear Test Ban Treaty (CTBT), the treaty remains in legal and political limbo. Legally, the treaty cannot enter into force without U.S. ratification. However, the Senate’s rejection does not strip the treaty of its force as an element of customary international law or alter the will of other signatories to see it enter into force. Politically, the debate over the virtues and flaws of the treaty remains as strong as ever, with some CTBT advocates pushing energetically for U.S. re-review while some opponents seek ways to “un-sign” the treaty. Meanwhile, despite the Senate’s rejection of the treaty, the United States acts in a manner consistent with its main obligations by maintaining a moratorium on explosive nuclear testing and providing financial support (as well as technical expertise) to the CTBT Organization (CTBTO)—in particular the CTBTO’s International Monitoring System (IMS). The CTBT may well remain in such limbo for a long time to come, unless some catalytic event comes along to re-shape national perceptions and priorities, one way or the other. The treaty’s future, whatever it might be, will be determined in part by how much perspectives might have changed in light of interim developments. Our purpose is to shed light on some of those developments and to assess their relevance. The 1999 ratification debate addressed a large number of issues. Rejection was driven by various policy and technical judgments. The purpose of this paper is to re-examine those technical judgments in light of what has been learned over the following two decades. On some technical matters, there was explicit recognition of extant uncertainties, as well as a conviction and hope that time would dispel them. To be doubly clear, our purpose is not to recommend a way forward on CTBT ratification. The policy judgments of 1999 and 2020 are beyond the scope of this paper. This technical re-examination proceeds as follows. It begins with a short background section to introduce the key technical matters in discussion in the U.S. ratification process. Then it moves systematically through each of those topics with a review of relevant background information, a discussion of key points in debate, an analysis of subsequent experience, and a current assessment. The historical analysis draws on many sources including the primary documents of the Congressional debate, the official article-by-article interpretation by the Department of State, and the statements of key players in the process. We also relied on key technical reports and publications by subject matter experts that have addressed in more detail many of the issues in our focus. These sources include the 2002 and 2012 reports of the National Academy of Sciences as well as publications by senior technical staff of Lawrence Livermore National Laboratory (LLNL) who were actively involved in the ratification debate, including then-Laboratory Director Bruce Tarter. We also conducted several interviews with government officials from the 1990s, past laboratory directors, and other members of the scientific community. But, except where individuals are specifically cited by name, the judgments offered here are our own and should not be attributed to any other individual or to LLNL as such.
Beam orientations in intensity-modulated radiation therapy (IMRT) is an important, but difficult NP-hard optimization problem. A considerable part of this difficulty is due to the essential prerequisite of defining, appropriately, counter-intuitive criteria for a mathematical plan evaluation that coincides with the clinical judgment of the considered plan. Moreover, the quality of beam directions depends heavily on their intensities. Usually a stochastic selector is utilized for optimizing beam orientations, and the intensities are thus calculated many thousands of times, each time for a different selection of beam directions, resulting in excessive time complexity. Therefore, selecting an appropriate set of beam directions in IMRT is still a time-consuming manual trial-and-error search procedure that depends on intuition and empirical knowledge. To overcome these difficulties, this work utilizes the concept of dose–volume histogram (DVH), which is one of the main recognized quantitative measurement tools used for plan judgment, and presents a DVH evaluation scheme that parallelizes plan evaluation with clinical plan judgment. This scheme is then utilized to optimize the intensities of each individual beam, enabling it to be represented by its DVHs, produced for the different irradiated volumes. Finally, an evolutionary algorithm is employed to search for the best combination of those DVHs reflecting the clinical optimal. The presented methods have been applied to three real clinical examples. In comparison to the standard equally spaced beam plans, improvements were reported in all three cases. In comparison with the greedy algorithm, improvements were reported in those examples requiring more than five beam directions.
This paper presents four key constructs developed in preparing IEEE Std P2411TM, Human Factors Engineering Guide for the Validation of System Designs and Integrated Systems Operations at Nuclear Facilities for submission. These constructs reflect judgments that responsible parties make in order to bring validation to closure. Such judgments weigh competing concerns and cost-benefits for stakeholders. Improved processes and methods may support such judgments but cannot alone eliminate uncertainty. Stating and clarifying these constructs aims to reduce uncertainty in the validation process, to better prepare the implementers and reviewers of future validation programs, both in the nuclear industry and beyond. Reasonable Confidence – A proof standard, as used to reach conclusions in a legal case. Legal proof standards are bounded between “preponderance of evidence” and “beyond reasonable doubt”. Reasonable confidence is comparable to an intermediate legal proof standard of “clear and convincing evidence.” Representative Test Set – A collection of scenarios of sufficient variety to represent the anticipated range of operational conditions, events, evolutions, and activities for validating the system(s) under test. Dispositive vs. Diagnostic Criteria – Relevant performance criteria are placed in one of two categories. Dispositive criteria are assessed, without exception, to determine whether a validation test passes or fails overall. Diagnostic criteria are assessed, subject to justified exception, to evaluate the quality or degree of some particular aspect of performance. Repeatability – Consistency in achieving passing results is to be demonstrated for each scenario in the representative test set; thus, the minimum number of formal repetitions for each scenario is two.
In validation, statistical methods routinely incorporate information from integral experiments to improve estimations of code biases and uncertainties. One source of high quality integral experiments is the ICSBEP Handbook. While uncertainties for criticality benchmarks are well characterized in the Handbook, information pertaining to the shared uncertainty/covariance between the cases is sparse, yet required for the aforementioned methods; correlations can result from using the same fuel for example. To remedy the lack of available covariance data in the Handbook, recent efforts have been undertaken, notably, within the framework of the NEA Expert Group on Uncertainty Analysis in Criticality Safety Assessment (UACSA) under the Working Party on Nuclear Criticality Safety (WPNCS) . The UACSA activities have focused primarily on comparing methods used to generate high quality covariance matrices for a small number of evaluations. Despite these efforts, within the Database for ICSBEP DICE only about 2% of the nearly 5000 experiments in the ICSBEP Handbook have correlation coefficients, so the amount of work required to generate the missing data appears daunting. And so, to fill in the gaps where no correlations are available, it is proposed to generate low fidelity correlations by isolating the largest components of uncertainty and determining the fraction shared between cases within an evaluation. Uncertainties can be shared when they are not independently measured, or the source of uncertainty is unchanged between cases. The degree to which an uncertainty is shared is linked to the experimental procedure; an approximation is made that in the absence of additional information, similar experimental procedures will lead to a similar fraction of shared uncertainty. Between experimental procedures there is some commonality. One method, proposed here, to help elucidate similar procedures is to encode each procedure into a decision tree grouped by uncertainty type. End nodes of the tree require judgment to assign the fraction of shared uncertainty between cases associated with the node. This can also be thought of as a set of rules, such that if experimental procedures and conditions occurred, then assign the shared uncertainty as some number between 0.0 and 1.0. One advantage of such a procedure is the formation of a transparent logical link between the judgment applied and the amount of shared uncertainty. This makes the results both more reproducible and easy to update. Generating correlation coefficients is reduced to developing decision trees that categorize how the experiments were done, and apply judgment of the degree of shared uncertainty for the end node of the tree. Trees need to be constructed after reading section 1 and 2 of the evaluation and some evaluations certainly do not contain enough information to create informative decision trees nor to assign an amount of shared uncertainty. These evaluations will not be considered here. (authors)
Abstract Identifying and quantifying preferential flow (PF) through soil—the rapid movement of water through spatially distinct pathways in the subsurface—is vital to understanding how the hydrologic cycle responds to climate, land cover, and anthropogenic changes. In recent decades, methods have been developed that use measured soil moisture time series to identify PF. Because they allow for continuous monitoring and are relatively easy to implement, these methods have become an important tool for recognizing when, where, and under what conditions PF occurs. The methods seek to identify a pattern or quantification that indicates the occurrence of PF. Most commonly, the chosen signature is either (1) a nonsequential response to infiltrated water, in which soil moisture responses do not occur in order of shallowest to deepest, or (2) a velocity criterion, in which newly infiltrated water is detected at depth earlier than is possible by nonpreferential flow processes. Alternative signatures have also been developed that have certain advantages but are less commonly utilized. Choosing among these possible signatures requires attention to their pertinent characteristics, including susceptibility to errors, possible bias toward false negatives or false positives, reliance on subjective judgments, and possible requirements for additional types of data. We review 77 studies that have applied such methods to highlight important information for readers who want to identify PF from soil moisture data and to inform those who aim to develop new methods or improve existing ones. Core Ideas Soil moisture data can be used to identify the occurrence of preferential flow (PF) and its initiating conditions. Various data‐analysis methods to identify PF differ in susceptibility to error, bias, and subjectivity. These methods can utilize vast amounts of data from soil moisture monitoring networks to develop understanding of when, where, and under what conditions PF occurs. Newly developed methods may lead to better accuracy and reliability, and reduce the need for subjective judgments. Plain Language Summary Preferential flow through soil occurs when a large amount of water is suddenly available, as during an intense storm. This type of flow moves rapidly through the soil in distinct narrow pathways rather than moving evenly throughout the body of soil, with major consequences for groundwater resources, ecosystems, spreading of contaminants, and other vital concerns. Methods of detecting preferential flow have been developed that utilize measurements of soil water content made by sensors installed at various depths. This measurement technology has been widely implemented, many locations now having datasets years in length, and various methods have been developed for using these to identify preferential flow. The various methods are based on different features in the soil moisture records and vary in their advantages and shortcomings. In this review, we explain and evaluate these methods, highlighting important information for their implementation to identify preferential flow from soil moisture data and for efforts to develop new methods or improve existing ones.
Understanding building occupancy is critical to a wide array of applications including natural hazards loss analysis, green building technologies, and population distribution modeling. Due to the expense of directly monitoring buildings, scientists rely in addition on a wide and disparate array of ancillary and open source information including subject matter expertise, survey data, and remote sensing information. These data are fused using data harmonization methods which refer to a loose collection of formal and informal techniques for fusing data together to create viable content for building occupancy estimation. In this paper, we add to the current state of the art by introducing the Population Data Tables (PDT), a Bayesian based informatics system for systematically arranging data and harmonization techniques into a consistent, transparent, knowledge learning framework that retains in the final estimation uncertainty emerging from data, expert judgment, and model parameterization. PDT probabilistically estimates ambient occupancy in units of people/1000ft2 for over 50 building types at the national and sub-national level with the goal of providing global coverage. The challenge of global coverage led to the development of an interdisciplinary geospatial informatics system tool that provides the framework for capturing, storing, and managing open source data, handling subject matter expertise, carrying out Bayesian analytics as well as visualizing and exporting occupancy estimation results. We present the PDT project, situate the work within the larger community, and report on the progress of this multi-year project.Understanding building occupancy is critical to a wide array of applications including natural hazards loss analysis, green building technologies, and population distribution modeling. Due to the expense of directly monitoring buildings, scientists rely in addition on a wide and disparate array of ancillary and open source information including subject matter expertise, survey data, and remote sensing information. These data are fused using data harmonization methods which refer to a loose collection of formal and informal techniques for fusing data together to create viable content for building occupancy estimation. In this paper, we add to the current state of the art by introducing the Population Data Tables (PDT), a Bayesian model and informatics system for systematically arranging data and harmonization techniques into a consistent, transparent, knowledge learning framework that retains in the final estimation uncertainty emerging from data, expert judgment, and model parameterization. PDT probabilistically estimates ambient occupancy in units of people/1000 ft 2 for over 50 building types at the national and sub-national level with the goal of providing global coverage. The challenge of global coverage led to the development of an interdisciplinary geospatial informatics system tool that provides the framework for capturing, storing, and managing open source data, handling subject matter expertise, carrying out Bayesian analytics as well as visualizing and exporting occupancy estimation results. We present the PDT project, situate the work within the larger community, and report on the progress of this multi-year project.
Traditional human reliability analysis (HRA) methods have difficulty dealing with the dynamic nature of factors such as time and rely on static and expert-judgment-based assessments of performance-shaping factors (PSFs) across limited levels. In this study, we introduce a mathematical approach for dynamically evaluating the experience and training PSF. Our proposed method integrates the psychological concept of the “forgetting curve” to evaluate how PSFs are impacted by the number of trainings and the time elapsed since training. To confirm the validity of the model, we provide experimental data fitted by identifying the quantitative relationship between training and human performance. This research enables dynamic and objective assessments, thus reducing reliance on subjective expert judgment and improving the accuracy of HRA.
Highlights: • Circulating components could be used for the non-invasive evaluation of renal tumor cryoablation therapeutic effects. • Different circulating components have distinct roles in the evaluation of renal tumor cryoablation. • Exosomal microRNAs may be more suitable for the early postoperative judgment of tumor elimination effects. The detection of peripheral circulating tumor-derived components, such as cell-free microRNAs, circulating microvesicles, and exosomal microRNAs, has been shown as a promising noninvasive strategy. However, the different roles of these components in tumor therapy evaluations have remained largely undefined. In this paper, we employed an in vivo model of the human clear cell renal cell carcinoma line Caki-1-bearing mice to evaluate the therapeutic effects of cryoablation, which is a new minimally invasive treatment for renal cell carcinoma. At different times after cryoablation, we found that the levels of the cell-free microRNAs miR-122, miR-155 and miR-210 were first increased and then decreased. Additionally, the number of large-sized microvesicles was increased after cryoablation, but the number of small-sized circulating microvesicles did not change. Furthermore, the exosomal microRNAs miR-126–3p, miR-17–5p, and miR-21–3p rapidly decreased one day after cryoablation, an effect that was well correlated with the treatment degree. Therefore, we suggest that these circulating components may have different levels of importance in the evaluation of the efficacy of renal cell cryoablation, furthermore, exosomal microRNAs may be more suitable for the early postoperative judgment of tumor elimination effects, which are worth further exploration in clinical practice.
Automated image processing techniques have the potential to aid in the performance evaluation of nuclear fuels by eliminating judgment calls that may vary from person-to-person or sample-to-sample. Analysis of in-core fuel performance is required for design and safety evaluations related to almost every aspect of the nuclear fuel cycle. This study presents a methodology for assessing the quality of uranium–molybdenum fuel images and describes image analysis routines designed for the characterization of several important microstructural properties. The analyses are performed in CellProfiler, an open-source program designed to enable biologists without training in computer vision or programming to automatically extract cellular measurements from large image sets. The quality metric scores an image based on three parameters: the illumination gradient across the image, the overall focus of the image, and the fraction of the image that contains scratches. The metric presents the user with the ability to ‘pass’ or ‘fail’ an image based on a reproducible quality score. Passable images may then be characterized through a separate CellProfiler pipeline, which enlists a variety of common image analysis techniques. The results demonstrate the ability to reliably pass or fail images based on the illumination, focus, and scratch fraction of the image, followed by automatic extraction of morphological data with respect to fission gas voids, interaction layers, and grain boundaries. - Graphical abstract: Display Omitted - Highlights: • A technique is developed to score U–10Mo FIB-SEM image quality using CellProfiler. • The pass/fail metric is based on image illumination, focus, and area scratched. • Automated image analysis is performed in pipeline fashion to characterize images. • Fission gas void, interaction layer, and grain boundary coverage data is extracted. • Preliminary characterization results demonstrate consistency of the algorithm.
Automated image processing routines have the potential to aid in the fuel performance evaluation process by eliminating bias in human judgment that may vary from person-to-person or sample-to-sample. This study presents several MATLAB based image analysis routines designed for fission gas void identification in post-irradiation examination of uranium molybdenum (U–Mo) monolithic-type plate fuels. Frequency domain filtration, enlisted as a pre-processing technique, can eliminate artifacts from the image without compromising the critical features of interest. This process is coupled with a bilateral filter, an edge-preserving noise removal technique aimed at preparing the image for optimal segmentation. Adaptive thresholding proved to be the most consistent gray-level feature segmentation technique for U–Mo fuel microstructures. The Sauvola adaptive threshold technique segments the image based on histogram weighting factors in stable contrast regions and local statistics in variable contrast regions. Once all processing is complete, the algorithm outputs the total fission gas void count, the mean void size, and the average porosity. The final results demonstrate an ability to extract fission gas void morphological data faster, more consistently, and at least as accurately as manual segmentation methods. - Highlights: •Automated image processing can aid in the fuel qualification process. •Routines are developed to characterize fission gas bubbles in irradiated U–Mo fuel. •Frequency domain filtration effectively eliminates FIB curtaining artifacts. •Adaptive thresholding proved to be the most accurate segmentation method. •The techniques established are ready to be applied to large scale data extraction testing.
A theory for controlling avalanche process of carrier during short pulse laser irradiation is proposed. We show that avalanche process of conduction band electrons (CBEs) is determined by the occupation number of phonons in dielectrics. The theory provides a way to suppress avalanche process and a direct judgment for the contribution of avalanche process and photon ionization process to the generation of CBEs. The obtained temperature dependent rate equation shows that the laser induced damage threshold of dielectrics, e.g., fused silica, increase nonlinearly with the decreases of temperature. Present theory predicts a new approach to improve the laser induced damage threshold of dielectrics.
Cyber defense requires decision making under uncertainty. Yet this critical area has not been a strong focus of research in judgment and decision-making. Future defense systems, which will rely on software-defined networks and may employ ‘moving target’ defenses, will increasingly automate lower level detection and analysis, but will still require humans in the loop for higher level judgment. We studied the decision making process and outcomes of 17 experienced network defense professionals who worked through a set of realistic network defense scenarios. We manipulated gain versus loss framing in a cyber defense scenario, and found significant effects in one of two focal problems. Defenders that began with a network already in quarantine (gain framing) used a quarantine system more than those that did not (loss framing). We also found some difference in perceived workload and efficacy. Alternate explanations of these findings and implications for network defense are discussed.
The moisture durability of an envelope component such as a wall or roof is difficult to predict. Moisture durability depends on all the construction materials used, as well as the climate, orientation, air tightness, and indoor conditions. Modern building codes require more insulation and tighter construction but provide little guidance about how to ensure these energy-efficient assemblies remain moisture durable. Furthermore, as new products and materials are introduced, builders are increasingly uncertain about the long-term durability of their building envelope designs. Oak Ridge National Laboratory and the US Department of Energy's Building America Program are applying a rule-based expert system methodology in a web tool to help designers determine whether a given wall design is likely to be moisture durable and provide expert guidance on moisture risk management specific to a wall design and climate. The expert system is populated with knowledge from both expert judgment and probabilistic hygrothermal simulation results.
Gender differences in cooperative choices and their neural correlates were investigated in a situation where reputation represented a crucial issue. Males and females were involved in an economic exchange (trust game) where economic and reputational payoffs had to be balanced in order to increase personal welfare. At the behavioral level, females showed a stronger reaction to negative reputation judgments that led to higher cooperation than males, measured by back transfers in the game. The neuroanatomical counterpart of this gender difference was found within the reward network (engaged in producing expectations of positive results) and reputation-related brain networks, such as the self-control network (engaged in strategically resisting the temptation to defect) and the mentalizing network (engaged in thinking about how one is viewed by others), in which the dorsolateral prefrontal cortex (DLPFC) and the medial (M)PFC respectively play a crucial role. Furthermore, both DLPFC and MPFC activity correlated with the amount of back transfer, as well as with the personality dimensions assessed with the Big-Five Questionnaire (BFQ-2). Males, according to their greater DLPFC recruitment and their higher level of the BFQ-2 subscale of Dominance, were more focused on implementing a profit-maximizing strategy, pursuing this target irrespectively of others’ judgments. On the contrary, females, according to their greater MPFC activity and their lower level of Dominance, were more focused on the reputation per se and not on the strategic component of reputation building. These findings shed light on the sexual dimorphism related to cooperative behavior and its neural correlates.
At the NRC's request, ORAU conducted surveys of the AAR Manufacturing site during the period of September 25 through September 27, 2012. The survey activities included walkover surveys and sampling activities. Once the survey team was onsite, the NRC personnel decided to forgo survey activities in the New Addition and the pickling area. Areas of the planned study boundary were inaccessible due to overgrowth/large pieces of concrete covering the soil surface; therefore, the study boundary was redefined. Gamma walkover scans of the site boundary and front yard identified multiple areas of elevated gamma radiation. As a result, two judgmental samples were collected. Sample results were above thorium background levels The answer to the PSQ relating to the relationship between thorium concentration in soil and NaI instrument response is Yes. NaI instrument response can be used as a predictor of Th-232 concentration in the 0 to 1 m layer. An R2 value of 0.79 was determined for the surface soil relationship, thus satisfying the DQOs. Moreover, the regression was cross-checked by comparing the predicted Th-232 soil core concentration to the average Th-232 concentration (Section 5.3.2). Based on the cross-check, the regression equation provides a reasonable estimate for the Th-232 concentration at the judgmental locations. Consideration must be given when applying this equation to other soil areas of the site. If the contamination was heterogeneously distributed, and not distributed in a discrete layer as it was in the study area, then using the regression equation to predict Th-232 concentration would not be applicable.