Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “expert elicitation”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 19 records

Machine Learning for the Validation of Expert-Elicited Causal Risk Diagrams

Exposure to spaceflight poses risk to human health in complex ways. To help manage this risk, the Human Systems Risk Board (HSRB) at the National Aeronautics and Space Administration (NASA) maintains a set of causal diagrams that attempt to explain how spaceflight hazards generate health risks and lead to adverse outcomes both in-mission, immediately post-mission, and over the long term. These causal risk diagrams are formulated as directed acyclic graphs (DAGs) and can function as knowledge graphs of connected risks and outcomes. These DAGs have proven useful for communication, and, through network analysis, have allowed for the identification of structurally important factors in the risk network. However, the utility these DAGs provide is directly proportional to their verisimilitude, making assessment of this trait using empirical data – whether from actual human spaceflight or various spaceflight analogue exposures and model organisms – a high priority. In this research we explore the use of machine learning algorithms to learn DAG structure from empirical data as a means of evaluating human-elicited DAG structures. To do so, we test several different graph structure-learning algorithms on data concerning changes in the bones of rats and mice after exposure to either spaceflight or a spaceflight analogue. We explore potential methods for indexing the similarity between each algorithm’s output DAG with all the others and with that of the expert-elicited DAG. We discuss next steps in this ongoing line of research and open science initiatives underway to complete them.

directed acyclic graphs↗

Quantifying Earth System Interactions for Sustainable Food Production Via Expert Elicitation

Several safe boundaries of critical Earth system processes have already been crossed due to human perturbations; not accounting for their interactions may further narrow the safe operating space for humanity. Using expert knowledge elicitation, we explored interactions among seven variables representing Earth system processes relevant to food production, identifying many interactions little explored in Earth system literature. We found that green water and land system change affect other Earth system processes strongly, while land, freshwater and ocean components of biosphere integrity are the most impacted by other Earth system processes, most notably blue water and biogeochemical flows. We also mapped a complex network of mechanisms mediating these interactions and created a future research prioritization scheme based on interaction strengths and existing knowledge gaps. Our study improves the understanding of Earth system interactions, with sustainability implications including improved Earth system modelling and more explicit biophysical limits for future food production.

Expert knowledge elicitation↗

Development of an Expert Judgement Elicitation and Calibration Methodology for Risk Analysis in Conceptual Vehicle Design

A comprehensive expert-judgment elicitation methodology to quantify input parameter uncertainty and analysis tool uncertainty in a conceptual launch vehicle design analysis has been developed. The ten-phase methodology seeks to obtain expert judgment opinion for quantifying uncertainties as a probability distribution so that multidisciplinary risk analysis studies can be performed. The calibration and aggregation techniques presented as part of the methodology are aimed at improving individual expert estimates, and provide an approach to aggregate multiple expert judgments into a single probability distribution. The purpose of this report is to document the methodology development and its validation through application to a reference aerospace vehicle. A detailed summary of the application exercise, including calibration and aggregation results is presented. A discussion of possible future steps in this research area is given.

Unal, Resit↗

A Modified Delphi Method to Accelerate Consensus Building in Expert Judgment Elicitation

The 2017 Earth Science Decadal Survey recommends the implementation of a novel Earth Observing mission to study Aerosols, Clouds, Convection, and Precipitation. The assessment of the candidate architectures under consideration requires the use of Expert Judgment Elicitation. Some of the assessment scores are obtained through consensus among the Science Leadership Team. A modified Delphi method was developed to accelerate the consensus building process and reduce the number of cycles required to converge. This paper discusses which elements of the traditional method were modified, how the method was applied, and the impact of the modifications on generating consensus.

Expert Judgement↗

A Modified Delphi Method to Accelerate Consensus Building in Expert Judgment Elicitation

The 2017 Earth Science Decadal Survey recommends the implementation of a novel Earth Observing mission to study Aerosols, Clouds, Convection, and Precipitation. The assessment of the candidate architectures under consideration requires the use of Expert Judgment Elicitation. Some of the assessment scores are obtained through consensus among the Science Leadership Team. A modified Delphi method was developed to accelerate the consensus building process and reduce the number of cycles required to converge. This paper discusses which elements of the traditional method were modified, how the method was applied, and the impact of the modifications on generating consensus.

Expert Judgment↗

Advancing Wildland Fire Response with NASA’s Second Shift Capabilities

Novel “Second-Shift” capabilities—leveraging Uncrewed Aerial Systems and Optionally Piloted Vehicles—are identified that could extend wildfire aerial logistics support into night or low-visibility conditions. Expert elicitations with subject-matter experts informed the development of these conceptual capabilities and identified key challenges in wildfire logistics operations. Furthermore, the potential of these capabilities could extend beyond aerial logistics support, encompassing aerial suppression, observation, and emergency extraction support for wildland firefighting. An approach to development, implementation and validation of the novel capabilities is described.

Wildfire logistics↗

Advancing Wildland Fire Response with NASA’s Second Shift Capabilities

Novel “Second-Shift” capabilities—leveraging Uncrewed Aerial Systems and Optionally Piloted Vehicles—are identified that could extend wildfire aerial logistics support into night or low-visibility conditions. Expert elicitations with subject-matter experts informed the development of these conceptual capabilities and identified key challenges in wildfire logistics operations. Furthermore, the potential of these capabilities could extend beyond aerial logistics support, encompassing aerial suppression, observation, and emergency extraction support for wildland firefighting. An approach to development, implementation and validation of the novel capabilities is described.

Wildfire logistics↗

Exit Presentation Fall 2013

Our current International Space Station Probabilistic Risk Assessment (ISS PRA) model assumes all collisions between a visiting vehicle (VV) and the ISS result in worst case loss of the ISS crew and the vehicle (LOCV). Drawing results from the Mir-Progress collision, we know this assumption is inaccurate because that collision did not lead to LOCV. Therefore the PRA team is conducting a study to determine the likelihood of LOCV when a collision occurs between a VV and the ISS. Kinetic energy is calculated and converted to pounds of TNT for the moving VVs when they collide with the ISS. Different scenarios are evaluated to obtain collision related data such as translational kinetic energy and rotational kinetic energy. These calculated data are integrated into the results from the expert elicitation performed on the Mir- Progress collision. As a result of this study, the PRA model will now calculate the probability of a VV collision with ISS, the probability that collision will result in Loss of Soyuz Crew (LOC) or Loss of ISS Crew and Vehicle (LOCV).

Yang, Qi Rong↗

Probabilistic 21st and 22nd Century Sea-Level Projections at a Global Network of Tide-Gauge Sites

Sea-level rise due to both climate change and non-climatic factors threatens coastal settlements, infrastructure, and ecosystems. Projections of mean global sea-level (GSL) rise provide insufficient information to plan adaptive responses; local decisions require local projections that accommodate different risk tolerances and time frames and that can be linked to storm surge projections. Here we present a global set of local sea-level (LSL) projections to inform decisions on timescales ranging from the coming decades through the 22nd century. We provide complete probability distributions, informed by a combination of expert community assessment, expert elicitation, and process modeling. Between the years 2000 and 2100, we project a very likely (90% probability) GSL rise of 0.5–1.2 m under representative concentration pathway (RCP) 8.5, 0.4–0.9 m under RCP 4.5, and 0.3–0.8 m under RCP 2.6. Site-to-site differences in LSL projections are due to varying non-climatic background uplift or subsidence, oceanographic effects, and spatially variable responses of the geoid and the lithosphere to shrinking land ice. The Antarctic ice sheet (AIS) constitutes a growing share of variance in GSL and LSL projections. In the global average and at many locations, it is the dominant source of variance in late 21st century projections, though at some sites oceanographic processes contribute the largest share throughout the century. LSL rise dramatically reshapes flood risk, greatly increasing the expected number of “1-in-10” and “1-in-100” year events.

sea level↗

Evolving Understanding of Antarctic Ice-Sheet Physics and Ambiguity in Probabilistic Sea-Level Projections

Mechanisms such as ice-shelf hydrofracturing and ice-cliff collapse may rapidly increase discharge from marine-based ice sheets. Here, we link a probabilistic framework for sea-level projections to a small ensemble of Antarctic ice-sheet (AIS) simulations incorporating these physical processes to explore their influence on global-mean sea-level (GMSL) and relative sea-level (RSL). We compare the new projections to past results using expert assessment and structured expert elicitation about AIS changes. Under high greenhouse gas emissions (Representative Concentration Pathway [RCP] 8.5), median projected 21st century GMSL rise increases from 79 to 146 cm. Without protective measures, revised median RSL projections would by 2100 submerge land currently home to 153 million people, an increase of 44 million. The use of a physical model, rather than simple parameterizations assuming constant acceleration of ice loss, increases forcing sensitivity: overlap between the central 90% of simulations for 2100 for RCP 8.5 (93-243 cm) and RCP 2.6 (26-98 cm) is minimal. By 2300, the gap between median GMSL estimates for RCP 8.5 and RCP 2.6 reaches >10 m, with median RSL projections for RCP 8.5 jeopardizing land now occupied by 950 million people (versus 167 million for RCP 2.6). The minimal correlation between the contribution of AIS to GMSL by 2050 and that in 2100 and beyond implies current sea-level observations cannot exclude future extreme outcomes. The sensitivity of post-2050 projections to deeply uncertain physics highlights the need for robust decision and adaptive management frameworks.

Kopp, Robert E.↗

Integrating the Social, Hydrological and Ecological Dimensions of Freshwater Health: The Freshwater Health Index

Degradation of freshwater ecosystems and the services they provide is a primary cause of increasing water insecurity, raising the need for integrated solutions to freshwater management. While methods for characterizing the multi-faceted challenges of managing freshwater ecosystems abound, they tend to emphasize either social or ecological dimensions and fall short of being truly integrative. This paper suggests that management for sustainability of freshwater systems needs to consider the linkages between human water uses, freshwater ecosystems and governance. We present a conceptualization of freshwater resources as part of an integrated social-ecological system and propose a set of corresponding indicators to monitor freshwater ecosystem health and to highlight priorities for management. We demonstrate an application of this new framework -the Freshwater Health Index (FHI) - in the Dongjiang River Basin in southern China, where stakeholders are addressing multiple and conflicting freshwater demands. By combining empirical and modeled datasets with surveys to gauge stakeholders' preferences and elicit expert information about governance mechanisms, the FHI helps stakeholders understand the status of freshwater ecosystems in their basin, how ecosystems are being manipulated to enhance or decrease water-related services, and how well the existing water re-source management regime is equipped to govern these dynamics over time. This framework helps to operationalize a truly integrated approach to water resource management by recognizing the interplay between governance, stakeholders, freshwater ecosystems and the services they provide

ecosystem services↗

Burning Embers: Towards More Transparent and Robust Climate-change Risk Assessments

The Intergovernmental Panel on Climate Change (IPCC) reports provide policy-relevant insights about climate impacts, vulnerabilities and adaptation through a process of peer-reviewed literature assessments underpinned by expert judgement. An iconic output from these assessments is the burning embers diagram, first used in the Third Assessment Report to visualize reasons for concern, which aggregate climate-change-related impacts and risks to various systems and sectors. These burning embers use colour transitions to show changes in the assessed level of risk to humans and ecosystems as a function of global mean temperature. In this Review, we outline the history and evolution of the burning embers and associated reasons for concern framework, focusing on the methodological approaches and advances. While the assessment framework and figure design have been broadly retained over time, refinements in methodology have occurred, including the consideration of different risks, use of confidence statements, more formalized protocols and standardized metrics. Comparison across reports reveals that the risk level at a given temperature has generally increased with each assessment cycle, reflecting accumulating scientific evidence. For future assessments, an explicit, transparent and systematic process of expert elicitation is needed to enhance comparability, quality and credibility of burning embers.

burning embers diagram↗

Application of Analytical Hierarchy Process for Narrowing Down Nep Candidate Reactor Designs

Nuclear electric propulsion (NEP)-powered vehicles have been contemplated for human Mars missions. The nuclear power system contemplates using a high temperature light-weight nuclear reactor for the production of electrical power in the range of 2-5 MWe with a 3-10 year service life. A myriad of technology options exist for achieving these mission objectives. The analytic hierarchy process (AHP), a multi-attribute decision method, is being used to narrow down the candidate designs. The AHP is a structured decision process that fuses model-supplied quantitative data with subjective assessments to facilitate decisions that involve multiple competing criteria. Proven end-to-end nuclear design and systems analysis tools will be used to provide quantitative performance data such as end-to-end system reliability, system robustness to recover from off-normal conditions, system specific weight (α in kg/kWe), and the ability to meet the service life-time and power level requirements. Three high assay low enriched uranium (HALEU) reactor concepts, namely, gas-, heat pipe- and pumped liquid-cooled nuclear cores coupled to a He/Xe gas or supercritical CO2- Brayton power conversion system are being modeled using the AHP to understand the trade-offs associated with these design combinations. In addition to the reactor and power conversion options, there are a myriad of additional components and subsystems – e.g., radiators, heat exchangers, and recuperators – that also figure into the evaluation process. The choice of the right overall system is a multidimensional problem that has to include not only quantitative data, but also the so called “external factors”, examples of which include the component and subsystem Technology Readiness Levels (TRLs), the associated Advancement Degrees of Difficulty (AD2), the cost and schedule required to achieve a technical maturity consistent with mission infusion, alignment with the priorities of NASA’s Space Nuclear Propulsion program, and alignment with other ongoing government and commercial investments in micro reactors. Focused expert elicitations form the basis for qualitative data set. As a final step, decision-makers individually express their opinions regarding the relative importance of the criteria and preferences among the alternatives through pairwise comparisons. The paper will describe the AHP approach, progress to-date applying it to the NEP human Mars mission problem, and preliminary results. Use of AHP provides sufficient flexibility for incorporating industry input at different stages as the technologies evolve through additional research and development. It is expected that the decision process will be ongoing and expanded to examine additional options and technology choices, culminating in a defensible set of candidate reactor concepts that will form the basis for developing a multi-year NEP technology maturation strategy. .

Dasari V Rao↗

Establishing the Framework for e-VTOL Flight Mission Scenario Development for Urban Air Mobility Research using Cognitive Task Analysis

This paper focuses on applying specific protocol aspects of Cognitive Task Analysis (CTA) methods to formulate non-routine flight mission use-case scenarios in support of a potential research study at the National Aeronautics and Space Administration (NASA) Langley Research Center under the Air Traffic Management – eXploration (ATM-X) Urban Air Mobility (UAM) sub-project. The objective of use-case scenarios is to provide a hypothetical situation to elicit expert knowledge and feedback in a specific subject matter area. Use-case scenarios in the field of aviation, including UAM, can help to support the safe integration of these vehicles into the National Airspace System (NAS) through the examination of airspace procedures during non-routine events. Inherent in this type of research is the need to study in-flight non-routine scenarios that the UAM vehicle could encounter. Situations categorized as non-routine events are any events that occur outside of the original flight plan such as a mechanical malfunction or other in-flight emergency requiring special procedures, which can include diversions to a specified, or non-specified, emergency landing area. In specific instances, some non-routine events could be considered as “routine” during in-flight operations, such as the need to perform a go-around due to a balked landing. As the rate of technological expansion surrounding UAM grows rapidly in modern aviation, so does the need for research that focuses on the safe integration of these vehicles including research with a focus on the methods and development of airspace operational procedures. Research rooted in CTA methods provides a framework for developing use-case scenarios appropriate for environments where mental demands are substantial. Substantial mental demands exist while operating in the NAS for pilots, airline dispatchers, and air traffic controllers among many others. With the focus on UAM in mind, it is critical to frame a baseline use-case scenario using information as it pertains to current-day airspace operations. According to Dr. Robert R. Hoffman’s “Protocols for Cognitive Task Analysis,” one of the first steps to applying CTA methods to research is the concept of “bootstrapping” in which the researcher(s) familiarize themselves with the domain that is being studied. To formulate the non-routine flight mission use-case scenarios for the purpose of this potential NASA ATM-X UAM research study, and to inform the baseline use-case scenario, the practice of bootstrapping was utilized to acquire knowledge of current-day airspace procedures.

Heidi S Glaudel↗

Systemic Risk From the Perspective of Climate, Environmental and Disaster Risk Science and Practice

Understanding and managing systemic risk is more important than ever due to our immense global connectivity (e.g., between sectors, such as food-health-water-energy, countries and continents, down to individuals). Despite the fact that the notion of systemic risk is several decades old, the term is used in diverse ways across different disciplines (e.g., financial systems, medicine, earth system sciences, disaster risk research and climate science). Triggered by the repercussions of the global financial crisis of the late 2000s, and more recently the COVID-19 pandemic, which are clear realization of systemic risk, the perception of systemic risk has often been focused on global and catastrophic or even existential risks. Systemic risk, however, can be seen as a feature of systems at all possible scales (e.g., global, national, regional, local) with system boundaries varying depending on the context. Addressing current societal challenges, such as climate change, in terms of systemic risk requires integrating different systems perspectives and fostering system thinking, while implementing key intergovernmental agendas, such as the Paris Agreement, the Sendai Framework for Disaster Risk Reduction and the Sustainable Development Goals. Based on insights gained and knowledge collected from an expert workshop, literature review and expert elicitation, we give an integrated perspective of climate, environmental and disaster risk science and practice on systemic risk as summarized in a Briefing Note to the International Science Council. We provide an overview of concepts of systemic risk that have evolved over time and identify commonalities across terminologies and perspectives associated with systemic risk used in different contexts. Key attributes of systemic risk are outlined without prescribing a single definition, and information and data requirements are discussed that are essential for a better and more actionable understanding of the systemic nature of risk. Finally, the opportunities to connect research and policy for addressing systemic risk are highlighted.

systemic risk↗

A Step-Wise Approach to Elicit Triangular Distributions

Adapt/combine known methods to demonstrate an expert judgment elicitation process that: 1.Models expert's inputs as a triangular distribution, 2.Incorporates techniques to account for expert bias and 3.Is structured in a way to help justify expert's inputs. This paper will show one way of "extracting" expert opinion for estimating purposes. Nevertheless, as with most subjective methods, there are many ways to do this.

Greenberg, Marc W.↗

Developing a Methodology for Eliciting Subjective Probability Estimates During Expert Evaluations of Safety Interventions: Application for Bayesian Belief Networks

The NASA Aviation Safety Program (AvSP) has defined several products that will potentially modify airline and/or ATC operations, enhance aircraft systems, and improve the identification of potential hazardous situations within the National Airspace System (NAS). Consequently, there is a need to develop methods for evaluating the potential safety benefit of each of these intervention products so that resources can be effectively invested to produce the judgments to develop Bayesian Belief Networks (BBN's) that model the potential impact that specific interventions may have. Specifically, the present report summarizes methodologies for improving the elicitation of probability estimates during expert evaluations of AvSP products for use in BBN's. The work involved joint efforts between Professor James Luxhoj from Rutgers University and researchers at the University of Illinois. The Rutgers' project to develop BBN's received funding by NASA entitled "Probabilistic Decision Support for Evaluating Technology Insertion and Assessing Aviation Safety System Risk." The proposed project was funded separately but supported the existing Rutgers' program.

Wiegmann, Douglas A.a↗