Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “expert elicitation”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 73 records · Page 4
All sourcesNASA NTRSDOE OSTI
DOE OSTI2020

Burning Embers: Towards more transparent and robust climate change risk assessments

The Intergovernmental Panel on Climate Change (IPCC) assessments provide policy-relevant insights about climate impacts, vulnerabilities, and adaptation through a process of peer-reviewed literature assessments underpinned by expert judgement. An iconic output from these assessments is the “burning embers” diagram. Burning embers were first used in the Third Assessment Report to visualize reasons for concern, which aggregate risks from various systems and sectors. In these diagrams, colour transitions show changes in the assessed level of risk to humans and ecosystems as a function of climate change indicated by increasing levels of global mean temperature. Here we review the approach to the construction of burning embers, its evolution across the various IPCC reports, and recent methodological advances in the elicitation process to reduce subjectivity in the expert judgement and improve the transparency of the assessment. We recommend the use of a standardized, transparent process of expert elicitation in the production of burning embers diagrams to enhance their quality, comparability, and credibility.

Zommers, Zinta↗
DOE OSTI2021

Molten Salt Reactor Fundamental Safety Function PIRT

The phenomenon identification and ranking table (PIRT) process was employed to evaluate the current capability to assess the ability of liquid-salt–fueled molten salt reactors to achieve their fundamental safety functions (FSFs). The PIRT process provides a structured mechanism to elicit and document expert opinions on the most important phenomena and the corresponding level of knowledge with regard to achieving the FSFs. The PIRT panel included reactor developers, accident progression evaluation tool developers, US Department of Energy (DOE) national laboratory technical staff, university researchers, and US Nuclear Regulatory Commission (NRC) staff. The information-gathering process concentrated on phenomena related to accidents in which the fuel salt—including any in the cover gas—has been released from the first barrier layer. This information-gathering process included identifying all potential accidents that could result in the release of a substantial amount of radionuclides that could breach or bypass the first barrier. The PIRT did not identify any previously unrecognized systemic vulnerabilities. However, the elicitation process identified several areas with low levels of knowledge and significant potential impact on accident progression. These areas either require additional research to improve the state of knowledge or additional design conservatism to accommodate the remaining unknowns.

21 SPECIFIC NUCLEAR REACTORS AND ASSOCIATED PLANTS↗
DOE OSTI2024

Independent Review of the Proof-of-Concept Cyber100 Compass Cybersecurity Risk Tool

The U.S. Department of Energy (DOE) Office of Cybersecurity, Energy Security, and Emergency Response (CESER), and Office of Electricity (OE) commissioned the National Renewable Energy Laboratory (NREL) to develop a method and tool to enable electric utilities to understand and manage the risk of cybersecurity events that can lead to physical effects like blackouts. This tool, called Cyber100 Compass, uses cybersecurity data elicited from cybersecurity experts, then incorporates that data into a tool designed to be usable by cybersecurity non-experts who understand the system itself. The tool estimates dollar-valued risks for a current or postulated future electric power digital control configuration, in order to enable utility risk planners to prioritize among proposed cybersecurity risk mitigation options. With the development of the Cyber100 Compass tool for quantification of future cyber-physical security risks, NREL has taken an initial bold step in the direction of enabling and indeed encouraging electric utilities to address the potential for cybersecurity incidents to produce detrimental physical effects related to electric power delivery. As part of the Cyber100 Compass development process, DOE funded NREL to seek out an independent technical review of the risk methodology embodied in the tool. NREL requested this review from Sandia National Laboratories, and made available to Sandia a very late version of the project report, as well as NREL personnel to provide clarification and to respond to questions. This paper provides the result of the independent review activity.

97 MATHEMATICS AND COMPUTING↗
NASA NTRS1991

A reusable knowledge acquisition shell: KASH

KASH (Knowledge Acquisition SHell) is proposed to assist a knowledge engineer by providing a set of utilities for constructing knowledge acquisition sessions based on interviewing techniques. The information elicited from domain experts during the sessions is guided by a question dependency graph (QDG). The QDG defined by the knowledge engineer, consists of a series of control questions about the domain that are used to organize the knowledge of an expert. The content information supplies by the expert, in response to the questions, is represented in the form of a concept map. These maps can be constructed in a top-down or bottom-up manner by the QDG and used by KASH to generate the rules for a large class of expert system domains. Additionally, the concept maps can support the representation of temporal knowledge. The high degree of reusability encountered in the QDG and concept maps can vastly reduce the development times and costs associated with producing intelligent decision aids, training programs, and process control functions.

Westphal, Christopher↗
NASA NTRS2001

Cognitive Task Analysis of Business Jet Pilots' Weather Flying Behaviors: Preliminary Results

This report presents preliminary findings from a cognitive task analysis (CTA) of business aviation piloting. Results describe challenging weather-related aviation decisions and the information and cues used to support these decisions. Further, these results demonstrate the role of expertise in business aviation decision-making in weather flying, and how weather information is acquired and assessed for reliability. The challenging weather scenarios and novice errors identified in the results provide the basis for experimental scenarios and dependent measures to be used in future flight simulation evaluations of candidate aviation weather information systems. Finally, we analyzed these preliminary results to recommend design and training interventions to improve business aviation decision-making with weather information. The primary objective of this report is to present these preliminary findings and to document the extended CTA methodology used to elicit and represent expert business aviator decision-making with weather information. These preliminary findings will be augmented with results from additional subjects using this methodology. A summary of the complete results, absent the detailed treatment of methodology provided in this report, will be documented in a separate publication.

Latorella, Kara↗
NASA NTRS1997

Towards a Methodology for Identifying Program Constraints During Requirements Analysis

Requirements analysis is the activity that involves determining the needs of the customer, identifying the services that the software system should provide and understanding the constraints on the solution. The result of this activity is a natural language document, typically referred to as the requirements definition document. Some of the problems that exist in defining requirements in large scale software projects includes synthesizing knowledge from various domain experts and communicating this information across multiple levels of personnel. One approach that addresses part of this problem is called context monitoring and involves identifying the properties of and relationships between objects that the system will manipulate. This paper examines several software development methodologies, discusses the support that each provide for eliciting such information from experts and specifying the information, and suggests refinements to these methodologies.

Romo, Lilly↗
NASA NTRS2014

Aviation Safety Risk Modeling: Lessons Learned From Multiple Knowledge Elicitation Sessions

Aviation safety risk modeling has elements of both art and science. In a complex domain, such as the National Airspace System (NAS), it is essential that knowledge elicitation (KE) sessions with domain experts be performed to facilitate the making of plausible inferences about the possible impacts of future technologies and procedures. This study discusses lessons learned throughout the multiple KE sessions held with domain experts to construct probabilistic safety risk models for a Loss of Control Accident Framework (LOCAF), FLightdeck Automation Problems (FLAP), and Runway Incursion (RI) mishap scenarios. The intent of these safety risk models is to support a portfolio analysis of NASA's Aviation Safety Program (AvSP). These models use the flexible, probabilistic approach of Bayesian Belief Networks (BBNs) and influence diagrams to model the complex interactions of aviation system risk factors. Each KE session had a different set of experts with diverse expertise, such as pilot, air traffic controller, certification, and/or human factors knowledge that was elicited to construct a composite, systems-level risk model. There were numerous "lessons learned" from these KE sessions that deal with behavioral aggregation, conditional probability modeling, object-oriented construction, interpretation of the safety risk results, and model verification/validation that are presented in this paper.

Luxhoj, J. T.↗
DOE OSTI2022

A Methodology for Evaluating Operator Usage of Machine Learning Recommendations for Power Grid Contingency Analysis

This work presents the application of a methodology to measure domain expert trust and workload, elicit feedback, and understand the technological usability and impact when a machine learning assistant is introduced into contingency analysis for real-time power grid simulation. The goal of this framework is to rapidly collect and analyze a broad variety of human factors data in order to accelerate the development and evaluation loop for deploying machine learning applications. We describe our methodology and analysis, and we discuss insights gained from a pilot participant about the current usability state of an early technology readiness level (TRL) artificial neural network (ANN) recommender.

29 ENERGY PLANNING, POLICY, AND ECONOMY↗
NASA NTRS1990

Computer-assisted knowledge acquisition for hypermedia systems

The usage of procedural and declarative knowledge to set up the structure or 'web' of a hypermedia environment is described. An automated knowledge acquisition tool was developed that helps a knowledge engineer elicit and represent an expert's knowledge involved in performing procedural tasks. The tool represents both procedural and prerequisite, declarative knowledge that supports each activity performed by the expert. This knowledge is output and subsequently read by a hypertext scripting language to generate the link between blank, but labeled cards. Each step of the expert's activity and each piece of supporting declarative knowledge is set up as an empty node. An instructional developer can then enter detailed instructional material concerning each step and declarative knowledge into these empty nodes. Other research is also described that facilitates the translation of knowledge from one form into a form more readily useable by computerized systems.

Steuck, Kurt↗
NASA NTRSDate not supplied

Pathfinding for Airspace with Autonomous Vehicles (PAAV) + m:N Tabletop

The Pathfinding for Airspace with Autonomous Vehicles (PAAV) project is investigating how to integrate increasingly autonomous aircraft into the current air traffic management system. The project aims to help develop airspace procedures and technologies that are scalable to future autonomous operations. The purpose of this brief is to inform a working group on past and future PAAV efforts. The working group specializes in "m:N" operations, which refers to remote operations where one or more ground-based pilots cooperatively control multiple unmanned aircraft. The PAAV team will detail an upcoming tabletop exercise that is designed to elicit feedback from subject matter experts on current barriers to m:N operations for unmanned cargo operations and potential solutions to those obstacles. The working group will be given an opportunity to provide feedback on the objectives and methodology to be used in the PAAV tabletop exercise.

autonomous↗
NASA NTRS2012

Safety Risk Knowledge Elicitation in Support of Aeronautical R and D Portfolio Management: A Case Study

Aviation is a problem domain characterized by a high level of system complexity and uncertainty. Safety risk analysis in such a domain is especially challenging given the multitude of operations and diverse stakeholders. The Federal Aviation Administration (FAA) projects that by 2025 air traffic will increase by more than 50 percent with 1.1 billion passengers a year and more than 85,000 flights every 24 hours contributing to further delays and congestion in the sky (Circelli, 2011). This increased system complexity necessitates the application of structured safety risk analysis methods to understand and eliminate where possible, reduce, and/or mitigate risk factors. The use of expert judgments for probabilistic safety analysis in such a complex domain is necessary especially when evaluating the projected impact of future technologies, capabilities, and procedures for which current operational data may be scarce. Management of an R&D product portfolio in such a dynamic domain needs a systematic process to elicit these expert judgments, process modeling results, perform sensitivity analyses, and efficiently communicate the modeling results to decision makers. In this paper a case study focusing on the application of an R&D portfolio of aeronautical products intended to mitigate aircraft Loss of Control (LOC) accidents is presented. In particular, the knowledge elicitation process with three subject matter experts who contributed to the safety risk model is emphasized. The application and refinement of a verbal-numerical scale for conditional probability elicitation in a Bayesian Belief Network (BBN) is discussed. The preliminary findings from this initial step of a three-part elicitation are important to project management practitioners as they illustrate the vital contribution of systematic knowledge elicitation in complex domains.

Shih, Ann T.↗
NASA NTRS2015

Development and Execution of the RUNSAFE Runway Safety Bayesian Belief Network Model

One focus area of the National Aeronautics and Space Administration (NASA) is to improve aviation safety. Runway safety is one such thrust of investigation and research. The two primary components of this runway safety research are in runway incursion (RI) and runway excursion (RE) events. These are adverse ground-based aviation incidents that endanger crew, passengers, aircraft and perhaps other nearby people or property. A runway incursion is the incorrect presence of an aircraft, vehicle or person on the protected area of a surface designated for the landing and take-off of aircraft; one class of RI events simultaneously involves two aircraft, such as one aircraft incorrectly landing on a runway while another aircraft is taking off from the same runway. A runway excursion is an incident involving only a single aircraft defined as a veer-off or overrun off the runway surface. Within the scope of this effort at NASA Langley Research Center (LaRC), generic RI, RE and combined (RI plus RE, or RUNSAFE) event models have each been developed and implemented as a Bayesian Belief Network (BBN). Descriptions of runway safety issues from the literature searches have been used to develop the BBN models. Numerous considerations surrounding the process of developing the event models have been documented in this report. The event models were then thoroughly reviewed by a Subject Matter Expert (SME) panel through multiple knowledge elicitation sessions. Numerous improvements to the model structure (definitions, node names, node states and the connecting link topology) were made by the SME panel. Sample executions of the final RUNSAFE model have been presented herein for baseline and worst-case scenarios. Finally, a parameter sensitivity analysis for a given scenario was performed to show the risk drivers. The NASA and LaRC research in runway safety event modeling through the use of BBN technology is important for several reasons. These include: 1) providing a means to clearly understand the cause and effect patterns leading to safety issues, incidents and accidents, 2) enabling the prioritization of specialty areas needing more attention to improve aviation safety, and 3) enabling the identification of gaps within NASA's Aviation Safety funding portfolio

Green, Lawrence L.↗
NASA NTRS1999

Institutional Memory Preservation at NASA Glenn Research Center

In this era of downsizing and deficit reduction, the preservation of institutional memory is a widespread concern for U.S. companies and governmental agencies. The National Aeronautical and Space Administration faces the pending retirement of many of the agency's long-term, senior engineers. NASA has a marvelous long-term history of success, but the agency faces a recurring problem caused by the loss of these engineers' unique knowledge and perspectives on NASA's role in aeronautics and space exploration. The current work describes a knowledge elicitation effort aimed at demonstrating the feasibility of preserving the more personal, heuristic knowledge accumulated over the years by NASA engineers, as contrasted with the "textbook" knowledge of launch vehicles. Work on this project was performed at NASA Glenn Research Center and elsewhere, and focused on launch vehicle systems integration. The initial effort was directed toward an historic view of the Centaur upper stage which is powered by two RL-10 engines. Various experts were consulted, employing a variety of knowledge elicitation techniques, regarding the Centaur and RL-10. Their knowledge is represented in searchable Web-based multimedia presentations. This paper discusses the various approaches to knowledge elicitation and knowledge representation employed, and assesses successes and challenges in trying to perform large-scale knowledge preservation of institutional memory. It is anticipated that strategies for knowledge elicitation and representation that have been developed in this grant will be utilized to elicit knowledge in a variety of domains including the complex heuristics that underly use of simulation software packages such as that being explored in the Expert System Architecture for Rocket Engine Numerical Simulators.

Coffey, J.↗
NASA NTRS1992

Generic domain models in software engineering

This paper outlines three research directions related to domain-specific software development: (1) reuse of generic models for domain-specific software development; (2) empirical evidence to determine these generic models, namely elicitation of mental knowledge schema possessed by expert software developers; and (3) exploitation of generic domain models to assist modelling of specific applications. It focuses on knowledge acquisition for domain-specific software development, with emphasis on tool support for the most important phases of software development.

Maiden, Neil↗
DOE OSTI2025

Secure Route: Roadway Risk Mapping for Transportation Planners

The secure transport of sensitive materials across U.S. road networks pose unique challenges for local, state, and federal agencies. Threats range from random events (e.g., accidents, medical emergencies, mechanical failures) to opportunistic or organized tactical assaults. Although the probability of such attacks is very low, the consequences of material loss to foreign states or terrorists can be catastrophic, qualifying these scenarios as “grey swan” events—low-probability, high-impact occurrences that are predictable but difficult to quantify. Traditional risk assessments struggle in these contexts, necessitating a shift toward subjective risk perception to inform planning. Risk perception in transport planning is shaped by various factors, including knowledge of adversarial capabilities, vehicle defenses, manifest details, and geographic features along the route. Geographic features such as bridges, tunnels, roadside elevation, and gaps in cellular coverage introduce vulnerabilities, while mitigative features include safe havens, police stations, and medical services. Temporal variables such as congestion, accidents, and weather further complicate route planning. Despite their importance, existing routing tools like Google Maps and commercial software do not explicitly account for geographic risk features, requiring planners to rely on personal familiarity with routes—a time-intensive, non-scalable approach. This work addresses these gaps by: (1) developing datasets that catalog geographic risk features along U.S. roadways, (2) eliciting risk perceptions from experienced transportation security experts, and (3) linking these perceptions to roadway conditions and geographic data. We implement these capabilities within Secure Route a novel mapping tool for classifying route segment risks associated with roadway conditions. This system provides transportation planners with an intuitive interface to assess and contextualize risk along potential routes, improving decision-making for secure transport. We present current progress in this effort and identify next steps.

Stewart, Robert [ORNL] (ORCID:0000000281867559)↗
NASA NTRS2011

Evaluating the Use of Remote Sensing Data in the USAID Famine Early Warning Systems Network

The US Agency for International Development (USAID) s Famine Early Warning System Network (FEWS NET) provides monitoring and early warning support to decision makers responsible for responding to food insecurity emergencies on three continents. FEWS NET uses satellite remote sensing and ground observations of rainfall and vegetation in order to provide information on drought, floods and other extreme weather events to decision makers. Previous research has presented results from a professional review questionnaire with FEWS NET expert end-users whose focus was to elicit Earth observation requirements. The review provided FEWS NET operational requirements and assessed the usefulness of additional remote sensing data. Here we analyzed 1342 food security update reports from FEWS NET. The reports consider the biophysical, socioeconomic, and contextual influences on the food security in 17 countries in Africa from 2000-2009. The objective was to evaluate the use of remote sensing information in comparison with other important factors in the evaluation of food security crises. The results show that all 17 countries use rainfall information, agricultural production statistics, food prices and food access parameters in their analysis of food security problems. The reports display large scale patterns that are strongly related to history of the FEWS NET program in each country. We found that rainfall data was used 84% of the time, remote sensing of vegetation 28% of the time, and gridded crop models 10%, reflecting the length of use of each product in the regions. More investment is needed in training personnel on remote sensing products to improve use of data products throughout the FEWS NET system.

Brown, Molly E.↗
NASA NTRS2012

Evaluating the Use of Remote Sensing Data in the U.S. Agency for International Development Famine Early Warning Systems Network

The U.S. Agency for International Development (USAID)'s Famine Early Warning System Network (FEWS NET) provides monitoring and early warning support to decision makers responsible for responding to food insecurity emergencies on three continents. FEWS NET uses satellite remote sensing and ground observations of rainfall and vegetation in order to provide information on drought, floods, and other extreme weather events to decision makers. Previous research has presented results from a professional review questionnaire with FEWS NET expert end-users whose focus was to elicit Earth observation requirements. The review provided FEWS NET operational requirements and assessed the usefulness of additional remote sensing data. We analyzed 1342 food security update reports from FEWS NET. The reports consider the biophysical, socioeconomic, and contextual influences on the food security in 17 countries in Africa from 2000 to 2009. The objective was to evaluate the use of remote sensing information in comparison with other important factors in the evaluation of food security crises. The results show that all 17 countries use rainfall information, agricultural production statistics, food prices, and food access parameters in their analysis of food security problems. The reports display large-scale patterns that are strongly related to history of the FEWS NET program in each country. We found that rainfall data were used 84% of the time, remote sensing of vegetation 28% of the time, and gridded crop models 10% of the time, reflecting the length of use of each product in the regions. More investment is needed in training personnel on remote sensing products to improve use of data products throughout the FEWS NET system.

USAID↗
NASA NTRS2025

Identifying Cognitive Capabilities Required for Optimal Exploration EVA Performance: A Cognitive Task Analysis

BACKGROUND Extravehicular activity (EVA) is one of the most dangerous and cognitively demanding actions that astronauts can execute, and the cognitive demands associated with future exploration EVA on the Moon and Mars are expected to be higher compared to EVA currently conducted from the International Space Station (ISS). Decrements in cognitive performance present an important risk to crew safety during exploration mission class EVA. Yet there is currently insufficient characterization of the cognitive capabilities required prior to, during, and following EVA. Furthermore, it is unclear which cognitive domains are most important for conducting mission critical decisions with crew safety implications. To address this gap, we conducted a cognitive task analysis of exploration EVA to characterize the cognitive capabilities, critical safety decisions, and contributing factors (e.g., lunar communications delay) important to monitor for optimal performance in future exploration EVA. This cognitive task analysis was conducted through interviews with astronauts and subject matter experts in EVA research and operations. Interviews focused on exploration EVA and elicited feedback on the cognitive capabilities required for specific EVA tasks and subtasks. The information from this cognitive task analysis will help close the gap in our understanding of the key cognitive capabilities required for safe decision-making during exploration mission class EVA on the Moon and Mars. METHOD We used an applied cognitive task analysis method1 over the course of interviews with a total of 15 NASA astronauts and subject matter experts in EVA. Each interview was led by a scientist with expertise in cognitive neuroscience from the Behavioral Health & Performance (BHP) Laboratory at NASA Johnson Space Center. Notes were taken by a research coordinator in the BHP Laboratory and interviews were recorded on Microsoft Teams to ensure the accuracy of notetaking. In the first interview protocol, participants were asked about the specific tasks and cognitive demands associated with EVA. This provided a high-level overview of the steps involved in the major tasks conducted during exploration EVA, as well as which of the steps require the most cognitive skill. Next, participants completed a knowledge audit, which employs a set of probes designed to describe types of domain knowledge of skill and elicit appropriate examples. In the second interview protocol, completed with a separate set of subject matter experts, interviewees were asked to complete a simulated incapacitated crew rescue (ICR) scenario2, which provided specific context that allowed probing around relevant issues such as situational awareness and potential errors. Experts were then asked to identify the knowledge, skills, and abilities (KSAs) underlying each EVA task and to provide ratings on the importance and cognitive demand of each KSA. Finally, participants also described the most likely and consequential critical safety incidents related to decrements in cognitive performance during exploration EVA and assessed the impact of lunar communication delay on cognitive performance. RESULTS & DISCUSSION Interviews for this cognitive task analysis are nearly complete and results will be presented in full at IWS 2025. Results will include a summary of all expert ratings of EVA tasks and subtasks, qualitative summaries of content from each interview part, and a discussion of future directions for products addressing cognitive performance monitoring and cognitive domain mapping in exploration EVA.

S R Anderson↗