Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “expert elicitation”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 37 records · Page 2
All sourcesNASA NTRSDOE OSTI
NASA NTRS1992

Min and max are the only continuous ampersand-, V-operations for finite logics

Experts usually express their degrees of belief in their statements by the words of a natural language (like 'maybe', 'perhaps', etc.). If an expert system contains the degrees of beliefs t(A) and t(B) that correspond to the statements A and B, and a user asks this expert system whether 'A&B' is true, then it is necessary to come up with a reasonable estimate for the degree of belief of A&B. The operation that processes t(A) and t(B) into such an estimate t(A&B) is called an &-operation. Many different &-operations have been proposed. Which of them to choose? This can be (in principle) done by interviewing experts and eliciting a &-operation from them, but such a process is very time-consuming and therefore, not always possible. So, usually, to choose a &-operation, the finite set of actually possible degrees of belief is extended to an infinite set (e.g., to an interval (0,1)), define an operation there, and then restrict this operation to the finite set. Only this original finite set is considered. It is shown that a reasonable assumption that an &-operation is continuous (i.e., that gradual change in t(A) and t(B) must lead to a gradual change in t(A&B)), uniquely determines min as an &-operation. Likewise, max is the only continuous V-operation. These results are in good accordance with the experimental analysis of 'and' and 'or' in human beliefs.

Kreinovich, Vladik↗
NASA NTRS2010

NextGen Future Safety Assessment Game

The successful implementation of the next generation infrastructure systems requires solid understanding of their technical, social, political and economic aspects along with their interactions. The lack of historical data that relate to the long-term planning of complex systems introduces unique challenges for decision makers and involved stakeholders which in turn result in unsustainable systems. Also, the need to understand the infrastructure at the societal level and capture the interaction between multiple stakeholders becomes important. This paper proposes a methodology in order to develop a holistic approach aiming to provide an alternative subject-matter expert (SME) elicitation and data collection method for future sociotechnical systems. The methodology is adapted to Next Generation Air Transportation System (NextGen) decision making environment in order to demonstrate the benefits of this holistic approach.

Ancel, Ersin↗
NASA NTRS2011

NextGen Future Safety Assessment Game

The successful implementation of the next generation infrastructure systems requires solid understanding of their technical, social, political and economic aspects along with their interactions. The lack of historical data that relate to the long-term planning of complex systems introduces unique challenges for decision makers and involved stakeholders which in turn result in unsustainable systems. Also, the need to understand the infrastructure at the societal level and capture the interaction between multiple stakeholders becomes important. This paper proposes a methodology in order to develop a holistic approach aiming to provide an alternative subject-matter expert (SME) elicitation and data collection method for future sociotechnical systems. The methodology is adapted to Next Generation Air Transportation System (NextGen) decision making environment in order to demonstrate the benefits of this holistic approach.

Ancel, Ersin↗
NASA NTRSDate not supplied

Ensuring Safe Decision-Making on the Moon and Mars: Cognitive Performance Assessment for Exploration Class Mission EVA

Extravehicular activity (EVA) is one of the most dangerous and cognitively demanding actions that astronauts can execute, and the cognitive demands associated with future partial gravity EVAs on the Moon and Mars are expected to be higher compared to microgravity EVAs currently conducted from the International Space Station. Decrements in cognitive performance present an important risk to crew safety during exploration mission class EVA. Yet there is currently insufficient data to characterize cognitive performance prior to, during, and following EVA. Furthermore, it is still unclear which cognitive domains are most important for conducting mission critical decisions with crew safety implications. To address this gap, we conducted a cognitive task analysis (CTA) of EVA to characterize the procedures, the cognitive demands required, and the critical safety decisions associated with decrements in cognitive performance. We conducted a cognitive task analysis with 15 astronauts and subject matter experts in EVA operations and research. Interviews focused on surface exploration EVA and elicited feedback from experts on the cognitive skills required for specific EVA tasks, including cognitive strategies, critical cues, and decision-making strategies. A cognitive demands table was assembled to consolidate and synthesize the information from all interviews. The information from this cognitive task analysis informs ongoing exploration EVA for Moon to Mars. This work identifies the specific cognitive challenges that astronauts are likely to encounter during surface exploration EVA, and provides the foundation for: (1) prioritized and targeted cognitive performance measurement and functional performance tests, (2) EVA simulation design at varying levels of cognitive workload, and (3) the development of training and other technologies that can improve safe decision-making and inform EVA planning on future spaceflight missions to the Moon and Mars.

Steven R Anderson↗
NASA NTRS1991

A reusable knowledge acquisition shell: KASH

KASH (Knowledge Acquisition SHell) is proposed to assist a knowledge engineer by providing a set of utilities for constructing knowledge acquisition sessions based on interviewing techniques. The information elicited from domain experts during the sessions is guided by a question dependency graph (QDG). The QDG defined by the knowledge engineer, consists of a series of control questions about the domain that are used to organize the knowledge of an expert. The content information supplies by the expert, in response to the questions, is represented in the form of a concept map. These maps can be constructed in a top-down or bottom-up manner by the QDG and used by KASH to generate the rules for a large class of expert system domains. Additionally, the concept maps can support the representation of temporal knowledge. The high degree of reusability encountered in the QDG and concept maps can vastly reduce the development times and costs associated with producing intelligent decision aids, training programs, and process control functions.

Westphal, Christopher↗
NASA NTRS2001

Cognitive Task Analysis of Business Jet Pilots' Weather Flying Behaviors: Preliminary Results

This report presents preliminary findings from a cognitive task analysis (CTA) of business aviation piloting. Results describe challenging weather-related aviation decisions and the information and cues used to support these decisions. Further, these results demonstrate the role of expertise in business aviation decision-making in weather flying, and how weather information is acquired and assessed for reliability. The challenging weather scenarios and novice errors identified in the results provide the basis for experimental scenarios and dependent measures to be used in future flight simulation evaluations of candidate aviation weather information systems. Finally, we analyzed these preliminary results to recommend design and training interventions to improve business aviation decision-making with weather information. The primary objective of this report is to present these preliminary findings and to document the extended CTA methodology used to elicit and represent expert business aviator decision-making with weather information. These preliminary findings will be augmented with results from additional subjects using this methodology. A summary of the complete results, absent the detailed treatment of methodology provided in this report, will be documented in a separate publication.

Latorella, Kara↗
NASA NTRS1997

Towards a Methodology for Identifying Program Constraints During Requirements Analysis

Requirements analysis is the activity that involves determining the needs of the customer, identifying the services that the software system should provide and understanding the constraints on the solution. The result of this activity is a natural language document, typically referred to as the requirements definition document. Some of the problems that exist in defining requirements in large scale software projects includes synthesizing knowledge from various domain experts and communicating this information across multiple levels of personnel. One approach that addresses part of this problem is called context monitoring and involves identifying the properties of and relationships between objects that the system will manipulate. This paper examines several software development methodologies, discusses the support that each provide for eliciting such information from experts and specifying the information, and suggests refinements to these methodologies.

Romo, Lilly↗
NASA NTRS2014

Aviation Safety Risk Modeling: Lessons Learned From Multiple Knowledge Elicitation Sessions

Aviation safety risk modeling has elements of both art and science. In a complex domain, such as the National Airspace System (NAS), it is essential that knowledge elicitation (KE) sessions with domain experts be performed to facilitate the making of plausible inferences about the possible impacts of future technologies and procedures. This study discusses lessons learned throughout the multiple KE sessions held with domain experts to construct probabilistic safety risk models for a Loss of Control Accident Framework (LOCAF), FLightdeck Automation Problems (FLAP), and Runway Incursion (RI) mishap scenarios. The intent of these safety risk models is to support a portfolio analysis of NASA's Aviation Safety Program (AvSP). These models use the flexible, probabilistic approach of Bayesian Belief Networks (BBNs) and influence diagrams to model the complex interactions of aviation system risk factors. Each KE session had a different set of experts with diverse expertise, such as pilot, air traffic controller, certification, and/or human factors knowledge that was elicited to construct a composite, systems-level risk model. There were numerous "lessons learned" from these KE sessions that deal with behavioral aggregation, conditional probability modeling, object-oriented construction, interpretation of the safety risk results, and model verification/validation that are presented in this paper.

Luxhoj, J. T.↗
NASA NTRS1990

Computer-assisted knowledge acquisition for hypermedia systems

The usage of procedural and declarative knowledge to set up the structure or 'web' of a hypermedia environment is described. An automated knowledge acquisition tool was developed that helps a knowledge engineer elicit and represent an expert's knowledge involved in performing procedural tasks. The tool represents both procedural and prerequisite, declarative knowledge that supports each activity performed by the expert. This knowledge is output and subsequently read by a hypertext scripting language to generate the link between blank, but labeled cards. Each step of the expert's activity and each piece of supporting declarative knowledge is set up as an empty node. An instructional developer can then enter detailed instructional material concerning each step and declarative knowledge into these empty nodes. Other research is also described that facilitates the translation of knowledge from one form into a form more readily useable by computerized systems.

Steuck, Kurt↗
NASA NTRSDate not supplied

Pathfinding for Airspace with Autonomous Vehicles (PAAV) + m:N Tabletop

The Pathfinding for Airspace with Autonomous Vehicles (PAAV) project is investigating how to integrate increasingly autonomous aircraft into the current air traffic management system. The project aims to help develop airspace procedures and technologies that are scalable to future autonomous operations. The purpose of this brief is to inform a working group on past and future PAAV efforts. The working group specializes in "m:N" operations, which refers to remote operations where one or more ground-based pilots cooperatively control multiple unmanned aircraft. The PAAV team will detail an upcoming tabletop exercise that is designed to elicit feedback from subject matter experts on current barriers to m:N operations for unmanned cargo operations and potential solutions to those obstacles. The working group will be given an opportunity to provide feedback on the objectives and methodology to be used in the PAAV tabletop exercise.

autonomous↗
NASA NTRS2012

Safety Risk Knowledge Elicitation in Support of Aeronautical R and D Portfolio Management: A Case Study

Aviation is a problem domain characterized by a high level of system complexity and uncertainty. Safety risk analysis in such a domain is especially challenging given the multitude of operations and diverse stakeholders. The Federal Aviation Administration (FAA) projects that by 2025 air traffic will increase by more than 50 percent with 1.1 billion passengers a year and more than 85,000 flights every 24 hours contributing to further delays and congestion in the sky (Circelli, 2011). This increased system complexity necessitates the application of structured safety risk analysis methods to understand and eliminate where possible, reduce, and/or mitigate risk factors. The use of expert judgments for probabilistic safety analysis in such a complex domain is necessary especially when evaluating the projected impact of future technologies, capabilities, and procedures for which current operational data may be scarce. Management of an R&D product portfolio in such a dynamic domain needs a systematic process to elicit these expert judgments, process modeling results, perform sensitivity analyses, and efficiently communicate the modeling results to decision makers. In this paper a case study focusing on the application of an R&D portfolio of aeronautical products intended to mitigate aircraft Loss of Control (LOC) accidents is presented. In particular, the knowledge elicitation process with three subject matter experts who contributed to the safety risk model is emphasized. The application and refinement of a verbal-numerical scale for conditional probability elicitation in a Bayesian Belief Network (BBN) is discussed. The preliminary findings from this initial step of a three-part elicitation are important to project management practitioners as they illustrate the vital contribution of systematic knowledge elicitation in complex domains.

Shih, Ann T.↗
NASA NTRS2015

Development and Execution of the RUNSAFE Runway Safety Bayesian Belief Network Model

One focus area of the National Aeronautics and Space Administration (NASA) is to improve aviation safety. Runway safety is one such thrust of investigation and research. The two primary components of this runway safety research are in runway incursion (RI) and runway excursion (RE) events. These are adverse ground-based aviation incidents that endanger crew, passengers, aircraft and perhaps other nearby people or property. A runway incursion is the incorrect presence of an aircraft, vehicle or person on the protected area of a surface designated for the landing and take-off of aircraft; one class of RI events simultaneously involves two aircraft, such as one aircraft incorrectly landing on a runway while another aircraft is taking off from the same runway. A runway excursion is an incident involving only a single aircraft defined as a veer-off or overrun off the runway surface. Within the scope of this effort at NASA Langley Research Center (LaRC), generic RI, RE and combined (RI plus RE, or RUNSAFE) event models have each been developed and implemented as a Bayesian Belief Network (BBN). Descriptions of runway safety issues from the literature searches have been used to develop the BBN models. Numerous considerations surrounding the process of developing the event models have been documented in this report. The event models were then thoroughly reviewed by a Subject Matter Expert (SME) panel through multiple knowledge elicitation sessions. Numerous improvements to the model structure (definitions, node names, node states and the connecting link topology) were made by the SME panel. Sample executions of the final RUNSAFE model have been presented herein for baseline and worst-case scenarios. Finally, a parameter sensitivity analysis for a given scenario was performed to show the risk drivers. The NASA and LaRC research in runway safety event modeling through the use of BBN technology is important for several reasons. These include: 1) providing a means to clearly understand the cause and effect patterns leading to safety issues, incidents and accidents, 2) enabling the prioritization of specialty areas needing more attention to improve aviation safety, and 3) enabling the identification of gaps within NASA's Aviation Safety funding portfolio

Green, Lawrence L.↗
NASA NTRS1999

Institutional Memory Preservation at NASA Glenn Research Center

In this era of downsizing and deficit reduction, the preservation of institutional memory is a widespread concern for U.S. companies and governmental agencies. The National Aeronautical and Space Administration faces the pending retirement of many of the agency's long-term, senior engineers. NASA has a marvelous long-term history of success, but the agency faces a recurring problem caused by the loss of these engineers' unique knowledge and perspectives on NASA's role in aeronautics and space exploration. The current work describes a knowledge elicitation effort aimed at demonstrating the feasibility of preserving the more personal, heuristic knowledge accumulated over the years by NASA engineers, as contrasted with the "textbook" knowledge of launch vehicles. Work on this project was performed at NASA Glenn Research Center and elsewhere, and focused on launch vehicle systems integration. The initial effort was directed toward an historic view of the Centaur upper stage which is powered by two RL-10 engines. Various experts were consulted, employing a variety of knowledge elicitation techniques, regarding the Centaur and RL-10. Their knowledge is represented in searchable Web-based multimedia presentations. This paper discusses the various approaches to knowledge elicitation and knowledge representation employed, and assesses successes and challenges in trying to perform large-scale knowledge preservation of institutional memory. It is anticipated that strategies for knowledge elicitation and representation that have been developed in this grant will be utilized to elicit knowledge in a variety of domains including the complex heuristics that underly use of simulation software packages such as that being explored in the Expert System Architecture for Rocket Engine Numerical Simulators.

Coffey, J.↗
NASA NTRS1992

Generic domain models in software engineering

This paper outlines three research directions related to domain-specific software development: (1) reuse of generic models for domain-specific software development; (2) empirical evidence to determine these generic models, namely elicitation of mental knowledge schema possessed by expert software developers; and (3) exploitation of generic domain models to assist modelling of specific applications. It focuses on knowledge acquisition for domain-specific software development, with emphasis on tool support for the most important phases of software development.

Maiden, Neil↗
NASA NTRS2011

Evaluating the Use of Remote Sensing Data in the USAID Famine Early Warning Systems Network

The US Agency for International Development (USAID) s Famine Early Warning System Network (FEWS NET) provides monitoring and early warning support to decision makers responsible for responding to food insecurity emergencies on three continents. FEWS NET uses satellite remote sensing and ground observations of rainfall and vegetation in order to provide information on drought, floods and other extreme weather events to decision makers. Previous research has presented results from a professional review questionnaire with FEWS NET expert end-users whose focus was to elicit Earth observation requirements. The review provided FEWS NET operational requirements and assessed the usefulness of additional remote sensing data. Here we analyzed 1342 food security update reports from FEWS NET. The reports consider the biophysical, socioeconomic, and contextual influences on the food security in 17 countries in Africa from 2000-2009. The objective was to evaluate the use of remote sensing information in comparison with other important factors in the evaluation of food security crises. The results show that all 17 countries use rainfall information, agricultural production statistics, food prices and food access parameters in their analysis of food security problems. The reports display large scale patterns that are strongly related to history of the FEWS NET program in each country. We found that rainfall data was used 84% of the time, remote sensing of vegetation 28% of the time, and gridded crop models 10%, reflecting the length of use of each product in the regions. More investment is needed in training personnel on remote sensing products to improve use of data products throughout the FEWS NET system.

Brown, Molly E.↗
NASA NTRS2012

Evaluating the Use of Remote Sensing Data in the U.S. Agency for International Development Famine Early Warning Systems Network

The U.S. Agency for International Development (USAID)'s Famine Early Warning System Network (FEWS NET) provides monitoring and early warning support to decision makers responsible for responding to food insecurity emergencies on three continents. FEWS NET uses satellite remote sensing and ground observations of rainfall and vegetation in order to provide information on drought, floods, and other extreme weather events to decision makers. Previous research has presented results from a professional review questionnaire with FEWS NET expert end-users whose focus was to elicit Earth observation requirements. The review provided FEWS NET operational requirements and assessed the usefulness of additional remote sensing data. We analyzed 1342 food security update reports from FEWS NET. The reports consider the biophysical, socioeconomic, and contextual influences on the food security in 17 countries in Africa from 2000 to 2009. The objective was to evaluate the use of remote sensing information in comparison with other important factors in the evaluation of food security crises. The results show that all 17 countries use rainfall information, agricultural production statistics, food prices, and food access parameters in their analysis of food security problems. The reports display large-scale patterns that are strongly related to history of the FEWS NET program in each country. We found that rainfall data were used 84% of the time, remote sensing of vegetation 28% of the time, and gridded crop models 10% of the time, reflecting the length of use of each product in the regions. More investment is needed in training personnel on remote sensing products to improve use of data products throughout the FEWS NET system.

USAID↗
NASA NTRS2025

Identifying Cognitive Capabilities Required for Optimal Exploration EVA Performance: A Cognitive Task Analysis

BACKGROUND Extravehicular activity (EVA) is one of the most dangerous and cognitively demanding actions that astronauts can execute, and the cognitive demands associated with future exploration EVA on the Moon and Mars are expected to be higher compared to EVA currently conducted from the International Space Station (ISS). Decrements in cognitive performance present an important risk to crew safety during exploration mission class EVA. Yet there is currently insufficient characterization of the cognitive capabilities required prior to, during, and following EVA. Furthermore, it is unclear which cognitive domains are most important for conducting mission critical decisions with crew safety implications. To address this gap, we conducted a cognitive task analysis of exploration EVA to characterize the cognitive capabilities, critical safety decisions, and contributing factors (e.g., lunar communications delay) important to monitor for optimal performance in future exploration EVA. This cognitive task analysis was conducted through interviews with astronauts and subject matter experts in EVA research and operations. Interviews focused on exploration EVA and elicited feedback on the cognitive capabilities required for specific EVA tasks and subtasks. The information from this cognitive task analysis will help close the gap in our understanding of the key cognitive capabilities required for safe decision-making during exploration mission class EVA on the Moon and Mars. METHOD We used an applied cognitive task analysis method1 over the course of interviews with a total of 15 NASA astronauts and subject matter experts in EVA. Each interview was led by a scientist with expertise in cognitive neuroscience from the Behavioral Health & Performance (BHP) Laboratory at NASA Johnson Space Center. Notes were taken by a research coordinator in the BHP Laboratory and interviews were recorded on Microsoft Teams to ensure the accuracy of notetaking. In the first interview protocol, participants were asked about the specific tasks and cognitive demands associated with EVA. This provided a high-level overview of the steps involved in the major tasks conducted during exploration EVA, as well as which of the steps require the most cognitive skill. Next, participants completed a knowledge audit, which employs a set of probes designed to describe types of domain knowledge of skill and elicit appropriate examples. In the second interview protocol, completed with a separate set of subject matter experts, interviewees were asked to complete a simulated incapacitated crew rescue (ICR) scenario2, which provided specific context that allowed probing around relevant issues such as situational awareness and potential errors. Experts were then asked to identify the knowledge, skills, and abilities (KSAs) underlying each EVA task and to provide ratings on the importance and cognitive demand of each KSA. Finally, participants also described the most likely and consequential critical safety incidents related to decrements in cognitive performance during exploration EVA and assessed the impact of lunar communication delay on cognitive performance. RESULTS & DISCUSSION Interviews for this cognitive task analysis are nearly complete and results will be presented in full at IWS 2025. Results will include a summary of all expert ratings of EVA tasks and subtasks, qualitative summaries of content from each interview part, and a discussion of future directions for products addressing cognitive performance monitoring and cognitive domain mapping in exploration EVA.

S R Anderson↗
NASA NTRSDate not supplied

Demonstration of Airline-Based Airborne Reroute Operations using Trajectory Option Sets and Third-Party Tools

A concept that enables advanced airborne reroute operations is proposed, built on NextGen capabilities called Airborne Reroute (ABRR) and Trajectory Option Set (TOS). The concept introduces new third-party services / tools for the airline dispatchers to generate airline-preferred trajectories in response to convective weather events. The airborne TOS routes are sent to the traffic flow managers, who can evaluate the feasibility of the trajectory options with the help of their own third-party services / tools which have been built for this concept. A virtual demonstration study was conducted to elicit feedback from air traffic and airline subject-matter-experts. The feedback was generally positive, both in the benefits of the concept and the feasibility / need for the new tools to enable the concept but their feedback was mixed on the feasibility of the new tools to be third-party services instead of being integrated into their core tools. Nevertheless, this concept explores and demonstrates an evolutionary pathway toward a service-oriented future that shifts the responsibilities and the capabilities of air traffic operations from the air traffic service providers to the airline industry and third-party vendors.

airborne reroutes↗