Engineering Papers⌕ Search

Engineering topics

Dorrit Billman

Publications and source records attributed to Dorrit Billman.

18 records
All sourcesNASA NTRSDOE OSTI
NASA NTRS2021

The Effects of Training and Flight Director Use on Pilot Monitoring Performance: A Sensemaking Approach

The need for improved pilot monitoring and awareness has been widely recognized, and training is a possible intervention. Based on our sensemaking-model of monitoring, we identified key properties of monitoring flight path. We designed scenarios with associated behavioral markers that provide measures of monitoring performance and a short training module emphasizing our proactive, anticipatory view of monitoring. Nineteen first officers from a major US airline participated in the training study. Each pilot flew in a simulator pretest, participated in a training session, and flew in a simulator posttest. We found modest but significant improvements in monitoring. The study collected video, simulator, and eyetracking data and also manipulated whether the Flight Director was on or off. Limitations and future directions are discussed.

monitoring↗
NASA NTRS2021

Methods for Evaluating the Effectiveness of Programs to Train Pilot Monitoring

This report provides a compendium of methods for evaluation of training programs. It addresses training programs for developing pilot monitoring skills and has a focus on monitoring flight path management. It presents the Sensemaking Model of Monitoring as a framework for organizing the targets of training, and thus also the types of change in performance useful for assessing program effectiveness. It provides guidance for designing program evaluations, which can be tailored to address the specific monitoring topics or content for an application to fleet or airline needs.

monitoring↗
NASA NTRS2020

Evaluation Issues for a Flight Deck Interface; CAST SE-210 Output 2: Report 4 of 6

This report is part of a series of reports that addresses flight deck design and evaluation, written as a response to loss of control accidents. In particular, this activity is directed at failures in airplane state awareness, in which the pilot loses awareness of the airplane’s energy state or attitude and enters an upset condition. Another report in this series of reports speaks directly to flight deck evaluation methods and metrics for the types of attention and awareness issues that were revealed from the airplane state awareness events. In this report, we describe a wide range of flight deck evaluation issues tied to flight crew performance. The objectives are to establish a framework for thinking about how the flight deck interface should support the performance of the flight crew, and to aid the Federal Aviation Administration (FAA) in identifying relevant human performance issues during the evaluation/certification process. Issues are broken out into sections that cover physical ergonomics, design for usability, data integration and display content, attention and task management, flight crew problem solving, and flight crew teaming. For each issue, we recommend specific ways that the flight deck interface should support the flight crew. For each, we also identify existing 14 CFR Part 25 rules and guidance that are relevant to the issue. This allows the FAA to determine what current regulatory materials can support them in raising the issue with the applicant.

commercial aviation↗
NASA NTRS2020

Identification of Scenarios for System Interface Design Evaluation: CAST SE-210 Output 2 Report 5 of 6

This report is one in a series of reports describing research to develop enhanced approaches to design and evaluation of flight deck interaction. The research was conducted to support the Commercial Aviation Safety Team (CAST) response to incidents and accidents caused by failure of the flight crew to maintain aircraft attitude or energy state awareness. This report partially responds to Safety Enhancement (SE) 210.2 (CAST, 2014) by describing development of operational scenarios for evaluation of flight crew interaction. Evaluations of human performance with an interface rely on asking humans to perform operational tasks in an appropriate operational context. The report includes the major sources of material for developing operational scenarios—which include system state, operational tasks, operational context, and safety events—and a method for developing an appropriate set of operational scenarios for interface evaluation.

commercial aviation↗
NASA NTRS2020

Analysis of Pilot Monitoring Skills and a Review of Training Effectiveness

The commercial aviation industry world-wide has identified a need for improved pilot monitoring and awareness (e.g., FAA, 2013, ICAO, 2016). More specifically, aviation safety data indicate that failures in pilots’ flight path management (FPM) monitoring and awareness have contributed to a range of undesired outcomes: accidents, major upsets, and non-compliance with air traffic control (ATC) guidance. The Federal Aviation Administration (FAA) has further stated that these types of FPM failures are likely to worsen with the increasingly complex air traffic control systems and FPM concepts proposed for NextGen (https://www.faa. gov/nextgen/what_is_nextgen/) operations (e.g., see Hah et al., 2017). Adding to this complexity is the introduction of increasingly automated aircraft systems that can increase monitoring burdens. One potential mitigation for this situation is to enhance pilot training for effective monitoring. NASA Ames Research Center was asked to identify and evaluate training approaches that have the potential to enhance pilots’ ability to effectively monitor for FPM (with the result of improved awareness). The focus of this work is to identify, develop or validate training guidance to improve pilot monitoring/awareness regarding FPM and mitigate the recent trend of accidents and incidents, especially loss of control (LOC) events. The result of this work should be input for improved industry standards and FAA guidance to reduce the risk of incidents and accidents due to inadequate pilot monitoring/awareness. This is the first of three reports that were developed for this project.

aviation human factors↗
NASA NTRS2019

Factors that Influenced Airplane State Awareness Accidents and Incidents: CAST SE-210 Output 2: Report 2 of 6

This report is part of a series of reports that address flight deck design and evaluation, written as a response to loss of control accidents. In particular, this activity is directed at failures in airplane state awareness in which the pilot loses awareness of the airplane’s energy state or attitude and enters an upset condition. The Commercial Aviation Safety Team (CAST) created a team to analyze a set of incidents and accidents associated with the flight crew’s loss of awareness of aircraft attitude or energy state. The CAST activity focused on a set of 18 loss of control (LOC) accidents and incidents. In those events, the pilot seemed to lose awareness of the state of the airplane, especially related to its energy state or its attitude. To understand the roots of that loss of awareness, we analyzed the events in relation to a set of factors that were present in many of them. We show how these factors contributed to lost awareness. Then, we use that understanding to identify potentially useful changes to the flight deck interface and how those flight deck changes could be evaluated.

commercial aviation↗
NASA NTRSDate not supplied

How Do Different Knowledge Frameworks Help Us Learn From Aviation Line Observations?

Human performance includes actions that increase safety, as well as actions that can reduce safety. Ensuring safety in complex dynamic operations like commercial aviation depends on the ability to institute appropriate responses based on what is learned from flightcrew performance and the contexts in which it occurs. To do this systemically at the organization level requires collecting data on flightcrew performance, developing effective approaches to analyzing those data, and understanding how to translate what has been learned into policies, procedures, and practice. Systematic observation of front-line operators is a vital source of human performance data. Much has been learned from such observations, including methodological principles. Most observations have been based on a framework focused on managing safety challenges and the ensuing unsafe events. A complementary perspective focuses on flexibility and actions that promote continued safe and effective operation. We consider lessons learned about observational methods from an established framework focused on undesired actions and how these might be extended for a framework focused on desired actions.

safety↗
NASA NTRSDate not supplied

Training Airline Pilots for Improved Flight Path Monitoring: The Sensemaking Model Framework

The importance and benefit of improved monitoring is increasingly recognized. Improved training may be a valuable intervention. Our study (conducted 2019) assessed and trained airline First Officers on flight path monitoring skills. The exploratory study assessed monitoring pre-training in a simulator session that included monitoring challenges (8 or 7 events). A 1-hour interactive training followed, based on the Sensemaking Model of Monitoring; it presented concepts and examples using a slide deck, discussion, and simple activities. Post-training assessment used scenarios with analogous monitoring challenges (7 or 8 events) but a different setting. Performance showed significant and relatively consistent improvement. Training monitoring as sensemaking merits further investigation.

training↗
NASA NTRSDate not supplied

Limitations on the Use of Eye-Tracking Data to Understand Operator Awareness

Safety events in commercial aviation have implicated failures of attention and awareness, and the need for improved attention and awareness has pushed trainers toward eye-tracking systems as a primary tool for evaluating monitoring performance. Further, in the last 15 years, eye-tracking technology has become easier to acquire and use in operationally realistic settings. Although we believe that eye-tracking can be a useful evaluation tool, this paper describes basic limitations on relying on eye-tracking as the sole means for evaluating attention and awareness. A set of other measures are offered that can better identify where performance breakdowns occur.

aviation↗
NASA NTRSDate not supplied

Cognitive Engineering in Training: Monitoring and Pilot-Automation Coordination in Complex Environments

This paper reports our investigation of flight path monitoring in aviation. We interviewed experienced pilots to understand the knowledge and skills underlying effective monitoring and we developed an example learning environment to improve these skills. We explore how design of pilot training and learning, like the design of interfaces and of the underlying automation, benefits from cognitive engineering methods and perspective. In aviation, monitoring and managing flight path are critical activities. The influences on flight path are complex and come from the autoflight system, from control actions by the pilot, and from external factors, including weather and Air Traffic Control (ATC). Indeed, inadequate flight path monitoring is a current aviation concern as it has been implicated in accidents and incidents. Effective piloting depends on strategies for noticing, understanding, and anticipating these influences to monitor and manage flight path. Lack of such skills reduces pilots' ability to maintain safety margin and resilience. Although flightdeck automation is intended to aid pilot understanding and prediction, the Fight Management Systems (FMS) can mislead as well as aid the pilot's understanding and projection of what will happen. In dynamic conditions, FMS predictions may be based on old or incomplete information. Understanding such vulnerabilities is an important part of pilot-autoflight coordination. The learning environment we developed is designed to help pilots proactively monitor and manage flight path. We consider how a broad cognitive engineering approach might inform the "what" and "how" of learning in dynamic work domains.

pilot-monitoring↗
NASA NTRSDate not supplied

What Can We Learn From Resilient Pilot Behaviors? The Case of Energy Management While Flying a Star

Recently, there has been increased interest in documenting flightcrew behaviors that contribute to safe operations. Instead of only capturing errors, new efforts are attempting to understand how pilots manage complexity and variability in the operational environment to ensure a safe mission. This approach highlights pilot responses to events and conditions that fall outside typical TEM threats; e.g., revised ATC clearances. This approach presents a two-sided coin: characterize flightcrew resilience /or/ generate insights regarding complexity in the operational environment that is not adequately managed by current flight deck interface designs, procedures, and training. To capture operational complexity, we have been analyzing flight path management tied to flying an RNAV STAR. Because ATC often requests revisions—e.g., descend late—and because RNAV STARs may not align with airplane performance limits, flightcrews need to monitor, anticipate threats to RNAV STAR compliance, and devise ways to accommodate unexpected challenges. In this paper, we identify general strategies that can support response adaptation and explore methods to facilitate training these strategies.

flight operations↗
NASA NTRSDate not supplied

Developing and Testing Two Interfaces for Supplemental Data Service Provider (SDSP) Tools to Support UAS Traffic Management (UTM)

Researchers conducted a usability study using two graphical user interfaces (GUIs) to explore how individuals interpret and interact with different preflight information displays, and to inform the development of Uncrewed Aircraft System (UAS) preflight planning predictive support tools to assess and mitigate flight hazards and risks. A series of preflight risk-assessment tasks were developed to evaluate participant performance using the Supplemental Data Service Provider-Consolidated Dashboard (SDSP-CD) and the Human Automation Team Interface System (HATIS) GUIs. Participants were trained to use both interfaces and their performance was evaluated. These evaluations focused on participants’ preflight planning activities. Objective data on performance tasks across different scenarios involving multi-UASs, as well as self-reports of interactions and subjective experiences using the GUIs were collected. Scores on the system usability scale (SUS) and on a simple task set were examined, as well as user feedback on open-ended questions, to inform development and identify potential improvements to the interfaces.

sUAAV interfaces↗
NASA NTRSDate not supplied

Developing and Testing Two Interfaces for Supplemental Data Service Provider (SDSP) Tools to Support UAS Traffic Management (UTM)

Researchers conducted a usability study using two graphical user interfaces (GUIs) to explore how individuals interpret and interact with different preflight information displays, and to inform the development of Uncrewed Aircraft System (UAS) preflight planning predictive support tools to assess and mitigate flight hazards and risks. A series of preflight risk-assessment tasks were developed to evaluate participant performance using the Supplemental Data Service Provider-Consolidated Dashboard (SDSP-CD) and the Human Automation Team Interface System (HATIS) GUIs. Participants were trained to use both interfaces and their performance was evaluated. These evaluations focused on participants’ preflight planning activities. Objective data on performance tasks across different scenarios involving multi-UASs, as well as self-reports of interactions and subjective experiences using the GUIs were collected. Scores on the system usability scale (SUS) and on a simple task set were examined, as well as user feedback on open-ended questions, to inform development and identify potential improvements to the interfaces.

sUAAV interfaces↗
NASA NTRSDate not supplied

Survey Development and Initial Data: Flight Context and Pilot Techniques in Everyday Flights

The aviation industry is recognizing that flight crews routinely contribute to system safety in ways that go beyond adherence to standard operating procedures (SOPs). Our research goals were to explore a) whether a survey could shed light on pilots' contributions to adaptation and resilience in everyday flights and b) relevant assessment methods. The survey focused on challenges faced by pilots in normal operations, and on the ways that pilots anticipate and monitor those challenges. We collected responses concerning revenue flights from two pilot groups; one group also provided responses concerning a simulated scenario. The results indicated that relatively few flights proceeded exactly as in the original flight plan. Pilots routinely anticipated and adapted to changing circumstances. We discuss some design and assessment challenges encountered for a survey on this topic, we provide 5 approaches to assessment, and we present example findings as illustrations. We hope assessment methods such as these will lead to useful surveys of resilience in flight.

pilot behavior↗
NASA NTRSDate not supplied

Limitations on the Use of Eye-Tracking Data to Understand Operator Awareness

In the last 20 years, a number of accidents and incidents in commercial aviation have pointed to poor flight crew awareness of basic flight path parameters (e.g., airspeed, bank, pitch). As a result, there is a desire to improve pilot situation awareness and how attention is allocated. Eye-tracking has been a commonly used measure of awareness; it can aid in understanding whether specific indications were fixated, and perhaps how a pilot gathered information (that is, which indications in which sequence). In this paper, we discuss limitations on what eye-tracking data can reveal about pilot awareness and understanding. First, previous studies (e.g., Sarter et al., 2007) have shown that fixation on an indication may not ensure awareness or understanding. Further, an operator may have awareness of information not fixated. Additional measures—such as self-report or control inputs—can help to better establish the extent of pilot awareness and understanding. Second, sequences of fixations (scan patterns) have also become a performance measure. While a small number of recognized scan patterns have been validated for a small set of parameters on the Primary Flight Display (PFD), scan patterns have not been identified to support the broader context of flight path management or flight operations. More important is to understand the full set of drivers underlying the larger pattern of eye fixations; this approach moves away from the idea of well-established scan patterns as a marker of skilled performance and gives a larger role to pilot cognition. Pilots have various reasons to direct attention to specific elements on the interface, such as: - feedback tied to control inputs - a check on compliance with flight path targets - a reaction to an alert or a call out - attempt to understand an unexpected indication - assess progress toward a flight path target The importance of cognition is further implicated in the finding that pilot interviews show that fixating typically is accompanied by expectation; generally, pilots have a strong expectation of what value or indication they will see, which allows more efficient integration of information and an ability to identify indications that suggest an alternative account of the current system state. We will describe a range of eye-tracking measures and how they should and should not be used.

aviation↗
NASA NTRSDate not supplied

The Contribution of Pilots to Resilience in Normal Operations: A Survey Approach

Much of our knowledge about human performance in flight safety has come from the analysis of undesired events, whether accidents, incidents, or crew behaviors identified via flight exceedance monitoring or observational techniques. In recent years, there has been an acknowledgement that operational personnel are not merely sources of “human error”, but also make a unique human contribution to safe outcomes. In a few celebrated cases, this takes the form of “heroic saves”, but on many more occasions, operational personnel contribute to safety through everyday, often-unnoticed actions that turn potentially hazardous situations into non-events. An emerging approach to safety, frequently referred to as “Safety II,” proposes that the positive human contribution is an important and largely untapped source of safety information. Some airlines have successfully trained observers to identify and record the positive behaviors exhibited by the crew over the course of a flight. In other cases, flight crew are interviewed about good practices or positive behaviors. However, each of these methods are relatively limited in scale and resource intensive. A survey could provide a relatively low-cost approach to systematically gather this information on a larger scale. The primary purpose of the research was to develop and assess a surveys methodology for assessing crews' activities in normal flights and the operational perturbations encountered during normal operations. We hope that such a survey could be both a research tool as well as a safety management aid for the aviation industry. We collected responses concerning revenue flights from two groups of airline pilots (N = 25 & N= 65). The results indicated that relatively few flights proceeded exactly as in the original flight plan. Pilots routinely anticipated and adapted to changing circumstances. We will review the challenges encountered in developing the survey and summarize preliminary findings from two administrations of the survey to airline pilots.

human contribution safety↗
NASA NTRSDate not supplied

The Contribution of Pilots to Resilience in Normal Operations. Part II: A closer look at briefings: Anticipation and Monitoring Also Known as Planning and Coordination

Much of our knowledge about human performance in flight safety has come from the analysis of undesired events, whether accidents, incidents, or crew behaviors identified via flight exceedance monitoring or observational techniques. In recent years, there has been an acknowledgement that operational personnel are not merely sources of “human error”, but also make a unique human contribution to safe outcomes. In a few celebrated cases, this takes the form of “heroic saves”, but on many more occasions, operational personnel contribute to safety through everyday, often-unnoticed actions that turn potentially hazardous situations into non-events. An emerging approach to safety, frequently referred to as “Safety II,” proposes that the positive human contribution is an important and largely untapped source of safety information. Some airlines have successfully trained observers to identify and record the positive behaviors exhibited by the crew over the course of a flight. In other cases, flight crew are interviewed about good practices or positive behaviors. However, each of these methods are relatively limited in scale and resource intensive. A survey could provide a relatively low-cost approach to systematically gather this information on a larger scale. The primary purpose of the research was to develop and assess a surveys methodology for assessing crews' activities in normal flights and the operational perturbations encountered during normal operations. We hope that such a survey could be both a research tool as well as a safety management aid for the aviation industry. We collected responses concerning revenue flights from two groups of airline pilots (N = 25 & N= 65). The results indicated that relatively few flights proceeded exactly as in the original flight plan. Pilots routinely anticipated and adapted to changing circumstances. We will review the challenges encountered in developing the survey and summarize preliminary findings from two administrations of the survey to airline pilots.

human contribution safety↗