Jolene Feldman | Engineering Papers

NASA NTRS2024

Testing of Advanced Capabilities to Enable In-time Safety Management and Assurance for Future Flight Operations

In order to refine an initial Concept of Operations, explore Concepts of Use, and expose/validate requirements for future In-Time Aviation Safety Management Systems (IASMS), testing architectures were created, along with a set of capabilities and underlying information exchange protocols. These systems were conceived and developed based on hazards associated with two envisioned urban area flight domains: (1) highly autonomous small uncrewed aerial systems (sUAS) operating at low altitudes, and (2) highly autonomous air taxis. The initial scope of this development is described in [1]; this report provides an update, focusing on the subsequent developments and test activities. As stated in [1], it is important to note that there are many capabilities already in use by the industry (or soon to be in use) that will play critical roles in future IASMS designs. Those reported here were developed to address a gap in the current state-of-the-art regarding specific hazards/risks, and/or to allow for investigation of the interplay between and across hazard types — particularly regarding how overall safety risk can be reduced or managed effectively. Results of testing and development activities are organized by the operational phase wherein a particular capability would be employed (i.e., preflight, in-flight, and post-flight/off-line). Pre-flight: A set of capabilities were developed to help mitigate safety risk prior to flight (e.g., during flight and mission planning). Results of testing summarize (1) validation activities to raise the Technology Readiness Level (TRL) and (2) evaluation activities where the capabilities were applied to flight/mission planning procedures and used by operators/pilots. For the latter, flight plans were automatically assessed, and operators/pilots were notified of hazardous flight segments so as to enable adjustment of the flight plan and re-evaluation, and/or to better inform go/no-go decisions. Capabilities addressed hazards associated with power consumption, third-party risk, wind, navigation system performance, radiofrequency interference, and proximity to geo-spatial threats (e.g., buildings, trees, and no-fly zones). In-flight: Flight experiments tested capabilities that detect and respond to hazards encountered during flight. In the first series, safety hazards were monitored and assessed onboard, and system-generated mitigation maneuvers were recorded (but not acted upon by the vehicle). In the second series, mitigation maneuver commands directed the aircraft in response to safety hazards (i.e., auto-mitigation). The sUAS used for testing is described in full, as is the test architecture, which included commercial avionics, research avionics, and onboard software designed to detect, assess, and respond to hazards. The onboard system was designed as a run-time assurance framework, consistent with [2] and supportive of both supervisory and automated modes. The primary functions included: real-time risk assessment (RTRA), auto-pilot monitoring, constraint monitoring, and contingency select/triggering. RTRA performs integrated risk assessment considering data from several hazard-related monitors (e.g., battery, motors, navigation, communications, population density, and loss-of-control). Post-flight/off-line: Data monitored and recorded during flights can enable IASMS capabilities that execute after flights have completed (or “off-line”). These include: (1) the ability to identify anomalies and trends that may only be observable when comparing data spanning a number of similar flights; (2) the ability to update and validate pre-flight and in-flight capabilities and any underlying models to improve their performance; (3) the ability to report anomalies/off-nominals that may indicate design changes or maintenance actions are needed; and (4) the ability for humans involved in operations to report safety-relevant observations to help in understanding the flight data and/or the operational context of a flight. Progress on three such capabilities is summarized; the first investigates anomaly detection given a limited set of flight logs and applies an approach previously used for space operations. The second explores what could be identified using a larger set of flight logs, including from web-based forums where flight logs are posted by sUAS autopilot users. The third creates a new means of collecting information on UAS incidents and accidents via the Aviation Safety Reporting System (ASRS).

sUAS↗

NASA NTRSDate not supplied

Reports of Resilient Performance: Investigating Operators' Descriptions of Safety-producing Behaviors in the Aviation Safety Reporting System

While many existing taxonomies and frameworks provide a common vocabulary for describing how human operators fail in the context of sociotechnical systems, at present, there is no common vocabulary to describe how humans succeed. Such a framework would facilitate systematically collecting and analyzing data on how human performance can produce safety, not just how it can reduce safety. One potentially rich source of currently available information for exploring desired performance is the reports submitted to NASA’s Aviation Safety Reporting System (ASRS). These de-identified, confidential, and voluntary narrative reports are submitted by pilots, controllers, ground operators, and others within aviation operations. While these reports are primarily submitted to describe safety risks, incidents, and problems, they also often describe how those risks were mitigated, and provide a window into aspects of everyday work in aviation. This paper describes an analysis of ASRS narratives to understand how operators talk about their own resilient behaviors during adverse safety conditions and events. Guided by Erik Hollnagel’s Resilience Assessment Grid framework (i.e., anticipate, monitor, respond, learn), we illustrate our approach and methodology with examples from reports. We also highlight some of the challenges and how further research is needed in developing a taxonomy of operators’ descriptions of resilient performance.

resilient behaviors↗

NASA NTRSDate not supplied

Reports of Resilient Performance: Investigating Operators' Descriptions of Safety-producing Behaviors in the Aviation Safety Reporting System

While many existing taxonomies and frameworks provide a common vocabulary for describing how human operators fail in the context of sociotechnical systems, at present, there is no common vocabulary to describe how humans succeed. Such a framework would facilitate systematically collecting and analyzing data on how human performance can produce safety, not just how it can reduce safety. One potentially rich source of currently available information for exploring desired performance is the reports submitted to NASA’s Aviation Safety Reporting System (ASRS). These de-identified, confidential, and voluntary narrative reports are submitted by pilots, controllers, ground operators, and others within aviation operations. While these reports are primarily submitted to describe safety risks, incidents, and problems, they also often describe how those risks were mitigated, and provide a window into aspects of everyday work in aviation. These reports can be searched in a variety of ways. This paper describes methods for systematically examining ASRS narratives to understand how operators talk about their own resilient behaviors during adverse safety conditions and events. Guided by Erik Hollnagel’s Resilience Assessment Grid framework (i.e., anticipate, monitor, respond, learn), various approaches and tools for such inquiries are described. The approach, process, challenges, and suggestions to building an operator-based description of resilient performance are discussed, and tools to facilitate data analysis to maximize learning from these reports are described.

ASRS narratives↗

NASA NTRSDate not supplied

Developing a Dashboard Interface to Display Assessment of Hazards and Risks to sUAS Flights

The Supplemental Data Services Provider-Consolidated Dashboard (SDSP-CD) is a graphical user interface (GUI) that displays the results of predictive tools in a single location. It is intended to be used in the preflight planning phase of an operation to allow users to proactively assess predicted flight hazards off-line; expanding an operator’s overall situational awareness of a flight plan and providing an opportunity for decision making and assessing the associated risks prior to flight. Hazard data and risk predictions are informative but can be complex to read and understand. However, presented visually and in relation to flight parameters (such as flight path), the nature and significance of hazards become much more evident. The SDSP-Consolidation Dashboard interface was designed to offer a means to present the results of hazard services in an easy to-use format. Two usability studies were run to explore what features might make a suite of hazard assessment services easy to use, and to assess the SDSP-CD interface. The first study evaluated the presentation of information on the GUI and the second evaluated users’ ability to understand and use the information. The studies gathered valuable information about how users approach a hazard assessment task and interpret information from the interface. Many suggestions were given for improving the interface’s information display, to allow users to more quickly understand and interpret the information being presented.

UAV display↗

NASA NTRSDate not supplied

Developing a Dashboard Interface to Display Assessment of Hazards and Risks to sUAS Flights

The Supplemental Data Services Provider-Consolidated Dashboard (SDSP-CD) is a graphical user interface (GUI) that displays the results of predictive tools in a single location. It is intended to be used in the prefight planning phase of an operation to allow users to proactively assess predicted flight hazards off-line; expanding an operator’s overall situational awareness of a flight plan and providing an opportunity for decision making and assessing the associated risks prior to flight. Hazard data and risk predictions are informative but can be complex to read and understand. However, presented visually and in relation to flight parameters (such as flight path), the nature and significance of hazards become much more evident. The SDSP-Consolidation Dashboard interface was designed to offer a means to present the results of hazard services in an easy to-use format. Two usability studies were run to explore what features might make a suite of hazard assessment services easy to use, and to assess the SDSP-CD interface. The first study evaluated the presentation of information on the GUI and the second evaluated users’ ability to understand and use the information. The studies gathered valuable information about how users approach a hazard assessment task and interpret information from the interface. Many suggestions were given for improving the interface’s information display, to allow users to more quickly understand and interpret the information being presented

UAV display↗

NASA NTRSDate not supplied

Extracting Lessons of Resilience Using Machine Mining of the ASRS Database

NASA’s Aviation Safety Reporting System (ASRS) database is the world's largest repository of voluntary, confidential safety information provided by aviation's frontline personnel, including pilots, air traffic controllers, mechanics, flight attendants, dispatchers, and other members of the aviation community and the public. The database contains close to 2 million narratives, many of which describe everyday situations in which people saved the day. In these situations, people’s resilient behavior solved a problem, dealt with a malfunction, and maintained a safe operation despite a serious perturbation. To be able to extract lessons of such resilience from this large database, the use of machine learning algorithms is being explored. In this report, we describe a comparison between two such algorithms: Perilog and Word2Vec. An identical search using both programs was done on a database containing approximately 470,000 ASRS reports submitted between 1988 and 2022. The comparison reveals some of the strength and weaknesses of each algorithm as well as the challenges inherent in using such algorithms to extract lessons of resilience from the ASRS database.

resilience↗

NASA NTRSDate not supplied

Developing and Testing Two Interfaces for Supplemental Data Service Provider (SDSP) Tools to Support UAS Traffic Management (UTM)

Researchers conducted a usability study using two graphical user interfaces (GUIs) to explore how individuals interpret and interact with different preflight information displays, and to inform the development of Uncrewed Aircraft System (UAS) preflight planning predictive support tools to assess and mitigate flight hazards and risks. A series of preflight risk-assessment tasks were developed to evaluate participant performance using the Supplemental Data Service Provider-Consolidated Dashboard (SDSP-CD) and the Human Automation Team Interface System (HATIS) GUIs. Participants were trained to use both interfaces and their performance was evaluated. These evaluations focused on participants’ preflight planning activities. Objective data on performance tasks across different scenarios involving multi-UASs, as well as self-reports of interactions and subjective experiences using the GUIs were collected. Scores on the system usability scale (SUS) and on a simple task set were examined, as well as user feedback on open-ended questions, to inform development and identify potential improvements to the interfaces.

sUAAV interfaces↗

NASA NTRSDate not supplied

Developing and Testing Two Interfaces for Supplemental Data Service Provider (SDSP) Tools to Support UAS Traffic Management (UTM)

Researchers conducted a usability study using two graphical user interfaces (GUIs) to explore how individuals interpret and interact with different preflight information displays, and to inform the development of Uncrewed Aircraft System (UAS) preflight planning predictive support tools to assess and mitigate flight hazards and risks. A series of preflight risk-assessment tasks were developed to evaluate participant performance using the Supplemental Data Service Provider-Consolidated Dashboard (SDSP-CD) and the Human Automation Team Interface System (HATIS) GUIs. Participants were trained to use both interfaces and their performance was evaluated. These evaluations focused on participants’ preflight planning activities. Objective data on performance tasks across different scenarios involving multi-UASs, as well as self-reports of interactions and subjective experiences using the GUIs were collected. Scores on the system usability scale (SUS) and on a simple task set were examined, as well as user feedback on open-ended questions, to inform development and identify potential improvements to the interfaces.

sUAAV interfaces↗

NASA NTRSDate not supplied

An Approach to Identifying Aspects of Positive Pilot Behavior within the Aviation Safety Reporting System

The National Airspace System (NAS) is constantly evolving as air traffic continues to ramp up to pre-pandemic numbers and projected to grow to unprecedented levels in the coming years. As well as increasing demand to the current system, emerging operations such as Unmanned Autonomous Systems are also expected to add to complexity in the airspace. To address these issues, the industry and government agencies supporting the NAS will need to rely upon additional automation and new technologies to address future operational requirements, while continuing to be a world-leading safe transportation system. As these new technologies are implemented, the system continues to rely on human pilots and controllers in the loop to monitor the system and intervene in situations the automation cannot handle. The goal of proactively addressing safety is of foremost concern to ensure passenger confidence. The industry has implemented various Safety Monitoring Systems to identify safety risks and proactively address them before they result in a serious incident or accident. One such program is the Aviation Safety Reporting System (ASRS). ASRS is a long-established system where pilots and controllers voluntarily and anonymously report safety incidents they experienced and observed during line operations by providing rich text narratives describing the events, the environment, and conditions leading to the safety event of concern. These narratives provide insight and context around events of interest and can be used to identify emerging problems. They can trigger investigations within Flight Operational Quality Assurance or Flight Data Monitoring programs. However, this process typically focuses on the adverse events and the unsafe aspects of the operations surrounding the reported or detected events. This perspective of investigating factors that went wrong around an adverse event is commonly referred to as Safety I. Alternatively, characterizing successful actions that operators perform every day under varying conditions that keep the system within safe operating bounds is a concept referred to as Safety II. The benefit of the Safety II view is that the scope is much larger than that of Safety I since a vast majority of the operations result in successful flights. Many of the successful techniques used to manage operational threats are not documented in standard operating procedures or taught during training. They are typically acquired over time by working with experienced pilots during line operations or in many cases after experiencing a problem for the first time and reacting to it in situ, drawing from years of experience to manage the threat. In an attempt to quantify these positive actions, we are proposing an approach to extracting key behaviors within ASRS reports that can support the Safety II concept. Our analysis assumes that ASRS reports contain some descriptions of corrective actions that operators performed to prevent a situation from leading to an accident. Leveraging recent advances in Natural Language Process modeling, we have developed an approach to extract positive sentiment from reports, embed these positive statements in a vector space where they can be numerically analyzed, and clustering these statements into similar contextual categories. From these contextualized categories we can attempt to summarized and distilled aspects of the positive behavior. The goal is to identify categories of behavior that describe consistent operator techniques that supports the Safety II concept. With this information, airlines may enable learning from these positive actions, or address procedures that need to be changed to avoid having pilots implement a workaround. These insights can provide a lens into what is “going right” in the operations that may otherwise not be known widely within the community. It is envisioned that this approach can be extended to other narrative programs such as Line Operation Safety Audit or Learning Improvement Team reports where similar observed behavior can be analyzed to extract positive actions and inform the overall operations.

NLP↗

NASA NTRSDate not supplied

Usability of Pre-Flight Planning Interfaces for Supplemental Data Service Provider Tools to Support Uncrewed Aircraft System Traffic Management

Small uncrewed aircraft systems (sUASs) operate in low-altitude, uncontrolled airspace – where support services for their operators (UASOs) are not currently provided. NASA’s System-Wide Safety (SWS) project is identifying the potential risks and hazards to sUAS operations to provide, inform, and improve the designs of In-time Aviation Safety Management Systems (IASMS). The IASMS will include a suite of data-driven tools that compile and analyze data collected from aviation systems and environmental sources to predict hazards, and provide information to allow operators to mitigate these risks (Young et al., 2020). These risk and hazard services can be run and displayed to operators on graphical user interfaces (GUIs), as they relate to a vehicle(s)’ route of flight. These interfaces offer both a means to present hazard service output and offer an opportunity to test user understanding of the information, user decision making, and the best ways to present such data to an operator. Based on these future technologies and intended missions, it is important to investigate interface requirements and evaluate how operators might use these tools. Presenting salient and meaningful risk assessment information to operators is necessary to increase situation awareness and ultimately safety. Building on previous research (Feldman et al., 2022), a usability study comparing two GUIs was conducted to explore how individuals interacted with different styles of information displays. A series of pre-flight hazard and risk-assessment tasks were developed to evaluate participant performance using the Supplemental Data Service Provider Consolidated Dashboard and the Human Automation Team Interface System interfaces. Participants were trained to use both GUIs and their performance was analysed across different scenarios involving multiple sUASs. Performance on simple tasks and the System Usability Scale scores were reported by Feldman et al., 2023. Additional analyses and evaluations on more complex tasks (e.g., risk assessment, prioritization), workload and response times are examined in this paper.

sUAV interfaces↗

NASA NTRSDate not supplied

An Approach to Identifying Aspects of Positive Pilot Behavior within the Aviation Safety Reporting System

The National Airspace System (NAS) is constantly evolving as air traffic continues to ramp up to pre-pandemic numbers and projected to grow to unprecedented levels in the coming years. As well as increasing demand to the current system, emerging operations such as Unmanned Autonomous Systems are also expected to add to complexity in the airspace. To address these issues, the industry and government agencies supporting the NAS will need to rely upon additional automation and new technologies to address future operational requirements, while continuing to be a world-leading safe transportation system. As these new technologies are implemented, the system continues to rely on human pilots and controllers in the loop to monitor the system and intervene in situations the automation cannot handle. The goal of proactively addressing safety is of foremost concern to ensure passenger confidence. The industry has implemented various Safety Monitoring Systems to identify safety risks and proactively address them before they result in a serious incident or accident. One such program is the Aviation Safety Reporting System (ASRS). ASRS is a long-established system where pilots and controllers voluntarily and anonymously report safety incidents they experienced and observed during line operations by providing rich text narratives describing the events, the environment, and conditions leading to the safety event of concern. These narratives provide insight and context around events of interest and can be used to identify emerging problems. They can trigger investigations within Flight Operational Quality Assurance or Flight Data Monitoring programs. However, this process typically focuses on the adverse events and the unsafe aspects of the operations surrounding the reported or detected events. This perspective of investigating factors that went wrong around an adverse event is commonly referred to as Safety I. Alternatively, characterizing successful actions that operators perform every day under varying conditions that keep the system within safe operating bounds is a concept referred to as Safety II. The benefit of the Safety II view is that the scope is much larger than that of Safety I since a vast majority of the operations result in successful flights. Many of the successful techniques used to manage operational threats are not documented in standard operating procedures or taught during training. They are typically acquired over time by working with experienced pilots during line operations or in many cases after experiencing a problem for the first time and reacting to it in situ, drawing from years of experience to manage the threat. In an attempt to quantify these positive actions, we are proposing an approach to extracting key behaviors within ASRS reports that can support the Safety II concept. Our analysis assumes that ASRS reports contain some descriptions of corrective actions that operators performed to prevent a situation from leading to an accident. Leveraging recent advances in Natural Language Process modeling, we have developed an approach to extract positive sentiment from reports, embed these positive statements in a vector space where they can be numerically analyzed, and clustering these statements into similar contextual categories. From these contextualized categories we can attempt to summarized and distilled aspects of the positive behavior. The goal is to identify categories of behavior that describe consistent operator techniques that supports the Safety II concept. With this information, airlines may enable learning from these positive actions, or address procedures that need to be changed to avoid having pilots implement a workaround. These insights can provide a lens into what is “going right” in the operations that may otherwise not be known widely within the community. It is envisioned that this approach can be extended to other narrative programs such as Line Operation Safety Audit or Learning Improvement Team reports where similar observed behavior can be analyzed to extract positive actions and inform the overall operations.

NLP↗

NASA NTRSDate not supplied

Usability of Pre-flight Planning Interfaces for Supplemental Data Service Provider Tools to Support Uncrewed Aircraft System Traffic Management

Small uncrewed aircraft systems (sUASs) operate in low-altitude, uncontrolled airspace – where support services for their operators (UASOs) are not currently provided. NASA’s System-Wide Safety (SWS) project is identifying the potential risks and hazards to sUAS operations to provide, inform, and improve the designs of In-time Aviation Safety Management Systems (IASMS). The IASMS will include a suite of data-driven tools that compile and analyze data collected from aviation systems and environmental sources to predict hazards, and provide information to allow operators to mitigate these risks (Young et al., 2020). These risk and hazard services can be run and displayed to operators on graphical user interfaces (GUIs), as they relate to a vehicle(s)’ route of flight. These interfaces offer both a means to present hazard service output and offer an opportunity to test user understanding of the information, user decision making, and the best ways to present such data to an operator. Based on these future technologies and intended missions, it is important to investigate interface requirements and evaluate how operators might use these tools. Presenting salient and meaningful risk assessment information to operators is necessary to increase situation awareness and ultimately safety. Building on previous research (Feldman et al., 2022), a usability study comparing two GUIs was conducted to explore how individuals interacted with different styles of information displays. A series of pre-flight hazard and risk-assessment tasks were developed to evaluate participant performance using the Supplemental Data Service Provider Consolidated Dashboard and the Human Automation Team Interface System interfaces. Participants were trained to use both GUIs and their performance was analysed across different scenarios involving multiple sUASs. Performance on simple tasks and the System Usability Scale scores were reported by Feldman et al., 2023. Additional analyses and evaluations on more complex tasks (e.g., risk assessment, prioritization), workload and response times are examined in this paper.

sUAV interfaces↗

NASA NTRSDate not supplied

Development of a Safety Hazards Risk Assessment Tool for Uncrewed Aircraft System Traffic Management during Preflight Planning

Tremendous growth in the uncrewed and remotely piloted vehicle market is expected in low-altitude, uncontrolled airspace, resulting in potential decreases in safety without systems that support monitoring, assessing, and mitigating risk. At NASA, the System-Wide Safety (SWS) project has been developing a suite of data-driven tools to predict hazards so that the potential risks that these hazards pose can be mitigated. Services to predict various hazards have been developed, including battery capacity, proximity to static obstacles, population risks, global positioning system signal strength, radio frequency spectrum interference risk, and vertiport congestion. These services can monitor hazards along a flight path and if any risks posed by these hazards exceed a threshold, the uncrewed aircraft system (UAS) fleet manager can be alerted to mitigate the risk by modifying the flight path, changing the scheduled departure or arrival times, and/or diverting the vehicle to an alternate vertiport. These services were originally developed to monitor and assess risks during flight, but they have been adapted to assess hazard risks prior to departure so that a fleet manager can evaluate the potential risks for a fleet of UAS along their planned flight paths. These services have been integrated into a prototype tool called the Supplemental Data Service Provider-Consolidated Dashboard (SDSP-CD), developed at NASA Ames Research Center. The tool consists of a dashboard which provides a comprehensive overview for a number of risks and a map display that shows the details of the hazards along each flight’s path. Based on the findings from three previous studies, the SDSP-CD has been updated with new design elements and functions. In this paper, we describe lessons learned from the previous studies, changes made to the interface, and the feedback received during a follow-up usability study. Overall, participants reported that there is a substantial benefit of having a fleet manager use a consolidated dashboard to assess hazards for the vehicles in their fleet and to provide situational awareness to potential risks so that they can be mitigated prior to flight. Once the SDSP-CD matures, it will need to be integrated into flight and mission planning tools. Some initial thoughts on how this integration should be accomplished are shared in this paper. Finally, the functional differences between preflight vs. in-flight risk assessment and the differences in fleet manager vs. UAS pilot roles that may require different information and user interactions are discussed.

preflight↗