Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “pilot learning”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 289 records · Page 16
All sourcesNASA NTRSDOE OSTI
NASA NTRS2002

Flight Evaluation of the Army/NASA Variable Stability Fly-by-Wire Rotorcraft Aircrew Systems Concept Airborne Laboratory (RASCAL) JUH-60A

NASA Ames Research Center and the U.S. Army Aeroflight dynamics Directorate (AFDD) have performed initial flight evaluations of the Research Flight Control System (RFCS) integrated into the Army/NASA Rotorcraft Aircrew Systems Concepts Airborne Laboratory (RASCAL) JUH-GOA. The highly modified JUH-GOA Black Hawk helicopter is a full authority, high bandwidth, variable stability, in-flight simulator designed to support development of advanced flight control, sensor, and integrated display and control technologies in a fail safe environment. Preparation for flight test required an extensive hazard analysis and ground testing to ensure proper system operation. A hardware in the loop development facility was utilized to evaluate control law stability following software changes, assess servo hardover upset conditions during manual and monitor disengagements and provide pilot familiarization of test techniques and software changes prior to flight. First engagement of the RFCS was conducted on 31 Aug 2001. RFCS transfer system operation, envelope expansion and a limited rate monitor evaluation have been completed with low bandwidth and model following control laws. The presentation will discuss the following - System overview including aircraft modifications and integrated development facilities used with the RASCAL facility. - Preliminary hazard identification and mitigation prior to flight test. - Ground testing used to qualify the RFCS transfer system and verify fault monitor operation. - Flight test results of low-bandwidth and model following control law evaluations including maneuver agility, control limitations, fault monitor reliability, and recovery from manual and monitor disengagement. - Lessons learned including test techniques using a passive three-axis sidearm controller, the value of the development facility in reducing risk and crew coordination issues related to the operation of a full authority, variable stability platform. - Future research and modifications planned for the RASCAL aircraft.

Dave Arterburn↗
NASA NTRS2002

Methods for semi-automated indexing for high precision information retrieval

OBJECTIVE: To evaluate a new system, ISAID (Internet-based Semi-automated Indexing of Documents), and to generate textbook indexes that are more detailed and more useful to readers. DESIGN: Pilot evaluation: simple, nonrandomized trial comparing ISAID with manual indexing methods. Methods evaluation: randomized, cross-over trial comparing three versions of ISAID and usability survey. PARTICIPANTS: Pilot evaluation: two physicians. Methods evaluation: twelve physicians, each of whom used three different versions of the system for a total of 36 indexing sessions. MEASUREMENTS: Total index term tuples generated per document per minute (TPM), with and without adjustment for concordance with other subjects; inter-indexer consistency; ratings of the usability of the ISAID indexing system. RESULTS: Compared with manual methods, ISAID decreased indexing times greatly. Using three versions of ISAID, inter-indexer consistency ranged from 15% to 65% with a mean of 41%, 31%, and 40% for each of three documents. Subjects using the full version of ISAID were faster (average TPM: 5.6) and had higher rates of concordant index generation. There were substantial learning effects, despite our use of a training/run-in phase. Subjects using the full version of ISAID were much faster by the third indexing session (average TPM: 9.1). There was a statistically significant increase in three-subject concordant indexing rate using the full version of ISAID during the second indexing session (p < 0.05). SUMMARY: Users of the ISAID indexing system create complex, precise, and accurate indexing for full-text documents much faster than users of manual methods. Furthermore, the natural language processing methods that ISAID uses to suggest indexes contributes substantially to increased indexing speed and accuracy.

Evaluation Studies↗
NASA NTRS2019

Leveraging Thousands of Contrail Observations from GLOBE Citizen Scientists

The GLOBE (Global Learning and Observations to Benefit the Environment) Program is NASA's largest and longest-operating citizen science program contributing Earth observations. Over 800,000 cloud observations have been reported worldwide since YEAR that include reports of short-lived, persistent, and persistent-spreading contrails. While contrails can be challenging to observe with space-borne platforms, humans are adept at spotting contrails from the ground. The NASA GLOBE Clouds team at NASA Langley Research Center in Hampton, Virginia matches cloud observations to multiple satellite platforms for comparison, including: NASA's CERES (Clouds and Earth's Radiant Energy System) instrument onboard Terra and Aqua, CALIPSO (Cloud-Aerosol Lidar and Infrared Pathfinder Satellite Observation), and geostationary satellites. A pilot project was started with select students in the United States to track airplanes above 25,000 ft and report airplane type, altitude, and report if a contrail was being or was not being produced. The objective of the pilot project was to establish if this is a scalable approach for building an international observational dataset documenting what types of airplanes are creating what types of contrails (short-lived, persistent, spreading) under what atmospheric conditions. Preliminary results of this pilot project will be presented.

Colón Robles, Marilé↗
NASA NTRS2016

Sleep Environment Recommendations for Future Spaceflight Vehicles

We conducted a comprehensive literature review summarizing optimal sleep hygiene parameters for lighting, temperature, airflow, humidity, comfort, intermittent and erratic sounds, and privacy and security in the sleep environment. We reviewed the design and use of sleep environments in a wide range of cohorts including among aquanauts, expeditioners, pilots, military personnel and ship operators. We also reviewed the specifications and sleep quality data arising from every NASA spaceflight mission, beginning with Gemini. Finally, we conducted structured interviews with individuals experienced in sleeping in non-traditional spaces including oilrig workers, Navy personnel, astronauts, and expeditioners. We also interviewed the engineers responsible for the design of the sleeping quarters presently deployed on the International Space Station. We found that the optimal sleep environment is cool, dark, quiet, and is perceived as safe and private. There are wide individual differences in the preferred sleep environment; therefore modifiable sleeping compartments are necessary to ensure all crewmembers are able to select personalized configurations for optimal sleep. It is possible to utilize lessons learned from prior spaceflight missions and from other industries in order to guide the design of an optimal sleep space suitable for long-duration spaceflight.

spaceflight↗
NASA NTRS1997

Walking to Olympus: An EVA Chronology

Spacewalkers enjoy a view of Earth once reserved for Apollo, Zeus, and other denizens of Mt. Olympus. During humanity's first extravehicular activity (EVA), Alexei Leonov floated above Gibraltar, the rock ancient seafarers saw as the gateway to the great unknown Atlantic. The symbolism was clear, Leonov stepped past a new Gibraltar when he stepped into space. More than 32 years and 154 EVAs later, Jerry Linenger conducted an EVA with Vladimir Tsibliyev as part of International Space Station Phase 1. They floated together above Gibraltar. Today the symbolism has new meaning: humanity is starting to think of stepping out of Earth orbit, space travel's new Gibraltar, and perhaps obtaining a new olympian view, a close-up look at Olympus Mons on Mars. Walking to Olympus: An EVA Chronology chronicles the 154 EVAs conducted from March 1965 to April 1997. It is intended to make clear the crucial role played by EVA in the history of spaceflight, as well as to chronicle the large body of EVA "lessons learned." Russia and the U.S. define EVA differently. Russian cosmonauts are said to perform EVA any time they are in vacuum in a space suit. A U.S. astronaut must have at least his head outside his spacecraft before he is said to perform an EVA. The difference is based in differing spacecraft design philoso- phies. Russian and Soviet spacecraft have always had a specialized airlock through which the EVA cosmonaut egressed, leaving the main habitable volume of the spacecraft pressurized. The U.S. Gemini and Apollo vehicles, on the other hand, depressurized their entire habitable volume for egress. In this document, we apply the Russian definition to Russian EVAS, and the U.S. definition to U.S. EVAS. Thus, for example, Gemini 4 Command Pilot James McDivitt does not share the honor of being first American spacewalker with Ed White, even though he was suited and in vacuum when White stepped out into space. Non-EVA spaceflights are listed in the chronology to provide context and to display the large num- ber of flights in which EVA played a role. This approach also makes apparent significant EVA gaps, for example, the U.S. gap between 1985 and 1991 following the Challenger accident. This NASA History Monograph is an edited extract from an extensive EVA Chronology and Reference Book being produced by the EVA Project Office, NASA Johnson Space Center, Houston, Texas. The larger work will be published as part of the NASA Formal Series in 1998. The authors gratefully acknowledge the assistance rendered by Max Ary, Ashot Bakunts, Gert-Jan Bartelds, Frank Cepollina, Andrew Chaikin, Phillip Clark, Richard Fullerton, Steven Glenn, Linda Godwin, Jennifer Green, Greg Harris, Clifford Hess, Jeffrey Hoffman, David Homan, Steven Hopkins, Nicholas Johnson, Eric Jones, Neville Kidger, Joseph Kosmo, Alexei Lebedev, Mark Lee, James LeBlanc, Dmitri Leshchenskii, Jerry Linenger, Igor Lissov, James McBarron, Clay McCullough, Joseph McMann, Story Musgrave, Dennis Newkirk, James Oberg, Joel Powell, Lee Saegesser, Andy Salmon, Glen Swanson, Joseph Tatarewicz, Kathy Thornton, Chris Vandenberg, Charles Vick, Bert Vis, David Woods, Mike Wright, John Young, and Keith Zimmerman. Special thanks to Laurie Buchanan, John Charles, Janet Kovacevich, Joseph Loftus, Sue McDonald, Martha Munies, Colleen Rapp, and Jerry Ross. Any errors remain the responsibility of the authors.

Portree, David S. F.↗
NASA NTRS2016

Finding Stuff Underwater: Open-Source Tools for Underwater Field Science

Since 2009, we have been developing and testing Exploration Ground Data Systems (xGDS). xGDS is a software suite, built on open-source web technologies, for supporting NASA terrestrial field science analog missions. We will describe our recent work supporting underwater mapping and dive operations at the Pavilion Lake Research Project (PLRP - www.pavilionlake.com). At PLRP we used open-source mapping technologies to meet an operational need to quickly guide divers and underwater vehicles to locations of interest, and the scientific goal of building a map showing the locations and characteristics of the microbialites at the bottom of the lake. We will discuss some of the benefits and lessons-learned from our recent transition from Google Earth to OpenLayers as the primary mapping engine to support our work. Well also explain the challenges we faced and solutions weve developed to support mapping and data collection in an underwater environment.Beyond their immediate value for advancing scientific research on Earth, NASA analogs explore new ideas for supporting scientists working in extreme environments (e.g. underwater) to develop best practices for future human missions to the Moon or Mars. Science at PLRP involved imaging, sampling and characterizing microbialites growing beneath the surface of an alpine lake in British Columbia. To support PLRP operations and science, xGDS helped plan data collection tasks and provided accurate tracking and mapping of the locations of the underwater assets (human-piloted submersibles, remotely operated vehicles (ROVs) and human divers) collecting imagery and samples from the lake. The data collected during our field work was registered with precursor bathymetry data collected by an autonomous underwater vehicle (AUV) and was globally located on a satellite base layer map of the area surrounding the lake.

Earth analogs↗
NASA NTRS2013

Improved CPAS Photogrammetric Capabilities for Engineering Development Unit (EDU) Testing

This paper focuses on two key improvements to the photogrammetric analysis capabilities of the Capsule Parachute Assembly System (CPAS) for the Orion vehicle. The Engineering Development Unit (EDU) system deploys Drogue and Pilot parachutes via mortar, where an important metric is the muzzle velocity. This can be estimated using a high speed camera pointed along the mortar trajectory. The distance to the camera is computed from the apparent size of features of known dimension. This method was validated with a ground test and compares favorably with simulations. The second major photogrammetric product is measuring the geometry of the Main parachute cluster during steady-state descent using onboard cameras. This is challenging as the current test vehicles are suspended by a single-point attachment unlike earlier stable platforms suspended under a confluence fitting. The mathematical modeling of fly-out angles and projected areas has undergone significant revision. As the test program continues, several lessons were learned about optimizing the camera usage, installation, and settings to obtain the highest quality imagery possible.

Ray, Eric S.↗
NASA NTRS2019

NASA KSC Intern Final Report - Virtual Reality in STEM Engagement

For this internship, I was a part of the multi-center NextGen STEM (Science, Technology, Engineering, and Mathematics) pilot. Specifically, I worked with the Developing Commercial Crew Program (CCP) Capabilities team that focuses on human spaceflight to the space station with NASA’s commercial partners. Composed of NASA employees and contractors (who are connected to KSC, JSC, and LaRC), they work to ensure that students will experience traditional classroom content and be immersed in emerging technologies in order to explore Commercial Crew missions and launch facilities. This group is in the process of creating educational products and resources to inspire the next generation to pursue STEM and to allow educators to have access to unique and enticing STEM activities. These products include: age-appropriate classroom lessons (including activity sheets and challenges) for kindergarten through twelfth grade; a CCP app and corresponding guide; learning experiences for students as well as educators; and virtual field trips and tours using NASA-created Virtual Reality (VR) videos and a custom VR app. My particular assignment was managing the VR equipment, developing an understanding of how to maximize the use of the equipment, creating documents that explain and convey the operations and procedures concerning the equipment (including lessons learned from equipment use), and training others to utilize the equipment effectively and correctly for conference events and group demonstrations. Over the course of this internship, I helped my team outline and learn essential procedures of operation for the VR equipment, solved technology problems that arose (due to the nature of creating unique products), and encouraged audiences at KSC and various conferences to consider using the CCP VR videos in their classrooms as an opportunity to engage and challenge the next generation to pursue STEM careers.

STEM↗
NASA NTRS1984

Predictions of Cockpit Simulator Experimental Outcome Using System Models

This study involved predicting the outcome of a cockpit simulator experiment where pilots used cockpit displays of traffic information (CDTI) to establish and maintain in-trail spacing behind a lead aircraft during approach. The experiments were run on the NASA Ames Research Center multicab cockpit simulator facility. Prior to the experiments, a mathematical model of the pilot/aircraft/CDTI flight system was developed which included relative in-trail and vertical dynamics between aircraft in the approach string. This model was used to construct a digital simulation of the string dynamics including response to initial position errors. The model was then used to predict the outcome of the in-trail following cockpit simulator experiments. Outcome included performance and sensitivity to different separation criteria. The experimental results were then used to evaluate the model and its prediction accuracy. Lessons learned in this modeling and prediction study are noted.

Sorensen, J. A.↗
NASA NTRS2002

Runway Safety Monitor Algorithm for Runway Incursion Detection and Alerting

The Runway Safety Monitor (RSM) is an algorithm for runway incursion detection and alerting that was developed in support of NASA's Runway Incursion Prevention System (RIPS) research conducted under the NASA Aviation Safety Program's Synthetic Vision System element. The RSM algorithm provides pilots with enhanced situational awareness and warnings of runway incursions in sufficient time to take evasive action and avoid accidents during landings, takeoffs, or taxiing on the runway. The RSM currently runs as a component of the NASA Integrated Display System, an experimental avionics software system for terminal area and surface operations. However, the RSM algorithm can be implemented as a separate program to run on any aircraft with traffic data link capability. The report documents the RSM software and describes in detail how RSM performs runway incursion detection and alerting functions for NASA RIPS. The report also describes the RIPS flight tests conducted at the Dallas-Ft Worth International Airport (DFW) during September and October of 2000, and the RSM performance results and lessons learned from those flight tests.

Green, David F., Jr.↗
NASA NTRS1988

Autonomous control systems - Architecture and fundamental issues

A hierarchical functional autonomous controller architecture is introduced. In particular, the architecture for the control of future space vehicles is described in detail; it is designed to ensure the autonomous operation of the control system and it allows interaction with the pilot and crew/ground station, and the systems on board the autonomous vehicle. The fundamental issues in autonomous control system modeling and analysis are discussed. It is proposed to utilize a hybrid approach to modeling and analysis of autonomous systems. This will incorporate conventional control methods based on differential equations and techniques for the analysis of systems described with a symbolic formalism. In this way, the theory of conventional control can be fully utilized. It is stressed that autonomy is the design requirement and intelligent control methods appear at present, to offer some of the necessary tools to achieve autonomy. A conventional approach may evolve and replace some or all of the `intelligent' functions. It is shown that in addition to conventional controllers, the autonomous control system incorporates planning, learning, and FDI (fault detection and identification).

Antsaklis, P. J.↗
NASA NTRS2024

Route-Recapturing State-Based Horizontal Maneuver Strategy for Automated Detect-and-Avoid

This report describes a novel approach to the development of a horizontal maneuver guidance strategy for Detect-and-Avoid systems. The maneuver guidance strategy provides a directive turn action that can be automatically executed by the vehicle’s auto-pilot system, taking into account the cost of recapturing the flight plan path. Pairwise conflict scenarios with non-accelerating intruders are simulated to validate the effectiveness of the maneuver guidance strategy. Initial results suggest the strategy is more effective for faster ownship than for slower ownship, which is unable to avoid conflict in certain scenarios against fast intruders. These findings indicate this novel approach shows great potential, but improvement to its performance is necessary and will be future work.

detect-and-avoid↗
NASA NTRS2003

Virtual Reality Training: "Cybersickness" and Effects on Sensorimotor Functions

The overall goal of this study is to examine the extent to which exposure to virtual reality (VR) systems produces motion sickness and disrupts sensorimotor functions. Two of the major problems in using VRs are: 1) potential "cybersickness", a form of motion sickness, and 2) maladaptive sensorimotor coordination following virtual environment (VE) training. It is likely that users will eventually adapt to any unpleasant perceptual experiences in a virtual environment. However the most critical problem for training applications is that sensorimotor coordination strategies learned in the VE may not be similar to the responses required in the real environment. This study will evaluate and compare responses to the two types of VR delivery systems (head-mounted display [HMD] and a dome-projection system [DOME]), two exposure duration periods (30 minutes or 60 minutes), and repeated exposures (3 sessions). Specific responses that we will examine include cybersickness severity and symptom patterns, and several sensorimotor functions (eye-hea.d and eye-head-hand coordination, and postural equilibrium). To date, all hardware and software acquisition, development, integration and testing has been completed. A database has been developed and tested for the input, management and storage of all questionnaire data. All data analysis scripts have been developed and tested. Data was collected from 20 subjects in a pilot study that was conducted to determine the amount of training necessary to achieve a stable performance level. Seven subjects are currently enrolled in the study designed to examine the effects of exposure to VE systems on postural control. Data has been collected from two subjects, and it is expected that the results from ten subjects will be presented.

Harm, Deborah L.↗
NASA NTRS1998

Learning from Automation Surprises and "Going Sour" Accidents: Progress on Human-Centered Automation

Advances in technology and new levels of automation on commercial jet transports has had many effects. There have been positive effects from both an economic and a safety point of view. The technology changes on the flight deck also have had reverberating effects on many other aspects of the aviation system and different aspects of human performance. Operational experience, research investigations, incidents, and occasionally accidents have shown that new and sometimes surprising problems have arisen as well. What are these problems with cockpit automation, and what should we learn from them? Do they represent over-automation or human error? Or instead perhaps there is a third possibility - they represent coordination breakdowns between operators and the automation? Are the problems just a series of small independent glitches revealed by specific accidents or near misses? Do these glitches represent a few small areas where there are cracks to be patched in what is otherwise a record of outstanding designs and systems? Or do these problems provide us with evidence about deeper factors that we need to address if we are to maintain and improve aviation safety in a changing world? How do the reverberations of technology change on the flight deck provide insight into generic issues about developing human-centered technologies and systems (Winograd and Woods, 1997)? Based on a series of investigations of pilot interaction with cockpit automation (Sarter and Woods, 1992; 1994; 1995; 1997a, 1997 b), supplemented by surveys, operational experience and incident data from other studies (e.g., Degani et al., 1995; Eldredge et al., 1991; Tenney et al., 1995; Wiener, 1989), we too have found that the problems that surround crew interaction with automation are more than a series of individual glitches. These difficulties are symptoms that indicate deeper patterns and phenomena concerning human-machine cooperation and paths towards disaster. In addition, we find the same kinds of patterns behind results from studies of physician interaction with computer-based systems in critical care medicine (e.g., Moll van Charante et al., 1993; Obradovich and Woods, 1996; Cook and Woods, 1996). Many of the results and implications of this kind of research are synthesized and discussed in two comprehensive volumes, Billings (1996) and Woods et al. (1994). This paper summarizes the pattern that has emerged from our research, related research, incident reports, and accident investigations. It uses this new understanding of why problems arise to point to new investment strategies that can help us deal with the perceived "human error" problem, make automation more of a team player, and maintain and improve safety.

Woods, David D.↗
NASA NTRS2018

Human Automation Teaming: Lessons Learned and Future Directions

Full autonomy seems to be the goal for system developers in almost every area of the economy. However, as we move from automated systems to autonomous systems, designers have needed to insert humans to oversee automation that has traditionally been brittle or incomplete. This creates its own problems as the operator is usually out of the loop when the automation hands over problems that it cannot handle. To better handle these situations, it has been proposed that we develop human automation teams that have shared goals and objectives to support task performance. This paper will describe an initial model of Human Automation Teaming (HAT) which has three elements: transparency, bi-directional communications, and human-directed execution. Transparency in our model is a method for giving insight into the reasoning behind automated recommendations and actions, bi-directional communication allows the operator to communicate directly with the automation, and finally the automation defers execution to the human. The model was implemented through a number of features on an electronic flight bag (EFB) which are described in the paper. The EFB was installed in a mid-fidelity flight simulator and used by 12 airline pilots to support diversion decisions during off-nominal flight scenarios. Pilots reported that working with the HAT automation made diversion decisions easier and reduced their workload. They also reported that the information provided about diversion airports was similar to what they would receive from ground dispatch, thus making coordination with dispatch easier and less time consuming. These HAT features engender more trust in the automation when appropriate, and less when not, allowing improved supervision of automated functions by flight crews.

Human-Autonomy Teaming↗
NASA NTRS2005

Aircraft Fault Detection and Classification Using Multi-Level Immune Learning Detection

This work is an extension of a recently developed software tool called MILD (Multi-level Immune Learning Detection), which implements a negative selection algorithm for anomaly and fault detection that is inspired by the human immune system. The immunity-based approach can detect a broad spectrum of known and unforeseen faults. We extend MILD by applying a neural network classifier to identify the pattern of fault detectors that are activated during fault detection. Consequently, MILD now performs fault detection and identification of the system under investigation. This paper describes the application of MILD to detect and classify faults of a generic transport aircraft augmented with an intelligent flight controller. The intelligent control architecture is designed to accommodate faults without the need to explicitly identify them. Adding knowledge about the existence and type of a fault will improve the handling qualities of a degraded aircraft and impact tactical and strategic maneuvering decisions. In addition, providing fault information to the pilot is important for maintaining situational awareness so that he can avoid performing an action that might lead to unexpected behavior - e.g., an action that exceeds the remaining control authority of the damaged aircraft. We discuss the detection and classification results of simulated failures of the aircraft's control system and show that MILD is effective at determining the problem with low false alarm and misclassification rates.

Wong, Derek↗
NASA NTRS2018

Impact of Advanced Synoptics and Simplified Checklists During Aircraft Systems Failures

Abstract—Natural human capacities are becoming increasingly mismatched to the enormous data volumes, processing capabilities, and decision speeds demanded in today’s aviation environment. Increasingly Autonomous Systems (IAS) are uniquely suited to solve this problem. NASA is conducting research and development of IAS - hardware and software systems, utilizing machine learning algorithms, seamlessly integrated with humans whereby task performance of the combined system is significantly greater than the individual components. IAS offer the potential for significantly improved levels of performance and safety that are superior to either human or automation alone. A human-in-the-loop test was conducted in NASA Langley’s Integration Flight Deck B-737-800 simulator to evaluate advanced synoptic pages with simplified interactive electronic checklists as an IAS for routine air carrier flight operations and in response to aircraft system failures. Twelve U.S. airline crews flew various normal and non-normal procedures and their actions and performance were recorded in response to failures. These data are fundamental to and critical for the design and development of future increasingly autonomous systems that can better support the human in the cockpit. Synoptic pages and electronic checklists significantly improved pilot responses to non-normal scenarios, but implementation of these aids and other intelligent assistants have barriers to implementation (e.g., certification cost) that must overcome.

Etherington, Timothy J.↗
NASA NTRSDate not supplied

A Survey Protocol to Assess Meaningfulness and Usefulness of Automated Topic Finding in the NASA Aviation Safety Reporting System

Context: The NASA Aviation Safety Reporting System (ASRS) is a voluntary confidential aviation safety reporting system. The ASRS receives reports from pilots, air traffic controllers, flight attendants and other involved in aviation operations. The reports are de-identified and coded by ASRS expert safety analysts and a short descriptive synopsis is written to describe the safety issue. The de-identified reports are then disseminated to the aviation community in a number of ways including entry into an online database, Safety Alert Bulletins and For Your Information Notices, and the CALLBACK newsletter. Key to these publications are the timely processing (de-identification, coding and summarization) of new reports, which is currently done by ASRS expert safety analysts. Thus, we believe topic modelling could decrease effort in ASRS, if topics are comprehensible. Aim: We propose a methodology to evaluate whether automated topic finding using topic modelling provides meaningful and useful topics. Method: We extend the total error survey methodology to evaluate user topic comprehension of machine learning outputs. To accomplish this we performed a literature review to identify existing methods and define a construct for topic comprehension, utilizing existing ASRS synopsis writing practices to more precisely define meaningfulness and usefulness. Results: A survey protocol was created that addresses the limitations of other survey protocols found in the literature review, which we found lacking in rationale and clear protocol definition. Conclusion: The surveying of user understanding in machine learning outputs presents challenges due to the explosion of parameters to control for and the lack of systematic approach presented in the literature. More reproducible work and survey protocols are needed in the literature and our work is one step towards that direction.

topic finding↗