Engineering Papers⌕ Search

Engineering topics

Carlos Paradis

Publications and source records attributed to Carlos Paradis.

At least 19 records
All sourcesNASA NTRSDOE OSTI
NASA NTRS2023

Towards Streamlining Auditing for Compliance With Requirements in Open-Source Software at NASA

Context: NASA requires all software to meet several requirements (NPR 7150.2) depending on software criticality. The instantiation of these requirements may vary per project; however, once decided upon, projects must undergo audits to evaluate compliance with these requirements. Aim: We propose that audit effort can be reduced when requirements are realized by leveraging commonly used open-source infrastructure for version control, issue tracking and continuous integration, and the generated records are analyzed using a repository mining software tool to quantify process compliance. Method: We perform a case study in the NASA-funded Copilot project, utilizing Kaiaulu, a repository mining software tool. We define four software compliance metrics based on the Copilot’s requirements, and analyze their impact on source code quality. Results: Our work demonstrates how it is possible to leverage existing open source tools and platforms to facilitate software certification and qualification, and to streamline the auditing process required even when stringent requirements must be enforced. Conclusion: Together, both project and tool can be utilized to visualize project compliance, and metrics can be defined to more easily identify process irregularities to minimize auditing efforts. Project Repository: github.com/Copilot-Language/copilot Tool Repository: github.com/sailuh/kaiaulu

code-quality↗
NASA NTRSDate not supplied

Augmenting Topic Finding in the NASA Aviation Safety Reporting System using Topic Modeling

Context: The NASA Aviation and Safety Reporting System (ASRS) provides various publications to the aviation community (including individual anonymous reports, Callback, Database Search Requests, Directline, and Alerting Messages). Key to these publications are the timely processing of new reports, which is currently done mostly manually by ASRS staff, and which the volume increases yearly. Aim: We investigate whether existing topic modelling techniques are suitable to ease some of the manual effort, and to enhance it with additional visual cues regarding the process of grouping, sense making and labeling incoming (and previous) reports. Method: We evaluate the applicability of WarpLDA topic modelling results combined with three visualization tools, the first two of which have been extended by us in this work for ASRS: Termite, TopicFlow, and LDAVis. Based on the identified limitations in these tools, we propose a methodology for improving them, and evaluate their outputs using ASRS as our test dataset. Results: The user interfaces of Termite, Topicflow and LDAVis were found insufficient for sense-making of the narratives. Moreover, concerns regarding the stability of results due to the inherent randomness of topic modelling, and the lack of a measurable approach for evaluation against the existing ASRS manual workflow were also noted. Conclusion: While many tools to topic modeling and visualization have been proposed, more work is necessary before they can be applied in practical situations to improve existing manual workflows. The methodology presented and applied in this work contribute towards this effort.

ASRS↗
NASA NTRSDate not supplied

Identifying Emerging Safety Threats Through Topic Modeling in the Aviation Safety Reporting System: A Covid-19 Study

The NASA Aviation Safety Reporting System (ASRS) is a voluntary, confidential aviation safety reporting system. The ASRS receives reports from pilots, air traffic controllers, flight attendants, and others involved in aviation operations. The reports are de-identified and coded by ASRS expert safety analysts, and a short descriptive synopsis is written to describe the safety issue. The de-identified reports are then disseminated to the aviation community in many ways, including via an online database, Safety Alert Bulletins, For Your Information Notices, and the CALLBACK newsletter. In this work, we consider whether we can improve the grouping, linking, and understanding of safety concerns through topic modeling. Specifically, we use topic modeling as a building block to identify emerging safety threats over time. This unsupervised approach, we argue, offers the flexibility to identify new emerging themes in this large dataset by constructing different timelines based on the content similarity of ASRS report narratives. This method's unsupervised nature improves upon related research, which is limited to pre-defined labels and therefore can not fully capture emerging safety threats. We apply our method to all ASRS reports in 2020 to assess if the generated timelines can highlight COVID-19 as it is emerging as a safety threat in incoming ASRS reports. We perform both a quantitative and qualitative evaluation of the automatically constructed timelines. The qualitative evaluation is performed by describing the evolution of top terms in the timelines, generated by our method, which we found explicitly convey the themes of COVID-19. Separately, we use a set of 1,213 COVID-19 reports from 2020 that were manually identified by ASRS analysts to quantitatively evaluate the COVID-19 reports distribution across the timelines. Our results have shown that COVID-19 emergence can be identified using the top terms that were generated by topic modeling. The top terms in topic modeling therefore can serve as a summary alternative to manually inspecting reports. Moreover, leveraging the manually identified COVID-19 reports, we found the manually identified timelines accounted for over 70% of the COVID-19 reports curated by the ASRS analysts, which demonstrates the potential of this approach for facilitating the understanding of safety concerns as they emerge and evolve. This method shows great potential to understand aerospace safety threats and other narrative- driven incident report databases.

ASRS↗
NASA NTRSDate not supplied

A Survey Protocol to Assess Meaningfulness and Usefulness of Automated Topic Finding in the NASA Aviation Safety Reporting System

Context: The NASA Aviation Safety Reporting System (ASRS) is a voluntary confidential aviation safety reporting system. The ASRS receives reports from pilots, air traffic controllers, flight attendants and other involved in aviation operations. The reports are de-identified and coded by ASRS expert safety analysts and a short descriptive synopsis is written to describe the safety issue. The de-identified reports are then disseminated to the aviation community in a number of ways including entry into an online database, Safety Alert Bulletins and For Your Information Notices, and the CALLBACK newsletter. Key to these publications are the timely processing (de-identification, coding and summarization) of new reports, which is currently done by ASRS expert safety analysts. Thus, we believe topic modelling could decrease effort in ASRS, if topics are comprehensible. Aim: We propose a methodology to evaluate whether automated topic finding using topic modelling provides meaningful and useful topics. Method: We extend the total error survey methodology to evaluate user topic comprehension of machine learning outputs. To accomplish this we performed a literature review to identify existing methods and define a construct for topic comprehension, utilizing existing ASRS synopsis writing practices to more precisely define meaningfulness and usefulness. Results: A survey protocol was created that addresses the limitations of other survey protocols found in the literature review, which we found lacking in rationale and clear protocol definition. Conclusion: The surveying of user understanding in machine learning outputs presents challenges due to the explosion of parameters to control for and the lack of systematic approach presented in the literature. More reproducible work and survey protocols are needed in the literature and our work is one step towards that direction.

topic finding↗
NASA NTRSDate not supplied

Identifying Emerging Safety Threats Through Topic Modeling in the Aviation Safety Reporting System: A COVID-19 Study

The NASA Aviation Safety Reporting System (ASRS) is a voluntary, confidential aviation safety reporting system. The ASRS receives reports from pilots, air traffic controllers, flight attendants, and others involved in aviation operations. The reports are de-identified and coded by ASRS expert safety analysts, and a short descriptive synopsis is written to describe the safety issue. The de-identified reports are then disseminated to the aviation community in many ways, including via an online database, Safety Alert Bulletins, For Your Information Notices, and the CALLBACK newsletter. In this work, we consider whether we can improve the grouping, linking, and understanding of safety concerns through topic modeling. Specifically, we use topic modeling as a building block to identify emerging safety threats over time. This unsupervised approach, we argue, offers the flexibility to identify new emerging themes in this large dataset by constructing different timelines based on the content similarity of ASRS report narratives. This method's unsupervised nature improves upon related research, which is limited to pre-defined labels and therefore can not fully capture emerging safety threats. We apply our method to all ASRS reports in 2020 to assess if the generated timelines can highlight COVID-19 as it is emerging as a safety threat in incoming ASRS reports. We perform both a quantitative and qualitative evaluation of the automatically constructed timelines. The qualitative evaluation is performed by describing the evolution of top terms in the timelines, generated by our method, which we found explicitly convey the themes of COVID-19. Separately, we use a set of 1,213 COVID-19 reports from 2020 that were manually identified by ASRS analysts to quantitatively evaluate the COVID-19 reports distribution across the timelines. Our results have shown that COVID-19 emergence can be identified using the top terms that were generated by topic modeling. The top terms in topic modeling therefore can serve as a summary alternative to manually inspecting reports. Moreover, leveraging the manually identified COVID-19 reports, we found the manually identified timelines accounted for over 70% of the COVID-19 reports curated by the ASRS analysts, which demonstrates the potential of this approach for facilitating the understanding of safety concerns as they emerge and evolve. This method shows great potential to understand aerospace safety threats and other narrative- driven incident report databases.

ASRS↗
NASA NTRSDate not supplied

Assessing the use of UAS-related terms in ASRS using Seed Topic Modeling

Context: The NASA Aviation Safety Reporting System (ASRS) is a voluntary confidential aviation safety reporting system. The ASRS receives reports from pilots, air traffic controllers, flight attendants and other involved in aviation operations. The reports are de-identified and coded by ASRS expert safety analysts. The de-identified reports are then disseminated to the aviation community in a number of ways including entry into an online database. Augmenting the discovery of topics of user interest in this online database would therefore be beneficial to the community it serves. Aim: We propose and execute an experiment to assess the use of seed term topic modeling using the database narratives to identify UAS reports. The use of seed term topic modeling would enable users to identify groups of related narratives associated to a topic of their interest. Method: We use a newly curated field in ASRS reports which identify UAS from non-UAS reports in combination of different set of UAS related terms to assess if seed topic modeling can be used in ASRS.

LDA↗
NASA NTRSDate not supplied

Visualizing Corridors in Terminal Airspace using Trajectory Clustering

Context: Advances in battery and automation technology have made routine air taxi and cargo transport in urban areas a business model that can be attained by emerging aviation innovators. The community vision and work to enable these novel operations is discussed using the term ‘Urban Air Mobility’ or UAM. Small, piloted, airspace vehicles that fly with a few passengers do operate in urban areas today, and these vehicles can be studied as an early proxy for this future UAM traffic. Aim: We seek to identify corridors already in daily operation and their properties. Method: We applied DBSCAN and HDBSCAN to Dallas Forth-Worth TRACON flight data to identify corridors in use, their density, and devised a method to annotate landing sites used in these corridors with site metadata. Results: While DBSCAN was unable to group similar trajectories, we we were able to successfully identify corridors using HDBSCAN, measure their density and annotate them. Conclusion: The applied method can successfully identify corridors in daily operation with additional metadata to help domain expert understand the intent of UAM corridors.

UAM, Trajectory, TRACON, Clustering, DBSCAN, HDBSC↗
NASA NTRSDate not supplied

Open Data Integration (ODIN): A Concurrent, Distributed Message-Based Architecture and Framework for Disaster Response

The Runtime for Airspace Concept Evaluation (RACE) is an open-source software architecture and framework to build configurable, highly concurrent and distributed message-based systems that offer scalable, low-latency performance on commodity hardware. RACE was used in commercial aviation applications to rapidly build systems that span several machines (including synchronized displays), interface existing hardware simulators and other live data feeds, and incorporate sophisticated visualization components such as NASA WorldWind. These RACE applications validated elements of the FAA’s System Wide Information Management (SWIM) Program, handling up to 1000 messages/sec from diverse sources (SFDPS, TFM-DATA, TAIS, ASDE-X, ITWS and local ADS) for 4,500 simultaneous flights tracked in the next-generation air transportation system’s digital backbone. We have since generalized RACE to support Open Data Integration (ODIN) applications outside aviation. Systems built with RACE/ODIN can be deployed in the field, on commodity hardware, and operate with limited or intermittent connectivity to the outside world. Our primary use case is a web-server with local/persistent data storage that runs within and only serves the stakeholder network (e.g. an incident command post). We are tailoring the RACE/ODIN system to support wildland fire management for the upcoming NASA Wildland Fire Safety Demonstration Series. RACE-ODIN is under consideration for application in the Scalable Traffic Management for Emergency Response Operations project, or STEReO, which aims to create a system that can be deployed during emergencies, to coordinate multiple elements of disaster response. Such data sources predominantly come from existing services on the internet (e.g. weather and satellite data, imported from so called "edge servers") but can also include dynamic (real-time) data from computer simulations and within the stakeholder network (such as aircraft and personnel tracking information). We will present the architecture and ODIN system demonstration incorporating local data from instrumented power-line towers, interpolated weather data and geospatial data from space-based platforms.

Joseph C Coughlan↗
NASA NTRSDate not supplied

Visualizing Corridors in Terminal Airspace Using Trajectory Clustering

Context: Advances in battery and automation technology have made routine air taxi and cargo transport in urban areas a business model that can be attained by emerging aviation innovators. The community vision and work to enable these novel operations is discussed using the term ‘Urban Air Mobility’ or UAM. Small, piloted, airspace vehicles that fly with a few passengers do operate in urban areas today, and these vehicles can be studied as an early proxy for this future UAM traffic. Aim: We seek to identify corridors already in daily operation and their properties. Method: We applied DBSCAN and HDBSCAN to Dallas Forth-Worth TRACON flight data to identify corridors in use, their density, and devised a method to annotate landing sites used in these corridors with site metadata. Results: While DBSCAN was unable to group similar trajectories, we we were able to successfully identify corridors using HDBSCAN, measure their density and annotate them. Conclusion: The applied method can successfully identify corridors in daily operation with additional metadata to help domain expert understand the intent of UAM corridors.

UAM Trajectory, TRACON, Clustering, DBSCAN, HDBSCA↗
NASA NTRSDate not supplied

Assessing the Use of UAS-Related Terms in ASRS Using Seed Topic Modeling

Context: The NASA Aviation Safety Reporting System (ASRS) is a voluntary confidential system that disseminates reports received from personnel involved in aviation operations after de-identifying them. These reports are used by the community to improve overall aviation system safety. Aim: We propose and execute an experiment to assess the use of seed term topic modeling over the database narratives to identify Unmanned Aircraft System (UAS) reports. The use of seed term topic modeling enables users to identify groups of conceptually similar narratives associated to a topic of their interest. Method: We use a collection of narratives, expert-selected words, and report metadata that separates UAS from non-UAS reports to assess if seed topic modeling can be used to improve ASRS searches. Results: For simpler queries, seed topic search observes a higher recall and lower precision than the existing DBOL (DataBase OnLine) search in operation. However, the best results are obtained when seed topic search is used as a search suggestion system to be executed on the DBOL. Conclusion: Utilizing a combination of both the existing method and the proposed method, users can expand their search vocabulary about subjects of interest while improving the quality of results.

Text Mining↗
NASA NTRSDate not supplied

Assessing the Use of UAS-Related Terms in ASRS using Seeds for Topic Modeling

Context: The NASA Aviation Safety Reporting System (ASRS) is a voluntary confidential system that disseminates reports received from personnel involved in aviation operations after de-identifying them. These reports are used by the community to improve overall aviation system safety. Aim: We propose and execute an experiment to assess the use of seed term topic modeling over the database narratives to identify Unmanned Aircraft System (UAS) reports. The use of seed term topic modeling enables users to identify groups of conceptually similar narratives associated to a topic of their interest. Method: We use a collection of narratives, expert-selected words, and report metadata that separates UAS from non-UAS reports to assess if seed topic modeling can be used to improve ASRS searches. Results: For simpler queries, seed topic search observes a higher recall and lower precision than the existing DBOL (DataBase OnLine) search in operation. However, the best results are obtained when seed topic search is used as a search suggestion system to be executed on the DBOL. Conclusion: Utilizing a combination of both the existing method and the proposed method, users can expand their search vocabulary about subjects of interest while improving the quality of results.

LDA↗
NASA NTRSDate not supplied

Textual and Network Analysis of Part 107 Waivers

Context: The management of hazards in sUAS operations is not as well defined as today's commercial operations despite sUAS widespread use. Part 107 waived operations' provisions, which manage hazards for higher risk operations that require approval, can offer insight to organizations establishing UAS Programs in managing their own operation hazards. Aim: We seek to understand how the FAA Part 107 waived operations manage hazards. Method: We used the constant comparative method to identify hazard mitigation textual categories from provisions and use networks to assess the dispersion of provisions and the identified categories across issued waivers. Results: Eight mitigation categories and twenty-four sub-categories were identified. Most provisions present in waivers are mostly reused in one waiver. Conclusion: While there is a broad range of provisions to control for hazard mitigations in the Part 107 issued waivers analyzed regulations, they require case-by-case modifications.

Urban Air Mobility↗
NASA NTRSDate not supplied

A Grounded Theory of UAS Reported Accidents

Context: The manufacture and operation of sUAS are not as regulated as today’s commercial operation, and their widespread use introduces new risks and hazards to the general public. Aim: Our intent is to understand the various processes that prevent or portend sUAS incidents or accidents and the influence that pilots’ expectations and training have on these processes. Method: We use classic grounded theory on sUAS reported accidents (and incidents). Results: We identified three categories, Control Interference, Reviewing and Reporting, which describe various processes surrounding UAS accidents based on the dataset analyzed. Conclusion: The use of grounded theory can offer a different perspective in the analysis of UAS accidents. In addition, the traceability between data sources and the method results facilitate result validation, and navigation. While the results are preliminary, the concepts can be expanded through the use of other accident datasets or used as basis for UAS accident surveys.

Carlos Paradis↗
NASA NTRSDate not supplied

Textual and Network Analysis of Title 14 CFR Part107 Waivers

Context: The management of hazards in small unmanned aircraft systems (sUAS) operations is not as well defined as today's commercial operations despite sUAS widespread use. FAA Title 14 Code of Federal Regulations (CFR) Part 107 waived operations' provisions, which manage hazards for higher risk operations that require approval, can offer insight to organizations establishing UAS Programs in managing their own operation hazards. Aim: We seek to understand how the Title 14 CFR Part 107 waived operations manage hazards. Method: We used the constant comparative method to identify hazard mitigation textual categories from provisions and use networks to assess the dispersion of provisions and the identified categories across issued waivers. Results: Eight mitigation categories and twenty-four sub-categories were identified. Most provisions present in waivers are mostly reused in one waiver. Conclusion: While there is a broad range of provisions to control for hazard mitigations in the Title 14 CFR Part 107 issued waivers analyzed regulations, they require case-by-case modifications.

Urban Air Mobility↗
NASA NTRSDate not supplied

A Grounded Theory of UAS Reported Accidents

Context: The manufacture and operation of sUAS are not as regulated as today’s commercial operation, and their widespread use introduces new risks and hazards to the general public. Aim: Our intent is to understand the various processes that prevent or portend sUAS incidents or accidents and the influence that pilots’ expectations and training have on these processes. Method: We use classic grounded theory on sUAS reported accidents (and incidents). Results: We identified three categories, Control Interference, Reviewing and Reporting, which describe various processes surrounding UAS accidents based on the dataset analyzed. Conclusion: The use of grounded theory can offer a different perspective in the analysis of UAS accidents. In addition, the traceability between data sources and the method results facilitate result validation, and navigation. While the results are preliminary, the concepts can be expanded through the use of other accident datasets or used as basis for UAS accident surveys.

Carlos Paradis↗