Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “Trustworthiness”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 37 records · Page 2
All sourcesNASA NTRSDOE OSTI
NASA NTRS1993

The Trustworthy Digital Camera: Restoring Credibility to the Photographic Image

The increasing sophistication of computers has made digital manipulation of photographic images (as well as other digitally-recorded artifacts, such as sound and video) incredibly easy to perform and, as time goes on, increasingly difficult to detect. Today, every picture appearing in newspapers and magazines has been digitally altered to some degree, with the severity varying from the trivial (cleaning up "noise" and removing distracting backgrounds) to the point of deception (articles of clothing removed, heads attached to other people's bodies, the complete rearrangement of city skylines). As the power, flexibility and ubiquity of image-altering computers continues to increase, the well-known adage that "the photograph doesn't lie" will continue to become an anachronism. A solution to this problem comes from the proposed Digital Signature Standard (DSS), which incorporates modern cryptographic techniques to authenticate electronic mail messages...

Friedman, Gary L.↗
NASA NTRSDate not supplied

Trustworthy Machine Learning for Damage Identification in Composites

A challenging opportunity in structural health monitoring of composite materials is using machine learning (ML) methods to classify acoustic emissions (AE) according to the damage mechanism that emitted the signal. Although a wide variety of ML frameworks have been developed, there is a distinct lack of ground truth datasets which has precluded any direct assessment of their accuracy. Here, we present a novel ground truth dataset gathered on simplified unidirectional SiC/SiC composite structures. Herein, AE is collected from minicomposites which are loaded to targeted percentages of the ultimate tensile stress. These minicomposites are then volumetrically imaged with XCT and individual damage events, along with the mechanism, are correlated to AE. We explore the signal features that allow for mechanism discrimination, along with the feasibility of both unsupervised and supervised frameworks for use in the online monitoring of composite structures.

Machine learning, acoustic emission, ceramic matri↗
NASA NTRS2021

ATTRACTOR: Autonomy Teaming and TRAjectories for Complex Trusted Operational Reliability

Autonomous systems (AS) are crucial to realizing the vision of new, complex transportation modes, such as advanced air mobility (AAM) and urban air mobility (UAM). A chief barrier to induction of AS into aviation is insufficient understanding of AS reliability in time-critical and safety-critical environments—an obstacle to certification. ATTRACTOR is aimed at building a basis for certification of classes of autonomous cyber-physical-human systems (CPHS) via establishing metrics and models of trustworthiness and trust in multi-agent team interactions, analyzable trajectories, explainability of computational algorithms (explainable artificial intelligence, or XAI), and persistent modeling and simulation, in the context of missions planning and operation. By “building a basis for certification,” we mean acquiring an understanding of when a system is trustworthy and developing computable means to estimate trustworthiness and trust in order to eventually inform functional requirements that contribute to certification. The outcomes are applicable not just to aviation but to all domains that rely on autonomous systems.

Autonomy↗
NASA NTRS2017

NASA's EOSDIS, Trust and Certification

NASA's Earth Observing System Data and Information System (EOSDIS) has been in operation since August 1994, managing most of NASA's Earth science data from satellites, airborne sensors, filed campaigns and other activities. Having been designated by the Federal Government as a project responsible for production, archiving and distribution of these data through its Distributed Active Archive Centers (DAACs), the Earth Science Data and Information System Project (ESDIS) is responsible for EOSDIS, and is legally bound by the Office of Management and Budgets circular A-130, the Federal Records Act. It must follow the regulations of the National Institute of Standards and Technologies (NIST) and National Archive and Records Administration (NARA). It must also follow the NASA Procedural Requirement 7120.5 (NASA Space Flight Program and Project Management). All these ensure that the data centers managed by ESDIS are trustworthy from the point of view of efficient and effective operations as well as preservation of valuable data from NASA's missions. Additional factors contributing to this trust are an extensive set of internal and external reviews throughout the history of EOSDIS starting in the early 1990s. Many of these reviews have involved external groups of scientific and technological experts. Also, independent annual surveys of user satisfaction that measure and publish the American Customer Satisfaction Index (ACSI), where EOSDIS has scored consistently high marks since 2004, provide an additional measure of trustworthiness. In addition, through an effort initiated in 2012 at the request of NASA HQ, the ESDIS Project and 10 of 12 DAACs have been certified by the International Council for Science (ICSU) World Data System (WDS) and are members of the ICSUWDS. This presentation addresses questions such as pros and cons of the certification process, key outcomes and next steps regarding certification. Recently, the ICSUWDS and Data Seal of Approval (DSA) organizations merged their Core Trustworthy Data Repositories Requirements and require that members be recertified every three years. Given the rigor with which NASA manages the ESDIS Project and the DAACs, the recertification through WDSDSA, while involving some additional work, is a relatively simple process.

Certification↗
NASA NTRS2018

A Persistent Simulation Environment for Autonomous Systems

The age of Autonomous Unmanned Aircraft Systems (AUAS) is creating new challenges for the accreditation and certification requiring new standards, policies and procedures that sanction whether a UAS is safe to fly. Establishing a basis for certification of autonomous systems via research into trust and trustworthiness is the focus of Autonomy Teaming and TRAjectories for Complex Trusted Operational Reliability (ATTRACTOR), a new NASA Convergent Aeronautics Solution (CAS) project. Simulation Environments to test and evaluate AUAS decision making may be a low-cost solution to help certify that various AUAS systems are trustworthy enough to be allowed to fly in current general and commercial aviation airspace. NASA is working to build a peer-to-peer persistent simulation (P3 Sim) environment. The P3 Sim will be a Massively Multiplayer Online (MMO) environment were AUAS avatars can interact with a complex dynamic environment and each other. The focus of the effort is to provide AUAS researchers a low-cost intuitive testing environment that will aid training for and assessment of decisions made by autonomous systems such as AUAS. This presentation focuses on the design approach and challenges faced in development of the P3 Sim Environment is support of investigating trustworthiness of autonomous systems.

Kelley, Benjamin N.↗
NASA NTRSDate not supplied

A Distributed Simulation-to-Flight Framework to Support Investigating Trust/Trustworthiness in Multi-Agent Systems

As autonomous systems continue to grow both in use and complexity, the necessity for robust and extensible simulation-to-flight frameworks is paramount for establishing an effective architecture for autonomous systems. Hardware test flights are time-consuming and cost prohibitive during early system design and development. Simulation environments can be useful tools to accelerate algorithm development and testing. However, transitions from simulation to flight (sim-to-flight) can be challenging, unless systems are designed with this transition in mind and with the necessary capabilities built into the architecture and framework. One of the objectives of Autonomy Teaming and TRAjectories for Complex Trusted Operational Reliability (ATTRACTOR) was to design and develop a distributed mixed-reality simulation environment to begin establishing a basis for certification of autonomous systems via research into trust and trustworthiness. ATTRACTOR’s objective was to construct computational concepts of trustworthiness and justifiable trust in multi-agent autonomous teams, to inform future certification of safety-critical and time-critical autonomous systems in aviation. In this paper, we present an autonomous systems architecture and development framework paired with a persistent distributed modeling and simulation (ModSim) environment for test and evaluation of autonomous systems. They were designed under ATTRACTOR in order to measure and establish trustworthiness and trust in single-and multi-agent human-machine systems whether these machines are fixed-wing general aviation, rotary-wing Unmanned Aerial Vehicles (UAVs), ground rovers, or even spacecraft. The Autonomous Entity Operational Network (AEON) framework enables autonomous system development with an easily extensible collection of libraries and plug-n-play nodes facilitated by the Data Distribution Service (DDS) communication protocol standard. The Baseline Environment for Autonomous Modeling (BEAM) simulation environment is a distributed mixed-reality Unity™-based environment built around the same DDS communication paradigm allowing for easy integration with AEON-based autonomous applications, enabling sim-to-flight with minimal configuration changes. Using AEON and BEAM, source code that runs in simulation ports directly to hardware and has successfully flown in the National Airspace System (NAS) at NASA LaRC many times over the lifetime of ATTRACTOR.

Benjamin N Kelley↗
NASA NTRSDate not supplied

Uncertainty Quantification using Deep Ensembles for Decision Making in Cyber-Physical-Human Systems

In this paper and its companion, Differential Equation Approximation Using Gradient-Boosted Quantile Regression, Robison et al., we examine an approach to quantifying model uncertainty with the aim of increasing the trustworthiness of computational models in human-machine interactions. In Differential Equation Approximation Using Gradient-Boosted Quantile Regression, we focus on gradient-boosted decision trees, while in this one, we give more details about deep ensembles. Uncertainty quantification is crucial for building trustworthy autonomous decision-making agents in human-machine teams. There are two types of uncertainties: aleatoric and epistemic. The former is related to the inherent stochasticity (noise) of the process, whereas the latter is associated with the lack of knowledge or representation capability of models, such as neural networks. By lack of knowledge, we mean the model’s inability to accurately predict outputs for all possible inputs. The aleatory uncertainty can be estimated fairly easily with, for example, filters, whereas epistemic uncertainty is challenging to compute. This paper uses deep ensembles to quantify both aleatory and epistemic uncertainty. It can act as an uncertainty-aware surrogate transition model for decision-making frameworks. "Uncertainty-aware" means that the surrogate transition model should make predictions along with confidence in those predictions. In the context of decision-making, the transition models are ordinary differential equations (ODEs). Since ODEs can be simulated to make one-step or multi-step predictions, a good surrogate model for them should perform reasonably well in both modes. In a multi-step approach, the trajectory sampling method TS∞ was used to propagate uncertainty over multiple steps. The cartpole dynamical system was selected to demonstrate the ability of deep ensembles as good surrogate transition models for decision-making frameworks. The deep ensembles modeled the dynamics of cartpole ODEs and made uncertainty-aware predictions in single-step and multi-step transition modes.

CPH systems↗
NASA NTRS2011

Toward a Framework for Systematic Error Modeling of NASA Spaceborne Radar with NOAA/NSSL Ground Radar-Based National Mosaic QPE

Characterization of the error associated to satellite rainfall estimates is a necessary component of deterministic and probabilistic frameworks involving space-born passive and active microwave measurement") for applications ranging from water budget studies to forecasting natural hazards related to extreme rainfall events. We focus here on the error structure of NASA's Tropical Rainfall Measurement Mission (TRMM) Precipitation Radar (PR) quantitative precipitation estimation (QPE) at ground. The problem is addressed by comparison of PR QPEs with reference values derived from ground-based measurements using NOAA/NSSL ground radar-based National Mosaic and QPE system (NMQ/Q2). A preliminary investigation of this subject has been carried out at the PR estimation scale (instantaneous and 5 km) using a three-month data sample in the southern part of US. The primary contribution of this study is the presentation of the detailed steps required to derive trustworthy reference rainfall dataset from Q2 at the PR pixel resolution. It relics on a bias correction and a radar quality index, both of which provide a basis to filter out the less trustworthy Q2 values. Several aspects of PR errors arc revealed and quantified including sensitivity to the processing steps with the reference rainfall, comparisons of rainfall detectability and rainfall rate distributions, spatial representativeness of error, and separation of systematic biases and random errors. The methodology and framework developed herein applies more generally to rainfall rate estimates from other sensors onboard low-earth orbiting satellites such as microwave imagers and dual-wavelength radars such as with the Global Precipitation Measurement (GPM) mission.

Kirstettier, Pierre-Emmanual↗
NASA NTRS2018

Towards Explainability of UAV-Based Convolutional Neural Networks for Object Classification

f autonomous systems using trust and trustworthiness is the focus of Autonomy Teaming and TRAjectories for Complex Trusted Operational Reliability (ATTRACTOR), a new NASA Convergent Aeronautical Solutions (CAS) Project. One critical research element of ATTRACTOR is explainability of the decision-making across relevant subsystems of an autonomous system. The ability to explain why an autonomous system makes a decision is needed to establish a basis of trustworthiness to safely complete a mission. Convolutional Neural Networks (CNNs) are popular visual object classifiers that have achieved high levels of classification performances without clear insight into the mechanisms of the internal layers and features. To explore the explainability of the internal components of CNNs, we reviewed three feature visualization methods in a layer-by-layer approach using aviation related images as inputs. Our approach to this is to analyze the key components of a classification event in order to generate component labels for features of the classified image at different layers of depths. For example, an airplane has wings, engines, and landing gear. These could possibly be identified somewhere in the hidden layers from the classification and these descriptive labels could be provided to a human or machine teammate while conducting a shared mission and to engender trust. Each descriptive feature may also be decomposed to a combination of primitives such as shapes and lines. We expect that knowing the combination of shapes and parts that create a classification will enable trust in the system and insight into creating better structures for the CNN.

Dolph, Chester V.↗
NASA NTRSDate not supplied

Analyzing Natural Language Context in Human-Machine Teaming using Supervised Machine Learning

Building a foundation for trustworthiness and trust verification in multi-asset teaming is the research challenge of Autonomy Teaming and TRAjectories for Complex Trusted Operational Reliability (ATTRACTOR). The Design Reference Mission (DRM) for ATTRACTOR is a search and rescue mission objective governed by a multi-member team consisting of human and machine operators. A crucial component to the effort is the communication between humans and autonomous agents throughout both planning and execution stages of the mission. Intuitive communication methods and modalities are posited as critical enablers for certifying trust and trustworthiness. This paper reports on the data collection and analysis conducted in support of the Human Informed Natural-language GANs Evaluation (HINGE)project to attain explainable and trusted communication between human-machine assets. Two identically curated image description datasets were acquired for HINGE, both consisting of two unique input modalities (typed vs. verbal) and retrieved in two distinct contexts (general vs. specific). The gathered datasets were assessed and compared using Parts-of-Speech (POS)features, sentence similarity metrics, and linguistic analysis. Then, the datasets were modeled and tested separately and in combination with one another using machine learning algorithms. The comparison and testing results reveal a superior dataset, by which a preferred context and input is understood, for generating image representations of missing persons using a Generative Adversarial Network (GAN).

Bryan A Barrows↗
NASA NTRSDate not supplied

A Distributed Simulation-to-Flight Framework to Support Investigating Trust/Trustworthiness in Multi-Agent Systems

As autonomous systems continue to grow both in use and complexity, the necessity for robust and extensible simulation-to-flight methods is paramount for establishing an effective architecture for autonomous systems. A fundamental objective of the ATTRACTOR (Autonomy Teaming and TRAjectories for Complex Trusted Operational Reliability) project was to design and develop a distributed mixed-reality simulation environment to begin establishing a basis for certification of autonomous systems via research into trust and trustworthiness. In this paper, we present an autonomous systems architecture and development framework paired with a persistent distributed modeling and simulation environment for test and evaluation of autonomous systems. The Autonomous Entity Operations Network (AEON) framework enables autonomous system development with an easily extensible collection of libraries and plug-n-play nodes facilitated by the Data Distribution Service (DDS) communication protocol standard. The Baseline Environment for Autonomous Modeling (BEAM) simulation environment is a distributed mixed-reality Unity™-based environment built around the same DDS communication paradigm allowing for easy integration with AEON-based autonomous applications. They were designed under ATTRACTOR in order to measure and establish trustworthiness and trust in single- and multi-agent human-machine systems whether these machines are fixed-wing general aviation, rotary-wing Unmanned Aerial Vehicles (UAVs), ground rovers, or even spacecraft. Together AEON and BEAM enable sim-to-flight with minimal configuration changes. By using AEON and BEAM, source code that runs in simulation ports directly to hardware and has successfully flown in the lab and in the National Airspace System (NAS) at NASA LaRC many times over the lifetime of ATTRACTOR.

Benjamin N Kelley↗