TRUST-AM: Trust and Reliability through Unified Simulation and Testing for Additive Manufacturing
Explore the source record for details and available documents.
Qualification↗
SEARCH · Engineering Papers
Results for “Trust”
Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.
Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.
A Meta-Analytic Approach to Investigating the Relationship Between Human-Automation Trust and Attention Allocation
Trust and attention allocation are pivotal determinants in human-automation interaction. However, there are scarce empirical findings regarding the relationship between trust and attention allocation. Observations from our previous work suggested there may be a negative correlation between trust in automation and eye movement towards automation, though no formal analysis of these data had been conducted to quantify this relationship. The present meta-analysis examined the relationship between three dimensions of trust in automation (performance, process, and purpose) and visual attention allocation to the automation. Specifically, we applied Cumming’s (2014) meta-analysis technique to combine evidence across three experiments. Results indicated a negative correlation between trust in automation and visual sampling of the automated system monitoring task for performance-based trust, but not for process- or purpose-based trust. These findings suggest that operators scanned the automation’s behavior less frequently when indicating higher performance-based trust towards the automation.
Human-Autonomy Teaming (HAT)↗
The Role of a Multidimensional Concept of Trust in the Performance of Global Virtual Teams
This paper focuses on the concept of trust as an important ingredient of effective global virtual team performance. Definitions of trust and virtual teams are presented. The concept of trust is developed from its unilateral application (trust, absence of trust) to a multidimensional concept including cognitive and affective components. The special challenges of a virtual team are then discussed with particular emphasis on how a multidimensional concept of trust impacts these challenges. Propositions suggesting the multidimensional concept of trust moderates the negative impacts of distance, cross cultural and organizational differences, the effects of electronically mediated communication, reluctance to share information and a lack of hi story/future on the performance of virtual teams are stated. The paper concludes with recommendations and a set of techniques to build both cognitive and affective trust in virtual teams.
Bodensteiner, Nan Muir↗
Designing and Training for Appropriate Trust in Increasingly Autonomous Advanced Air Mobility Operations: A Mental Model Approach: Version 1
To enable effective human-autonomy teaming (HAT) in Advanced Air Mobility (AAM) operations, the current paper presents a theoretical framework to design and train for appropriate trust in automation. The novel contribution of this work resides in connecting the construct of trust to mental models and showing how this method could be used to enable emerging HAT concepts such as Adaptive Trust Calibration. To contextualize this framework, in section 2 we discuss simplified vehicle operations (SVO) and remote vehicle operations (RVO), which are leading operational concepts within AAM. In section 3 we describe our perspective on automation and increasingly autonomous systems and present a brief discussion on human-automation interaction and human-autonomy teaming. In section 4 we provide a detailed discussion on the construct of trust in automation. In section 5 we present a framework that associates mental models with trust through principles of transparent design. Finally, in section 6 we present three descriptive models for designing and training for appropriate trust in increasingly autonomous systems.
Human-Autonomy Teaming↗
Serious Gaming for Building a Basis of Certification via Trust and Trustworthiness of Autonomous Systems
Autonomous systems governed by a variety of adaptive and nondeterministic algorithms are being planned for inclusion into safety-critical environments, such as unmanned aircraft and space systems in both civilian and military applications. However, until autonomous systems are proven and perceived to be capable and resilient in the face of unanticipated conditions, humans will be reluctant or unable to delegate authority, remaining in control aided by machine-based information and decision support. Proving capability, or trustworthiness, is a necessary component of certification. Perceived capability is a component of trust. Trustworthiness is an attribute of a cyber-physical system that requires context-driven metrics to prove and certify. Trust is an attribute of the agents participating in the system and is gained over time and multiple interactions through trustworthy behavior and transparency. Historically, artificial intelligence and machine learning systems provide answers without explanation - without a rationale or insight into the machine “thinking”. In order to function as trusted teammates, machines must be able to explain their decisions and actions. This transparency is a product of both content and communication. NASA’s Autonomy Teaming & TRAjectories for Complex Trusted Operational Reliability (ATTRACTOR) project seeks to build a basis for certification of autonomous systems via establishing metrics for trustworthiness and trust in multi-agent team interactions, using AI (Artificial Intelligence) explainability and persistent modeling and simulation, in the context of mission planning and execution, with analyzable trajectories. Inspired by Massively Multiplayer Online Role Playing Games (MMORPG) and Serious Gaming, the proposed ATTRACTOR modeling and simulation environment is similar to online gaming environments in which player (aka agent) participants interact with each other, affect their environment, and expect the simulation to persist and change regardless of any individual agent’s active participation. This persistent simulation environment will accommodate individual agents, groups of self-organizing agents, and large-scale infrastructure behavior. The effects of the emerging adaptation and coevolution can be observed and measured to building a basis of measurable trustworthiness and trust, toward certification of safety-critical autonomous systems.
Allen, B. Danette↗
Adapting a Trusted AI Framework to Space Mission Autonomy
As artificial intelligence (AI) is increasingly pro- posed for new and future capabilities in space missions, the question of how to trust AI-enabled space autonomy has been explored. Recently, a collaboration between The Aerospace Corporation (Aerospace) and NASA’s Jet Propulsion Labora- tory (JPL) investigated how Aerospace’s Trusted AI Frame- work could be applied to two JPL projects that planned on lev- eraging AI for critical autonomous tasks. This combined effort led to many insights in the practical implementation of trusted AI along with considerable updates to the Trusted AI Frame- work that tailored its topic threads to space exploration. This document cohesively summarizes the enhanced framework as tailored to space missions as well as estimation of the level of trust required as a function of mission criticality and key stakeholders. The goal of this work is to provide a set of best practices to inform autonomy researchers, flight engineers, mission and proposal reviewers, and instrument and mission principal investigators (PI’s) to drive AI-based autonomy that maximizes trust and lowers the barriers to mission adoption for both science and engineering applications.
Amini, Rashied↗
Effects of Transparency on Pilot Trust and Agreement in the Autonomous Constrained Flight Planner
We performed a human-in-the-loop study to explore the role of transparency in engendering trust and reliance within highly automated systems. Specifically, we examined how transparency impacts trust in and reliance upon the Autonomous Constrained Flight Planner (ACFP), a critical automated system being developed as part of NASA's Reduced Crew Operations (RCO) Concept. The ACFP is designed to provide an enhanced ground operator, termed a super dispatcher, with recommended diversions for aircraft when their primary destinations are unavailable. In the current study, 12 commercial transport rated pilots who played the role of super dispatchers were given six time-pressured all land scenarios where they needed to use the ACFP to determine diversions for multiple aircraft. Two factors were manipulated. The primary factor was level of transparency. In low transparency scenarios the pilots were given a recommended airport and runway, plus basic information about the weather conditions, the aircraft types, and the airport and runway characteristics at that and other airports. In moderate transparency scenarios the pilots were also given a risk evaluation for the recommended airport, and for the other airports if they requested it. In the high transparency scenario additional information including the reasoning for the risk evaluations was made available to the pilots. The secondary factor was level of risk, either high or low. For high-risk aircraft, all potential diversions were rated as highly risky, with the ACFP giving the best option for a bad situation. For low-risk aircraft the ACFP found only low-risk options for the pilot. Both subjective and objective measures were collected, including rated trust, whether the pilots checked the validity of the automation recommendation, and whether the pilots eventually flew to the recommended diversion airport. Key results show that: 1) Pilots trust increased with higher levels of transparency, 2) Pilots were more likely to verify ACFPs recommendations with low levels of transparency and when risk was high, 3) Pilots were more likely to explore other options from the ACFP in low transparency conditions and when risk was high, and 4) Pilots decision to accept or reject ACFPs recommendations increased as a function of the transparency in the explanation. The finding that higher levels of transparency was coupled with higher levels of trust, a lower need to verify other options, and higher levels of agreement with ACFP recommendations, confirms the importance of transparency in aiding reliance on automated recommendations. Additional analyses of qualitative data gathered from subjects through surveys and during debriefing interviews also provided the basis for new design recommendations for the ACFP.
trust↗
The Role of Trust and Usability in Enabling Spaceflight Crew Autonomy
Future long duration exploration missions will require an increased use of onboard automated systems as spaceflight crews venture further than before and have longer communications delays with ground support. The design of these systems must support appropriate crew trust and have sufficient usability to enable spaceflight crew autonomy or risk being misused while crews wait to communicate with ground support. We evaluated trust & usability in our self-scheduling tool, Playbook, for crew mission timelines. Data was collected in a controlled lab experiment with 31 participants. Participants in the study conducted two tasks: scheduling, where participants were responsible for scheduling a majority of a day's operational tasks, and rescheduling, where participants were provided a schedule and asked to reschedule higher priority activities. We found a significant correlation between system trust and usability, irrespective of self-scheduling tasks. We conclude that system usability may play a bigger role in how trust is learned while conducting novel crew autonomy tasks such as self-scheduling. Future research should investigate the role of usability to encourage appropriate trust in onboard automated crew systems and enable crew autonomy.
crew autonomy↗
Effects of Secondary Task Modality and Processing Code on Automation Trust and Utilization During Simulated Airline Luggage Screening
The purpose of this research was to examine the impact of environmental distractions on human trust and utilization of automation during the process of visual search. Participants performed a computer-simulated airline luggage screening task with the assistance of a 70% reliable automated decision aid (called DETECTOR) both with and without environmental distractions. The distraction was implemented as a secondary task in either a competing modality (visual) or non-competing modality (auditory). The secondary task processing code either competed with the luggage screening task (spatial code) or with the automation's textual directives (verbal code). We measured participants' system trust, perceived reliability of the system (when a target weapon was present and absent), compliance, reliance, and confidence when agreeing and disagreeing with the system under both distracted and undistracted conditions. Results revealed that system trust was lower in the visual-spatial and auditory-verbal conditions than in the visual-verbal and auditory-spatial conditions. Perceived reliability of the system (when the target was present) was significantly higher when the secondary task was visual rather than auditory. Compliance with the aid increased in all conditions except for the auditory-verbal condition, where it decreased. Similar to the pattern for trust, reliance on the automation was lower in the visual-spatial and auditory-verbal conditions than in the visual-verbal and auditory-spatial conditions. Confidence when agreeing with the system decreased with the addition of any kind of distraction; however, confidence when disagreeing increased with the addition of an auditory secondary task but decreased with the addition of a visual task. A model was developed to represent the research findings and demonstrate the relationship between secondary task modality, processing code, and automation use. Results suggest that the nature of environmental distractions influence interaction with automation via significant effects on trust and system utilization. These findings have implications for both automation design and operator training.
Phillips, Rachel↗
Accessible Telemetry Streams using a Zero Trust Architecture for the Flight Operations Directorate
As a result of information technology based work becoming increasingly distributed, unique challenges have been presented within the realm of defined network perimeters, namely with respect to secure access to resources. Historically, and from a simplistic abstract perspective, the common approach has been to adopt the, so-called, moat model whereby a physical network perimeter (or interconnected perimeters) is defined to encapsulate resources behind a boundary protected by a firewall. Users are provisioned access through a virtual private network (VPN) and may be further constrained to resources through specific firewall allow and disallow rulesets. Virtual Private Networks and firewall rulesets lead to common problems, particularly at scale and, as a result, perimeter-less architectures provided over the public internet are increasingly becoming prevalent, particularly with its more popular implementation, the Zero Trust Architecture. We present a proposed implementation of the Zero Trust Architecture with a particular concrete example utilizing a de-perimeterized network that requires authentication and authorization for each action between nodes and does not operate within an implicit trust boundary. It should be noted that this paper is not an attempt at providing comprehensive resolutions for the specific problem space with respect to perimeter based security and is more directed at providing information with regard to our proposed implementation of a Zero Trust Architecture for the Flight Operations Directorate. We direct the reader to our Introduction and Background section for more details on specific documentation and where it can be located as it relates to de-perimeterization and Zero Trust.
Paul Shoemaker↗
Space Trusted Autonomy Readiness Levels
Technology Readiness Levels are a mainstay for organizations that fund, develop, test, acquire, or use technologies. Technology Readiness Levels provide a standardized assessment of a technology’s maturity and enable consistent comparison among technologies. They inform decisions throughout a technology’s development life cycle, from concept, through development, to use. A variety of alternative Readiness Levels have been developed, including Algorithm Readiness Levels, manufacturing Readiness Levels, Human Readiness Levels, Commercialization Readiness Levels, Machine Learning Readiness Levels, and Technology Commitment Levels. However, while Technology Readiness Levels have been increasingly applied to emerging disciplines, there are unique challenges to assessing the rapidly developing capabilities of autonomy. This paper adopts the moniker of Space Trusted Autonomy Readiness Levels to identify a two-dimensional scale of readiness and trust appropriate for the special challenges of assessing autonomy technologies that seek space use. It draws inspiration from other readiness levels’ definitions, and from the rich field of trust and trustworthiness. The Space Trusted Autonomy Readiness Levels were developed by a collaborative Space Trusted Autonomy subgroup, which was created from The Space Science and Technology Partnership Forum between the United States Space Force, the National Aeronautics and Space Administration, and the National Reconnaissance Office.
Kerianne L Hobbs↗
What is the Role of Usability and Trust in Autonomy?
Usability encompasses learnability, efficiency, memorability, effectiveness, and satisfaction. NASA’s standards for usability acceptance criteria focus on interfaces that help operators achieve their tasks efficiently, effectively, and with satisfaction. However, discussions on usability, especially regarding future highly automated and autonomous systems, rarely include trust. As NASA plans for long-duration exploration missions, it envisions astronauts operating more independently from Mission Control on Earth. This independence will drive the development of these highly automated and autonomous systems that astronauts will use daily. To prepare for this future, our team has developed a scheduling and execution software tool that facilitates self-scheduling, allowing astronauts to independently manage their own schedule without Mission Control’s involvement. Over many years, we have developed, matured, and evaluated our software tool in extreme environments, prioritizing user-centered design and high usability. These evaluations have included multiple campaigns in NASA analogs, including NEEMO, BASALT, and HERA, as well as technology demonstrations onboard the International Space Station. Our recent research on software interfaces for future astronaut autonomy revealed a strong correlation between usability and trust measures. In a controlled lab experiment, we asked novice users to perform a complex scheduling task, during which the software immediately validated the schedule’s constraints and checked for violations. We collected usability (User Experience Questionnaire, UEQ) and trust (Trust in Automated Systems scale, TAS) measures; significant, strong, and moderate correlations emerged between several of the UEQ metrics and TAS. These results support the argument for investing in usability early to enable and sustain trust in highly automated and autonomous systems.
usability↗
Reliable Design Versus Trust
This presentation focuses on reliability and trust for the users portion of the FPGA design flow. It is assumed that the manufacturer prior to hand-off to the user tests FPGA internal components. The objective is to present the challenges of creating reliable and trusted designs. The following will be addressed: What makes a design vulnerable to functional flaws (reliability) or attackers (trust)? What are the challenges for verifying a reliable design versus a trusted design?
Field Programmable Gate Aray (FPGA)↗
Public Trust and Acceptance for Concepts of Remotely Operated Urban Air Mobility Transportation
There is building interest within industry and government to enable Urban Air Mobility (i.e., air-taxies). One concept envisions remotely piloted aircraft, yet it is unclear how this will impact public trust and acceptance. Method: Two hundred participants read vignettes describing remotely-piloted UAM operations and then responded to a series of questionnaires. The study employed a one-way between-subjects design manipulating five levels of Pilot-in-Command Distance: Onboard Pilot; Remote Control Pilot; Dedicated Remote Operator; Remote Operator; System Manager. Results: The Remote Control Pilot group indicated they would be less likely than the Onboard Pilot Group to use UAM, based on the mediating effect of trust in the automation. The Remote Control Pilot and Remote Operator groups indicated they would be less likely to use UAM than the Onboard Pilot group, based on the mediating effect of trust in the remote pilot/operator. Conclusion: Trust in UAM automation and remote pilots/operators will likely affect public acceptance of UAM.
MTurk↗
ATTRACTOR: Toward Trustworthy and Trusted Autonomous Systems
The question of what it means and what it takes for an autonomous system to consider another autonomous system justifiably trustworthy must be addressed by all who seek to integrate intelligent machine agents into real-world operations. A satisfactory answer to this question is an essential component in accepting autonomous machine decision-making in safety-critical and time-critical environments, such as aviation. Historically, simulation platforms for test and evaluation of complex systems have proven to be effective in assessing performance and contributing to decisions on the fitness of systems to operate in current general and commercial aviation airspace. Moreover, simulations have informed the definition of safety-critical constraints. However, as machine systems progressively take on responsibilities for decision-making traditionally supplied by humans, simulations require enhancement. Mixed reality simulation that integrates real-world platforms and data or high-fidelity simulation data in a sim-to-flight paradigm provides insight into agent interaction and the rationale behind autonomous agent decision-making as well as the capacity for seamless integrated implementation, testing, and operation of systems. Strong simulation capabilities are especially important in the presence of algorithms that hold great promise in decision-making yet increase the uncertainty in the system. Autonomy Teaming and TRAjectories for Complex Trusted Operational Reliability (ATTRACTOR) is a subproject of NASA’s Convergent Aeronautics Solutions (CAS) Project. ATTRACTOR’s objective is to build a basis for understanding trust and trustworthiness in multi-agent autonomous teams, and thus to inform future certification of safety-critical and time-critical autonomous systems in aviation. Because the concepts of trust and trustworthiness must be addressed in a context, ATTRACTOR has chosen Search and Rescue (SAR) in dynamic and unstructured environments, with emphasis on search, as its design reference mission (DRM). During dynamic planning and execution of trajectory-based operations, autonomous agents determine their trajectories given an assigned mission or missions and call for assistance from an appropriate teammate when needed. This experience along with the attendant human-machine and machine-machine interactions, serve as a platform for developing approaches to identifying and measuring trustworthiness and increasing trust. In this paper, we give an overview of some of ATTRACTOR’s research and development activities, findings, and ongoing work.
ATTRACTOR↗
TRUST: TDRSS Resource User Support Tool
TRUST-TDRSS (Tracking Data and Relay Satellite System) Resource User Support Tool is presented in the form of the viewgraphs. The following subject areas are covered: TRUST development cycle; the TRUST system; scheduling window; ODM/GCMR window; TRUST architecture; surpass; and summary.
Sparn, Thomas P.↗
Operator Trust Function for Predicted Drone Arrival
To realize the full benefit from autonomy, systems will have to react to unknown events and uncertain dynamic environments. The resulting number of behaviors is essentially infinite; thus, the system is effectively non-deterministic but an operator needs to understand and trust the actions of the autonomous vehicles. This research began to tackle non-deterministic systems and trust by beginning to develop a user trust function based on intent information displayed and the prescribed bounds on allowable behaviors/actions of the non-deterministic system. Linear regression shows promise on being able to predict a person’s confidence of the machine’s prediction. Linear regression techniques indicated that subject characteristics, scenario difficulty, the experience with the system, and confidence earlier in the scenario account for approximately 60% of the variation in confidence ratings. This paper details the specifics of the liner regression model – essentially a trust function – for predicting a person’s confidence.
Trujillo, Anna C.↗
Enabling Advanced Air Mobility Operations through Appropriate Trust in Human-Autonomy Teaming: Foundational Research Approaches and Applications
Emerging Advanced Air Mobility(AAM)operations will be enabled by increasingly autonomous systems, requiring technologies to take on more responsibilities and fundamentally altering traditional human-automation interaction paradigms. The growing reliance on higher levels of automation will necessitate research to identify capabilities and principles that facilitate humans and machines working and thinking better together, i.e., human-autonomy teaming (HAT). Trust is an inherent requirement in effective teams because when members work interdependently, those agents (human, automation) must be willing to accept a level of risk to rely upon each other to reach goals and contribute to team tasks. This work provides an initial approach to enabling AAM operations through appropriate trust within HAT. The main contributions of this approach resides in connecting the construct of trust to mental models. Using the outlined mental model approach, we propose novel HAT strategies, such as Adaptive Trust Calibration, and preview planned research activities derived from this approach. Additionally, we propose several practical applications that can currently be employed by AAM development communities.
Eric T Chancey↗