Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “verification and validation”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 829 records · Page 46
All sourcesNASA NTRSDOE OSTI
NASA NTRS2019

Architecture of a faul-tolerant and verifiable outer planet flyby

Flyby missions to outer solar system objects require a very high confidence of since there are no second chances. The quantity, quality, and diversity of science measurements to be made in a matter of hours or days must justify the mission cost. Accordingly, the mission architecture must include a high degree of redundancy, large design margins, and an ability to correct or operate through anomalies while at the same time being deterministic enough for confidence in system verification and validation. In this paper we present a system architecture as part of a proposed mission to Triton, called “Trident”.

Mitchell, Karl L.↗
NASA NTRSDate not supplied

ADVANCES IN PLUME-SURFACE INTERACTION SIMULATION CAPABILITY UNDER LUNAR LANDING CONDITIONS

The Fluid Dynamics Branch at theNASA Marshall Space Flight Center has assembled a portfolio of simulation tools to predict Plume-SurfaceInteraction (PSI) environments during extra-terrestrial propulsive landings. Particular focus is on engineering support for lunar landers such as robotic CommercialLunar Payload Services (CLPS) and the HumanLander System (HLS). Extension of existing PSI simulation capabilities are required to accurately capture the complex plume flow conditions and surface soil particle composition effects that arise in the lunar environment. The required model development and implementation, and extensive verification and validation of the key simulation tools, Loci/Chem [1]and Loci/Boltzmann [2], and Loci/GGFS [3] are now performed under a NASA Space Technology MissionDirectorate (STMD) funded multi-year GameChanging Development (GCD) project. The project will implement and mature these new modeling features and verify and validate them for the Mars andMoon environments with the aid of existing and new experiments to be performed under this project

Plume Surface Interaction↗
NASA NTRS2020

Risk considerations for autonomy software

This paper summarizes key findings related to methods for the risk considerations of autonomy software. Existing methods for Verification and Validation (V&V) of autonomy software are summarized and a method for the assurance of autonomy software is suggested and demonstrated on a command execution use case. Risk and reliability are defined in the context of autonomy and an approach for risk assessment of autonomy is presented using an example use case. Key insights regarding areas of uncertainty for autonomy are provided, along with a suggested architecture for the systematic consideration of reliability within the context of a given autonomous planner.

Meshkat, Leila↗
NASA NTRSDate not supplied

A Survey of Thermocouple Anomalies: Mechanisms, Interpretation, and Mitigation

Thermocouples (TCs) are a critical diagnostic sensor for high enthalpy environments including thermal protection systems (TPS) for atmospheric re-entry, hypersonic wind tunnels, and arc jet test facilities. They are used for both evaluating facility environmental conditions (enthalpy, heat flux) as well as for measuring in-situ material thermal response under test conditions (thermal conductivity, heat capacity, B’, etc.). Consequently, accurate TC measurements are fundamental for both material and facility verification and validation (V&V). However, a number of TC anomalies have been observed in both ground tests and flight instrumentation. These anomalies are caused via various coupling mechanisms: electrostatic gradients, temperature-dependent electrical shorts, radio pickup, magnetic induction, and pressure-dependent leaks. All of these mechanisms can, if unaddressed, result in obviously compromised results or worse, non-physical measurements which appear reasonable. This work will survey known anomaly mechanisms and discuss ongoing efforts to quantify and mitigate anomaly mechanisms including anomaly modeling, diagnostics, and new TC designs/development.

Thermocouple Anomalies↗
NASA NTRSDate not supplied

The System Modeling and Analysis of Resiliency in STEReO (SMARt-STEReO)

NASA's Scalable Traffic Management for Emergency Response Operations (STEReO) project aims to leverage Unmanned Aerial Systems (UAS) and UAS Traffic Management (UTM) to improve asset coordination and overall emergency response. One application of STEReO is wildfire response, which is the focus of this research. In order to implement the operations described in the STEReO project, these additions must have tangible benefits and proven safety. To this end, the System Modeling and Analysis of Resiliency in STEReO (SMARt-STEReO) project constructs a simulation model, developed through the Python modeling and resiliency analysis package fmdtools. The model describes wildfire response operations, including current operational concepts and emerging concepts utilizing UAS as described in STEReO. While previous simulation models focus primarily on fire propagation with some models including emergency response intervention, SMART-STEReO evaluates the system performance and resilience benefits gained by the addition of UAS and UTM. Due to the novelty and complexity of the model, initial model verification and validation efforts are conducted and a detailed description of the model is provided. Preliminary results from experimental analysis on the SMARt-STEReO model indicate that when compared to current operations, the addition of UAS in wildfire operations results in improved response efforts, in terms of fewer acres burned, as well as improved system resilience in response to a given fault.

Sequoia Andrade↗
NASA NTRSDate not supplied

Statistical learning framework for safety and failure analysis of a DNN-based autonomous aircraft system

Deep Neural Networks (DNNs) and Machine Learning technology is increasingly used for safety-critical applications in the Aerospace domain. To ensure safe operations, the DNN and the system must undergo rigorous verification and validation, including advanced statistical analyses. Performance and safety of the DNN and system behavior must not only be analyzed for the nominal case, but under numerous off-nominal and failure cases. In this paper we will describe how our statistical learning framework SYSAI can efficiently perform such analyses using the tool’s unique combination of advanced learning modeling and statistical analysis techniques. SYSAI can effectively explore the high-dimensional state and failure space of the system under test; geometrical shape detection of safety regions and boundaries support explainability of the results to the designer. In this paper, we report experiments and results obtained with a vision-based DNN control system (ACT) that is capable of autonomously steering an aircraft down a runway.

Yuning He↗
NASA NTRSDate not supplied

Lessons Learned in Systems Engineering Availability and Recommendations for Mission Technical Leaders

In spaceflight missions at Goddard Space Flight Center (GSFC), the Mission System Engineer (MSE) is the technical leader of the overall engineering team and also is the Independent Engineering Technical Authority. The responsibility of this role includes the definition of the mission design architecture, concept of operations, and mission requirements, management of risk throughout the development, and verification and validation of the final system performance and function amongst other duties. This responsibility inherently requires time management, enabling focus on a balanced development with appropriate risk. Time is the most valuable resource of the MSE. The system engineer’s availability to interact with the development team (often product or component design leads and technicians, often in different worksites) to discover and mitigate mission risks during development is key to mission success. This paper presents examples from Lunar Reconnaissance Orbiter, Landsat 9, and Neutron star Interior Composition ExploreR (NICER) which represent in-house and out-of-house hardware builds. These examples demonstrate how interactions between the Mission Systems Engineers and other project and partner engineers result in discovery of critical risks, leading to early mitigation with significant cost and performance savings. These three missions would have suffered test failures or on-orbit failures had their MSEs not set aside time to visit engineers and technicians that were working on key pieces of space flight hardware. Availability is more than just time; it is openness to listen to concerns and questions. It begins by building a level of trust in the team that it is safe to ask questions or share concerns without the fear of blame or additional workload. It also requires enabling informal conversations (over lunch, coffee, in the clean room, or at the team members desk, etc.) where key information can be exchanged, and team members may even provide an easy-to-implement mitigation idea for another subsystem. Availability is a highly valuable commodity and completely non-obvious to protect and optimize. The natural tendency of engineers is to keep themselves busy with solving problems that they know about. This paper is encouraging MSEs to resist this tendency to try and solve all the complex problems themselves and actively devote daily time to learning and solving problems that are found with informal communications with other team members.

Lessons Learned↗
NASA NTRSDate not supplied

NESC GN&C TDT Workshop on 2D Image Motion Optical Transfer Functions, Pointing Performance Analysis, and Requirements

This course was prepared by the author in support of the NASA GN&C Technical Discipline Team (TDT). The purpose of this two-day course is to inform practicing GN&C engineers, and other system and subsystem engineers, including payload engineers and mission analysts, of methodologies to analyze the pointing performance of spacecraft and to write unambiguous pointing requirements that are relevant to optical payload performance. It is hoped that the seminar will contribute to a best-practices manual and contribute to accepted and uniform means of requirements definition, validation, and verification. The course material is designed to address the needs of not only practicing GN&C engineers, but also system and subsystem engineers of other disciplines, payload engineers, mission analysts, and even astronomers who need to understand how they interface to a GN&C subsystem. Much revision is the direct result of active student participation during the presentations and from feedback obtained through course evaluation forms.

2D Image↗
NASA NTRSDate not supplied

Impact, a Tool Suite for Crew Health and Performance System Trade Analyses and Decision Support - Status of Development

Mission planners, systems engineers, and clinicians that support crew health and performance face very difficult choices on upcoming exploration missions. Given that there will be a heavily constrained mass and volume allocation for a medical system on these missions, what medical capability should be manifested to minimize both medical risk and mission risk? Given that not all promising research and technology proposals can be funded, how can proposals be prioritized so that those funded research investments produce the maximum benefit in reducing overall medical risk? The Informing Mission Planning via Analysis of Complex Tradespaces (IMPACT) project seeks to answer these kinds of questions and others to support upcoming exploration missions. IMPACT enables risk-informed and evidence-based trade space analysis for future space vehicles, missions, and systems. This presentation will discuss the long-term HRP and ExMC vision for the larger ecosystem of tools, which include an updated medical database (consisting of an Evidence Library for medical conditions and a medical item database (MedID) for medical resources), dynamic Probabilistic Risk Assessment (PRA) capabilities, System Modeling Language (SysML) models, and contextual data visualizations of output data. IMPACT is the result of a multi-center collaborative effort. The trade space analyses performed by IMPACT can directly inform mission, vehicle, and habitat development by quantifying medical risk, given a design reference mission, crew attributes and a set of medical capabilities. This presentation will update the audience on the development status of the tool suite, its constituent parts and the plans for when it will deploy as an operational product, currently scheduled for the end of FY22. Recent development successes on the IMPACT project include the ability to cluster medical resources into medical capabilities and mutually-dependent bundles, the ability to perform trade analyses on different medical sets, different DRMs, and different crew complements, and the ability to specify mission segments as periods of time assigned to one or more members of the crew during which unique mission events occur (e.g., extravehicular activities, surface operations, gravity well adaptations, etc.). IMPACT is gearing up for its verification and validation phase to compile the data package necessary for a successful Transition to Operations (TtO), per the guidance in NPR 8900.1B .

IMPACT↗
NASA NTRS2020

A Reinforcement Learning Framework for Space Missions in Unknown Environments

A land-and-traverse mission to icy worlds such as Europa and Enceladus is challenging due to lack of prior knowledge regarding the terrain conditions. Previous work [1] showed that rovers with high degrees of freedom (DoF) can achieve robust traversal by leveraging redundant modes for mobility to counter terrain uncertainty (e.g. walking, driving, or inch-worming). This paper presents a generic and scalable reinforcement learning scheme for enabling on-board decision making on rovers to automatically switch between modes of traversal based on online performance feedback. The objective is to maximize energy efficiency, minimize operator input and successfully negotiate unstructured terrain conditions without relying on exhaustive prior knowledge. The proposed methodology is well grounded in the literature on reinforcement learning and has been adapted to address conformance to validation and verification requirements and JPL flight operations history of using per-sol prescribed sequences for a space mission.

Tavallali, Peyman↗
NASA NTRS2020

Demonstrating Assurance of Model-Based Fault Diagnosis Systems on an Operational Mission

Developers of robotic scientific and commercial spacecraft are trending towards use of onboard autonomous capabilities for responding quickly to dynamic environments and rapidly changing situations. These capabilities need to know the state of the spacecraft’s health. Model-based fault diagnosis (MBFD) is an approach to estimating health by continuously verifying accurate behavior and diagnosing off-nominal behavior. Proper functioning of MBFD depends on 1) the quality of the diagnostic system model that is analyzed and compared to commands and onboard measurements to estimate a system’s health state, and 2) the correct functionality of the diagnosis engine interrogating the model and comparing its analyses to observed system behavior. Our goal is to develop Verification and Validation (V&V) techniques for MBFD to provide future missions sufficient confidence in its functionality and performance to deploy it on the systems they develop. Our work has been focused on infusing the techniques we developed earlier to an operational mission. First, we are constructing diagnostic models of a spacecraft attitude control system and updating our diagnostic engine so they can be demonstrated aboard the Arcsecond Space Telescope Enabling Research in Astrophysics (ASTERIA) mission, an operational spacecraft for which experiments in autonomy are being planned and executed, using the V&V techniques we have previously developed to assure they are both correct and complete. Since it is nearing the end of its life, ASTERIA provides a unique opportunity to demonstrate MBFD since the monitored components are expected to fail. Our demonstration will give system developers additional confidence to make timely, informed MBFD deployment decisions. Second, we will be completing performance assessments of the diagnostic engine/diagnostic model ensemble both on the flight system and ground-based testbeds to gain confidence in MBFD’s ability to run successfully in a spacecraft’s resource-constrained environment without adversely affecting other on-board activities. Finally, we are capturing our experience in preparing this demonstration in a set of checklists and guidance documents. Current practice includes high-level institutional guidance documents and standards, but at a high level of abstraction that does not necessarily address specific MBFD concerns. The purpose of the new checklists is to provide future mission developers clear, unambiguous, procedure-oriented guidance on assuring MBFD. This paper describes our work in these areas. For the first area, we describe the diagnostic models and updated diagnostic engine that will be used for the on-board demonstration. We describe how the V&V techniques we developed earlier are used to assure model and engine correctness and completeness. For the second area, we identify the performance measurement and assessment techniques used to characterize the diagnostic engine and diagnostic models, and discuss the effect of measured performance on overall mission operation. Finally, we present the checklist and guidance documents and describe how they meet the goals of providing system developers with clear, unambiguous, procedure-oriented guidance on MBFD assurance. We show how the techniques we have developed map into those artifacts.

Nikora, Allen↗
NASA NTRS2020

Robotics System Process and Concept for On-orbit Assembly for Potential Mars Sample Return

Proposed Mars Sample Return (MSR) missions would require on-orbit assembly of containment vessels to meet backward Planetary Protection requirements and transfer of the sample container through various stations and positions. Some operations would have to be performed autonomously, and others would require ground-in-loop decision-making stages and verification processes. One concept design for an Earth Return Orbiter (ERO) Capture, Contain, and Return System (CCRS) Transfer Mechanism (TM) is a multi-Degree of Freedom (DOF) manipulator that utilizes a passive End Effector (EE) to assist in containment vessel assembly. To converge on a feasible design, a robotic system process has been instantiated. This process is composed of three main phases: robotic problem definition (operating environment, operations/functions, system goals), robotic solution selection (trade studies on the number of degrees of freedom, number of mechanisms, types of mechanisms), robotic solution design, implementation, and verification and validation (kinematic configuration, robotic and kinematic analysis and topology optimization of components). As a final product of this process, a half-scale functional prototype of the TM was developed to demonstrate the end-to-end operation capability.

Strahle, Jackson W↗
NASA NTRS2020

Enabling Assurance in the MBSE Environment

A number of specific benefits that fit within the hallmarks of effective development are realized with implementation of model-based approaches to systems and assurance. Model Based Systems Engineering (MBSE) enabled by standardized modeling languages (e.g., SysML®) is at the core. These benefits in the context of spaceflight system challenges can include [1]: • Improved management of complex development • Reduced risk in the development process • Improved cost management • Improved design decisions With appropriate modeling techniques the assurance community also can improve early oversight and insight into project development. NASA has shown the basic constructs of SysML in an MBSE environment offer several key advantages, within a Model Based Mission Assurance (MBMA) initiative [2, 3]. These include the following: • Model viewpoints that promote rapid and systematic assessment of requirements coverage, hazard tagging and risk management • Embedded safety assessments for launch vehicles • Deployment of model assisted development of reliability products - Failure Modes and Effects Analyses (FMEAs) and Fault Trees • Test Planning • Validation and Verification of complex functions • Support of Assurance Case development for complex systems In addition, while there are benefits to be harvested, there is a realization that these do not come without effort and cost. Enabling model-based approaches requires structure, not only in an organizational context, but in a modeling context as well. There can be a steep learning curve and costs associated to train skilled modelers. But, on the other hand, not all of the assurance community need to be modelers. Models themselves must conform to ontologies that enable assurance. This places constraints upon the models and modelers. Optimums have yet to be developed where resources and constraints on modeling must be traded off in the organization and modeling efforts for projects. A number of barriers need to be overcome, as well, which pose challenges to the developers of the software that supports MBSE/MBMA. Information and data must be made to flow seamlessly through the life cycle. Because there is a wide variety of tools used in the community, to avoid the problems of the past of silos, delays, and diverging interests, information should flow among these tools to support the “single source of truth” paradigm of MBSE. This will greatly facilitate MBMA and advancement of assurance functions.

Evans, John W.↗
NASA NTRS2020

Integrating the Science Opportunity Analyzer with a Reusable Opportunity Search Framework

In our interactions with the Science Opportunity Analyzer (SOA) software, we recognized how its ability to search for geometric events in space is a need for robotic space missions in general. To satisfy this need, we propose the Tychonis framework, which is built upon the principles of: (1) separations of concerns, (2) extensibility, (3) reusability, and (4) independent verification and validation. Tychonis’ separation of concerns results in the availability of different constructs to model geometric events and search for them. These constructs can be extended by users as needed and reused across missions without changes. Given the low coupling between concerns, and the fact Tychonis can be augmented in isolation, its constructs can be validated independently from other pieces of software. This paper elaborates on these topics and presents an integration case study between Tychonis and SOA that relies on the concept of dynamic integration. Dynamic integration entails that augmentations of the framework are reflected automatically in the host application without any changes to the host application’s code. The SOA-Tychonis integration case study can be extrapolated to other tools that need to search for geometric events as the pattern repeats across implementations: (i) the manipulation of a user interface to model opportunities, (ii) the execution of algorithms to search for opportunities, and (iii) the presentation of search results to users. Overall, Tychonis’ is a story about how the application of proven software principles and good design choices can reduce risk and cost to space missions

Soria, Manel↗
NASA NTRSDate not supplied

Design and Testing of an Approach to Automated In-Flight Safety Risk Management for sUAS Operations

An onboard risk management automation design is presented based on run-time assurance principles, as well as the concept for In-Time Aviation Safety Management Systems (IASMS) as described by the National Academies. The automation is designed to operate independently of the autopilot and perform real-time risk assessment spanning multiple classes of hazards, predict constraint violations, and track autopilot states. In the event of elevated risk conditions or predicted constraint violations, the automation will select from a set of available contingencies and trigger autopilot mode changes if necessary to mitigate risk exposure. The onboard automation also informs the remote operator/pilot of what the independent monitor is observing and any contingency decisions or actions that may arise during flight. Details of an implementation of this design and results of verification and validation activities, as required to meet stringent NASA software and system assurance standards, are also presented. This includes simulation and flight testing using small unmanned aircraft systems.

Ersin Ancel↗
NASA NTRSDate not supplied

Establishing the Assurance Efficacy of Automated Risk Mitigation Strategies

Verification and validation of increasingly autonomous aviation systems is a major challenge. Traditional techniques for the assurance of high-confidence, safety-critical systems are not equipped to handle the complexity, uncertainty, and lack of predictability inherent in non-deterministic systems. Techniques such as run time monitoring, formal methods, and testing and simulation have been applied to some effect, but it is difficult to properly assess the success of such measures. The authors propose the concept of Assurance Efficacy to address this gap. Assurance Efficacy is seen as a parameter, criteria, or perspective by which to evaluate, identify and explore safety risk mitigation strategies and operational assurance architectures. Validation of the utility of this concept through flight testing is a first step in determining its potential role in assessing the overall safety of complex, increasingly autonomous systems that cannot be fully assured in the design phase.

system safety↗
NASA NTRSDate not supplied

NASA GeneLab RNASeq Consensus Pipeline: A Nextflow Implementation

The NASA GeneLab project (genelab.nasa.gov) seeks to accelerate space biology research through cataloging and democratizing omics data. Since raw omics data is largely inaccessible to non-bioinformaticians, GeneLab works with the scientific community to develop standard processing pipelines to generate and publish processed data. Unlike raw data, processed data has greater immediate value to a wide range of users with varying technical backgrounds and computational capabilities. Standardizing processing workflows is essential to match the pace of raw data generation, ensure reproducibility, and enable standardized processed data for comparison across datasets. Previously, GeneLab developed a standardized pipeline for processing RNAseq data, referred to as the ‘GeneLab RNAseq Consensus Pipeline (RCP)’, in collaboration with GeneLab’s Analysis Working Groups. The work presented here is a Nextflow implementation of GeneLab’s RCP that automates and accelerates data processing of RNASeq datasets hosted on GeneLab. In addition to the core data processing, the workflow also includes staging of GeneLab raw data and a robust verification and validation (V&V) program that runs after each processing step to identify errors in real-time, stop additional downstream computation, and preserve computational resources. The workflow, including the staging and V&V functionality, is open source for others to reuse and modify at https://github.com/nasa/GeneLab_Data_Processing/tree/master/RNAseq.

Jonathan Dejesus Oribello↗