Results for “verification and validation” — Page 53

NASA NTRS1991

Meteorological radar methods for validating space observations of precipitation

Meteorological approaches to verification of space measurements of rainfall are examined; validation of Tropical Rainfall Measuring Mission (TRMM) observations is expected to depend significantly on ground-based radars. Two methods of comparison are initially contemplated. TRMM rainfall data over time periods of a month for large areas (500 x 500 km) are averaged and compared with similarly averaged ground truth measurements. Both the rainfall and height distribution data from TRMM are compared with the instantaneous values observed at one or more 'ground truth' stations and from airborne radar and radiometers as available.

Thiele, Otto W.↗

NASA NTRS1989

A verification library for multibody simulation software

A multibody dynamics verification library, that maintains and manages test and validation data is proposed, based on RRC Robot arm and CASE backhoe validation and a comparitive study of DADS, DISCOS, and CONTOPS that are existing public domain and commercial multibody dynamic simulation programs. Using simple representative problems, simulation results from each program are cross checked, and the validation results are presented. Functionalities of the verification library are defined, in order to automate validation procedure.

Kim, Sung-Soo↗

NASA NTRS2024

NASA Engineering and Safety Center Technical Bulletin No. 24-04: 6DOF Check Cases

In 2015, the NESC released benchmark Earth-based check-cases for well specified, rigid-body, six-degree-of-freedom (6DOF) aero/spacecraft models to promote consistent and accurate flight simulations across multiple Agency tools and facilities. Recently, the NESC expanded upon that effort to add Lunar-based check-cases to support new lunar exploration initiatives. This study produced a smaller, focused set of cases that exercise new and unique features of missions in the lunar environment in comparison with 8 high-fidelity NASA simulation tools and provides a measure of validation for simulations supporting Human Landing Systems.

Flight Mechanics↗

NASA NTRS1978

Pretest information for a test to validate plume simulation procedures (FA-17)

The results of an effort to plan a final verification wind tunnel test to validate the recommended correlation parameters and application techniques were presented. The test planning effort was complete except for test site finalization and the associated coordination. Two suitable test sites were identified. Desired test conditions were shown. Subsequent sections of this report present the selected model and test site, instrumentation of this model, planned test operations, and some concluding remarks.

Hair, L. M.↗

NASA NTRS1995

The Specification-Based Validation of Reliable Multicast Protocol

Reliable Multicast Protocol (RMP) is a communication protocol that provides an atomic, totally ordered, reliable multicast service on top of unreliable IP multicasting. In this report, we develop formal models for RMP using existing automated verification systems, and perform validation on the formal RMP specifications. The validation analysis help identifies some minor specification and design problems. We also use the formal models of RMP to generate a test suite for conformance testing of the implementation. Throughout the process of RMP development, we follow an iterative, interactive approach that emphasizes concurrent and parallel progress of the implementation and verification processes. Through this approach, we incorporate formal techniques into our development process, promote a common understanding for the protocol, increase the reliability of our software, and maintain high fidelity between the specifications of RMP and its implementation.

Wu, Yunqing↗

NASA NTRS1995

The specification-based validation of reliable multicast protocol: Problem Report

Reliable Multicast Protocol (RMP) is a communication protocol that provides an atomic, totally ordered, reliable multicast service on top of unreliable IP multicasting. In this report, we develop formal models for RMP using existing automated verification systems, and perform validation on the formal RMP specifications. The validation analysis help identifies some minor specification and design problems. We also use the formal models of RMP to generate a test suite for conformance testing of the implementation. Throughout the process of RMP development, we follow an iterative, interactive approach that emphasizes concurrent and parallel progress of implementation and verification processes. Through this approach, we incorporate formal techniques into our development process, promote a common understanding for the protocol, increase the reliability of our software, and maintain high fidelity between the specifications of RMP and its implementation.

Wu, Yunqing↗

NASA NTRS2013

Model-Based Design and Validation of Cellularized Satellites for the DARPA Phoenix Program

No abstract available

systems engineering↗

NASA NTRSDate not supplied

Validation of Image-Based Neural Network Controllersthrough Adaptive Stress Testing

Neural networks have become state-of-the-art for computer vision problems because of their ability to efficiently model complex functions from large amounts of data. While neural networks can be shown to perform well empirically fora variety of tasks, their performance is difficult to guarantee.Neural network verification tools have been developed that can certify robustness with respect to a given input image; however,for neural network systems used in closed-loop controllers,robustness with respect to individual images does not address multi-step properties of the neural network controller and itsenvironment. Furthermore, neural network systems interacting in the physical world and using natural images are operating in a black-box environment, making formal verification in-tractable. This work combines the adaptive stress testing (AST)framework with neural network verification tools to search for the most likely sequence of image disturbances that cause the neural network controlled system to reach a failure. Anautonomous aircraft taxi application is presented, and results show that the AST method finds failures with more likely image disturbances than baseline methods. Further analysis of AST results revealed an explainable cause of the failure, giving insight into the problematic scenarios that should be addressed.

Adaptive Stress Testing, Marabou, Deep Neural Netw↗

NASA NTRSDate not supplied

In-Space Modular Assembly: An Approach for Reliable, Affordable, Precision Space Apertures

In-space assembly will revolutionize the creation, upgrade, and evolution of future space systems. In-space assembly represents an alternative deployment strategy that is not constrained by the requirement of using a single launch vehicle and enables a greater freedom of design for the initial emplacement of assets and their evolution over time. In-space assembly enables assets, such as observatories and science platforms, to become persistent, evolving over time like their terrestrial counterparts. Also, in-space assembly provides a direct path for utilization of in-space manufactured components designed exclusively for the operational environment. To highlight the advantages of an in-space assembly approach, the modular assembly of a 3 m to 4 m precision optical aperture based on thin meniscus technology coupled with structurally efficient TriTruss modules is presented. The 3 m to 4 m aperture stows compactly within two standard ride share slots (0.61 m by 0.71 m by 0.97 m). Placing instruments and the robotic system used for assembly in an adjacent ride share slot enables a capable observatory to be placed into service via modest ride share opportunities. Further, recent hardware assembly tests of similar modules and progress toward hardware tests to validate the overall architecture via diffraction limited testing will be summarized.

Design↗

NASA NTRSDate not supplied

In-Space Modular Assembly: An Approach for Reliable, Affordable, Precision Space Apertures

In-space assembly will revolutionize the creation, upgrade, and evolution of future space systems. In-space assembly represents an alternative deployment strategy that is not constrained by the requirement of using a single launch vehicle and enables a greater freedom of design for the initial emplacement of assets and their evolution over time. In-space assembly enables assets, such as observatories and science platforms, to become persistent, evolving over time like their terrestrial counterparts. Also, in-space assembly provides a direct path for utilization of in-space manufactured components designed exclusively for the operational environment. To highlight the advantages of an in-space assembly approach, the modular assembly of a 3 m to 4 m precision optical aperture based on thin meniscus technology coupled with structurally efficient TriTruss modules is presented. The 3 m to 4 m aperture stows compactly within two standard ride share slots (0.61 m by 0.71 m by 0.97 m). Placing instruments and the robotic system used for assembly in an adjacent ride share slot enables a capable observatory to be placed into service via modest ride share opportunities. Further, recent hardware assembly tests of similar modules and progress toward hardware tests to validate the overall architecture via diffraction limited testing will be summarized.

Design↗

NASA NTRS2004

The Role of Integrated Modeling in the Design and Verification of the James Webb Space Telescope

The James Web Space Telescope (JWST) is a large, infrared-optimized space telescope scheduled for launch in 2011. System-level verification of critical optical performance requirements will rely on integrated modeling to a considerable degree. In turn, requirements for accuracy of the models are significant. The size of the lightweight observatory structure, coupled with the need to test at cryogenic temperatures, effectively precludes validation of the models and verification of optical performance with a single test in 1-g. Rather, a complex series of steps are planned by which the components of the end-to-end models are validated at various levels of subassembly, and the ultimate verification of optical performance is by analysis using the assembled models. This paper describes the critical optical performance requirements driving the integrated modeling activity, shows how the error budget is used to allocate and track contributions to total performance, and presents examples of integrated modeling methods and results that support the preliminary observatory design. Finally, the concepts for model validation and the role of integrated modeling in the ultimate verification of observatory are described.

Mosier, Gary E.↗

NASA NTRS2017

Verification of the Icarus Material Response Tool

Due to the complex physics encountered during reentry, material response solvers are used for two main purposes: improve the understanding of the physical phenomena; and design and size thermal protection systems (TPS). Icarus, is a three dimensional, unstructured material response tool that is intended to be used for design while maintaining the flexibility to easily implement physical models as needed. Because TPS selection and sizing is critical, it is of the utmost importance that the design tools be extensively verified and validated before their use. Verification tests aim at insuring that the numerical schemes and equations are implemented correctly by comparison to analytical solutions and grid convergence tests.

ablation↗

NASA NTRS2013

SILHIL Replication of Electric Aircraft Powertrain Dynamics and Inner-Loop Control for V&V of System Health Management Routines

Software-in-the-loop and Hardware-in-the-loop testing of failure prognostics and decision making tools for aircraft systems will facilitate more comprehensive and cost-effective testing than what is practical to conduct with flight tests. A framework is described for the offline recreation of dynamic loads on simulated or physical aircraft powertrain components based on a real-time simulation of airframe dynamics running on a flight simulator, an inner-loop flight control policy executed by either an autopilot routine or a human pilot, and a supervisory fault management control policy. The creation of an offline framework for verifying and validating supervisory failure prognostics and decision making routines is described for the example of battery charge depletion failure scenarios onboard a prototype electric unmanned aerial vehicle.

Verification and Validation↗

NASA NTRS2024

Development Testing and Analysis of the Integrated Gateway-ESPRIT Bipropellant Refuelling System

The Gateway will be humanity’s first space station orbiting the Moon completed by NASA in partnership with ESA and other US and international partners. The Gateway will provide critical support to sustainable human exploration on the moon through the Artemis program. To enable the Lunar Gateway to complete its mission, on orbit refuelling is essential. The ESPRIT Refuelling Module (ERM) will provide the capability to transfer propellants, MMH and MON-3, through the Habitation and Logistics Outpost (HALO) to the Power and Propulsion Element (PPE). The transfer of these propellants carries known risks and hazards. These hazards include overpressure of the propellant lines during priming sequences between modules and during refuelling pause operations. To support the early system development and to mitigate these risks, a collaborative test program between NASA, ESA and Thales Alenia Space was completed at the Thales Alenia Space test facility in Harwell, UK. This test program integrated fluidic breadboards of the ERM, HALO and PPE modules. The objectives of the test program were to demonstrate and characterise critical performance and transient operations, inform refuelling concept of operations and to calibrate and validate numerical models of the refuelling subsystem in EcosimPro. To support the completion of this final objective a detailed model of the integrated breadboard was developed in EcosimPro and key steady-state and transient test cases were simulated. As an industry first, a National Institute of Standards and Technology (NIST) database correlation for Hydrofluoroether (HFE-7000) was used as a mixed oxides of nitrogen (MON-3) simulant with EcosimPro and European Space Propulsion System Simulation (ESPSS) libraries to result in a more accurate analysis for transient phenomenon such as priming. The test and simulation data showed good agreement validating the model for further system analysis as the ERM design progresses.

Refuelling↗

NASA NTRS2020

The GPM Ground Validation Program

We present a detailed overview of the structure and activities associated with the NASA-led ground-validation component of the NASA-JAXA Global Precipitation Measurement (GPM) mission. The overarching philosophy and approaches for NASA's GV program are presented with primary focus placed on aspects of direct validation and a summary of physical validation campaigns and results. We describe a spectrum of key instruments, methods, field campaigns and data products developed and used by NASA's GV team to verify GPM level-2 precipitation products in rain and snow. We describe the tools and analysis framework used to confirm that NASA's Level-I science requirements for GPM are met by the GPM Core Observatory. Examples of routine validation activities related to verification of Integrated MultisatellitE Retrievals for GPM (IMERG) products for two different regions of the globe (Korea and the US) are provided, and a brief analysis related to IMERG performance in the extreme rainfall event associated with Hurricane Florence is discussed.

precipitation↗

NASA NTRSDate not supplied

Exploitation of a Validation Hierarchy for Modeling and Simulation

Across engineering there is an evolving need to increase reliance on physics-based simulation to develop, design and optimize engineering systems. This increased reliance on modeling and simulation has highlighted a growing need to transform the confidence that modeling and simulation analysts have in their results into credibility for systems engineers to design and field systems more quickly and with less physical testing. For isolated components of a complex system, where a single discipline may drive product design, this is less of a concern as the relationship is often straightforward and easy to explain. However, when these isolated components are integrated, and are expected to operate in a multi-disciplinary context in which safety critical systems are involved, new concepts and model assurance standards are required. In this paper we address this challenge by showing how a model validation hierarchy can be exploited to identify those model validation experiments that will contribute most to increasing confidence and credibility of modeling and simulation predictions. The approach that is adopted contains four main steps. The first step is the construction of a model validation hierarchy that links subsystems, assemblies, and components to a hierarchy of physical experiments that can be used support model validation. This hierarchy connects the concerns of systems engineers to those of the modeling and simulation analyst in a clear and logical way. The structure and content of this hierarchy is then used in a second step to establish which physical phenomena have the greatest impact on overall system performance metrics. A gap analysis technique, based upon modeling and simulation concerns, is then used to prioritize the important physical phenomenon. Unfortunately, a common outcome of such gap analyses is the identification of many important gaps and so, in the final step of our process, we advocate the use of a global sensitivity analysis as a means to complete the prioritization.

Verification and Validation↗

NASA NTRSDate not supplied

Exploitation of a Validation Hierarchy for Modeling and Simulation

Across engineering there is an evolving need to increase reliance on physics-based simulation to develop, design and optimize engineering systems. This increased reliance on modeling and simulation has highlighted a growing need to transform the confidence that modeling and simulation analysts have in their results into credibility for systems engineers to design and field systems more quickly and with less physical testing. For isolated components of a complex system, where a single discipline may drive product design, this is less of a concern as the relationship is often straightforward and easy to explain. However, when these isolated components are integrated, and are expected to operate in a multi-disciplinary context in which safety critical systems are involved, new concepts and model assurance standards are required. In this paper we address this challenge by showing how a model validation hierarchy can be exploited to identify those model validation experiments that will contribute most to increasing confidence and credibility of modeling and simulation predictions. The approach that is adopted contains four main steps. The first step is the construction of a model validation hierarchy that links subsystems, assemblies, and components to a hierarchy of physical experiments that can be used support model validation. This hierarchy connects the concerns of systems engineers to those of the modeling and simulation analyst in a clear and logical way. The structure and content of this hierarchy is then used in a second step to establish which physical phenomena have the greatest impact on overall system performance metrics. A gap analysis technique, based upon modeling and simulation concerns, is then used to prioritize the important physical phenomenon. Unfortunately, a common outcome of such gap analyses is the identification of many important gaps and so, in the final step of our process, we advocate the use of a global sensitivity analysis as a means to complete the prioritization.

Verification and Validation↗

NASA NTRS2020

Adaptive Stress Testing: Finding Likely Failure Events with Reinforcement Learning

Finding the most likely path to a set of failure states is important to the analysis of safety-critical systems that operate over a sequence of time steps, such as aircraft collision avoidance systems and autonomous cars. In many applications such as autonomous driving, failures cannot be completely eliminated due to the complex stochastic environment in which the system operates.As a result, safety validation is not only concerned about whether a failure can occur, but also discovering which failures are most likely to occur. This article presents adaptive stress testing (AST), a framework for finding the most likely path to a failure event in simulation. We consider a general black box setting for partially observable and continuous-valued systems operating in an environment with stochastic disturbances. We formulate the problem as a Markov decision process and use reinforcement learning to optimize it. The approach is simulation-based and does not require internal knowledge of the system, making it suitable for black-box testing of large systems. We present different formulations depending on whether the state is fully observable or partially observable. In the latter case, we present a modified Monte Carlo tree search algorithm that only requires access to the pseudorandom number generator of the simulator to overcome partial observability. We also present an extension of the framework, called differential adaptive stress testing (DAST), that can find failures that occur in one system but not in another. This type of differential analysis is useful in applications such as regression testing, where we are concerned with finding areas of relative weakness compared to a baseline. We demonstrate the effectiveness of the approach on an aircraft collision avoidance application, where a prototype aircraft collision avoidance system is stress tested to find the most likely scenarios of near mid-air collision.

Verification and Validation↗