Results for “VERIFICATION” — Page 65

NASA NTRS2020

Parsing-based Approaches for Verification and Recognition of Hierarchical Plans

Hierarchical Task Networks were proposed as a method todescribe plans by decomposition of tasks to sub-tasks untilprimitive tasks, actions, are obtained. Plan verification assumesa complete plan as input, and the objective is findinga task that decomposes to this plan. In plan recognition, aprefix of the plan is given and the objective is finding a taskthat decomposes to the (shortest) plan with the given prefix.This paper describes how to verify and recognize plans usinga common method known from formal grammars, by parsing.

Bartak, Roman↗

NASA NTRS2020

A Novel Parsing-based Approach for Verification of Hierarchical Plans

Hierarchical Task Networks were proposed as amethod to describe plans by decomposition of tasks to subtasksuntil primitive tasks, actions, are obtained. Valid plans –sequences of actions – must adhere both to causal dependenciesbetween the actions and to the structure given by the decompositionof the goal task. Plan verification aims at finding if a givenplan is valid, that is, if it is causally consistent and it can beobtained by decomposition of some task. The paper describes anovel parsing-based approach for hierarchical plan verificationthat is orders of magnitude faster than existing methods.

Bercher, Pascal↗

NASA NTRS2020

Actor-based Runtime Verification with MESA

This work presents a runtime verification approach implemented in the tool MESA (MEssage-based System Analysis) which allows for using concurrent monitors to check for properties specified in data parameterized temporal logic and state machines. The tool is implemented as an internal Scala DSL. We employ the actor programming model to implement MESA where monitors are captured by concurrent actors that communicate via messaging. The paper presents a case study in which MESA is used to effectively monitor a large number of flights from live US airspace data streams. We also perform an empirical study by conducting experiments using monitoring systems with different num- bers of concurrent monitors and different layers of indexing on the data contained in events. The paper describes the experiments, evaluates the results, and discusses challenges faced during the study. The evaluation shows the value of combining concurrency with indexing to handle data rich events.

Mehlitz, Peter↗

NASA NTRS2022

NASA Orbital Debris Engineering Model (ORDEM) 3.1: Model Verification and Validation

The NASA Orbital Debris Engineering Model (ORDEM) 3.1 Model Verification and Validation (V&V) document accompanies the delivery of the latest ORDEM 3.1 model (Vavrin & Manis, 2019) and provides a detailed description of the V&V activities used to verify that the model was built correctly and validate the model against independent, real world sources of data obtained from sampling the orbital debris environment. This ORDEM 3.1 Model V&V document, along with the related ORDEM 3.1 Model Process document – which covers details of the mathematical, statistical, and physical basis of the model – are intended to inform credibility assessments, risk analyses, uncertainty characterizations, and other applications derived from use of the model by the ORDEM 3.1 user community.

Timothy Kennedy↗

NASA NTRS2021

Europa Clipper Magnetometer Thermal Design Evolution, Validation, and Verification

Europa Clipper Spacecraft is exposed to a high radiation environment and is limited with power. This makes the thermal design of Europa Clipper magnetometer more challenging. The design is required to be highly insulated and at the same time magnetically clean. This paper describes the thermal design evolution of Europa Clipper Magnetometer. A thermal vacuum test is conducted in Europa Clipper orbit’s relevant environment to characterize the requirements and estimate the power consumption. A flight-like engineering model magnetometer from the University of California at Los Angeles (UCLA) is used for this test. Magnetically clean Cernox temperature measurement sensors with the accuracy of ±0.1ºC are used to measure the temperature value inside the sensor head and across the main frame interfaces. This level of accuracy in temperature measurement is required to capture the temperature gradient across the sensor head in a relevant environment which impacts the magnetometer measurement accuracy. Test data is used to correlate the thermal model and characterize the heat leak across the interfaces. Validated thermal model is used as a verification tool for power consumption and thermal requirements.

Caron, Ryan↗

NASA NTRSDate not supplied

Analysis of the Value Added When Deploying a Model-Based Approach for the Validation and Verification of the Medical Database Software

The Medical Database (MD) is a virtual repository consisting of two software components: Medical Item Database (MedID) and the Evidence Library (EL). MedID consists of engineering data and associated information for specific medical resource items (e.g., pharmaceutical, medical devices, and supporting components), while the EL is a tool which provides all of the medical evidence necessary. The MD will 1) serve as the single “source of truth” for the Informing Mission Planning via Analysis of Complex Tradespaces (IMPACT) tool suite for both medical evidence and medical resource engineering data and 2) will be used in conjunction with the IMPACT tool suite to inform research prioritizations and perform systematic trade study evaluations to aid stakeholders in making informed decisions regarding simulated human spaceflight missions. The MD project used a Model-Based Systems Engineering (MBSE) approach to support all life cycles of the software development, while in parallel the human factors engineering team used modeling to support Human Centered Design (HCD) strategies in an effort to improve software usability. HCD is a frequently used approach in design frameworks that develops resolutions to complexities and challenges by involving the human perspective in all steps of the problem-solving process. By integrating the model-based approaches used for systems engineering and human factors activities, the project is able to leverage the model-based artifacts originally created for HCD activities for system level and human factors validation. In this presentation, our team highlights the value added when leveraging these model-based artifacts to support the on-going verification and validation activities.

C. Laing↗

NASA NTRS2023

Adaptive Independent Verification and Validation (IV&V) Reduces Risk of Software Impacting Safety in Artemis Missions

The National Aeronautics and Space Administration (NASA) is asking more of its human spaceflight programs than ever before through the collective Artemis Missions. The NASA Independent Verification and Validation (IV&V) Program contributes to NASA’s human spaceflight goals by providing IV&V services for NASA’s critical spacecraft and ground software. The IV&V Program is tasked with providing assurance from both individual and integrated mission software perspectives. The Artemis IV&V organization is actively supporting six distinct development efforts: Orion, the Space Launch System (SLS), Exploration Ground Systems (EGS), Mission Control Center (MCC), the Lunar Gateway, and the Human Landing System (HLS), representing a wide diversity of developer organizations, management structures, and development approaches. With much of this extremely complex flight and ground software being essential to human safety both on the ground and in space, Artemis IV&V is likewise challenged to provide more value-added assurance to future Artemis missions within a constrained budget. To meet this challenge, Artemis IV&V employs a variety of novel and evolving “Adaptive IV&V” approaches for planning and executing IV&V analysis to increase both the efficiency and effectiveness of the IV&V Program’s assurance activities, and to address the difficulties imposed by assuring software for a large, highly integrated, multi-mission enterprise managed and executed by physically and organizationally distinct programs. Instilling agile principles like iterative planning cycles, self-organizing teams, and regular retrospectives, into IV&V planning and execution has led to a more rapid turnaround of a minimum viable assurance product and allowed for increased alignment of assurance activities with development progress. Adopting an assurance case methodology has led to greater consistency and clearer communication of assurance design and provided a foundation for long-term maintenance of assurance plans, products, and results across missions. The IV&V-developed Assurance / Safety Case Analytical Network (A-SCAN) framework and tool has enabled the quantification and tracking of system/software risk and confidence. These confidence measures provide a means to repeatedly express the impact of planned and completed assurance work and the remaining residual risk. Applied as part of a “Follow-the-Risk” organizational ethos, this allows consistent rightsizing of analysis rigor and intensity commensurate with the perceived risk of defects, as well as appropriate targeting of the highest risk areas of the software to find safety issues before they can manifest. Finally, the development of the IV&V Advanced Risk Reduction Integrated Software Test and Operations Tri-program Lightweight Environment (ARRISTOTLE), an integrated software-only simulation of Orion, SLS, and EGS systems, has made it possible to independently test integrated pad and flight scenarios and inject faults to observe how the Artemis multi-program, mission software behaves in degraded modes and in response to hazards. These adaptive IV&V investments have enabled Artemis IV&V to become more efficient and effective in IV&V planning and execution and respond more readily to changes in the risk landscape, increasing the breadth and depth of risk reduction possible within the available resources. Residual risk tracking allows IV&V to communicate more effectively with stakeholders, both internal and external at all levels, and inform key decision-making personnel. This evolving assurance design approach provides IV&V surety that work is performed in the highest risk, most value-added areas of the software, to keep our astronauts and ground crews safe and ensure mission success.

Gerek A Whitman↗

NASA NTRS2021

Maneuver Design Implementation and Verification for the Mars 2020 Mission

On February 18, 2021, the Perseverance rover landed in Jezero Crater on Mars, transported by the Mars 2020 spacecraft on a nearly seven-month journey to the Red Planet. The execution of three propulsive maneuvers during the interplanetary cruise phase was required to remove the launch-injection bias and deliver the spacecraft to the Mars atmospheric entry point. This paper focuses on the maneuver implementation and verification process between the navigation and spacecraft teams. Additionally, this paper discusses the execution error models that were used to determine maneuver performance and delivery accuracy at the atmospheric entry interface point.

Kruizinga, Gerhard↗

NASA NTRS2022

Visual Odometry Thinking While Driving for the Curiosity Mars Rover's Three-Year Test Campaign: Impact of Evolving Constraints on Verification and Validation

Over the first 9 years of the Mars Science Laboratory (MSL) Curiosity rover's surface mission, more than 87\% of its driving has been performed using Visual Odometry (VO). The benefits of using VO during driving are that it minimizes rover position uncertainty and can be used to monitor wheel slip, halting a drive if excessive wheel slip is occurring. The VO implementation on board Curiosity acquires and processes VO images in between drive steps while the rover is stationary. A VO Thinking While Driving (VTWD) flight software capability has been developed which enables the processing of VO images during rover driving, increasing the distance Curiosity can drive with VO in a given time period up to as much as 1.75x total distance. Verification and Validation (V\&V) of the capability has been challenging due to impacts from the COVID-19 pandemic and unavailability of the JPL Mars Yard outdoor test site. The VTWD V\&V test procedures were modified to use a small indoor space with Mars-like terrain. This paper describes the 3 year V\&V effort under challenging conditions to approve the VTWD capability for use on the Curiosity rover.

Rankin, Arturo↗

NASA NTRS2022

The SWOT Mission Environmental Verification Program

The objective of the Surface Water and Ocean Topography (SWOT) Mission is to make global measurements of sea surface and terrestrial water heights. The SWOT Observatory consists of the Spacecraft Bus, supplied by CNES, and the Payload Module, provided by JPL. The Spacecraft Bus is a carrier of the Payload Module, which includes all the science instruments. The Spacecraft Bus components have been environmentally verified by CNES and the Payload Module components verified by JPL. The Payload Module was delivered to CNES in June 2021 for integration with the Spacecraft Bus to form the Observatory, which is in the process of undergoing a full set of environmental tests before launch. This paper focuses on the environmental test and analysis program at the assembly (unit), subsystem, and module level for the Payload Module hardware that JPL is responsible for verifying. Even though an Earth-orbiting mission like SWOT is not considered to be at extreme environments, SWOT’s environmental verification program does illustrate the standard System Integration and Test (SIT) process for hardware integration and environmental testing to ensure all hardware have been properly qualified to meet all mission environments. Typical space environmental test and analysis campaign includes acoustics vibration, shock, thermal vacuum, venting/pressure, radiation, electrostatic discharge, EMC/EMI/magnetics, and meteoroids. Environmental test and analysis metrics are presented in this paper. This paper also shows the complex integration process that involves multi-organizations and the latest status on the Observatory environmental test campaign.

Man, Kin F.↗

NASA NTRS2022

Robotics Verification and Validation Strategies for Perseverance Rover Sampling and Caching

The Mars 2020 Sampling and Caching Subsystem(SCS) is the most complex robotic system ever fielded on a MarsRover. It includes a 5 degree-of-freedom Robotic Arm, coringdrill, gas Dust Removal Tool, interfaces for two turret-mountedinstruments, and an Adaptive Caching Assembly (ACA). TheACA is itself a complex robotic system, containing hardware tosupport docking and bit exchange, a 3 degree-of-freedomSample Handling Assembly for manipulating sample tubes,storage for several drill bits and sample tubes, and mechanismsto support observing and sealing samples collected by the drill.To successfully verify and validate the SCS hardware andsoftware and its integration with the Mars 2020 flight systemseveral key strategies were employed.The SCS Verification and Validation (V&V) program utilizedmultiple test venues with tiered levels of fidelity. These includedsimulation and visualization software environments, low fidelitydevelopment testbeds, testbeds with high fidelity SCS hardwareand commercial off-the shelf avionics, integrated systemtestbeds with flight-like avionics, and environmental testbedscapable of simulating Martian surface temperature andpressure. Multiple units of each SCS hardware componentmoved fluidly between test venues to accomplish myriadstandalone and coordinated test objectives. Test preparationand executions were performed by a diverse team of engineerswith training and technical ownership tailored for individualexperience and role. Despite significant differences between testvenues, the SCS V&V team established efficient and consistentprocesses and tools for procedure development, test execution,and data review that enabled personnel, as well as technicalproducts such as sequences and parameter configurations, toflow between venues effectively. A series of benchmark testsprovided evidence of performance consistency as elements weretransferred between venues and as system capability evolved.This paper provides an overview of the SCS V&V program andexplores several overarching strategies that enabled successfuloperation in the face of unprecedented complexity. Keyoutcomes of the SCS validation effort are summarized, alongwith lessons learned and beneficial integrations of validationtool and process innovations into Mars surface operations.

Brooks, Sawyer↗

NASA NTRSDate not supplied

Assumption Generation for the Verification of Learning-Enabled Autonomous Systems

Providing safety guarantees for autonomous systems is difficultas these systems operate in complex environments that require the use of learning-enabled components, such as deep neural networks (DNNs) for visual perception. DNNs are hard to analyze due to their size (they can have thousands or millions of parameters), lack of formal specifications (DNNs are typically learnt from labeled data, in the absence of any formal or informal requirements), and sensitivity to small changes in the environment. We present an assume-guarantee style compositional approach for the formal verification of system-level safety properties of such autonomous systems. Our insight is that we can analyze the system in the absence of the DNN perception components by automatically synthesizing assumptions on the DNN behaviour that guarantee the satisfaction of the required safety properties. The synthesized assumptions are the weakest in the sense that they characterize the output sequences of all the possible DNNs that, plugged into the autonomous system, guarantee the required safety properties. The assumptions can be leveraged as run-time monitors over a deployed DNN to guarantee the safety of the overall system; they can also be mined to extract local specifications for use during training and testing of DNNs. We illustrate our approach on a case study taken from the autonomous airplanes domain that uses a complex DNN for perception

Autonomous systems↗

NASA NTRS2023

On-Board Deployment Event Verification for GOES-R Spacecraft

As is common with many spacecraft designs, the GOES-R vehicles require a series of deployment events to transition from the launch configuration to the operational configuration. Rather than implementing additional sensors to verify various deployments, the GOES-R program developed an alternate approach that uses existing gyro rate sensing. The approach includes two pieces: the first is a new onboard shock detection capability to confirm initiation of individual deployment events, and the second is a ground-based dynamics verification step to confirm completion of deployment events. We first present the algorithm that detects shock events for various deployment devices along with the parameter tuning performed during ground tests. We then show inflight performance of the shock detection algorithm. While shock detection is useful for observing initiation of deployment events, completion of some deployment events cannot be determined by shock detection alone. For these events, such as solar panel latch-up and deployable boom extension, the program developed dynamics models for the deployment transient responses. High-rate gyro data were recorded for these events, which allow the ground team to verify that the appendages were fully deployed. We show the predictive models for these events and corresponding flight results.

GeoXO↗

NASA NTRS2023

Adaptive Independent Verification and Validation (IV&V) Reduces Risk of Software Impacting Safety in Artemis Missions

The National Aeronautics and Space Administration (NASA) is asking more of its human spaceflight programs than ever before through the collective Artemis Missions. The NASA Independent Verification and Validation (IV&V) Program contributes to NASA’s human spaceflight goals by providing IV&V services for NASA’s critical spacecraft and ground software. The IV&V Program is tasked with providing assurance from both individual and integrated mission software perspectives. The Artemis IV&V organization is actively supporting six distinct development efforts: Orion, the Space Launch System (SLS), Exploration Ground Systems (EGS), Mission Control Center (MCC), the Lunar Gateway, and the Human Landing System (HLS), representing a wide diversity of developer organizations, management structures, and development approaches. With much of this extremely complex flight and ground software being essential to human safety both on the ground and in space, Artemis IV&V is likewise challenged to provide more value-added assurance to future Artemis missions within a constrained budget. To meet this challenge, Artemis IV&V employs a variety of novel and evolving “Adaptive IV&V” approaches for planning and executing IV&V analysis to increase both the efficiency and effectiveness of the IV&V Program’s assurance activities, and to address the difficulties imposed by assuring software for a large, highly integrated, multi-mission enterprise managed and executed by physically and organizationally distinct programs. Instilling agile principles like iterative planning cycles, self-organizing teams, and regular retrospectives, into IV&V planning and execution has led to a more rapid turnaround of a minimum viable assurance product and allowed for increased alignment of assurance activities with development progress. Adopting an assurance case methodology has led to greater consistency and clearer communication of assurance design and provided a foundation for long-term maintenance of assurance plans, products, and results across missions. The IV&V-developed Assurance / Safety Case Analytical Network (A-SCAN) framework and tool has enabled the quantification and tracking of system/software risk and confidence. These confidence measures provide a means to repeatedly express the impact of planned and completed assurance work and the remaining residual risk. Applied as part of a “Follow-the-Risk” organizational ethos, this allows consistent rightsizing of analysis rigor and intensity commensurate with the perceived risk of defects, as well as appropriate targeting of the highest risk areas of the software to find safety issues before they can manifest. Finally, the development of the IV&V Advanced Risk Reduction Integrated Software Test and Operations Tri-program Lightweight Environment (ARRISTOTLE), an integrated software-only simulation of Orion, SLS, and EGS systems, has made it possible to independently test integrated pad and flight scenarios and inject faults to observe how the Artemis multi-program, mission software behaves in degraded modes and in response to hazards. These adaptive IV&V investments have enabled Artemis IV&V to become more efficient and effective in IV&V planning and execution and respond more readily to changes in the risk landscape, increasing the breadth and depth of risk reduction possible within the available resources. Residual risk tracking allows IV&V to communicate more effectively with stakeholders, both internal and external at all levels, and inform key decision-making personnel. This evolving assurance design approach provides IV&V surety that work is performed in the highest risk, most value-added areas of the software, to keep our astronauts and ground crews safe and ensure mission success.

Gerek Whitman↗

NASA NTRS2023

The Design, Verification and Performance of the James Webb Space Telescope

The James Webb Space Telescope (JWST) is NASA’s flagship mission successor to the highly successful Hubble Space Telescope. It is an infrared observatory featuring a cryogenic 6.6 m aperture, deployable Optical Telescope Element (OTE) with a payload of four science instruments (SIs) assembled into an Integrated Science Instrument Module (ISIM) that provide imagery and spectroscopy in the near-infrared band between 0.6 and 5 μm and in the mid-infrared band between 5 and 28.1 μm. JWST was successfully launched on 2021 December 25 aboard an Ariane 5 launch vehicle. All 50 major deployments were successfully completed on 2022 January 8. The observatory performed all midcourse correction maneuvers and achieved its operational mission orbit around the Sun–Earth second Lagrange point (L2). All commissioning and calibration activities have been completed, and JWST has begun its science mission. This paper will provide a description of the driving requirements and their technical challenges, the engineering processes involved in the design formulation, the resulting observatory design, the verification programs that proved it to be flightworthy, and the measured on-orbit performance of the observatory. Since companion papers will describe the details of the OTE and SIs, this paper will concentrate on describing the key features of the observatory architecture that accommodates these elements, particularly those features and capabilities associated with accommodating the radiometric and image-quality performance.

Galaxy evolution↗

NASA NTRSDate not supplied

RST CGI: Final Verification and Calibration of Prism and Polarizer Flight Units

As part of its technology demonstration, the Nancy Grace Roman Space Telescope Coronagraph Instrument (CGI) will demonstrate point source spectroscopy and polarization measurements of disks. The spectroscopy mode is a zero-deviation Amici prism and slit which is deployed to the planet position after an imaging detection. The Wollaston prism polarization optics allow for imaging two orthogonal polarization states simultaneously. The CGI spectral characterization modes, designed and built at Goddard Space Flight Center (GSFC), have a spectral resolution of R50 in two 15% bandpass centered at 660nm and 730nm. There are duplicate Wollaston prism channels, clocked 45 degrees with respect to one another to completely characterize the instrument polarization effects and recover the Stokes parameters of the targeted disk. The Wollaston design and optical elements are a contribution by the Japanese Aerospace Exploration Agency, with final alignment and testing being done at GSFC. The spectroscopy mode is optimized to target Methane absorption features around 730nm, keeping the spectral resolution as low as possible to improve the signal-to-noise ratio and hence reduce detection time. We highlight the requirements for these modes, the ground-to-orbit calibration process, and the operations required to use a deployable slit on a CGI point source in the presence of pointing error. We also provide further detail on the optomechanical design, testing results from the final as-built flight units, verification process, and performance of the as-built flight assemblies.

high contrast imaging↗

NASA NTRSDate not supplied

Verification and Validation of Progressive Damage and Failure Analysis Methods for Intralaminar Failure Modes in Thermoplastic Composites

Current progressive damage and failure analysis (PDFA) methods were developed around thermoset material systems and their physical behavior. However, the applicability of these analysis methods has not been evaluated for use in modeling thermoplastic material systems. As part of the Hi-Rate Composite Aircraft Manufacturing (HiCAM) Project sponsored by NASA, intralaminar failure characterization was completed on a thermoplastic material system, AS4D/PEKK-FC, with off-axis tension (OAT) and off-axis compression (OAC) testing. The experiments were able to capture matrix non-linear shear behavior and the failure envelope of the material system. Two PDFA methods, LS-DYNA MAT299 and NASA’s CompDam-DGD, were then employed to simulate individual coupon behavior and simulation results were compared to published verification and validation guidelines from the CMH-17 crashworthiness working group. The results found that the two modeling methods provided suitable representation of material behavior of thermoplastics.

PDFA↗

NASA NTRSDate not supplied

Formal Verification, Distributed Computing, and Path Planning Algorithms

The safety- and mission-critical nature of much of the work done at NASA requires algorithms and software to be exceedingly reliable. Formal methods techniques are one way of ensuring this high level of robustness. This talk will discuss the development and formal verification of autonomous aircraft path planning algorithms related to the Bellman-Ford shortest path algorithm, including consideration of distributed computation of the algorithm.

Formal Methods↗