Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “software verification and validation”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 73 records · Page 4
All sourcesNASA NTRSDOE OSTI
NASA NTRS1985

A practical experience with independent verification and validation

One approach to reducing software cost and increasing reliability is the use of an independent verification and validation (IV & V) methodology. The Software Engineering Laboratory (SEL) applied the IV & V methodology to two medium-size flight dynamics software development projects. Then, to measure the effectiveness of the IV & V approach, the SEL compared these two projects with two similar past projects, using measures like productivity, reliability, and maintain ablilty. Results indicated that the use of the IV & V methodology did not help the overall process nor improve the product in these cases.

Page, Gerald↗
NASA NTRS2018

LADEE Simulation for Mission Operations

The Lunar Atmosphere Dust Environment Explorer (LADEE) model-based spacecraft simulator has been discussed previously at the Workshop on Spacecraft Flight Software including the verification and validation of the flight software, hardware integration of payloads, and multi-domain simulation. In addition to flight software development and testing the LADEE simulator was used by Mission Operations to develop and test spacecraft command scripts, train operators during mission simulations, verification of all tactical command sequence files uploaded to spacecraft during flight.This presentation will discuss the experience of the LADEE operations team using the spacecraft simulator including implementation, processes and lessons learned. We will also discuss a specific instance where the simulator was used in operations to debug and design a software fix for a spacecraft anomaly experienced with the star tracker.

Operations↗
NASA NTRS2023

Adaptive Independent Verification and Validation (IV&V) Reduces Risk of Software Impacting Safety in Artemis Missions

The National Aeronautics and Space Administration (NASA) is asking more of its human spaceflight programs than ever before through the collective Artemis Missions. The NASA Independent Verification and Validation (IV&V) Program contributes to NASA’s human spaceflight goals by providing IV&V services for NASA’s critical spacecraft and ground software. The IV&V Program is tasked with providing assurance from both individual and integrated mission software perspectives. The Artemis IV&V organization is actively supporting six distinct development efforts: Orion, the Space Launch System (SLS), Exploration Ground Systems (EGS), Mission Control Center (MCC), the Lunar Gateway, and the Human Landing System (HLS), representing a wide diversity of developer organizations, management structures, and development approaches. With much of this extremely complex flight and ground software being essential to human safety both on the ground and in space, Artemis IV&V is likewise challenged to provide more value-added assurance to future Artemis missions within a constrained budget. To meet this challenge, Artemis IV&V employs a variety of novel and evolving “Adaptive IV&V” approaches for planning and executing IV&V analysis to increase both the efficiency and effectiveness of the IV&V Program’s assurance activities, and to address the difficulties imposed by assuring software for a large, highly integrated, multi-mission enterprise managed and executed by physically and organizationally distinct programs. Instilling agile principles like iterative planning cycles, self-organizing teams, and regular retrospectives, into IV&V planning and execution has led to a more rapid turnaround of a minimum viable assurance product and allowed for increased alignment of assurance activities with development progress. Adopting an assurance case methodology has led to greater consistency and clearer communication of assurance design and provided a foundation for long-term maintenance of assurance plans, products, and results across missions. The IV&V-developed Assurance / Safety Case Analytical Network (A-SCAN) framework and tool has enabled the quantification and tracking of system/software risk and confidence. These confidence measures provide a means to repeatedly express the impact of planned and completed assurance work and the remaining residual risk. Applied as part of a “Follow-the-Risk” organizational ethos, this allows consistent rightsizing of analysis rigor and intensity commensurate with the perceived risk of defects, as well as appropriate targeting of the highest risk areas of the software to find safety issues before they can manifest. Finally, the development of the IV&V Advanced Risk Reduction Integrated Software Test and Operations Tri-program Lightweight Environment (ARRISTOTLE), an integrated software-only simulation of Orion, SLS, and EGS systems, has made it possible to independently test integrated pad and flight scenarios and inject faults to observe how the Artemis multi-program, mission software behaves in degraded modes and in response to hazards. These adaptive IV&V investments have enabled Artemis IV&V to become more efficient and effective in IV&V planning and execution and respond more readily to changes in the risk landscape, increasing the breadth and depth of risk reduction possible within the available resources. Residual risk tracking allows IV&V to communicate more effectively with stakeholders, both internal and external at all levels, and inform key decision-making personnel. This evolving assurance design approach provides IV&V surety that work is performed in the highest risk, most value-added areas of the software, to keep our astronauts and ground crews safe and ensure mission success.

Gerek A Whitman↗
NASA NTRS2023

Adaptive Independent Verification and Validation (IV&V) Reduces Risk of Software Impacting Safety in Artemis Missions

The National Aeronautics and Space Administration (NASA) is asking more of its human spaceflight programs than ever before through the collective Artemis Missions. The NASA Independent Verification and Validation (IV&V) Program contributes to NASA’s human spaceflight goals by providing IV&V services for NASA’s critical spacecraft and ground software. The IV&V Program is tasked with providing assurance from both individual and integrated mission software perspectives. The Artemis IV&V organization is actively supporting six distinct development efforts: Orion, the Space Launch System (SLS), Exploration Ground Systems (EGS), Mission Control Center (MCC), the Lunar Gateway, and the Human Landing System (HLS), representing a wide diversity of developer organizations, management structures, and development approaches. With much of this extremely complex flight and ground software being essential to human safety both on the ground and in space, Artemis IV&V is likewise challenged to provide more value-added assurance to future Artemis missions within a constrained budget. To meet this challenge, Artemis IV&V employs a variety of novel and evolving “Adaptive IV&V” approaches for planning and executing IV&V analysis to increase both the efficiency and effectiveness of the IV&V Program’s assurance activities, and to address the difficulties imposed by assuring software for a large, highly integrated, multi-mission enterprise managed and executed by physically and organizationally distinct programs. Instilling agile principles like iterative planning cycles, self-organizing teams, and regular retrospectives, into IV&V planning and execution has led to a more rapid turnaround of a minimum viable assurance product and allowed for increased alignment of assurance activities with development progress. Adopting an assurance case methodology has led to greater consistency and clearer communication of assurance design and provided a foundation for long-term maintenance of assurance plans, products, and results across missions. The IV&V-developed Assurance / Safety Case Analytical Network (A-SCAN) framework and tool has enabled the quantification and tracking of system/software risk and confidence. These confidence measures provide a means to repeatedly express the impact of planned and completed assurance work and the remaining residual risk. Applied as part of a “Follow-the-Risk” organizational ethos, this allows consistent rightsizing of analysis rigor and intensity commensurate with the perceived risk of defects, as well as appropriate targeting of the highest risk areas of the software to find safety issues before they can manifest. Finally, the development of the IV&V Advanced Risk Reduction Integrated Software Test and Operations Tri-program Lightweight Environment (ARRISTOTLE), an integrated software-only simulation of Orion, SLS, and EGS systems, has made it possible to independently test integrated pad and flight scenarios and inject faults to observe how the Artemis multi-program, mission software behaves in degraded modes and in response to hazards. These adaptive IV&V investments have enabled Artemis IV&V to become more efficient and effective in IV&V planning and execution and respond more readily to changes in the risk landscape, increasing the breadth and depth of risk reduction possible within the available resources. Residual risk tracking allows IV&V to communicate more effectively with stakeholders, both internal and external at all levels, and inform key decision-making personnel. This evolving assurance design approach provides IV&V surety that work is performed in the highest risk, most value-added areas of the software, to keep our astronauts and ground crews safe and ensure mission success.

Gerek Whitman↗
NASA NTRS1998

Towards a Theory for Integration of Mathematical Verification and Empirical Testing

From the viewpoint of a project manager responsible for the V&V (verification and validation) of a software system, mathematical verification techniques provide a possibly useful orthogonal dimension to otherwise standard empirical testing. However, the value they add to an empirical testing regime both in terms of coverage and in fault detection has been difficult to quantify. Furthermore, potential cost savings from replacing testing with mathematical verification techniques cannot be realized until the tradeoffs and synergies can be formulated. Integration of formal verification with empirical testing is also difficult because the idealized view of mathematical verification providing a correctness proof with total coverage is unrealistic and does not reflect the limitations imposed by computational complexity of mathematical techniques. This paper first describes a framework based on software reliability and formalized fault models for a theory of software design fault detection - and hence the utility of various tools for debugging. It then describes a utility model for integrating mathematical and empirical techniques with respect to fault detection and coverage analysis. It then considers the optimal combination of black-box testing, white-box (structural) testing, and formal methods in V&V of a software system. Using case studies from NASA software systems, it then demonstrates how this utility model can be used in practice.

Lowry, Michael↗
NASA NTRS1996

The Application of V&V within Reuse-Based Software Engineering

Verification and Validation (V&V) is performed during application development for many systems, especially safety-critical and mission-critical systems. The V&V process is intended to discover errors as early as possible during the development process. Early discovery is important in order to minimize the cost and other impacts of correcting these errors. In reuse-based software engineering, decisions on the requirements, design and even implementation of domain assets can can be made prior to beginning development of a specific system. in order to bring the effectiveness of V&V to bear within reuse-based software engineering. V&V must be incorporated within the domain engineering process.

Addy, Edward↗
NASA NTRSDate not supplied

Analysis of the Value Added When Deploying a Model-Based Approach for the Validation and Verification of the Medical Database Software

The Medical Database (MD) is a virtual repository consisting of two software components: Medical Item Database (MedID) and the Evidence Library (EL). MedID consists of engineering data and associated information for specific medical resource items (e.g., pharmaceutical, medical devices, and supporting components), while the EL is a tool which provides all of the medical evidence necessary. The MD will 1) serve as the single “source of truth” for the Informing Mission Planning via Analysis of Complex Tradespaces (IMPACT) tool suite for both medical evidence and medical resource engineering data and 2) will be used in conjunction with the IMPACT tool suite to inform research prioritizations and perform systematic trade study evaluations to aid stakeholders in making informed decisions regarding simulated human spaceflight missions. The MD project used a Model-Based Systems Engineering (MBSE) approach to support all life cycles of the software development, while in parallel the human factors engineering team used modeling to support Human Centered Design (HCD) strategies in an effort to improve software usability. HCD is a frequently used approach in design frameworks that develops resolutions to complexities and challenges by involving the human perspective in all steps of the problem-solving process. By integrating the model-based approaches used for systems engineering and human factors activities, the project is able to leverage the model-based artifacts originally created for HCD activities for system level and human factors validation. In this presentation, our team highlights the value added when leveraging these model-based artifacts to support the on-going verification and validation activities.

C. Laing↗
NASA NTRS1996

A Framework for Performing V&V within Reuse-Based Software Engineering

Verification and validation (V&V) is performed during application development for many systems, especially safety-critical and mission-critical systems. The V&V process is intended to discover errors, especially errors related to critical processing, as early as possible during the development process. Early discovery is important in order to minimize the cost and other impacts of correcting these errors. In order to provide early detection of errors, V&V is conducted in parallel with system development, often beginning with the concept phase. In reuse-based software engineering, however, decisions on the requirements, design and even implementation of domain assets can be made prior to beginning development of a specific system. In this case, V&V must be performed during domain engineering in order to have an impact on system development. This paper describes a framework for performing V&V within architecture-centric, reuse-based software engineering. This framework includes the activities of traditional application-level V&V, and extends these activities into domain engineering and into the transition between domain engineering and application engineering. The framework includes descriptions of the types of activities to be performed during each of the life-cycle phases, and provides motivation for the activities.

Addy, Edward A.↗
NASA NTRSDate not supplied

Emulation of Core Flight System Applications for Flight Software Development and Validation

The Mars Sample Return (MSR) campaign is an unprecedented attempt in the return of Martian samples back to Earth. The ascent from the surface will be performed by the Mars Ascent Vehicle (MAV), a critical element in the mission that National Aeronautics and Space Administration (NASA) Marshall Space Flight Center (MSFC) is developing. To this end, innovations in flight software development, verification, and validation are occurring. The MAV flight computer will run Core Flight System (cFS), an open-source software environment developed by NASA Goddard Space Flight Center (GSFC). NASA Marshall’s MAV Mission and Fault Management (M&FM) Team has implemented an emulation of two applications of this architecture: Limit Checker and Stored Command. Using an emulation of the functionalities of these applications allows for rapid prototyping of table-based algorithms. Further, M&FM is leveraging an in-house, low-fidelity but high-throughput State Analysis Model (SAM), an integrated MATLAB Stateflow Plant and Software model. This model is run in parallel with the cFS emulation for full flyout testing of the M&FM algorithms, verification of intent of these algorithms, and for future auto-generation of application-ingestible M&FM tables. The tables can then be delivered to the MAV Flight Software (FSW) team in a seamless process, reducing the cost of traditional FSW development and the risk of starting M&FM FSW development at later points in the NASA program life cycle.

Cody Wheeler↗
NASA NTRSDate not supplied

Emulation of Core Flight System Applications for Flight Software Development and Validation

The Mars Sample Return (MSR) campaign is an unprecedented attempt in the return of Martian samples back to Earth. The ascent from the surface will be performed by the Mars Ascent Vehicle (MAV), a critical element in the mission that National Aeronautics and Space Administration (NASA) Marshall Space Flight Center (MSFC) is developing. To this end, innovations in flight software development, verification, and validation are occurring. The MAV flight computer will run Core Flight System (cFS), an open-source software environment developed by NASA Goddard Space Flight Center (GSFC). NASA Marshall’s MAV Mission and Fault Management (M&FM) Team has implemented an emulation of two applications of this architecture: Limit Checker and Stored Command. Using an emulation of the functionalities of these applications allows for rapid prototyping of table-based algorithms. Further, M&FM is leveraging an in-house, low-fidelity but high-throughput State Analysis Model (SAM), an integrated MATLAB Stateflow Plant and Software model. This model is run in parallel with the cFS emulation for full flyout testing of the M&FM algorithms, verification of intent of these algorithms, and for future auto-generation of application-ingestible M&FM tables. The tables can then be delivered to the MAV Flight Software (FSW) team in a seamless process, reducing the cost of traditional FSW development and the risk of starting M&FM FSW development at later points in the NASA program life cycle.

Cody Wheeler↗
NASA NTRS2013

Validation and Verification of LADEE Models and Software

The Lunar Atmosphere Dust Environment Explorer (LADEE) mission will orbit the moon in order to measure the density, composition and time variability of the lunar dust environment. The ground-side and onboard flight software for the mission is being developed using a Model-Based Software methodology. In this technique, models of the spacecraft and flight software are developed in a graphical dynamics modeling package. Flight Software requirements are prototyped and refined using the simulated models. After the model is shown to work as desired in this simulation framework, C-code software is automatically generated from the models. The generated software is then tested in real time Processor-in-the-Loop and Hardware-in-the-Loop test beds. Travelling Road Show test beds were used for early integration tests with payloads and other subsystems. Traditional techniques for verifying computational sciences models are used to characterize the spacecraft simulation. A lightweight set of formal methods analysis, static analysis, formal inspection and code coverage analyses are utilized to further reduce defects in the onboard flight software artifacts. These techniques are applied early and often in the development process, iteratively increasing the capabilities of the software and the fidelity of the vehicle models and test beds.

Gundy-Burlet, Karen↗
NASA NTRS2020

Hardware Verification and Validation for a Navigation Sensor Software Model in Support of Flight Vehicle Performance Analysis

… or, “It’s in the details, how to make complicated software perform like complicated hardware.” In attempts to minimize development time and quickly build an operational vehicle, NASA’s Space Launch System (SLS) has had to be intentional about integrated testing. Constraints on budget and schedule have required balance between testing needs and the desire for an integrated flight vehicle as soon as possible. To provide key insights early in design and analysis cycles, a large amount of effort has shifted into maturing and validating models at the component level with integrated testing as a means to validate their integration. In terms of SLS Navigation, this, and the model-based design approach have pushed explicit requirements for sensor models to be validated against flight hardware to high precision. This paper covers the approach taken to verify and validate the models for the two key navigation sensors on the SLS vehicle, the Redundant Inertial Navigation Sensor and the Rate Gyro Assembly. These models are used in performance evaluation, fault detection, and operations development extensively. Using a mix of data from hardware vendor documentation and testing reports, limited in-house testing, and integration activities, these models were able to be validated against flight hardware at multiple levels, from the internal software design to statistical behavior at the raw sensor and integrated box levels. The high level of insight into the hardware elements is instrumental to support flight certification activities and building confidence in SLS Navigation capability. Focused testing enabled additional insight and validation that proved invaluable and the resulting insights were used to focus and mature models. Additionally, of having validated performance-based hardware models enables a wide breadth of activities including detailed fault detection studies and integration into future vehicle frameworks, such as an upper stage and provide a valuable asset to continued SLS analysis and design.

Evan J Anzalone↗
NASA NTRS2020

Hardware Verification and Validation for a Navigation Sensor Software Model in Support of Flight Vehicle Performance Analysis

… or, “It’s in the details, how to make complicated software perform like complicated hardware.” In attempts to minimize development time and quickly build an operational vehicle, NASA’s Space Launch System (SLS) has had to be intentional about integrated testing. Constraints on budget and schedule have required balance between testing needs and the desire for an integrated flight vehicle as soon as possible. To provide key insights early in design and analysis cycles, a large amount of effort has shifted into maturing and validating models at the component level with integrated testing as a means to validate their integration. In terms of SLS Navigation, this, and the model-based design approach have pushed explicit requirements for sensor models to be validated against flight hardware to high precision. This paper covers the approach taken to verify and validate the models for the two key navigation sensors on the SLS vehicle, the Redundant Inertial Navigation Sensor and the Rate Gyro Assembly. These models are used in performance evaluation, fault detection, and operations development extensively. Using a mix of data from hardware vendor documentation and testing reports, limited in-house testing, and integration activities, these models were able to be validated against flight hardware at multiple levels, from the internal software design to statistical behavior at the raw sensor and integrated box levels. The high level of insight into the hardware elements is instrumental to support flight certification activities and building confidence in SLS Navigation capability. Focused testing enabled additional insight and validation that proved invaluable and the resulting insights were used to focus and mature models. Additionally, of having validated performance-based hardware models enables a wide breadth of activities including detailed fault detection studies and integration into future vehicle frameworks, such as an upper stage and provide a valuable asset to continued SLS analysis and design.

Thomas Park↗
NASA NTRS2006

Partial Automation of Requirements Tracing

Requirements Tracing on Target (RETRO) is software for after-the-fact tracing of textual requirements to support independent verification and validation of software. RETRO applies one of three user-selectable information-retrieval techniques: (1) term frequency/inverse document frequency (TF/IDF) vector retrieval, (2) TF/IDF vector retrieval with simple thesaurus, or (3) keyword extraction. One component of RETRO is the graphical user interface (GUI) for use in initiating a requirements-tracing project (a pair of artifacts to be traced to each other, such as a requirements spec and a design spec). Once the artifacts have been specified and the IR technique chosen, another component constructs a representation of the artifact elements and stores it on disk. Next, the IR technique is used to produce a first list of candidate links (potential matches between the two artifact levels). This list, encoded in Extensible Markup Language (XML), is optionally processed by a filtering component designed to make the list somewhat smaller without sacrificing accuracy. Through the GUI, the user examines a number of links and returns decisions (yes, these are links; no, these are not links). Coded in XML, these decisions are provided to a "feedback processor" component that prepares the data for the next application of the IR technique. The feedback reduces the incidence of erroneous candidate links. Unlike related prior software, RETRO does not require the user to assign keywords, and automatically builds a document index.

Hayes, Jane↗
NASA NTRS1993

An assessment of space shuttle flight software development processes

In early 1991, the National Aeronautics and Space Administration's (NASA's) Office of Space Flight commissioned the Aeronautics and Space Engineering Board (ASEB) of the National Research Council (NRC) to investigate the adequacy of the current process by which NASA develops and verifies changes and updates to the Space Shuttle flight software. The Committee for Review of Oversight Mechanisms for Space Shuttle Flight Software Processes was convened in Jan. 1992 to accomplish the following tasks: (1) review the entire flight software development process from the initial requirements definition phase to final implementation, including object code build and final machine loading; (2) review and critique NASA's independent verification and validation process and mechanisms, including NASA's established software development and testing standards; (3) determine the acceptability and adequacy of the complete flight software development process, including the embedded validation and verification processes through comparison with (1) generally accepted industry practices, and (2) generally accepted Department of Defense and/or other government practices (comparing NASA's program with organizations and projects having similar volumes of software development, software maturity, complexity, criticality, lines of code, and national standards); (4) consider whether independent verification and validation should continue. An overview of the study, independent verification and validation of critical software, and the Space Shuttle flight software development process are addressed. Findings and recommendations are presented.

Source record↗