Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “verification and validation”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 127 records · Page 7
All sourcesNASA NTRSDOE OSTI
NASA NTRS2023

Adaptive Independent Verification and Validation (IV&V) Reduces Risk of Software Impacting Safety in Artemis Missions

The National Aeronautics and Space Administration (NASA) is asking more of its human spaceflight programs than ever before through the collective Artemis Missions. The NASA Independent Verification and Validation (IV&V) Program contributes to NASA’s human spaceflight goals by providing IV&V services for NASA’s critical spacecraft and ground software. The IV&V Program is tasked with providing assurance from both individual and integrated mission software perspectives. The Artemis IV&V organization is actively supporting six distinct development efforts: Orion, the Space Launch System (SLS), Exploration Ground Systems (EGS), Mission Control Center (MCC), the Lunar Gateway, and the Human Landing System (HLS), representing a wide diversity of developer organizations, management structures, and development approaches. With much of this extremely complex flight and ground software being essential to human safety both on the ground and in space, Artemis IV&V is likewise challenged to provide more value-added assurance to future Artemis missions within a constrained budget. To meet this challenge, Artemis IV&V employs a variety of novel and evolving “Adaptive IV&V” approaches for planning and executing IV&V analysis to increase both the efficiency and effectiveness of the IV&V Program’s assurance activities, and to address the difficulties imposed by assuring software for a large, highly integrated, multi-mission enterprise managed and executed by physically and organizationally distinct programs. Instilling agile principles like iterative planning cycles, self-organizing teams, and regular retrospectives, into IV&V planning and execution has led to a more rapid turnaround of a minimum viable assurance product and allowed for increased alignment of assurance activities with development progress. Adopting an assurance case methodology has led to greater consistency and clearer communication of assurance design and provided a foundation for long-term maintenance of assurance plans, products, and results across missions. The IV&V-developed Assurance / Safety Case Analytical Network (A-SCAN) framework and tool has enabled the quantification and tracking of system/software risk and confidence. These confidence measures provide a means to repeatedly express the impact of planned and completed assurance work and the remaining residual risk. Applied as part of a “Follow-the-Risk” organizational ethos, this allows consistent rightsizing of analysis rigor and intensity commensurate with the perceived risk of defects, as well as appropriate targeting of the highest risk areas of the software to find safety issues before they can manifest. Finally, the development of the IV&V Advanced Risk Reduction Integrated Software Test and Operations Tri-program Lightweight Environment (ARRISTOTLE), an integrated software-only simulation of Orion, SLS, and EGS systems, has made it possible to independently test integrated pad and flight scenarios and inject faults to observe how the Artemis multi-program, mission software behaves in degraded modes and in response to hazards. These adaptive IV&V investments have enabled Artemis IV&V to become more efficient and effective in IV&V planning and execution and respond more readily to changes in the risk landscape, increasing the breadth and depth of risk reduction possible within the available resources. Residual risk tracking allows IV&V to communicate more effectively with stakeholders, both internal and external at all levels, and inform key decision-making personnel. This evolving assurance design approach provides IV&V surety that work is performed in the highest risk, most value-added areas of the software, to keep our astronauts and ground crews safe and ensure mission success.

Gerek Whitman↗
NASA NTRS1992

Independent verification and validation for Space Shuttle flight software

The Committee for Review of Oversight Mechanisms for Space Shuttle Software was asked by the National Aeronautics and Space Administration's (NASA) Office of Space Flight to determine the need to continue independent verification and validation (IV&V) for Space Shuttle flight software. The Committee found that the current IV&V process is necessary to maintain NASA's stringent safety and quality requirements for man-rated vehicles. Therefore, the Committee does not support NASA's plan to eliminate funding for the IV&V effort in fiscal year 1993. The Committee believes that the Space Shuttle software development process is not adequate without IV&V and that elimination of IV&V as currently practiced will adversely affect the overall quality and safety of the software, both now and in the future. Furthermore, the Committee was told that no organization within NASA has the expertise or the manpower to replace the current IV&V function in a timely fashion, nor will building this expertise elsewhere necessarily reduce cost. Thus, the Committee does not recommend moving IV&V functions to other organizations within NASA unless the current IV&V is maintained for as long as it takes to build comparable expertise in the replacing organization.

Source record↗
NASA NTRS1995

Verification and Validation of Simulation Models for High-Speed Flow Fields

This paper is largely based on the paper titled "Guide to Credible Computational Fluid Dynamics Simulations (Invited)" (AIAA Paper 95-2225). The significance of computational fluid dynamics (CFD) simulations depends solely on their credibility. A customer of CFD products -- simulations and software -- expects that these products are credible for the intended use. Simulation model verification and validation are critical in establishing the credibility of simulations and in certifying simulation software. The only thing that matters in establishing the credibility is uncertainty, not veracity or validity. The sensitivity-uncertainty analysis is the key to the establishment of this credibility. Assessing the credibility of complex simulation results poses a significant challenge. Terminology, concepts, framework, and guidelines are presented for addressing this challenge. Verification assesses whether the problem is solved correctly and estimates the level of computational accuracy of simulations; validation determines whether the right problem is solved and assesses the level of validity of the simulation model by estimating the degree to which simulations accurately represent reality. These concepts and the related guidelines are discussed with examples from high-speed flow fields.

Mehta, Unmeel B.↗
NASA NTRS1999

Performing Verification and Validation in Reuse-Based Software Engineering

The implementation of reuse-based software engineering not only introduces new activities to the software development process, such as domain analysis and domain modeling, it also impacts other aspects of software engineering. Other areas of software engineering that are affected include Configuration Management, Testing, Quality Control, and Verification and Validation (V&V). Activities in each of these areas must be adapted to address the entire domain or product line rather than a specific application system. This paper discusses changes and enhancements to the V&V process, in order to adapt V&V to reuse-based software engineering.

Addy, Edward A.↗
NASA NTRS2000

Verification and Validation of Autonomy Software at NASA

Autonomous software holds the promise of new operation possibilities, easier design and development and lower operating costs. However, as those system close control loops and arbitrate resources on board with specialized reasoning, the range of possible situations becomes very large and uncontrollable from the outside, making conventional scenario-based testing very inefficient. Analytic verification and validation (V&V) techniques, and model checking in particular, can provide significant help for designing autonomous systems in a more efficient and reliable manner, by providing a better coverage and allowing early error detection. This article discusses the general issue of V&V of autonomy software, with an emphasis towards model-based autonomy, model-checking techniques and concrete experiments at NASA.

Pecheur, Charles↗
NASA NTRS2000

Verification and Validation of Autonomy Software at NASA

Autonomous software holds the promise of new operation possibilities, easier design and development, and lower operating costs. However, as those system close control loops and arbitrate resources on-board with specialized reasoning, the range of possible situations becomes very large and uncontrollable from the outside, making conventional scenario-based testing very inefficient. Analytic verification and validation (V&V) techniques, and model checking in particular, can provide significant help for designing autonomous systems in a more efficient and reliable manner, by providing a better coverage and allowing early error detection. This article discusses the general issue of V&V of autonomy software, with an emphasis towards model-based autonomy, model-checking techniques, and concrete experiments at NASA.

Pecheur, Charles↗
NASA NTRS1991

Translating expert system rules into Ada code with validation and verification

The purpose of this ongoing research and development program is to develop software tools which enable the rapid development, upgrading, and maintenance of embedded real-time artificial intelligence systems. The goals of this phase of the research were to investigate the feasibility of developing software tools which automatically translate expert system rules into Ada code and develop methods for performing validation and verification testing of the resultant expert system. A prototype system was demonstrated which automatically translated rules from an Air Force expert system was demonstrated which detected errors in the execution of the resultant system. The method and prototype tools for converting AI representations into Ada code by converting the rules into Ada code modules and then linking them with an Activation Framework based run-time environment to form an executable load module are discussed. This method is based upon the use of Evidence Flow Graphs which are a data flow representation for intelligent systems. The development of prototype test generation and evaluation software which was used to test the resultant code is discussed. This testing was performed automatically using Monte-Carlo techniques based upon a constraint based description of the required performance for the system.

Becker, Lee↗
NASA NTRS2022

Europa Clipper Payload Verification and Validation: Avionics-Instrument Interface Test Campaign

NASA's Europa Clipper mission will investigate Jupiter's icy moon Europa using a payload suite consisting of nine instruments to address a range of scientific objectives concerning Europa's habitability. As the project proceeds past its Critical Design Review, confidence is being built in the system's ability to achieve mission objectives through the implementation of a rigorous payload verification and validation (V&V) program. As part of this payload V&V program, instrument box-level testing was performed by the payload team to verify select instrument-avionics interface requirements. This testing was performed at JPL using the avionics testbed's Bulk Data Storage Emulator (BDSEM) with visiting instrument Test Models. This paper summarizes the Data Link test campaign involving roughly four days of functional testing per instrument, including planning, testing methods, types of issues found, and the requirement closure process. Detail is also provided on the development, deployment, and validation of a standardized analysis tool used in data reviews. This testing verified requirements related to commanding rates, loss of link, packet format, clock counters, loopback test capability, and SpaceWire jitter and skew margins. Additional risk reduction testing of basic commanding, counter behavior, science data collection and transfer, and interface swapping was also performed. Because the BDSEM venue was not originally designed to be a run for record venue, the process of characterizing venue fidelity and establishing suitability for requirement closure using data collected in this venue will also be addressed.In order to close requirements, an extensible tool was developed to post-process instrument command and telemetry data from their original binary to a human-readable format and give visibility to errors detected within the data, such as packets with Cyclic Redundancy Check errors. This tool, called payload-packet-parser, is a Python 3.9 command line tool built using a variety of open-source Python libraries. Payload-packet-parser was designed to support parsing command and telemetry packets for all Europa Clipper instruments and additional analysis tools were developed for verification of specific information interface requirements. This test campaign, including post-processing using a single parsing and verification toolset, allowed for early interface testing, alleviating testing burdens on instrument teams and buying down risk on the instrument-avionics interface by finding hardware and software issues and idiosyncrasies prior to integration with system test venues. Over twenty issues were discovered across the payload, resulting in software updates and instrument rework well in advance of any system impacts. This paper concludes with an assessment of benefits and costs of this type of testing and lessons learned.

Montanez, Leticia↗
NASA NTRS2013

Model Verification and Validation Concepts for a Probabilistic Fracture Assessment Model to Predict Cracking of Knife Edge Seals in the Space Shuttle Main Engine High Pressure Oxidizer

Physics-based models are routinely used to predict the performance of engineered systems to make decisions such as when to retire system components, how to extend the life of an aging system, or if a new design will be safe or available. Model verification and validation (V&V) is a process to establish credibility in model predictions. Ideally, carefully controlled validation experiments will be designed and performed to validate models or submodels. In reality, time and cost constraints limit experiments and even model development. This paper describes elements of model V&V during the development and application of a probabilistic fracture assessment model to predict cracking in space shuttle main engine high-pressure oxidizer turbopump knife-edge seals. The objective of this effort was to assess the probability of initiating and growing a crack to a specified failure length in specific flight units for different usage and inspection scenarios. The probabilistic fracture assessment model developed in this investigation combined a series of submodels describing the usage, temperature history, flutter tendencies, tooth stresses and numbers of cycles, fatigue cracking, nondestructive inspection, and finally the probability of failure. The analysis accounted for unit-to-unit variations in temperature, flutter limit state, flutter stress magnitude, and fatigue life properties. The investigation focused on the calculation of relative risk rather than absolute risk between the usage scenarios. Verification predictions were first performed for three units with known usage and cracking histories to establish credibility in the model predictions. Then, numerous predictions were performed for an assortment of operating units that had flown recently or that were projected for future flights. Calculations were performed using two NASA-developed software tools: NESSUS(Registered Trademark) for the probabilistic analysis, and NASGRO(Registered Trademark) for the fracture mechanics analysis. The goal of these predictions was to provide additional information to guide decisions on the potential of reusing existing and installed units prior to the new design certification.

Pai, Shantaram S.↗
NASA NTRS2021

Europa Clipper Payload Verification and Validation: Early Architecture and Implementation

NASA’s Europa Clipper mission will investigate the Jovian icy moon Europa with a sophisticated payload consisting of a suite of nine instruments. Clipper’s observations will help provide answers to a broad range of scientific objectives related to Europa’s habitability. As the project proceeds through Phase C, we are building confidence in the system’s ability to achieve these science goals by implementing a rigorous payload verification and validation (V&V) program. This paper details the planned structure and implementation of the Europa Clipper payload V&V program as it exists leading up to the project critical design review (CDR), before the bulk of instrument- or payload- level V&V activities have begun. We first explain the context for payload V&V on Europa Clipper, including its interfaces with other parts of the project system. We then describe the taxonomy of V&V concepts and their interrelationships, followed by a discussion on how this framework allows us to codify interfaces, processes, and ideas that are key to a successful V&V program. We focus on key aspects of the planning phases of V&V, especially establishing a complete set of verification items and accurately mapping them into appropriate verification activities. The second half of the paper describes the logistics of implementing this framework onto our actual payload system, including the processes and tools we use to manage and communicate agreements, changes, and risks across the project. We conclude with a reflection on the unique drivers and challenges of the V&V program concept development so far and the work that is planned for phase D implementation.

Srivastava, Priyanka↗
NASA NTRS2010

Tropospheric Airborne Meteorological Data Reporting (TAMDAR) Sensor Validation and Verification on National Oceanographic and Atmospheric Administration (NOAA) Lockheed WP-3D Aircraft

As part of the National Aeronautics and Space Administration's Aviation Safety and Security Program, the Tropospheric Airborne Meteorological Data Reporting project (TAMDAR) developed a low-cost sensor for aircraft flying in the lower troposphere. This activity was a joint effort with support from Federal Aviation Administration, National Oceanic and Atmospheric Administration, and industry. This paper reports the TAMDAR sensor performance validation and verification, as flown on board NOAA Lockheed WP-3D aircraft. These flight tests were conducted to assess the performance of the TAMDAR sensor for measurements of temperature, relative humidity, and wind parameters. The ultimate goal was to develop a small low-cost sensor, collect useful meteorological data, downlink the data in near real time, and use the data to improve weather forecasts. The envisioned system will initially be used on regional and package carrier aircraft. The ultimate users of the data are National Centers for Environmental Prediction forecast modelers. Other users include air traffic controllers, flight service stations, and airline weather centers. NASA worked with an industry partner to develop the sensor. Prototype sensors were subjected to numerous tests in ground and flight facilities. As a result of these earlier tests, many design improvements were made to the sensor. The results of tests on a final version of the sensor are the subject of this report. The sensor is capable of measuring temperature, relative humidity, pressure, and icing. It can compute pressure altitude, indicated air speed, true air speed, ice presence, wind speed and direction, and eddy dissipation rate. Summary results from the flight test are presented along with corroborative data from aircraft instruments.

Tsoucalas, George↗
NASA NTRS1994

Experiences in improving the state of the practice in verification and validation of knowledge-based systems

Knowledge-based systems (KBS's) are in general use in a wide variety of domains, both commercial and government. As reliance on these types of systems grows, the need to assess their quality and validity reaches critical importance. As with any software, the reliability of a KBS can be directly attributed to the application of disciplined programming and testing practices throughout the development life-cycle. However, there are some essential differences between conventional software and KBSs, both in construction and use. The identification of these differences affect the verification and validation (V&V) process and the development of techniques to handle them. The recognition of these differences is the basis of considerable on-going research in this field. For the past three years IBM (Federal Systems Company - Houston) and the Software Technology Branch (STB) of NASA/Johnson Space Center have been working to improve the 'state of the practice' in V&V of Knowledge-based systems. This work was motivated by the need to maintain NASA's ability to produce high quality software while taking advantage of new KBS technology. To date, the primary accomplishment has been the development and teaching of a four-day workshop on KBS V&V. With the hope of improving the impact of these workshops, we also worked directly with NASA KBS projects to employ concepts taught in the workshop. This paper describes two projects that were part of this effort. In addition to describing each project, this paper describes problems encountered and solutions proposed in each case, with particular emphasis on implications for transferring KBS V&V technology beyond the NASA domain.

Culbert, Chris↗
NASA NTRS2016

Development, Verification and Validation of Enclosure Radiation Capabilities in the CHarring Ablator Response (CHAR) Code

With the recent development of multi-dimensional thermal protection system (TPS) material response codes including the capabilities to account for radiative heating is a requirement. This paper presents the recent efforts to implement such capabilities in the CHarring Ablator Response (CHAR) code developed at NASA's Johnson Space Center. This work also describes the different numerical methods implemented in the code to compute view factors for radiation problems involving multiple surfaces. Furthermore, verification and validation of the code's radiation capabilities are demonstrated by comparing solutions to analytical results, to other codes, and to radiant test data.

Salazar, Giovanni↗
NASA NTRSDate not supplied

Analysis of the Value Added When Deploying a Model-Based Approach for the Validation and Verification of the Medical Database Software

The Medical Database (MD) is a virtual repository consisting of two software components: Medical Item Database (MedID) and the Evidence Library (EL). MedID consists of engineering data and associated information for specific medical resource items (e.g., pharmaceutical, medical devices, and supporting components), while the EL is a tool which provides all of the medical evidence necessary. The MD will 1) serve as the single “source of truth” for the Informing Mission Planning via Analysis of Complex Tradespaces (IMPACT) tool suite for both medical evidence and medical resource engineering data and 2) will be used in conjunction with the IMPACT tool suite to inform research prioritizations and perform systematic trade study evaluations to aid stakeholders in making informed decisions regarding simulated human spaceflight missions. The MD project used a Model-Based Systems Engineering (MBSE) approach to support all life cycles of the software development, while in parallel the human factors engineering team used modeling to support Human Centered Design (HCD) strategies in an effort to improve software usability. HCD is a frequently used approach in design frameworks that develops resolutions to complexities and challenges by involving the human perspective in all steps of the problem-solving process. By integrating the model-based approaches used for systems engineering and human factors activities, the project is able to leverage the model-based artifacts originally created for HCD activities for system level and human factors validation. In this presentation, our team highlights the value added when leveraging these model-based artifacts to support the on-going verification and validation activities.

C. Laing↗
NASA NTRS2017

Unmanned Aircraft Systems Detect and Avoid System: End-to-End Verification and Validation Simulation Study of Minimum Operations Performance Standards for Integrating Unmanned Aircraft into the National Airspace System

As Unmanned Aircraft Systems (UAS) make their way to mainstream aviation operations within the National Airspace System (NAS), research efforts are underway to develop a safe and effective environment for their integration into the NAS. Detect and Avoid (DAA) systems are required to account for the lack of "eyes in the sky" due to having no human on-board the aircraft. The technique, results, and lessons learned from a detailed End-to-End Verification and Validation (E2-V2) simulation study of a DAA system representative of RTCA SC-228's proposed Phase I DAA Minimum Operational Performance Standards (MOPS), based on specific test vectors and encounter cases, will be presented in this paper.

Ghatas, Rania W.↗
NASA NTRS2017

The Effects of Severity of Losses of Well Clear on Minimum Operations Performance Standards End-To-End Verification and Validation Simulation Study for Integrating Unmanned Aircraft Systems into the National Airspace System Using Detect and Avoid Systems

As Unmanned Aircraft Systems (UAS) make their way to mainstream aviation operations within the National Airspace System (NAS), research efforts are underway to develop a safe and effective environment for their integration into the NAS. Detect and Avoid (DAA) systems are required to account for the lack of “eyes in the sky” due to having no human on-board the aircraft. The technique, results, and lessons learned from a detailed End-to-End Verification and Validation (E2-V2) simulation study of a DAA system representative of RTCA Special Committee(SC)-228’s proposed Phase I DAA Minimum Operational Performance Standards (MOPS), based on specific test vectors and encounter cases, will be presented in this paper.

Ghatas, Rania W.↗
NASA NTRS2017

Applying Standard Independent Verification and Validation (IVV) Techniques Within an Agile Framework: Is There a Compatibility Issue?

Agile methods have gained wide acceptance over the past several years, to the point that they are now a standard management and execution approach for small-scale software development projects. While conventional Agile methods are not generally applicable to large multi-year and mission-critical systems, Agile hybrids are now being developed (such as SAFe) to exploit the productivity improvements of Agile while retaining the necessary process rigor and coordination needs of these projects. From the perspective of Independent Verification and Validation (IVV), however, the adoption of these hybrid Agile frameworks is becoming somewhat problematic. Hence, we find it prudent to question the compatibility of conventional IVV techniques with (hybrid) Agile practices.This paper documents our investigation of (a) relevant literature, (b) the modification and adoption of Agile frameworks to accommodate the development of large scale, mission critical systems, and (c) the compatibility of standard IVV techniques within hybrid Agile development frameworks. Specific to the latter, we found that the IVV methods employed within a hybrid Agile process can be divided into three groups: (1) early lifecycle IVV techniques that are fully compatible with the hybrid lifecycles, (2) IVV techniques that focus on tracing requirements, test objectives, etc. are somewhat incompatible, but can be tailored with a modest effort, and (3) IVV techniques involving an assessment requiring artifact completeness that are simply not compatible with hybrid Agile processes, e.g., those that assume complete requirement specification early in the development lifecycle.

Agile↗