Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “verification and validation”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 217 records · Page 12
All sourcesNASA NTRSDOE OSTI
NASA NTRS2000

Verification and Validation of Autonomy Software at NASA

Autonomous software holds the promise of new operation possibilities, easier design and development, and lower operating costs. However, as those system close control loops and arbitrate resources on-board with specialized reasoning, the range of possible situations becomes very large and uncontrollable from the outside, making conventional scenario-based testing very inefficient. Analytic verification and validation (V&V) techniques, and model checking in particular, can provide significant help for designing autonomous systems in a more efficient and reliable manner, by providing a better coverage and allowing early error detection. This article discusses the general issue of V&V of autonomy software, with an emphasis towards model-based autonomy, model-checking techniques, and concrete experiments at NASA.

Pecheur, Charles↗
NASA NTRS1991

Translating expert system rules into Ada code with validation and verification

The purpose of this ongoing research and development program is to develop software tools which enable the rapid development, upgrading, and maintenance of embedded real-time artificial intelligence systems. The goals of this phase of the research were to investigate the feasibility of developing software tools which automatically translate expert system rules into Ada code and develop methods for performing validation and verification testing of the resultant expert system. A prototype system was demonstrated which automatically translated rules from an Air Force expert system was demonstrated which detected errors in the execution of the resultant system. The method and prototype tools for converting AI representations into Ada code by converting the rules into Ada code modules and then linking them with an Activation Framework based run-time environment to form an executable load module are discussed. This method is based upon the use of Evidence Flow Graphs which are a data flow representation for intelligent systems. The development of prototype test generation and evaluation software which was used to test the resultant code is discussed. This testing was performed automatically using Monte-Carlo techniques based upon a constraint based description of the required performance for the system.

Becker, Lee↗
NASA NTRS2022

Europa Clipper Payload Verification and Validation: Avionics-Instrument Interface Test Campaign

NASA's Europa Clipper mission will investigate Jupiter's icy moon Europa using a payload suite consisting of nine instruments to address a range of scientific objectives concerning Europa's habitability. As the project proceeds past its Critical Design Review, confidence is being built in the system's ability to achieve mission objectives through the implementation of a rigorous payload verification and validation (V&V) program. As part of this payload V&V program, instrument box-level testing was performed by the payload team to verify select instrument-avionics interface requirements. This testing was performed at JPL using the avionics testbed's Bulk Data Storage Emulator (BDSEM) with visiting instrument Test Models. This paper summarizes the Data Link test campaign involving roughly four days of functional testing per instrument, including planning, testing methods, types of issues found, and the requirement closure process. Detail is also provided on the development, deployment, and validation of a standardized analysis tool used in data reviews. This testing verified requirements related to commanding rates, loss of link, packet format, clock counters, loopback test capability, and SpaceWire jitter and skew margins. Additional risk reduction testing of basic commanding, counter behavior, science data collection and transfer, and interface swapping was also performed. Because the BDSEM venue was not originally designed to be a run for record venue, the process of characterizing venue fidelity and establishing suitability for requirement closure using data collected in this venue will also be addressed.In order to close requirements, an extensible tool was developed to post-process instrument command and telemetry data from their original binary to a human-readable format and give visibility to errors detected within the data, such as packets with Cyclic Redundancy Check errors. This tool, called payload-packet-parser, is a Python 3.9 command line tool built using a variety of open-source Python libraries. Payload-packet-parser was designed to support parsing command and telemetry packets for all Europa Clipper instruments and additional analysis tools were developed for verification of specific information interface requirements. This test campaign, including post-processing using a single parsing and verification toolset, allowed for early interface testing, alleviating testing burdens on instrument teams and buying down risk on the instrument-avionics interface by finding hardware and software issues and idiosyncrasies prior to integration with system test venues. Over twenty issues were discovered across the payload, resulting in software updates and instrument rework well in advance of any system impacts. This paper concludes with an assessment of benefits and costs of this type of testing and lessons learned.

Montanez, Leticia↗
NASA NTRS2013

Model Verification and Validation Concepts for a Probabilistic Fracture Assessment Model to Predict Cracking of Knife Edge Seals in the Space Shuttle Main Engine High Pressure Oxidizer

Physics-based models are routinely used to predict the performance of engineered systems to make decisions such as when to retire system components, how to extend the life of an aging system, or if a new design will be safe or available. Model verification and validation (V&V) is a process to establish credibility in model predictions. Ideally, carefully controlled validation experiments will be designed and performed to validate models or submodels. In reality, time and cost constraints limit experiments and even model development. This paper describes elements of model V&V during the development and application of a probabilistic fracture assessment model to predict cracking in space shuttle main engine high-pressure oxidizer turbopump knife-edge seals. The objective of this effort was to assess the probability of initiating and growing a crack to a specified failure length in specific flight units for different usage and inspection scenarios. The probabilistic fracture assessment model developed in this investigation combined a series of submodels describing the usage, temperature history, flutter tendencies, tooth stresses and numbers of cycles, fatigue cracking, nondestructive inspection, and finally the probability of failure. The analysis accounted for unit-to-unit variations in temperature, flutter limit state, flutter stress magnitude, and fatigue life properties. The investigation focused on the calculation of relative risk rather than absolute risk between the usage scenarios. Verification predictions were first performed for three units with known usage and cracking histories to establish credibility in the model predictions. Then, numerous predictions were performed for an assortment of operating units that had flown recently or that were projected for future flights. Calculations were performed using two NASA-developed software tools: NESSUS(Registered Trademark) for the probabilistic analysis, and NASGRO(Registered Trademark) for the fracture mechanics analysis. The goal of these predictions was to provide additional information to guide decisions on the potential of reusing existing and installed units prior to the new design certification.

Pai, Shantaram S.↗
NASA NTRS2021

Europa Clipper Payload Verification and Validation: Early Architecture and Implementation

NASA’s Europa Clipper mission will investigate the Jovian icy moon Europa with a sophisticated payload consisting of a suite of nine instruments. Clipper’s observations will help provide answers to a broad range of scientific objectives related to Europa’s habitability. As the project proceeds through Phase C, we are building confidence in the system’s ability to achieve these science goals by implementing a rigorous payload verification and validation (V&V) program. This paper details the planned structure and implementation of the Europa Clipper payload V&V program as it exists leading up to the project critical design review (CDR), before the bulk of instrument- or payload- level V&V activities have begun. We first explain the context for payload V&V on Europa Clipper, including its interfaces with other parts of the project system. We then describe the taxonomy of V&V concepts and their interrelationships, followed by a discussion on how this framework allows us to codify interfaces, processes, and ideas that are key to a successful V&V program. We focus on key aspects of the planning phases of V&V, especially establishing a complete set of verification items and accurately mapping them into appropriate verification activities. The second half of the paper describes the logistics of implementing this framework onto our actual payload system, including the processes and tools we use to manage and communicate agreements, changes, and risks across the project. We conclude with a reflection on the unique drivers and challenges of the V&V program concept development so far and the work that is planned for phase D implementation.

Srivastava, Priyanka↗
NASA NTRS2010

Tropospheric Airborne Meteorological Data Reporting (TAMDAR) Sensor Validation and Verification on National Oceanographic and Atmospheric Administration (NOAA) Lockheed WP-3D Aircraft

As part of the National Aeronautics and Space Administration's Aviation Safety and Security Program, the Tropospheric Airborne Meteorological Data Reporting project (TAMDAR) developed a low-cost sensor for aircraft flying in the lower troposphere. This activity was a joint effort with support from Federal Aviation Administration, National Oceanic and Atmospheric Administration, and industry. This paper reports the TAMDAR sensor performance validation and verification, as flown on board NOAA Lockheed WP-3D aircraft. These flight tests were conducted to assess the performance of the TAMDAR sensor for measurements of temperature, relative humidity, and wind parameters. The ultimate goal was to develop a small low-cost sensor, collect useful meteorological data, downlink the data in near real time, and use the data to improve weather forecasts. The envisioned system will initially be used on regional and package carrier aircraft. The ultimate users of the data are National Centers for Environmental Prediction forecast modelers. Other users include air traffic controllers, flight service stations, and airline weather centers. NASA worked with an industry partner to develop the sensor. Prototype sensors were subjected to numerous tests in ground and flight facilities. As a result of these earlier tests, many design improvements were made to the sensor. The results of tests on a final version of the sensor are the subject of this report. The sensor is capable of measuring temperature, relative humidity, pressure, and icing. It can compute pressure altitude, indicated air speed, true air speed, ice presence, wind speed and direction, and eddy dissipation rate. Summary results from the flight test are presented along with corroborative data from aircraft instruments.

Tsoucalas, George↗
NASA NTRS1994

Experiences in improving the state of the practice in verification and validation of knowledge-based systems

Knowledge-based systems (KBS's) are in general use in a wide variety of domains, both commercial and government. As reliance on these types of systems grows, the need to assess their quality and validity reaches critical importance. As with any software, the reliability of a KBS can be directly attributed to the application of disciplined programming and testing practices throughout the development life-cycle. However, there are some essential differences between conventional software and KBSs, both in construction and use. The identification of these differences affect the verification and validation (V&V) process and the development of techniques to handle them. The recognition of these differences is the basis of considerable on-going research in this field. For the past three years IBM (Federal Systems Company - Houston) and the Software Technology Branch (STB) of NASA/Johnson Space Center have been working to improve the 'state of the practice' in V&V of Knowledge-based systems. This work was motivated by the need to maintain NASA's ability to produce high quality software while taking advantage of new KBS technology. To date, the primary accomplishment has been the development and teaching of a four-day workshop on KBS V&V. With the hope of improving the impact of these workshops, we also worked directly with NASA KBS projects to employ concepts taught in the workshop. This paper describes two projects that were part of this effort. In addition to describing each project, this paper describes problems encountered and solutions proposed in each case, with particular emphasis on implications for transferring KBS V&V technology beyond the NASA domain.

Culbert, Chris↗
NASA NTRS2016

Development, Verification and Validation of Enclosure Radiation Capabilities in the CHarring Ablator Response (CHAR) Code

With the recent development of multi-dimensional thermal protection system (TPS) material response codes including the capabilities to account for radiative heating is a requirement. This paper presents the recent efforts to implement such capabilities in the CHarring Ablator Response (CHAR) code developed at NASA's Johnson Space Center. This work also describes the different numerical methods implemented in the code to compute view factors for radiation problems involving multiple surfaces. Furthermore, verification and validation of the code's radiation capabilities are demonstrated by comparing solutions to analytical results, to other codes, and to radiant test data.

Salazar, Giovanni↗
NASA NTRSDate not supplied

Analysis of the Value Added When Deploying a Model-Based Approach for the Validation and Verification of the Medical Database Software

The Medical Database (MD) is a virtual repository consisting of two software components: Medical Item Database (MedID) and the Evidence Library (EL). MedID consists of engineering data and associated information for specific medical resource items (e.g., pharmaceutical, medical devices, and supporting components), while the EL is a tool which provides all of the medical evidence necessary. The MD will 1) serve as the single “source of truth” for the Informing Mission Planning via Analysis of Complex Tradespaces (IMPACT) tool suite for both medical evidence and medical resource engineering data and 2) will be used in conjunction with the IMPACT tool suite to inform research prioritizations and perform systematic trade study evaluations to aid stakeholders in making informed decisions regarding simulated human spaceflight missions. The MD project used a Model-Based Systems Engineering (MBSE) approach to support all life cycles of the software development, while in parallel the human factors engineering team used modeling to support Human Centered Design (HCD) strategies in an effort to improve software usability. HCD is a frequently used approach in design frameworks that develops resolutions to complexities and challenges by involving the human perspective in all steps of the problem-solving process. By integrating the model-based approaches used for systems engineering and human factors activities, the project is able to leverage the model-based artifacts originally created for HCD activities for system level and human factors validation. In this presentation, our team highlights the value added when leveraging these model-based artifacts to support the on-going verification and validation activities.

C. Laing↗
NASA NTRS2017

Unmanned Aircraft Systems Detect and Avoid System: End-to-End Verification and Validation Simulation Study of Minimum Operations Performance Standards for Integrating Unmanned Aircraft into the National Airspace System

As Unmanned Aircraft Systems (UAS) make their way to mainstream aviation operations within the National Airspace System (NAS), research efforts are underway to develop a safe and effective environment for their integration into the NAS. Detect and Avoid (DAA) systems are required to account for the lack of "eyes in the sky" due to having no human on-board the aircraft. The technique, results, and lessons learned from a detailed End-to-End Verification and Validation (E2-V2) simulation study of a DAA system representative of RTCA SC-228's proposed Phase I DAA Minimum Operational Performance Standards (MOPS), based on specific test vectors and encounter cases, will be presented in this paper.

Ghatas, Rania W.↗
NASA NTRS2017

The Effects of Severity of Losses of Well Clear on Minimum Operations Performance Standards End-To-End Verification and Validation Simulation Study for Integrating Unmanned Aircraft Systems into the National Airspace System Using Detect and Avoid Systems

As Unmanned Aircraft Systems (UAS) make their way to mainstream aviation operations within the National Airspace System (NAS), research efforts are underway to develop a safe and effective environment for their integration into the NAS. Detect and Avoid (DAA) systems are required to account for the lack of “eyes in the sky” due to having no human on-board the aircraft. The technique, results, and lessons learned from a detailed End-to-End Verification and Validation (E2-V2) simulation study of a DAA system representative of RTCA Special Committee(SC)-228’s proposed Phase I DAA Minimum Operational Performance Standards (MOPS), based on specific test vectors and encounter cases, will be presented in this paper.

Ghatas, Rania W.↗
NASA NTRS2017

Applying Standard Independent Verification and Validation (IVV) Techniques Within an Agile Framework: Is There a Compatibility Issue?

Agile methods have gained wide acceptance over the past several years, to the point that they are now a standard management and execution approach for small-scale software development projects. While conventional Agile methods are not generally applicable to large multi-year and mission-critical systems, Agile hybrids are now being developed (such as SAFe) to exploit the productivity improvements of Agile while retaining the necessary process rigor and coordination needs of these projects. From the perspective of Independent Verification and Validation (IVV), however, the adoption of these hybrid Agile frameworks is becoming somewhat problematic. Hence, we find it prudent to question the compatibility of conventional IVV techniques with (hybrid) Agile practices.This paper documents our investigation of (a) relevant literature, (b) the modification and adoption of Agile frameworks to accommodate the development of large scale, mission critical systems, and (c) the compatibility of standard IVV techniques within hybrid Agile development frameworks. Specific to the latter, we found that the IVV methods employed within a hybrid Agile process can be divided into three groups: (1) early lifecycle IVV techniques that are fully compatible with the hybrid lifecycles, (2) IVV techniques that focus on tracing requirements, test objectives, etc. are somewhat incompatible, but can be tailored with a modest effort, and (3) IVV techniques involving an assessment requiring artifact completeness that are simply not compatible with hybrid Agile processes, e.g., those that assume complete requirement specification early in the development lifecycle.

Agile↗
DOE OSTI2022

Verification and Validation of the Stakeholder Tool for Assessing Radioactive Transportation (START) – 22323

The U.S. Department of Energy (DOE) Office of Integrated Waste Management is planning for the eventual transportation, storage, and disposal of spent nuclear fuel (SNF) and high-level radioactive waste (HLW) from nuclear power plant and DOE sites. The Stakeholder Tool for Assessing Radioactive Transportation (START) is a web-based, geospatial decision-support tool developed for evaluating routing options and other aspects of transporting SNF and HLW, covering rail, truck, barge, and intermodal infrastructure and operations in the continental United States. The verification and validation (V&V) process is intended to independently assess START to provide confidence in the ability of START to accurately provide intended results. The V&V process checks the START tool using a variety of methods, ranging from independent hand calculations to comparison of START performance and results to those of other codes. The V&V activity was conducted independently from the START development team with opportunities to provide feedback and collaborate throughout the process. The V&V analyzed attributes of transportation routes produced by START, including route distance and both population and population density captured within buffer zones around routes. Population in the buffer zone, population density in the buffer zone, and route distance were all identified as crucial outputs of the START code and were subject to V&V tasks. Some of the improvements identified through the V&V process were standardizing the underlying population data in START, changing the projection of the population raster data, and changes to the methodology used for population density to improve its applicability for expected users. This collaboration also led to suggested improvements to some of the underlying shape file segments within START. A formal series of test routes went through the V&V process from START version 3.2.1 following the implementation of the recommended improvements to START. The V&V of the buffer zone population and distance reported in START were achieved by utilizing a custom workflow developed in QGIS. This platform was chosen because it is fully independent of START. The buffer zone population and route distances in START showed excellent agreement to independent V&V test results. Over 200 route test cases were run; in all cases, the percent difference in the population within the buffer zone was less than +/- 5%, and the majority of cases were below +/- 1%. These cases were also duplicated in ArcMap, which is Environmental Systems Research Institute’s (ESRI) GIS desktop application. The ArcMap results also show good agreement; however, they cannot be considered independent of START because START also uses ESRI software. The V&V tasks are currently addressing population density and estimated radiation dose in START as well as implementing further automation of the V&V process to allow larger suites of test routes to be run more easily and quickly by less experienced GIS users. These improvements to the V&V process will make it a valuable tool for the development team and the stakeholders so that each new iteration of START can easily be checked for quality and consistency. This paper and presentation will describe the V&V process, findings and implications, and ongoing V&V activities.

START, DOE, Transporation, dose, GIS↗
NASA NTRS1996

Verification and Validation of KBS with Neural Network Components

Artificial Neural Network (ANN) play an important role in developing robust Knowledge Based Systems (KBS). The ANN based components used in these systems learn to give appropriate predictions through training with correct input-output data patterns. Unlike traditional KBS that depends on a rule database and a production engine, the ANN based system mimics the decisions of an expert without specifically formulating the if-than type of rules. In fact, the ANNs demonstrate their superiority when such if-then type of rules are hard to generate by human expert. Verification of traditional knowledge based system is based on the proof of consistency and completeness of the rule knowledge base and correctness of the production engine.These techniques, however, can not be directly applied to ANN based components.In this position paper, we propose a verification and validation procedure for KBS with ANN based components. The essence of the procedure is to obtain an accurate system specification through incremental modification of the specifications using an ANN rule extraction algorithm.

Wen, Wu↗
NASA NTRS2001

RIACS Workshop on the Verification and Validation of Autonomous and Adaptive Systems

The long-term future of space exploration at NASA is dependent on the full exploitation of autonomous and adaptive systems: careful monitoring of missions from earth, as is the norm now, will be infeasible due to the sheer number of proposed missions and the communication lag for deep-space missions. Mission managers are however worried about the reliability of these more intelligent systems. The main focus of the workshop was to address these worries and hence we invited NASA engineers working on autonomous and adaptive systems and researchers interested in the verification and validation (V&V) of software systems. The dual purpose of the meeting was to: (1) make NASA engineers aware of the V&V techniques they could be using; and (2) make the V&V community aware of the complexity of the systems NASA is developing.

Pecheur, Charles↗
NASA NTRS2006

Verification and Validation of NASA-Supported Enhancements to PECAD's Decision Support Tools

The NASA Applied Sciences Directorate (ASD), part of the Earth-Sun System Division of NASA's Science Mission Directorate, has partnered with the U.S. Department of Agriculture (USDA) to enhance decision support in the area of agricultural efficiency-an application of national importance. The ASD integrated the results of NASA Earth science research into USDA decision support tools employed by the USDA Foreign Agricultural Service (FAS) Production Estimates and Crop Assessment Division (PECAD), which supports national decision making by gathering, analyzing, and disseminating global crop intelligence. Verification and validation of the following enhancements are summarized: 1) Near-real-time Moderate Resolution Imaging Spectroradiometer (MODIS) products through PECAD's MODIS Image Gallery; 2) MODIS Normalized Difference Vegetation Index (NDVI) time series data through the USDA-FAS MODIS NDVI Database; and 3) Jason-1 and TOPEX/Poseidon lake level estimates through PECAD's Global Reservoir and Lake Monitor. Where possible, each enhanced product was characterized for accuracy, timeliness, and coverage, and the characterized performance was compared to PECAD operational requirements. The MODIS Image Gallery and the GRLM are more mature and have achieved a semi-operational status, whereas the USDA-FAS MODIS NDVI Database is still evolving and should be considered

McKellipo, Rodney↗
DOE OSTI2026

Georgetown University – SYSM 5630 Systems Integration Verification and Validation : Todd Noste

At Lawrence Livermore National Laboratory in the National Ignition Facility Optics Group, we use the systems engineering approach for project management and as a design tool. Systems engineering is used in a graded approach to design and project management that is based on risk, informing how much rigor to apply. The tools and techniques from systems engineering offer a framework to organize projects with everyone speaking the same language to provide consistent and repeatable project success that satisfies the stakeholders’ needs and meets the mission. The classes have provided a framework with tools for communicating system design, requirements, verification and validation, and an operational context.

42 ENGINEERING↗
NASA NTRS2009

Verification and Validation of Adaptive and Intelligent Systems with Flight Test Results

F-15 IFCS project goals are: a) Demonstrate Control Approaches that can Efficiently Optimize Aircraft Performance in both Normal and Failure Conditions [A] & [B] failures. b) Advance Neural Network-Based Flight Control Technology for New Aerospace Systems Designs with a Pilot in the Loop. Gen II objectives include; a) Implement and Fly a Direct Adaptive Neural Network Based Flight Controller; b) Demonstrate the Ability of the System to Adapt to Simulated System Failures: 1) Suppress Transients Associated with Failure; 2) Re-Establish Sufficient Control and Handling of Vehicle for Safe Recovery. c) Provide Flight Experience for Development of Verification and Validation Processes for Flight Critical Neural Network Software.

Burken, John J.↗