Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “verification and validation”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 163 records · Page 9
All sourcesNASA NTRSDOE OSTI
NASA NTRS2010

Space Weather Models and Their Validation and Verification at the CCMC

The Community Coordinated l\lodeling Center (CCMC) is a US multi-agency activity with a dual mission. With equal emphasis, CCMC strives to provide science support to the international space research community through the execution of advanced space plasma simulations, and it endeavors to support the space weather needs of the CS and partners. Space weather support involves a broad spectrum, from designing robust forecasting systems and transitioning them to forecasters, to providing space weather updates and forecasts to NASA's robotic mission operators. All of these activities have to rely on validation and verification of models and their products, so users and forecasters have the means to assign confidence levels to the space weather information. In this presentation, we provide an overview of space weather models resident at CCMC, as well as of validation and verification activities undertaken at CCMC or through the use of CCMC services.

Hesse, Michael↗
DOE OSTI2021

Verification and Validation of Systems in which AI is a Key Element

Many systems are being considered in which artificial intelligence (AI) will be a key element. Failure of an AI element can lead to system failure, hence the need for AI verification and validation (V&V). This article addresses V&V of such a system, focusing on the issues created by characteristics of AI that make V&V challenging. The element(s) containing AI capabilities is treated as a subsystem and V&V is conducted on that subsystem and its interfaces with other elements of the system under study, just as V&V would be conducted on other subsystems. That is, the high-level definitions of verification and of validation do not change for systems containing one or more AI elements. However, AI V&V challenges require approaches and solutions beyond those for conventional or traditional (those without AI elements) systems. This article provides an overview of how machine learning components/subsystems “fit” in the systems engineering framework (Section 1), identifies characteristics of AI subsystems that create challenges in their V&V (Section 2), describes those challenges (Section 3) and provides some potential solutions (Section 4).

Pullum, Laura↗
DOE OSTI2025

Using a Large Language Model as a Building Block to Generate Usable Validation and Verification Suite for OpenMP

In the HPC area, both hardware and software move quickly. Often new hardware is developed and deployed, the corresponding software stack, including compilers and other tools, are under active development while leading edge software developers are working to port and tune their applications, all at the same time. While the software ecosystem is in flux, one of the key challenges for users is obtaining insight into the state of implementation of key features in the programming languages and models their applications are using – whether they have been implemented, and whether the implementation conforms to the specification, especially for newly implemented features (less tested by widespread use). OpenMP is one of the most prominent shared memory programming models used for on-node programming in HPC. With the shift towards accelerators (such as GPUs and FPGAs) and heterogeneous programming OpenMP features are getting more complex. It is natural to ask whether generative AI approaches, and large language models (LLMs) in particular, can help in producing validation and verification test suites to allow users better and faster insights into the availability and correctness of OpenMP features of interest. In this work, we explore the use of ChatGPT-4 to generate a suite of tests for OpenMP features. We have chosen a set of directives and clauses, a total of 78 combinations, which first appeared in OpenMP 3.0 (released in May 2008) but are also relevant for accelerators. We prompted ChatGPT to generate tests in the C and Fortran languages, for both host (CPU) and device (accelerator). On the Summit super-computer using the GNU implementation, we found that, of the 78 generated tests 67 C tests and 43 Fortran tests compiled successfully and fewer than those executed to completion. On further analysis we show that not all generated tests are valid. We document the process, results, and provide detailed analysis regarding the quality of tests generated. With the aim of providing input to a production quality validation and verification suite, we manually implement the corrections required to make the tests valid according to the current OpenMP specification. We quantify this effort as small, medium, or large, and record the lines of code changed to correct the invalid tests. With the corrected tests we validate recent implementations from HPE, AMD, and GNU on the Frontier supercomputer. Our experiment and subsequent analysis show that although LLMs are capable of producing HPC specific codes, they are limited by their understanding of the deeper semantics and restrictions of programming models such as OpenMP. Unsurprisingly more commonly used features have better support, while some OpenMP 3.0 directives such as sections and tasking are not universally supported on accelerators. We demonstrate that successful compilation and execution to completion are inadequate metrics for evaluating generated code and that, at this time, commodity LLMs require expert intervention for code verification. This points to gaps in the training data that is currently available for HPC. We demonstrate that with "small" effort 37% of generated invalid C tests and 63% of generated invalid Fortran tests could be corrected. This improves productivity of test generation as we circumvent writing from scratch and the common programming errors associated with it.

Pophale, Swaroop [ORNL] (ORCID:0000000185446367)↗
NASA NTRS1988

Knowledge based system verification and validation as related to automation of space station subsystems: Rationale for a knowledge based system lifecycle

The role of verification and validation (V and V) in software has been to support and strengthen the software lifecycle and to ensure that the resultant code meets the standards of the requirements documents. Knowledge Based System (KBS) V and V should serve the same role, but the KBS lifecycle is ill-defined. The rationale of a simple form of the KBS lifecycle is presented, including accommodation to certain critical KBS differences from software development.

Richardson, Keith↗
NASA NTRS1988

Knowledge based system verification and validation as related to automation of Space Station subsystems - Rationale for a knowledge based system lifecycle

The role of verification and validation (V and V) in software has been to support and strengthen the software lifecycle and to ensure that the resultant code meets the standards of the requirements documents. Knowledge-based system (KBS) V and V should serve the same role, but the KBS lifecycle is ill-defined. The rationale of a simple form of the KBS lifecycle is presented, including accommodation to certain critical KBS differences from software development.

Richardson, Keith↗
NASA NTRS1989

Evidence flow graph methods for validation and verification of expert systems

The results of an investigation into the use of evidence flow graph techniques for performing validation and verification of expert systems are given. A translator to convert horn-clause rule bases into evidence flow graphs, a simulation program, and methods of analysis were developed. These tools were then applied to a simple rule base which contained errors. It was found that the method was capable of identifying a variety of problems, for example that the order of presentation of input data or small changes in critical parameters could affect the output from a set of rules.

Becker, Lee A.↗
NASA NTRS1992

Expert system verification and validation study. ES V/V guidelines/workshop conference summary

The intent of the workshop was to start moving research on the verification and validation (V&V) of knowledge based systems (KBSs) in the direction of providing tangible 'products' that a KBS developer could use. In the near term research will focus on identifying the kinds of experiences encountered during KBS development of 'real' KBSs. These will be stored in a repository and will serve as the foundation for the rest of the activities described here. One specific approach to be pursued is 'benchmarking'. With this approach, a KBS developer can use either 'canned' KBSs with seeded errors or existing KBSs with known errors to evaluate a given tool's ability to satisfactorily identify errors.

French, Scott W.↗
NASA NTRSDate not supplied

Optimizing Urban Air Mobility Research Through Bi-Directional Integration Processes for Effective Verification and Validation

Urban Air Mobility (UAM), a subset of Advanced Air Mobility (AAM), aims to revolutionize metropolitan transportation. This paper explores an engineering process framework within National Aeronautics and Space Administration (NASA)’s Air Mobility Pathfinders (AMP) project, which is focused on the safe scaling and seamless integration of UAM operations within the National Airspace System (NAS). Progressing through three phases — Initial, Midterm, and Mature — the Federal Aviation Administration (FAA)’s UAM Concept of Operations (ConOps) describes a proposed path for the evolution of UAM operations, presenting multifaceted challenges in technology, regulation, and stakeholder engagement. With this ConOps as a basis, the AMP project is executing research, development, test, and evaluation activities aimed at delivering validated reference architectures for the Midterm phase. This paper illuminates the pivotal role of the systems engineering process in managing these technical efforts. Robust verification and validation methods, along with systematic bi-directional integration process, enhance the ability to conduct research into the design and evolution of safe, efficient, and reliable UAM systems. In this paper a systematic framework will be presented and shown to facilitate interactions between UAM stakeholders and to effectively manage the complex architecture of UAM operations in the NAS.

National Airspace System↗
NASA NTRS1992

Expert system verification and validation study: ES V/V Workshop

The primary purpose of this document is to build a foundation for applying principles of verification and validation (V&V) of expert systems. To achieve this, some V&V as applied to conventionally implemented software is required. Part one will discuss the background of V&V from the perspective of (1) what is V&V of software and (2) V&V's role in developing software. Part one will also overview some common analysis techniques that are applied when performing V&V of software. All of these materials will be presented based on the assumption that the reader has little or no background in V&V or in developing procedural software. The primary purpose of part two is to explain the major techniques that have been developed for V&V of expert systems.

French, Scott↗
NASA NTRS2005

Verification and Validation of NASA-Supported Enhancements to Decision Support Tools of PECAD

This section of the evaluation report summarizes the verification and validation (V&V) of recently implemented, NASA-supported enhancements to the decision support tools of the Production Estimates and Crop Assessment Division (PECAD). The implemented enhancements include operationally tailored Moderate Resolution Imaging Spectroradiometer (MODIS) products and products of the Global Reservoir and Lake Monitor (GRLM). The MODIS products are currently made available through two separate decision support tools: the MODIS Image Gallery and the U.S. Department of Agriculture (USDA) Foreign Agricultural Service (FAS) MODIS Normalized Difference Vegetation Index (NDVI) Database. Both the Global Reservoir and Lake Monitor and MODIS Image Gallery provide near-real-time products through PECAD's CropExplorer. This discussion addresses two areas: 1. Assessments of the standard NASA products on which these enhancements are based. 2. Characterizations of the performance of the new operational products.

Ross, Kenton W.↗
NASA NTRS2006

Verification and Validation of NASA-Supported Enhancements to the Near Real Time Harmful Algal Blooms Observing System (HABSOS)

This report discusses verification and validation (V&V) assessment of Moderate Resolution Imaging Spectroradiometer (MODIS) ocean data products contributed by the Naval Research Laboratory (NRL) and Applied Coherent Technologies (ACT) Corporation to National Oceanic Atmospheric Administration s (NOAA) Near Real Time (NRT) Harmful Algal Blooms Observing System (HABSOS). HABSOS is a maturing decision support tool (DST) used by NOAA and its partners involved with coastal and public health management.

Spruce, Joseph P.↗
NASA NTRS1997

Formal Methods for Verification and Validation of Partial Specifications: A Case Study

This paper describes our work exploring the suitability of formal specification methods for independent verification and validation (IV&V) of software specifications for large, safety critical systems. An IV&V contractor often has to perform rapid analysis on incomplete specifications, with no control over how those specifications are represented. Lightweight formal methods show significant promise in this context, as they offer a way of uncovering major errors, without the burden of full proofs of correctness. We describe a case study of the use of partial formal models for V&V of the requirements for Fault Detection Isolation and Recovery on the space station. We conclude that the insights gained from formalizing a specification are valuable, and it is the process of formalization, rather than the end product that is important. It was only necessary to build enough of the formal model to test the properties in which we were interested. Maintenance of fidelity between multiple representations of the same requirements (as they evolve) is still a problem, and deserves further study.

Easterbrook, Steve↗
DOE OSTI2023

Verification and Validation for Energy System Co-Design and Simulation

This project was only active for two months and was terminated due to a re-organization of the sponsoring initiative. The original purpose of the project was to develop and apply verification and validation (V&V) methodologies for the Energy System Co-Design with Multiple Objectives and Power Electronics (E-COMP) Initiative. This report summarizes the project and the work that was completed in the shortened time frame.

42 ENGINEERING↗
DOE OSTI2023

Verification and Validation of the PLTEMP/ANL Code for Thermal-Hydraulic Analysis of Experimental and Test Reactors, Volume 1

This is Volume 1 of a two-volume document that collects the verification and validation (V&V) works done for the PLTEMP/ANL code during the years of its development and improvement. Volume 1 describes the V&V of sixteen capabilities of the PLTEMP/ANL code that were identified by research reactor analysts as frequently used in their thermal-hydraulic analysis. Volume 2 describes the V&V of developments and improvements since the release of PLTEMP/ANL Version 4.3. Each chapter of the document focuses on verifying or validating a specific part of the software that calculates a specific phenomenon, e.g., channel flow calculation, coolant property calculation, heat transfer calculation, and flow instability calculation. Software verification is performed by comparing the code with hand calculation, Microsoft spreadsheet calculation, Mathematica calculation, or MATLAB calculation. The software validation is done by comparing the code with experimental data or a widely tested code like the RELAP5 code. In addition, some PLTEMP/ANL V&V works that are available in the open literature are simply cited in Volume 1 of the document. PLTEMP/ANL has been used in the safety analysis reports of several US and foreign research reactors licensed and converted from highly enriched uranium fuel to low-enriched uranium fuel. A list of such reactors is given in Volume 1 of the document.

22 GENERAL STUDIES OF NUCLEAR REACTORS↗
DOE OSTI2023

Verification and Validation of the PLTEMP/ANL Code for Thermal-Hydraulic Analysis of Experimental and Test Reactors, Volume 2

This is Volume 2 of a two-volume document that collects the verification and validation (V&V) works done for the PLTEMP/ANL code during the years of its development and improvement. Volume 1 describes the V&V of sixteen capabilities of the PLTEMP/ANL code that were identified by research reactor analysts as frequently used in their thermal-hydraulic analysis. Volume 2 describes the V&V of developments and improvements since the release of PLTEMP/ANL Version 4.3. Each chapter of the document focuses on verifying or validating a specific part of the software that calculates a particular phenomenon, e.g., channel flow calculation, coolant property calculation, heat transfer calculation, and flow instability calculation. Software verification is performed by comparing the code with a hand calculation, Microsoft spreadsheet calculation, Mathematica calculation, or MATLAB calculation. The software validation is achieved by comparing the code with experimental data or a widely tested code like the RELAP5 code. In addition, some PLTEMP/ANL V&V works that are available in the open literature are simply referenced in Volume 1 of the document. PLTEMP/ANL has been used in safety analysis reports of several US and foreign research reactors licensed and converted from highly enriched uranium fuel to low-enriched uranium fuel. A list of such reactors is given in Volume 1 of the document.

22 GENERAL STUDIES OF NUCLEAR REACTORS↗
NASA NTRS1988

Evidence flow graph methods for validation and verification of expert systems

This final report describes the results of an investigation into the use of evidence flow graph techniques for performing validation and verification of expert systems. This was approached by developing a translator to convert horn-clause rule bases into evidence flow graphs, a simulation program, and methods of analysis. These tools were then applied to a simple rule base which contained errors. It was found that the method was capable of identifying a variety of problems, for example that the order of presentation of input data or small changes in critical parameters could effect the output from a set of rules.

Becker, Lee A.↗
NASA NTRS1990

Expert system verification and validation study. Phase 2: Requirements identification. Delivery 1: Updated survey report

The purpose is to report the state-of-the-practice in Verification and Validation (V and V) of Expert Systems (ESs) on current NASA and Industry applications. This is the first task of a series which has the ultimate purpose of ensuring that adequate ES V and V tools and techniques are available for Space Station Knowledge Based Systems development. The strategy for determining the state-of-the-practice is to check how well each of the known ES V and V issues are being addressed and to what extent they have impacted the development of Expert Systems.

Source record↗