Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “software validation”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 91 records · Page 5

Using Colored Stochastic Petri Net (CS-PN) software for protocol specification, validation, and evaluation

The specification, verification, validation, and evaluation, which make up the different steps of the CS-PN software are outlined. The colored stochastic Petri net software is applied to a Wound/Wait protocol decomposable into two principal modules: request or couple (transaction, granule) treatment module and wound treatment module. Each module is specified, verified, validated, and then evaluated separately, to deduce a verification, validation and evaluation of the complete protocol. The colored stochastic Petri nets tool is shown to be a natural extension of the stochastic tool, adapted to distributed systems and protocols, because the color conveniently takes into account the numerous sites, transactions, granules and messages.

Zenie, Alexandre↗

Software unit testing in Ada environment

A validation procedure for the Ada binding of the Graphical Kernel System (GKS) is being developed. PRIOR Data Sciences is also producing a version of the GKS written in Ada. These major software engineering projects will provide an opportunity to demonstrate a sound approach for software testing in an Ada environment. The GKS/Ada validation capability will be a collection of test programs and data, and test management guidelines. These products will be used to assess the correctness, completeness, and efficiency of any GKS/Ada implementation. The GKS/Ada developers will be able to obtain the validation software for their own use. It is anticipated that this validation software will eventually be taken over by an independent standards body to provide objective assessments of GKS/Ada implementations, using an approach similar to the validation testing currently applied to Ada compilers. In the meantime, if requested, this validation software will be used to assess GKS/Ada products. The second project, implementation of GKS using the Ada language, is a conventional software engineering tasks. It represents a large body of Ada code and has some interesting testing problems associated with automatic testing of graphics routines. Here the normal test practices which include automated regression testing, independent quality assistance, test configuration management, and the application of software quality metrics will be employed. The software testing methods emphasize quality enhancement and automated procedures. Ada makes some aspects of testing easier, and introduces some concerns. These issues are addressed.

Warnock, Glenn↗

Obtaining Valid Safety Data for Software Safety Measurement and Process Improvement

We report on a preliminary case study to examine software safety risk in the early design phase of the NASA Constellation spaceflight program. Our goal is to provide NASA quality assurance managers with information regarding the ongoing state of software safety across the program. We examined 154 hazard reports created during the preliminary design phase of three major flight hardware systems within the Constellation program. Our purpose was two-fold: 1) to quantify the relative importance of software with respect to system safety; and 2) to identify potential risks due to incorrect application of the safety process, deficiencies in the safety process, or the lack of a defined process. One early outcome of this work was to show that there are structural deficiencies in collecting valid safety data that make software safety different from hardware safety. In our conclusions we present some of these deficiencies.

Basili, Victor r.↗

Simulation verification techniques study: Simulation performance validation techniques document

Techniques and support software for the efficient performance of simulation validation are discussed. Overall validation software structure, the performance of validation at various levels of simulation integration, guidelines for check case formulation, methods for real time acquisition and formatting of data from an all up operational simulator, and methods and criteria for comparison and evaluation of simulation data are included. Vehicle subsystems modules, module integration, special test requirements, and reference data formats are also described.

Duncan, L. M.↗

Toward Certification of Machine-Learning Systems for Low Criticality Airborne Applications

The exceptional progress in the field of machine learning (ML) in recent years has attracted a lot of interest in using this technology in aviation. Possible airborne applications of ML include safety-critical functions, which must be developed in compliance with rigorous certification standards of the aviation industry. Current certification standards for the aviation industry were developed prior to the ML renaissance without taking specifics of ML technology into account. There are some fundamental incompatibilities between traditional design assurance approaches and certain aspects of ML-based systems. In this paper, we analyze the current airborne certification standards and show that all objectives of the standards can be achieved for a low-criticality ML-based system if certain assumptions about ML development workflow are applied.

Avionics↗

Toward Design Assurance of Machine-Learning Airborne Systems

In recent years, Artificial Intelligence (AI) systems, enabled by Machine Learning (ML)technology, have demonstrated impressive progress and provides historic opportunities for the aviation industry. However, several key aspects of ML technology are not compatible with existing design assurance standards and make certification problematic. In this paper, we present a case study of a visual system with a Deep Neural Network (DNN) intended to detect and identify airport runway signs. Different use cases and variants of this system exhibit different levels of criticality ranging from design assurance level (DAL) D to B. We use the case study to illustrate the challenges of certification according to the current standards, such asDO-178C. We present the system design, data generation, training, and verification in detail and describe how the design assurance objectives can be met for a DAL D variant of the system. We also discuss gaps and potential approaches for the higher design assurance levels.

Avionics↗

Validation of CFD/Heat Transfer Software for Turbine Blade Analysis

I am an intern in the Turbine Branch of the Turbomachinery and Propulsion Systems Division. The division is primarily concerned with experimental and computational methods of calculating heat transfer effects of turbine blades during operation in jet engines and land-based power systems. These include modeling flow in internal cooling passages and film cooling, as well as calculating heat flux and peak temperatures to ensure safe and efficient operation. The branch is research-oriented, emphasizing the development of tools that may be used by gas turbine designers in industry. The branch has been developing a computational fluid dynamics (CFD) and heat transfer code called GlennHT to achieve the computational end of this analysis. The code was originally written in FORTRAN 77 and run on Silicon Graphics machines. However the code has been rewritten and compiled in FORTRAN 90 to take advantage of more modem computer memory systems. In addition the branch has made a switch in system architectures from SGI's to Linux PC's. The newly modified code therefore needs to be tested and validated. This is the primary goal of my internship. To validate the GlennHT code, it must be run using benchmark fluid mechanics and heat transfer test cases, for which there are either analytical solutions or widely accepted experimental data. From the solutions generated by the code, comparisons can be made to the correct solutions to establish the accuracy of the code. To design and create these test cases, there are many steps and programs that must be used. Before a test case can be run, pre-processing steps must be accomplished. These include generating a grid to describe the geometry, using a software package called GridPro. Also various files required by the GlennHT code must be created including a boundary condition file, a file for multi-processor computing, and a file to describe problem and algorithm parameters. A good deal of this internship will be to become familiar with these programs and the structure of the GlennHT code. Additional information is included in the original extended abstract.

Kiefer, Walter D.↗

Verification and Validation in a Rapid Software Development Process

The high cost of software production is driving development organizations to adopt more automated design and analysis methods such as rapid prototyping, computer-aided software engineering (CASE) tools, and high-level code generators. Even developers of safety-critical software system have adopted many of these new methods while striving to achieve high levels Of quality and reliability. While these new methods may enhance productivity and quality in many cases, we examine some of the risks involved in the use of new methods in safety-critical contexts. We examine a case study involving the use of a CASE tool that automatically generates code from high-level system designs. We show that while high-level testing on the system structure is highly desirable, significant risks exist in the automatically generated code and in re-validating releases of the generated code after subsequent design changes. We identify these risks and suggest process improvements that retain the advantages of rapid, automated development methods within the quality and reliability contexts of safety-critical projects.

Callahan, John R.↗

Experimental Evaluation of Verification and Validation Tools on Martian Rover Software

To achieve its science objectives in deep space exploration, NASA has a need for science platform vehicles to autonomously make control decisions in a time frame that excludes intervention from Earth-based controllers. Round-trip light-time is one significant factor motivating autonomy capability, another factor is the need to reduce ground support operations cost. An unsolved problem potentially impeding the adoption of autonomy capability is the verification and validation of such software systems, which exhibit far more behaviors (and hence distinct execution paths in the software) than is typical in current deepspace platforms. Hence the need for a study to benchmark advanced Verification and Validation (V&V) tools on representative autonomy software. The objective of the study was to access the maturity of different technologies, to provide data indicative of potential synergies between them, and to identify gaps in the technologies with respect to the challenge of autonomy V&V. The study consisted of two parts: first, a set of relatively independent case studies of different tools on the same autonomy code, second a carefully controlled experiment with human participants on a subset of these technologies. This paper describes the second part of the study. Overall, nearly four hundred hours of data on human use of three different advanced V&V tools were accumulated, with a control group that used conventional testing methods. The experiment simulated four independent V&V teams debugging three successive versions of an executive controller for a Martian Rover. Defects were carefully seeded into the three versions based on a profile of defects from CVS logs that occurred in the actual development of the executive controller. The rest of the document is structured a s follows. In section 2 and 3, we respectively describe the tools used in the study and the rover software that was analyzed. In section 4 the methodology for the experiment is described; this includes the code preparation, seeding of defects, participant training and experimental setup. Next we give a qualitative overview of how the experiment went from the point of view of each technology; model checking (section 5), static analysis (section 6), runtime analysis (section 7) and testing (section 8). The find section gives some preliminary quantitative results on how the tools compared.

Brat, Guillaume↗

Preliminary design of the redundant software experiment

The goal of the present experiment is to characterize the fault distributions of highly reliable software replicates, constructed using techniques and environments which are similar to those used in comtemporary industrial software facilities. The fault distributions and their effect on the reliability of fault tolerant configurations of the software will be determined through extensive life testing of the replicates against carefully constructed randomly generated test data. Each detected error will be carefully analyzed to provide insight in to their nature and cause. A direct objective is to develop techniques for reducing the intensity of coincident errors, thus increasing the reliability gain which can be achieved with fault tolerance. Data on the reliability gains realized, and the cost of the fault tolerant configurations can be used to design a companion experiment to determine the cost effectiveness of the fault tolerant strategy. Finally, the data and analysis produced by this experiment will be valuable to the software engineering community as a whole because it will provide a useful insight into the nature and cause of hard to find, subtle faults which escape standard software engineering validation techniques and thus persist far into the software life cycle.

Campbell, Roy↗

Advanced flight software reconfiguraton

Information is given in viewgraph form on advanced flight software reconfiguration. Reconfiguration is defined as identifying mission and configuration specific requirements, controlling mission and configuration specific data, binding this information to the flight software code to perform specific missions, and the release and distribution of the flight software. The objectives are to develop, demonstrate, and validate advanced software reconfiguration tools and techniques; to demonstrate reconfiguration approaches on Space Station Freedom (SSF) onboard systems displays; and to interactively test onboard systems displays, flight software, and flight data.

Porcher, Bryan↗

Experimental Evaluation of Verification and Validation Tools on Martian Rover Software

We report on a study to determine the maturity of different verification and validation technologies (V&V) on a representative example of NASA flight software. The study consisted of a controlled experiment where three technologies (static analysis, runtime analysis and model checking) were compared to traditional testing with respect to their ability to find seeded errors in a prototype Mars Rover. What makes this study unique is that it is the first (to the best of our knowledge) to do a controlled experiment to compare formal methods based tools to testing on a realistic industrial-size example where the emphasis was on collecting as much data on the performance of the tools and the participants as possible. The paper includes a description of the Rover code that was analyzed, the tools used as well as a detailed description of the experimental setup and the results. Due to the complexity of setting up the experiment, our results can not be generalized, but we believe it can still serve as a valuable point of reference for future studies of this kind. It did confirm the belief we had that advanced tools can outperform testing when trying to locate concurrency errors. Furthermore the results of the experiment inspired a novel framework for testing the next generation of the Rover.

Brat, Guillaume↗

The validation by measurement theory of proposed object-oriented software metrics

Moving software development into the engineering arena requires controllability, and to control a process, it must be measurable. Measuring the process does no good if the product is not also measured, i.e., being the best at producing an inferior product does not define a quality process. Also, not every number extracted from software development is a valid measurement. A valid measurement only results when we are able to verify that the number is representative of the attribute that we wish to measure. Many proposed software metrics are used by practitioners without these metrics ever having been validated, leading to costly but often useless calculations. Several researchers have bemoaned the lack of scientific precision in much of the published software measurement work and have called for validation of software metrics by measurement theory. This dissertation applies measurement theory to validate fifty proposed object-oriented software metrics (Li and Henry, 1993; Chidamber and Kemerrer, 1994; Lorenz and Kidd, 1994).

Neal, Ralph D.↗

The Validation by Measurement Theory of Proposed Object-Oriented Software Metrics

Moving software development into the engineering arena requires controllability, and to control a process, it must be measurable. Measuring the process does no good if the product is not also measured, i.e., being the best at producing an inferior product does not define a quality process. Also, not every number extracted from software development is a valid measurement. A valid measurement only results when we are able to verify that the number is representative of the attribute that we wish to measure. Many proposed software metrics are used by practitioners without these metrics ever having been validated, leading to costly but often useless calculations. Several researchers have bemoaned the lack of scientific precision in much of the published software measurement work and have called for validation of software metrics by measurement theory. This dissertation applies measurement theory to validate fifty proposed object-oriented software metrics.

Neal, Ralph D.↗

The Validation by Measurement Theory of Proposed Object-Oriented Software Metrics

Moving software development into the engineering arena requires controllability, and to control a process, it must be measurable. Measuring the process does no good if the product is not also measured, i.e., being the best at producing an inferior product does not define a quality process. Also, not every number extracted from software development is a valid measurement. A valid measurement only results when we are able to verify that the number is representative of the attribute that we wish to measure. Many proposed software metrics are used by practitioners without these metrics ever having been validated, leading to costly but often useless calculations. Several researchers have bemoaned the lack of scientific precision in much of the published software measurement work and have called for validation of software metrics by measurement theory. This dissertation applies measurement theory to validate fifty proposed object-oriented software metrics.

Neal, Ralph D.↗

BRAINSTACK – A Platform for Artificial Intelligence & Machine Learning Collaborative Experiments on a Nano-Satellite

As the space economy continues to expand through increasingly easy access to advanced and inexpensive technology, space missions themselves have become more ambitious with exploration targets growing ever distant while simultaneously requiring larger guidance and communication budgets. These conflicting desires of distance and control drive the need for advanced on-board intelligent decision making to reduce communication and control limitations by automating as many mission functions as possible in-situ. While the amount of research on such Artificial Intelligence and Machine Learning (AI/ML) software modules has grown exponentially, the capacity to experimentally validate such software modules in space in a rapid and inexpensive format has not. To this end, the Nano Orbital Workshop (NOW) group at NASA Ames Research Center has been at the forefront of performing initial flight evaluation tests of ‘commercially’ available bleeding-edge computational platforms via what is programmatically referred to as the BrainStack on the TechEdSat (TES-n) flight series. This on-orbit computational platform provides an evaluation laboratory where advanced software experiments are pre-loaded into memory prior to launch, then executed as payloads during mission operations with results reported back and program tweaks or new training sets uploaded as needed. Processors selected as part of the BrainStack are of ideal size, packaging, and power consumption for easy integration into a cube satellite structure. These experiments have included the evaluation of small, high-performance GPUs and, more recently, neuromorphic processors, in LEO operations. Neuromorphic processors are of particular interest due to their superior power efficiency over GPUs in intelligent automation applications. The first TES-n flight test of an Intel first-generation Loihi neuromorphic processor launched on TES-13, January 13, 2022, and continues to operate in orbit despite no significant modifications to harden the processor against the space environment. The Intel Loihi Gen-1 on TES-13 is characterized by a 14nm 128-core Spiking Neural Network (SNN) able to support on-chip training. The processor is packaged in the Kapoho Bay USB module, providing a relatively straight-forward interface to the bus avionics system. The Kapoho Bay was in turn managed by an Intel Pentium single-board computer to handle scheduling of the software application payloads and communications with the satellite’s primary computer. The recently released Intel Loihi Gen-2, able to support integer-valued spike payloads and produced using 7nm process, will form part of the continually evolving BrainStack in the upcoming three TES-n/NOW flights. The Kapoho Point unit will incorporate eight Loihi-2 processors, enabling neural networks of up to one million neurons and one billion synapsis. Additionally, it is planned to measure the radiation environment these processors experience to understand any degradation or computational artifacts caused by long term space radiation exposure on these novel architectures. This evolving flexible and collaborative environment involving various research teams across NASA and other organizations is intended to be a convenient orbital test platform from which many anticipated future space automation applications may be initially tested.

Artificial Intelligence↗