Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “software quality, testing”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 37 records · Page 2
All sourcesNASA NTRSDOE OSTI
DOE OSTI2023

Spring 2023 Verification Presentation to Headquarters [Slides]

Testing codes is important, and more tests are needed. Verification testing is different than software-quality assurance, like regression suites. Tests should be simple, but not too simple, and tests should be code agnostic. Developing verification methods is still open research. Testing single-physics code pieces in isolation may lead to bad results when they are coupled. Good verification tests must include multiple physics models, multiple materials, and be performed in multiple dimensions.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2023

Automated Credibility Assessments of User Features in Scientific Software

Scientific software (SciSoft) is complex, often containing a mixture of production capabilities co-mingled with features under active research and development. Furthermore, SciSoft is often developed over decades by non-computer scientists who may not have a strong background in or prioritize software architecture design, testing, and quality (e.g., test coverage). These conditions lead to difficulty in understanding which software components or functions implement what user-facing features and therefore those features’ software quality pedigree. This lack of understanding poses challenges in assessing readiness and credibility of user features, and often relies on a SciSoft subject matter expert’s (SME) laborious investigation and assertion. This final report of a one-year Computing and Information Sciences Lab Directed Research and Development project presents a general framework for modeling SciSoft architecture as a direct relationship between user features and the software components/functions that implement them. Our approach leverages automated labeling of the SciSoft’s regression test suite and employs machine learning algorithms to construct the architecture model. We demonstrate this framework on the Solid Mechanics component of the SIERRA multi-physics engineering analysis suite developed at Sandia National Laboratories.

97 MATHEMATICS AND COMPUTING↗
NASA NTRS1981

The quality assurance role in the Deep Space Network

The quality assurance section provides skilled personnel who perform the following functions: quality hardware engineering, quality software engineering, inspection and testing, and an independent assessment to quality, engineering and project requirements. The quality assurance section provides specifications, workmanship standards, process specifications and quality assurance procedures in support of DSN equipment. In addition, the quality assurance section reviews DSN engineering documents to ensure that appropriate and adequate quality provisions are included.

Sirpilla, R. L.↗
NASA NTRS1987

Space station software reliability analysis based on failures observed during testing at the multisystem integration facility

Quality of software not only is vital to the successful operation of the space station, it is also an important factor in establishing testing requirements, time needed for software verification and integration as well as launching schedules for the space station. Defense of management decisions can be greatly strengthened by combining engineering judgments with statistical analysis. Unlike hardware, software has the characteristics of no wearout and costly redundancies, thus making traditional statistical analysis not suitable in evaluating reliability of software. A statistical model was developed to provide a representation of the number as well as types of failures occur during software testing and verification. From this model, quantitative measure of software reliability based on failure history during testing are derived. Criteria to terminate testing based on reliability objectives and methods to estimate the expected number of fixings required are also presented.

Tamayo, Tak Chai↗
DOE OSTI2022

CTF Theory Manual: Version 4.3

Coolant-Boiling in Rod Arrays—Two Fluids (COBRA-TF) is a thermal/hydraulic (T/H) simulation code designed for light water reactor (LWR) vessel analysis. It uses a two-fluid, three-field (i.e., fluid film, fluid drops, and vapor) modeling approach. Both subchannel and three-dimensional Cartesian forms of its governing equations are available for the solution. The code was originally developed by Pacific Northwest Laboratory in 1980, and had been used and modified by several institutions over the last few decades. COBRA-TF also found use at the Pennsylvania State University (PSU) by the Reactor Dynamics and Fuel Modeling Group (RDFMG) and has been improved, updated, and subsequently rebranded as CTF. CTF was later adopted in the early 2010s by Oak Ridge National Laboratory (ORNL) for use in the Consortium for Advanced Simulation of Light Water Reactors (CASL) program, which led to a significant advancement of the code software quality, modeling accuracy, testing systems, and capabilities for improved support of modeling of common LWR nominal and transient behavior. As part of the improvement process, it was necessary to generate sufficient documentation for the public domain code which had lacked such material upon being adopted by RDFMG. This document serves as a theory manual for CTF, detailing the many two-phase heat transfer, drag, and important accident scenario models contained in the code, as well as the numerical solution process utilized. Additional documents available in the CTF documentation suite include the user manual and verification and validation manual.

22 GENERAL STUDIES OF NUCLEAR REACTORS↗
DOE OSTI2023

CTF Theory Manual: Version 4.4

Coolant-Boiling in Rod Arrays – Two Fluids (COBRA-TF) is a thermal/hydraulic (T/H) simulation code designed for light-water reactor (LWR) vessel analysis. It uses a two-fluid, three-field (i.e., fluid film, fluid drops, and vapor) modeling approach. Both subchannel and 3D Cartesian forms of its governing equations are available for the solution. The code was originally developed by Pacific Northwest Laboratory in 1980 and has been used and modified by several institutions over the last few decades. COBRA-TF also found use at the Pennsylvania State University (PSU) by the Reactor Dynamics and Fuel Modeling Group (RDFMG) and has been improved, updated, and subsequently rebranded as CTF. CTF was later adopted in the early 2010s by Oak Ridge National Laboratory (ORNL) for use in the Consortium for Advanced Simulation of Light Water Reactors (CASL) program, which led to a significant advancement of the code’s software quality, modeling accuracy, testing systems, and capabilities for improved support in modeling common LWR nominal and transient behavior. As part of the improvement process, it was necessary to generate sufficient documentation for the public domain code which had lacked such material upon being adopted by RDFMG. This document serves as a theory manual for CTF, detailing the many two-phase heat transfer, drag, and important accident scenario models contained in the code, as well as the numerical solution process utilized. Additional documents available in the CTF documentation suite include the user manual and the verification and validation manual.

22 GENERAL STUDIES OF NUCLEAR REACTORS↗
NASA NTRS1991

Improving the Software Development Process Using Testability Research

Software testability is the the tendency of code to reveal existing faults during random testing. This paper proposes to take software testability predictions into account throughout the development process. These predictions can be made from formal specifications, design documents, and the code itself. The insight provided by software testability is valuable during design, coding, testing, and quality assurance. We further believe that software testability analysis can play a crucial role in quantifying the likelihood that faults are not hiding after testing does not result in any failures for the current version.

Voas, Jeffrey M.↗
DOE OSTI2022

SCO#1197 Addendum #4 OSR Characterization Database (Test Report)

A test plan was developed and approved in June of 2022. Testing was successfully performed to verify the functions of Version 2.0 of the OSR Characterization database. Checks verified that data remains consistent to tests previously performed using Microsoft Access® 2003, 2010 and 2016.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2021

BeeSwarm: Enabling Parallel Scaling Performance Measurement in Continuous Integration for HPC Applications

Testing is one of the most important steps in software development–it ensures the quality of software. Continuous Integration (CI) is a widely used testing standard that can report software quality to the developer in a timely manner during development progress. Performance, especially scalability, is another key factor for High Performance Computing (HPC) applications. There are many existing profiling and performance tools for HPC applications, but none of these are integrated into CI tools. In this work, we propose BeeSwarm, an HPC container based parallel scaling performance system that can be easily applied to the current CI test environments. BeeSwarm is mainly designed for HPC application developers who need to monitor how their applications can scale on different compute resources. We demonstrate BeeSwarm using a multi-physics HPC application with Travis CI, GitLab CI and GitHub Actions while using ChameleonCloud and Google Compute Engine as the compute backends. Finally, our results show that BeeSwarm can be used for scalability and performance testing of HPC applications.

97 MATHEMATICS AND COMPUTING↗
NASA NTRS2014

Autonomous Real Time Requirements Tracing

One of the more challenging aspects of software development is the ability to verify and validate the functional software requirements dictated by the Software Requirements Specification (SRS) and the Software Detail Design (SDD). Insuring the software has achieved the intended requirements is the responsibility of the Software Quality team and the Software Test team. The utilization of Timeliner-TLX(sup TM) Auto- Procedures for relocating ground operations positions to ISS automated on-board operations has begun the transition that would be required for manned deep space missions with minimal crew requirements. This transition also moves the auto-procedures from the procedure realm into the flight software arena and as such the operational requirements and testing will be more structured and rigorous. The autoprocedures would be required to meet NASA software standards as specified in the Software Safety Standard (NASASTD- 8719), the Software Engineering Requirements (NPR 7150), the Software Assurance Standard (NASA-STD-8739) and also the Human Rating Requirements (NPR-8705). The Autonomous Fluid Transfer System (AFTS) test-bed utilizes the Timeliner-TLX(sup TM) Language for development of autonomous command and control software. The Timeliner-TLX(sup TM) system has the unique feature of providing the current line of the statement in execution during real-time execution of the software. The feature of execution line number internal reporting unlocks the capability of monitoring the execution autonomously by use of a companion Timeliner-TLX(sup TM) sequence as the line number reporting is embedded inside the Timeliner-TLX(sup TM) execution engine. This negates I/O processing of this type data as the line number status of executing sequences is built-in as a function reference. This paper will outline the design and capabilities of the AFTS Autonomous Requirements Tracker, which traces and logs SRS requirements as they are being met during real-time execution of the targeted system. It is envisioned that real time requirements tracing will greatly assist the movement of autoprocedures to flight software enhancing the software assurance of auto-procedures and also their acceptance as reliable commanders.

Plattsmier, George↗
NASA NTRS2014

Autonomous Real Time Requirements Tracing

One of the more challenging aspects of software development is the ability to verify and validate the functional software requirements dictated by the Software Requirements Specification (SRS) and the Software Detail Design (SDD). Insuring the software has achieved the intended requirements is the responsibility of the Software Quality team and the Software Test team. The utilization of Timeliner-TLX(sup TM) Auto-Procedures for relocating ground operations positions to ISS automated on-board operations has begun the transition that would be required for manned deep space missions with minimal crew requirements. This transition also moves the auto-procedures from the procedure realm into the flight software arena and as such the operational requirements and testing will be more structured and rigorous. The autoprocedures would be required to meet NASA software standards as specified in the Software Safety Standard (NASASTD- 8719), the Software Engineering Requirements (NPR 7150), the Software Assurance Standard (NASA-STD-8739) and also the Human Rating Requirements (NPR-8705). The Autonomous Fluid Transfer System (AFTS) test-bed utilizes the Timeliner-TLX(sup TM) Language for development of autonomous command and control software. The Timeliner- TLX(sup TM) system has the unique feature of providing the current line of the statement in execution during real-time execution of the software. The feature of execution line number internal reporting unlocks the capability of monitoring the execution autonomously by use of a companion Timeliner-TLX(sup TM) sequence as the line number reporting is embedded inside the Timeliner-TLX(sup TM) execution engine. This negates I/O processing of this type data as the line number status of executing sequences is built-in as a function reference. This paper will outline the design and capabilities of the AFTS Autonomous Requirements Tracker, which traces and logs SRS requirements as they are being met during real-time execution of the targeted system. It is envisioned that real time requirements tracing will greatly assist the movement of autoprocedures to flight software enhancing the software assurance of auto-procedures and also their acceptance as reliable commanders

Plattsmier, George I.↗
NASA NTRS1991

Development of a calibrated software reliability model for flight and supporting ground software for avionic systems

The object of this project was to develop and calibrate quantitative models for predicting the quality of software. Reliable flight and supporting ground software is a highly important factor in the successful operation of the space shuttle program. The models used in the present study consisted of SMERFS (Statistical Modeling and Estimation of Reliability Functions for Software). There are ten models in SMERFS. For a first run, the results obtained in modeling the cumulative number of failures versus execution time showed fairly good results for our data. Plots of cumulative software failures versus calendar weeks were made and the model results were compared with the historical data on the same graph. If the model agrees with actual historical behavior for a set of data then there is confidence in future predictions for this data. Considering the quality of the data, the models have given some significant results, even at this early stage. With better care in data collection, data analysis, recording of the fixing of failures and CPU execution times, the models should prove extremely helpful in making predictions regarding the future pattern of failures, including an estimate of the number of errors remaining in the software and the additional testing time required for the software quality to reach acceptable levels. It appears that there is no one 'best' model for all cases. It is for this reason that the aim of this project was to test several models. One of the recommendations resulting from this study is that great care must be taken in the collection of data. When using a model, the data should satisfy the model assumptions.

Lawrence, Stella↗
NASA NTRS1991

Software component quality evaluation

The paper describes a software inspection process that can be used to evaluate the quality of software components. Quality criteria, process application, independent testing of the process and proposed associated tool support are covered. Early results indicate that this technique is well suited for assessing software component quality in a standardized fashion. With automated machine assistance to facilitate both the evaluation and selection of software components, such a technique should promote effective reuse of software components.

Clough, A. J.↗
NASA NTRS1998

Testing, Requirements, and Metrics

The criticality of correct, complete, testable requirements is a fundamental tenet of software engineering. Also critical is complete requirements based testing of the final product. Modern tools for managing requirements allow new metrics to be used in support of both of these critical processes. Using these tools, potential problems with the quality of the requirements and the test plan can be identified early in the life cycle. Some of these quality factors include: ambiguous or incomplete requirements, poorly designed requirements databases, excessive or insufficient test cases, and incomplete linkage of tests to requirements. This paper discusses how metrics can be used to evaluate the quality of the requirements and test to avoid problems later. Requirements management and requirements based testing have always been critical in the implementation of high quality software systems. Recently, automated tools have become available to support requirements management. At NASA's Goddard Space Flight Center (GSFC), automated requirements management tools are being used on several large projects. The use of these tools opens the door to innovative uses of metrics in characterizing test plan quality and assessing overall testing risks. In support of these projects, the Software Assurance Technology Center (SATC) is working to develop and apply a metrics program that utilizes the information now available through the application of requirements management tools. Metrics based on this information provides real-time insight into the testing of requirements and these metrics assist the Project Quality Office in its testing oversight role. This paper discusses three facets of the SATC's efforts to evaluate the quality of the requirements and test plan early in the life cycle, thus preventing costly errors and time delays later.

Rosenberg, Linda↗
NASA NTRS1978

Standardized development of computer software. Part 2: Standards

This monograph contains standards for software development and engineering. The book sets forth rules for design, specification, coding, testing, documentation, and quality assurance audits of software; it also contains detailed outlines for the documentation to be produced.

Tausworthe, R. C.↗
NASA NTRS1994

Test/score/report: Simulation techniques for automating the test process

A Test/Score/Report capability is currently being developed for the Transportable Payload Operations Control Center (TPOCC) Advanced Spacecraft Simulator (TASS) system which will automate testing of the Goddard Space Flight Center (GSFC) Payload Operations Control Center (POCC) and Mission Operations Center (MOC) software in three areas: telemetry decommutation, spacecraft command processing, and spacecraft memory load and dump processing. Automated computer control of the acceptance test process is one of the primary goals of a test team. With the proper simulation tools and user interface, the task of acceptance testing, regression testing, and repeatability of specific test procedures of a ground data system can be a simpler task. Ideally, the goal for complete automation would be to plug the operational deliverable into the simulator, press the start button, execute the test procedure, accumulate and analyze the data, score the results, and report the results to the test team along with a go/no recommendation to the test team. In practice, this may not be possible because of inadequate test tools, pressures of schedules, limited resources, etc. Most tests are accomplished using a certain degree of automation and test procedures that are labor intensive. This paper discusses some simulation techniques that can improve the automation of the test process. The TASS system tests the POCC/MOC software and provides a score based on the test results. The TASS system displays statistics on the success of the POCC/MOC system processing in each of the three areas as well as event messages pertaining to the Test/Score/Report processing. The TASS system also provides formatted reports documenting each step performed during the tests and the results of each step. A prototype of the Test/Score/Report capability is available and currently being used to test some POCC/MOC software deliveries. When this capability is fully operational it should greatly reduce the time necessary to test a POCC/MOC software delivery, as well as improve the quality of the test process.

Hageman, Barbara H.↗
NASA NTRS2017

NASA's Space Launch System Takes Shape

Major hardware and software for NASA's Space Launch System (SLS) began rolling off assembly lines in 2016, setting the stage for critical testing in 2017 and the launch of a major new capability for deep space human exploration. SLS continues to pursue a 2018 first launch of Exploration Mission 1 (EM-1). At NASA's Michoud Assembly Facility near New Orleans, LA, Boeing completed welding of structural test and flight liquid hydrogen tanks, and engine sections. Test stands for core stage structural tests at NASA's Marshall Space Flight Center, Huntsville, AL. neared completion. The B2 test stand at NASA's Stennis Space Center, MS, completed major structural renovation to support core stage green run testing in 2018. Orbital ATK successfully test fired its second qualification solid rocket motor in the Utah desert and began casting the motor segments for EM-1. Aerojet Rocketdyne completed its series of test firings to adapt the heritage RS-25 engine to SLS performance requirements. Production is under way on the first five new engine controllers. NASA also signed a contract with Aerojet Rocketdyne for propulsion of the RL10 engines for the Exploration Upper Stage. United Launch Alliance delivered the structural test article for the Interim Cryogenic Propulsion Stage to MSFC for tests and construction was under way on the flight stage. Flight software testing at MSFC, including power quality and command and data handling, was completed. Substantial progress is planned for 2017. Liquid oxygen tank production will be completed at Michoud. Structural testing at Marshall will get under way. RS-25 hotfire testing will verify the new engine controllers. Core stage horizontal integration will begin. The core stage pathfinder mockup will arrive at the B2 test stand for fit checks and tests. EUS will complete preliminary design review. This paper will discuss the technical and programmatic successes and challenges of 2016 and look ahead to plans for 2017.

Askins, Bruce↗