Software Testing - Auditing Software Quality Assurance Plans
This report is an assurance plan for the software being audited for quality.
SEARCH · Engineering Papers
Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.
Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.
This report is an assurance plan for the software being audited for quality.
Papers are presented on the following topics: measurement of software technology, recent studies of the Software Engineering Lab, software management tools, expert systems, error seeding as a program validation technique, software quality assurance, software engineering environments (including knowledge-based environments), the Distributed Computing Design System, and various Ada experiments.
Requirements have always been acknowledged as the backbone of any system. However, in many past development efforts, requirements were paid little heed. At NASA, in recent years, the hue and cry for project development has been "Faster, Better, Cheaper and Safer". This has impacted the way we develop software; it has increased the risks to quality, safety and reliability. At NASA, the Software Assurance Technology Center (SATC) is working with projects to emphasize the criticality of requirements throughout development, not just in the initial phases. This emphasis is on requirements relationship to all aspects of quality, including reliability and safety. In this presentation, we will look at some of these relationships through the eyes of quality.
The Geochemist’s Workbench® (GWB) Release 10.0.10 is a geochemical modeling suite for simulating the reactive fate and transport of radionuclides and other metals in aqueous systems including groundwater, surface water, soil, and sediment. At the Savannah River National Laboratory, GWB plays an important role in the preparation of Performance Assessments involving cementitious materials. The software classification for GWB is Level “C.” GWB was purchased from Aqueous Solutions LLC and came into existence outside of the Software Quality Assurance procedures; therefore, it is considered “existing/acquired” software. This Software Quality Assurance Plan addresses verification techniques and the required life cycle components for Level “C” existing/acquired software (i.e., “Evaluation,”“Configuration Control,” and “Cyber Security Analysis”) with other elements optionally applied using a graded approach.
This letter report highlights the Software Quality Assurance (SQA) tasks that concentrated on the continued operation of the implemented framework of the SQA program along with the maintenance and general support activities of SAPHIRE for the duration of June 2019 – October 2019.
Automated testing of DYNA3D/ParaDyn plot files was added to the DYNA3D/ParaDyn software quality assurance (SQA) test suite. The new capability extracts select data from the plot files generated during each verification run and compares it to the same baseline answers used to verify the problem. Deviations between baseline answers and plot file values are reported in the same manner as solution discrepancies, and differences in precision levels between the baseline answers and plot file results are accounted for. The new testing leverages the existing SQA test suite framework and test problems and the Python Mili reader and minimally increases the overall run time (< 5%) of the SQA test suite. This new capability provides incremental end-toend testing of the most common DYNA3D/ParaDyn simulation workflows.
Complex Electronics (CE) now perform tasks that were previously handled in software, such as communication protocols. Many methods used to develop software bare a close resemblance to CE development. Field Programmable Gate Arrays (FPGAs) can have over a million logic gates while system-on-chip (SOC) devices can combine a microprocessor, input and output channels, and sometimes an FPGA for programmability. With this increased intricacy, the possibility of software-like bugs such as incorrect design, logic, and unexpected interactions within the logic is great. With CE devices obscuring the hardware/software boundary, we propose that mature software methodologies may be utilized with slight modifications in the development of these devices. Software Process Assurance for Complex Electronics (SPACE) is a research project that used standardized S/W Assurance/Engineering practices to provide an assurance framework for development activities. Tools such as checklists, best practices and techniques were used to detect missing requirements and bugs earlier in the development cycle creating a development process for CE that was more easily maintained, consistent and configurable based on the device used.
The basic idea underlying this paper is that the conventional understanding of the role of a Software Quality Assurance (SQA) engineer is unduly limited. This is because few have asked who the customers of a SQA engineer are. Once you do this, you can better define what tasks a SQA engineer should perform, as well as identify the knowledge and skills that such a person should have. The consequence of doing this is that a SQA engineer can provide greater value to his or her customers. It is the position of this paper that a SQA engineer providing significant value to his or her customers must not only assume the role of an auditor, but also that of a software and systems engineer. This is because software engineers and their managers particularly value contributions that directly impact products and their development. These ideas are summarized as lessons learned, based on my experience at Jet Propulsion Laboratory (JPL).
nal Aeronautics and Space Administration (NASA) have tens of thousands of networked computer systems and applications. Software Security vulnerabilities present risks such as lost or corrupted data, information the3, and unavailability of critical systems. These risks represent potentially enormous costs to NASA. The NASA Code Q research initiative 'Reducing Software Security Risk (RSSR) Trough an Integrated Approach '' offers, among its capabilities, formal verification of software security properties, through the use of model based verification (MBV) to address software security risks. [1,2,3,4,5,6] MBV is a formal approach to software assurance that combines analysis of software, via abstract models, with technology, such as model checkers, that provide automation of the mechanical portions of the analysis process. This paper will discuss: The need for formal analysis to assure software systems with respect to software and why testing alone cannot provide it. The means by which MBV with a Flexible Modeling Framework (FMF) accomplishes the necessary analysis task. An example of FMF style MBV in the verification of properties over the Secure Socket Layer (SSL) communication protocol as a demonstration.
This document includes the information pertaining to the practices of the Emergency Response Software Development Team (ERSDT) for the RASCAL software project. The content includes information regarding: 1) Software Quality Assurance. 2) Project Management. 3) Configuration Management. 4) Development Standards. 5) Third-Party Software. 6) Verification and Validation. The information contained in this report is considered living documentation. The information contained in this report was assembled from multiple documents and content management systems.
Complex Electronics (CE) are now programmed to perform tasks that were previously handled in software, such as communication protocols. Many of the methods used to develop software bare a close resemblance to CE development. For instance, Field Programmable Gate Arrays (FPGAs) can have over a million logic gates while system-on-chip (SOC) devices can combine a microprocessor, input and output channels, and sometimes an FPGA for programmability. With this increased intricacy, the possibility of software-like bugs such as incorrect design, logic, and unexpected interactions within the logic is great. Since CE devices are obscuring the hardware/software boundary, we propose that mature software methodologies may be utilized with slight modifications in the development of these devices. Software Process Assurance for Complex Electronics (SPACE) is a research project that looks at using standardized S/W Assurance/Engineering practices to provide an assurance framework for development activities. Tools such as checklists, best practices and techniques can be used to detect missing requirements and bugs earlier in the development cycle creating a development process for CE that will be more easily maintained, consistent and configurable based on the device used.
This report fulfills the requirements for the Consortium for the Advanced Simulation of Light-Water Reactors (CASL) milestone L2:RTM.P14.02, “SQA and Verification for MPACT Library Generation,” by documenting the current status of the software quality, verification, and acceptance testing of nuclear data libraries for MPACT. It provides a brief overview of the library generation process, from general-purpose evaluated nuclear data files (ENDF/B) to a problem-dependent cross section library for modeling of light-water reactors (LWRs). The software quality assurance (SQA) programs associated with each of the software used to generate the nuclear data libraries are discussed; specific tests within the SCALE/AMPX and VERA/XSTools repositories are described. The methods and associated tests to verify the quality of the library during the generation process are described in detail. The library generation process has been automated to a degree to (1) ensure that it can be run without user intervention and (2) to ensure that the library can be reproduced. Finally, the acceptance testing process that will be performed by representatives from the Radiation Transport Methods (RTM) Focus Area prior to the production library’s release is described in detail.
Ideally, quality is designed into software, just as quality is designed into hardware. However, when dealing with legacy systems, demonstrating that the software meets required quality standards may be difficult to achieve. As the need to demonstrate the quality of existing software was recognized at Los Alamos National Laboratory (LANL), an effort was initiated to uncover and demonstrate that legacy software met the required quality standards. This effort led to the development of a reverse engineering approach referred to as software archaeology. This paper documents the software archaeology approaches used at LANL to document legacy software systems. A case study for the Robotic Integrated Packaging System (RIPS) software is included.
Digital instrumentation and control (DI&C) systems monitor and control parameters in nuclear power plants. Ensuring their safety is a critical part of ensuring overall plant safety. Nuclear power plant licensing generates thousands of safety documents that could be organized more effectively using a safety assurance case (SAC). We conducted a literature survey of SACs and created a SAC framework for DI&C software using Goal Structuring Notation (GSN). This framework focuses on four software development processes: management & assurance, pre-developed software (PDS) qualification, the Software Development Life Cycle (SDLC), and the Machine Learning Development Life Cycle (MLDLC). We organized our framework using a novel level structure that can be applied to other SACs to improve their clarity. Finally, we demonstrate how our framework can be incorporated as part of a SAC for a larger reactor system.
The System Analysis Module (SAM) is an advanced and modern system analysis tool being developed at Argonne National Laboratory under the U.S. DOE Office of Nuclear Energy’s Nuclear Energy Advanced Modeling and Simulation (NEAMS) program. As a modern-day software, SAM development included efforts to follow best practices in software development. These best practices include version control using git, independent reviews of development activities, and detailed descriptions of developments and bug fixes using the GitLab issue and merge request system. In Fiscal Year 2023 (FY23), the SAM development team set out to formalize a Software Quality Assurance (SQA) program that allowed industry partners to credit the informal steps being taken by the SAM development team to ensure the quality of the software. As part of formalizing an SQA program, a SQA Plan (SQAP) was developed, implemented, and assessed. The SAM SQAP targets compliance with NQA-1-2008/2009 Addenda. The SQAP builds on the MOOSE SQAP and the Argonne SSQAPP while accounting for the needs of the multi-organization SAM development team. Modifications to the SAM repository structure, including a new testing system, updated test cases, and the development of an internal website, facilitate the implementation of the SAM SQAP. The initial assessment of the SAM SQAP indicated that the SAM SQAP was adequately and effectively implemented and additional work was required to improve the compliance of the SAM SQAP with NQA-1 2008/2009 Addenda. Several improvements to the SQAP have already been drafted to address these assessments and future work is planned to further improve the SAM SQAP in support of end-user needs.
The purpose of this document is for the verification and validation results of ANSYS Mechanical on an ORNL workstation. The ANSYS Mechanical verification and validation was performed in accordance with the ANSYS Mechanical software quality assurance plan.
The purpose of this document is to show the verification and validation results of ANSYS LSDYNA 2023R1 an ORNL workstation. The ANSYS LSDYNA verification and validation was done in accordance with the ANSYS LSDYNA 2023R1 software quality assurance plan.
Programmable Logic (PLC, FPGA, ASIC) devices are hybrids - hardware devices that are designed and programmed like software. As such, they fall in an assurance gray area. Programmable Logic is usually tested and verified as hardware, and the software aspects are ignored, potentially leading to safety or mission success concerns. The objective of this proposal is to first determine where and how Programmable Logic (PL) is used within NASA and document the current methods of assurance. Once that is known, raise awareness of the PL software aspects within the NASA engineering community and provide guidance for the use and assurance of PL form a software perspective.