Results for “software quality, testing”

DOE OSTI2024

ORNL Package Testing Program Software Quality Assurance Plan

The Oak Ridge National Laboratory (ORNL) Package Testing Program (PTP) uses commercial off-the-shelf (COTS) software in performing data collection of thermal test results for package designs that contain radioactive materials. Specifically, this software is used to collect temperature data from the furnace, packages, and ambient air to prepare and execute the thermal test specified in 10 CFR 71.73, “Thermal Test.” This software quality assurance (SQA) plan sets forth the guidelines, standards, and procedures that shall be used to provide SQA for PTP software applications. This is a living document that will be maintained for the lifecycle of the PTP program. The SQA plan follows the requirements set forth in ORNL Standards Based Management System (SBMS): Information Technology; Subject Area: Software Quality Assurance. When applicable to the requirements as described in ORNL SBMS, Software Quality Assurance, the software shall be listed in the ORNL Software Registration System (SRS). Exemptions to this SBMS are COTS and firmware that are not modified; spreadsheet applications and personal productivity tools that do not have a utility or safety application, research applications, legacy software, system software, vendor-supplied software used to interface with the vendor’s services, software used within the organization to facilitate processing or management of information, and software developed for applications not specific to the US Department of Energy (DOE).

97 MATHEMATICS AND COMPUTING↗

DOE OSTI2026

Exploring Sustainability in Scientific Software through Code Quality & Test Coverage Metrics

Context: Scientific open-source software (SciOSS) plays a foundational role in research and engineering, yet its long-term sustainability has often been overlooked and remains a significant concern. Objective: This study investigates the long-term sustainability of SciOSS through code and test quality metrics. Method: We analyze CASS Software Portfolio projects, classifying them by sustainability and comparing their code structure, test coverage, and links between code quality and testing across the dataset. Results: Sustainable projects show higher, more consistent test coverage and clearer code-test correlations, while unsustainable ones show weaker patterns. Overall, test coverage is low in scientific software, and high complexity and coupling reduce testability. Conclusion: In this study, we present a practical, data-driven approach for assessing sustainability in scientific software, offering a foundation for evaluating long-term software health and supporting future efforts in quality assurance and sustainability monitoring.

Md mushfiqur rahman, Sheikh [University of Tenness↗

DOE OSTI2021

Implementation of Plot File Testing in the DYNA3D/ParaDyn Software Quality Assurance Suite

Automated testing of DYNA3D/ParaDyn plot files was added to the DYNA3D/ParaDyn software quality assurance (SQA) test suite. The new capability extracts select data from the plot files generated during each verification run and compares it to the same baseline answers used to verify the problem. Deviations between baseline answers and plot file values are reported in the same manner as solution discrepancies, and differences in precision levels between the baseline answers and plot file results are accounted for. The new testing leverages the existing SQA test suite framework and test problems and the Python Mili reader and minimally increases the overall run time (< 5%) of the SQA test suite. This new capability provides incremental end-toend testing of the most common DYNA3D/ParaDyn simulation workflows.

42 ENGINEERING↗

DOE OSTI2022

FY2021 Improvements to the New CTH Code Verification & Validation Test Suite

Over the past few years, the CTH multiphysics hydrocode has overhauled its software quality and testing processes, implementing current best practices in software quality and building a robust V&V test suite comprised of traditional hydrocode verification problems, including ASC Tri-Lab Test Suite and Enhanced Tri-Lab Test Suite problems, as well as validation problems for some of CTH’s most frequently used equations of state, materials models, and other key capabilities. Substantial progress towards building this new test suite was made in FY19 and FY20. In FY21, the test suite has been expanded to include verification and validation tests of the Steinberg-Guinan-Lund (ST) viscoplastic model and the Johnson Cook (JFRAC) fracture model. Additionally, two new verification tests were added, covering hydrodynamics and high explosive (HE) modeling capabilities: the Kidder Gaussian density problem and the Escape of HE Products (EHEP) problem from the Tri-Lab Test Suite. This report discusses each of these test problems in detail. Verification test results are compared to analytic solutions. Validation test results are compared to experimental data. Wherever possible, convergence or mesh refinement studies are included. Additionally, while implementing the Kidder verification problem, a bug was identified that affects the use of tables to initialize pressure or density in 1D or 2D calculations. A brief discussion of the bug and its fix is included. CTH demonstrates good performance overall on the new test suite problems. Simulation results showed good agreement with analytic solutions for the Kidder problem, with convergence rates ranging between 1.8 and sub-linear, and relatively good agreement for the EHEP problem, though convergence rates for pressure and density were nearly 0. The ST and JFRAC strain rate loading verification tests show good agreement with analytic solutions. Likewise, CTH simulation results show good agreement with experimental validation data, including Taylor rod impact testing, for the materials tested. Future V&V work will focus on adding 2D and 3D versions of existing verification tests as well as adding validation tests of other frequently used capabilities such as other fracture models.

42 ENGINEERING↗

DOE OSTI2023

Verification and Regression Testing of a Physically Stabilized Layered Solid Element Formulation in DYNA3D/Paradyn

Recent modeling of filament-wound composite structures drove the need for a new element type within the finite element code DYNA3D/Paradyn. This new layered solid element was implemented and designed to capture the kinematics and constitutive be havior of various lamina layers defined with arbitrary orientations and volume fractions within the element to accurately model a laminated composite material. The new el ement uses a single integration point in each of the lamina layers defined to capture the constitutive behavior, and a novel physical stabilization routine is used to prevent hourglassing while mitigating shear and volumetric locking. This report documents the verification testing and the regression tests added to the DYNA3D/Paradyn software quality assurance test suite to assess the proper implementation of the new layered solid element. There were a total of 27 tests added the DYNA3D test suite, which consist of problems using the layered solid element with a single layer or multiple layers for isotropic and orthotropic material models. These tests are a combination of simple kinematically driven patch tests, beam bending tests, plate bending tests, and more complicated problems. Ultimately, the testing done verifies that the element behaves as expected and is implemented correctly.

42 ENGINEERING↗

DOE OSTI2021

Continuous Integration, In-Code Documentation, and Automation for Nuclear Quality Assurance Conformance

The Multiphysics Object Oriented Simulation Environment (MOOSE) is an open-source, finite element framework for solving highly coupled sets of nonlinear equations. The development of the framework and applications occurs concurrently using an agile, continuous-integration software package. Included in the framework is an in-code, extensible documentation system. Using these two tools in union with the repository management tools GitHub and GitLab, a software quality plan was created and followed such that MOOSE and a MOOSE-based application (BISON) have been shown to meet the American Society of Mechanical Engineers’ Nuclear Quality Assurance-1 standard. The approach relies heavily on automation for both testing and documentation. The resulting effort demonstrates that a rigorous software quality plan may be implemented that incurs a minimal impact on day-to-day development of the software, satisfying the stringent guidelines necessary to operate the software in a safety function within a nuclear facility.

97 MATHEMATICS AND COMPUTING↗

DOE OSTI2023

Verification Problems for Smooth Step Amplitude Load Curves in DYNA3D/Paradyn

This report documents the addition of three new verification tests in the LOADCURVE directory of the DYNA3D/Paradyn Software Quality Assurance test suite. Each test consists of a single element, where the velocities of each node are specified by either the newly added smooth step tabular load curve or another load curve option. The first test assesses the initialization and interpolation of the newly inputted load curve option through tabulated abscissa-ordinate pairs of data. The second test uses the same set of abscissa-ordinate data points and applies offset and scaling parameters available within the load curve definition. The third test defines the smooth step load curve in an original input deck, and assesses its correct redefinition using a restart file. The simulation velocities are compared to their true values at discrete points in time, and each test is verified up to numerical precision. These results confirm that the smooth step load curve option is functioning correctly and as intended.

97 MATHEMATICS AND COMPUTING↗

DOE OSTI2021

Verification Testing For Solid-Element Material Models 11-19 in DYNA3D/ParaDyn

This technical report documents the creation and implementation of verification tests for solid-element material models 11 through 19 available in DYNA3D/ParaDyn. The verification tests covered all aspects of each material model, except for the Weibull distribution functionality in material models 15 and 19. General test cases were created to verify the elastic and plastic behavior of the material models. Other additional tests were developed to examine the intricacies of each material model. Each test involved the use a kinematic load case and specification of material parameters necessary to activate corresponding features of the material model. The load cases prescribed the full time history of the kinematic motion for the solid elements, and these loads are independent of the material model or element formulation. When possible, closed form solutions were then derived for each verification test in a continuum setting. The DYNA3D simulations for each test were carried out over a unit time interval, t ϵ [0, 1], and the as implemented DYNA3D response was compared to the closed form solutions evaluated at discrete points in time. A relative error measure was determined for each test to justify the proper implementation of the material model. The relative errors comparing the DYNA3D solution to the analytical solution were, in general, on the order of machine precision except where noted. This signifies the proper implementation of solid-element material models 11-19. In the development of these verification tests, six bugs were found and fixed in the source code. Additionally, this work generated eighteen DYNA3D input decks and answer extraction scripts in the DYNA3D/ParaDyn Software Quality Assurance test suite, which are comprised of a total of 285 solid-element tests. Testing for each material model utilizes two input decks and answer extraction scripts, where one focuses on the linear elastic response and the other examines the inelastic and remaining functionalities of the material model. In total, this work added 285 individual verification test problems in the DYNA3D/ParaDyn test suite.

42 ENGINEERING↗

DOE OSTI2020

Status Report on IES Plug-and-Play Framework

This report discusses the status of the flexible plug-and-play framework development currently ongoing that aims to integrate Modelica/Dymola with the Risk Analysis and Virtual ENviroment (RAVEN) software in terms of both Functional Mock-Up Interface (FMI)/Functional Mock-Up Unit (FMU) construction and repository structures that aim to ease the sharing and simulation of complex dynamic models. This report provides discusses the FMI/FMU adaptors that have been created within the HYBRID repository to allow users to quickly export models as FMUs. Several examples are shown that highlight the step-by-step process of converting an existing Modelica model into an FMU for use within the Dymola platform. Simulation results demonstrate that, while minor differences may occur, the overall control, trends, and solution integrity is maintained between standard Modelica simulation and FMU simulation results. However, it is worth noting that, for small systems, the FMU results have a slower simulation time than the Modelica only simulation. Using this process, a company can provide models that contain proprietary information to entities without disclosing any of the information about the model that could be considered business sensitive. Such an ability would allow institutions to bypass the necessity of “whitewashing” data. In addition to the investigative work being conducted on FMUs and FMIs, a series of updates to the hybrid repository has been completed. These updates include the addition of Modelica system-level regression tests and software quality assurance documentation that ensure that modifications to the Modelica models do not alter system-level model results. Overall, extensive work has been completed on developing FMUs and FMIs from existing models, understanding the requirements and limitations of FMUs, and opensourcing the HYBRID repository with an integrated regression system.

99 GENERAL AND MISCELLANEOUS↗

DOE OSTI2021

Development of the IES Plug-and-Play Framework

This report discusses the status of the flexible plug-and-play framework development currently ongoing that aims to integrate Modelica/Dymola with the Risk Analysis and Virtual ENvironment (RAVEN) software in terms of both Functional Mock-Up Interface (FMI)/Functional Mock-Up Unit (FMU) construction and repository structures that aim to ease the sharing and simulation of complex dynamic models. This report aims to provide an overview of all the performed activities resolving around the deployment of methods, software infrastructures, guidelines and workflow for the construction and usage of models, encapsulated using the FMI/FMU protocols and standards. In particular, the report is organized in three main macro-subjects, which are connected to each other: - FMI/FMU adaptors for modelica models - HYBRID repository new structure and open-source deployment - RAVEN FMI/FMU exporting capabilities and Artificial Intelligence (AI)-based analysis acceleration. The first part of the report discusses the FMI/FMU adaptors that have been created within the HYBRID repository to allow users to quickly export models, such as FMUs. Several examples are shown that highlight the step-by-step process of converting an existing Modelica model into an FMU for use within the Dymola platform. Simulation results demonstrate that, while minor differences may occur, the overall control, trends, and solution integrity are maintained between standard Modelica simulation and FMU simulation results. However, it is worth noting that, for small systems, the FMU results have a slower simulation time than the Modelica only simulation. Using this process, a company can provide models that contain proprietary information to entities without disclosing any of the information about the model that could be considered business sensitive. Such an ability would allow institutions to bypass the necessity of “whitewashing” data. In the second part of the report, the new structure of the HYBRID repository is discussed with a major focus on the series of updates that has been completed. These updates include the addition of Modelica system-level regression tests and software quality assurance documentation that ensure that modifications to the Modelica models do not alter system-level model results. The third and final part of the report aims to report the work that has been performed for the deployment of methods and workflows for the construction of RAVEN AI-based models compliant with the FMI/FMU standard. Such development represents the key for the deployment of the concept of “Flexible ecosystem” since it allows for the replacement of high-fidelity modelica models (or any other FMI/FMU compliant model) with RAVEN generated AI surrogate models. Overall, extensive work has been completed on developing FMUs and FMIs from existing models, understanding the requirements and limitations of FMUs, and open-sourcing the HYBRID repository with an integrated regression system.

42 ENGINEERING↗

DOE OSTI2021

Development of the IES Plug-and-Play Framework

This report discusses the status of the flexible plug-and-play framework development currently ongoing that aims to integrate Modelica/Dymola with the Risk Analysis and Virtual ENvironment (RAVEN) software in terms of both Functional Mock-Up Interface (FMI)/Functional Mock-Up Unit (FMU) construction and repository structures that aim to ease the sharing and simulation of complex dynamic models. This report aims to provide an overview of all the performed activities resolving around the deployment of methods, software infrastructures, guidelines and workflow for the construction and usage of models, encapsulated using the FMI/FMU protocols and standards. In particular, the report is organized in three main macro-subjects, which are connected to each other: - FMI/FMU adaptors for modelica models - HYBRID repository new structure and open-source deployment - RAVEN FMI/FMU exporting capabilities and Artificial Intelligence (AI)-based analysis acceleration. The first part of the report discusses the FMI/FMU adaptors that have been created within the HYBRID repository to allow users to quickly export models, such as FMUs. Several examples are shown that highlight the step-by-step process of converting an existing Modelica model into an FMU for use within the Dymola platform. Simulation results demonstrate that, while minor differences may occur, the overall control, trends, and solution integrity are maintained between standard Modelica simulation and FMU simulation results. However, it is worth noting that, for small systems, the FMU results have a slower simulation time than the Modelica only simulation. Using this process, a company can provide models that contain proprietary information to entities without disclosing any of the information about the model that could be considered business sensitive. Such an ability would allow institutions to bypass the necessity of “whitewashing” data. In the second part of the report, the new structure of the HYBRID repository is discussed with a major focus on the series of updates that has been completed. These updates include the addition of Modelica system-level regression tests and software quality assurance documentation that ensure that modifications to the Modelica models do not alter system-level model results. The third and final part of the report aims to report the work that has been performed for the deployment of methods and workflows for the construction of RAVEN AI-based models compliant with the FMI/FMU standard. Such development represents the key for the deployment of the concept of “Flexible ecosystem” since it allows for the replacement of high-fidelity modelica models (or any other FMI/FMU compliant model) with RAVEN generated AI surrogate models. Overall, extensive work has been completed on developing FMUs and FMIs from existing models, understanding the requirements and limitations of FMUs, and open-sourcing the HYBRID repository with an integrated regression system.

14 SOLAR ENERGY↗

DOE OSTI2023

Verification Testing of Body Forces due to a Prescribed Angular Velocity in DYNA3D/Paradyn

This report documents the verification testing and regression testing done on a new feature in DYNA3D/Paradyn, which allows users to prescribe body force loads based upon an angular velocity. This feature is unique in that the direction of the angular velocity vector follows the unit vector formed by two coordinate points associated with two nodes or the average coordinates of two separate small collection of nodes. The angular velocity direction will follow the directional vector defined by these nodes while the angular velocity magnitude is defined by a load curve. A simple single element verification test was performed to determine the correct implementation of this feature, and two separate regression tests were added to the DYNA3D Software Quality Assur ance test suite. The nodal positions, velocities, and accelerations from the solution of the single element verification test compare well to analytically derived values of those nodal quantities. The regression tests serve as good examples of this new feature’s use case and were consequently added to the SQA test suite to ensure that further modifications of the DYNA3D source code do not unintentionally change the generated baseline answers.

42 ENGINEERING↗

DOE OSTI2023

MARVEL 90% Final Design Report

This document provides documentation of the Microreactor Applications Research Validation and Evaluation Project’s (MARVEL) 90% Final Design, as required by U.S. Department of Energy (DOE) Standard-1189, “Integration of Safety into the Design Process." Per DOE-STD-1189-2016, the 90% Final Design documentation focuses on design completion, at a level capable of supporting procurement, construction, testing, and operation. At this phase, the design organization finalizes the hazards and accident analyses, Fire Hazard Analysis (FHA), security vulnerability assessments, and other supporting analyses for design completion. The objective of this report is to provide a high-level summary of the design thus far and provide references including, but not limited to, the following design deliverables: • Complete final drawings, specifications and commercial grade dedications that may be released for bid and/or construction. • Clearly defined testing plans for the safety and functionality of all subsystems. • Quality Assurance Program for Design, Testing and Procurement. • Software Quality Assurance Plan. • Code of Record (COR), applicable design requirements including codes and standards. • Final design that meets all the requirements stipulated in the COR. • Final design review, consisting of final validation of comment resolution from previous reviews, and a review of any additional developments since the last review. • Updated Safety Design Strategy. • Hazard Analysis. • Fire Hazard Analysis. • Accident analysis. • Security vulnerability assessment. • Current and detailed cost estimate. • Current construction schedule, and • Risk & Opportunities Assessment.

21 SPECIFIC NUCLEAR REACTORS AND ASSOCIATED PLANTS↗

DOE OSTI2023

Spring 2023 Verification Presentation to Headquarters [Slides]

Testing codes is important, and more tests are needed. Verification testing is different than software-quality assurance, like regression suites. Tests should be simple, but not too simple, and tests should be code agnostic. Developing verification methods is still open research. Testing single-physics code pieces in isolation may lead to bad results when they are coupled. Good verification tests must include multiple physics models, multiple materials, and be performed in multiple dimensions.

97 MATHEMATICS AND COMPUTING↗

DOE OSTI2023

Automated Credibility Assessments of User Features in Scientific Software

Scientific software (SciSoft) is complex, often containing a mixture of production capabilities co-mingled with features under active research and development. Furthermore, SciSoft is often developed over decades by non-computer scientists who may not have a strong background in or prioritize software architecture design, testing, and quality (e.g., test coverage). These conditions lead to difficulty in understanding which software components or functions implement what user-facing features and therefore those features’ software quality pedigree. This lack of understanding poses challenges in assessing readiness and credibility of user features, and often relies on a SciSoft subject matter expert’s (SME) laborious investigation and assertion. This final report of a one-year Computing and Information Sciences Lab Directed Research and Development project presents a general framework for modeling SciSoft architecture as a direct relationship between user features and the software components/functions that implement them. Our approach leverages automated labeling of the SciSoft’s regression test suite and employs machine learning algorithms to construct the architecture model. We demonstrate this framework on the Solid Mechanics component of the SIERRA multi-physics engineering analysis suite developed at Sandia National Laboratories.

97 MATHEMATICS AND COMPUTING↗

DOE OSTI2022

CTF Theory Manual: Version 4.3

Coolant-Boiling in Rod Arrays—Two Fluids (COBRA-TF) is a thermal/hydraulic (T/H) simulation code designed for light water reactor (LWR) vessel analysis. It uses a two-fluid, three-field (i.e., fluid film, fluid drops, and vapor) modeling approach. Both subchannel and three-dimensional Cartesian forms of its governing equations are available for the solution. The code was originally developed by Pacific Northwest Laboratory in 1980, and had been used and modified by several institutions over the last few decades. COBRA-TF also found use at the Pennsylvania State University (PSU) by the Reactor Dynamics and Fuel Modeling Group (RDFMG) and has been improved, updated, and subsequently rebranded as CTF. CTF was later adopted in the early 2010s by Oak Ridge National Laboratory (ORNL) for use in the Consortium for Advanced Simulation of Light Water Reactors (CASL) program, which led to a significant advancement of the code software quality, modeling accuracy, testing systems, and capabilities for improved support of modeling of common LWR nominal and transient behavior. As part of the improvement process, it was necessary to generate sufficient documentation for the public domain code which had lacked such material upon being adopted by RDFMG. This document serves as a theory manual for CTF, detailing the many two-phase heat transfer, drag, and important accident scenario models contained in the code, as well as the numerical solution process utilized. Additional documents available in the CTF documentation suite include the user manual and verification and validation manual.

22 GENERAL STUDIES OF NUCLEAR REACTORS↗

DOE OSTI2023

CTF Theory Manual: Version 4.4

Coolant-Boiling in Rod Arrays – Two Fluids (COBRA-TF) is a thermal/hydraulic (T/H) simulation code designed for light-water reactor (LWR) vessel analysis. It uses a two-fluid, three-field (i.e., fluid film, fluid drops, and vapor) modeling approach. Both subchannel and 3D Cartesian forms of its governing equations are available for the solution. The code was originally developed by Pacific Northwest Laboratory in 1980 and has been used and modified by several institutions over the last few decades. COBRA-TF also found use at the Pennsylvania State University (PSU) by the Reactor Dynamics and Fuel Modeling Group (RDFMG) and has been improved, updated, and subsequently rebranded as CTF. CTF was later adopted in the early 2010s by Oak Ridge National Laboratory (ORNL) for use in the Consortium for Advanced Simulation of Light Water Reactors (CASL) program, which led to a significant advancement of the code’s software quality, modeling accuracy, testing systems, and capabilities for improved support in modeling common LWR nominal and transient behavior. As part of the improvement process, it was necessary to generate sufficient documentation for the public domain code which had lacked such material upon being adopted by RDFMG. This document serves as a theory manual for CTF, detailing the many two-phase heat transfer, drag, and important accident scenario models contained in the code, as well as the numerical solution process utilized. Additional documents available in the CTF documentation suite include the user manual and the verification and validation manual.

22 GENERAL STUDIES OF NUCLEAR REACTORS↗

DOE OSTI2022

SCO#1197 Addendum #4 OSR Characterization Database (Test Report)

A test plan was developed and approved in June of 2022. Testing was successfully performed to verify the functions of Version 2.0 of the OSR Characterization database. Checks verified that data remains consistent to tests previously performed using Microsoft Access® 2003, 2010 and 2016.

97 MATHEMATICS AND COMPUTING↗