Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “software verification”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 1,027 records · Page 57

Software Development Technologies for Reactive, Real-Time, and Hybrid Systems: Summary of Research

This research is directed towards the implementation of a comprehensive deductive-algorithmic environment (toolkit) for the development and verification of high assurance reactive systems, especially concurrent, real-time, and hybrid systems. For this, we have designed and implemented the STCP (Stanford Temporal Prover) verification system. Reactive systems have an ongoing interaction with their environment, and their computations are infinite sequences of states. A large number of systems can be seen as reactive systems, including hardware, concurrent programs, network protocols, and embedded systems. Temporal logic provides a convenient language for expressing properties of reactive systems. A temporal verification methodology provides procedures for proving that a given system satisfies a given temporal property. The research covered necessary theoretical foundations as well as implementation and application issues.

Manna, Zohar↗

Time and Frequency-Domain Cross-Verification of SLS 6DOF Trajectory Simulations

The SLS GNC team and its partners have developed several time- and frequency-based simulations for development and analysis of the proposed SLS launch vehicle. The simulations differ in fidelity and some have unique functionality that allows them to perform specific analyses. Some examples of the purposes of the various models are: trajectory simulation, multi-body separation, Monte Carlo, hardware in the loop, loads, and frequency domain stability analyses. While no two simulations are identical, many of the models are essentially six degree-of-freedom (6DOF) representations of the SLS plant dynamics, hardware implementation, and flight software. Thus at a high level all of those models should be in agreement. Comparison of outputs from several SLS trajectory and stability analysis tools are ongoing as part of the program's current verification effort. The purpose of these comparisons is to highlight modeling and analysis differences, verify simulation data sources, identify inconsistencies and minor errors, and ultimately to verify output data as being a good representation of the vehicle and subsystem dynamics. This paper will show selected verification work in both the time and frequency domain from the current design analysis cycle of the SLS for several of the design and analysis simulations. In the time domain, the tools that will be compared are MAVERIC, CLVTOPS, SAVANT, STARS, ARTEMIS, and POST 2. For the frequency domain analysis, the tools to be compared are FRACTAL, SAVANT, and STARS. The paper will include discussion of these tools including their capabilities, configurations, and the uses to which they are put in the SLS program. Determination of the criteria by which the simulations are compared (matching criteria) requires thoughtful consideration, and there are several pitfalls that may occur that can severely punish a simulation if not considered carefully. The paper will discuss these considerations and will present a framework for responding to these issues when they arise. For example, small event timing differences can lead to large differences in mass properties if the criteria are to measure those properties at the same time, or large differences in altitude if the criteria are to measure those properties when the simulation experiences a staging event. Similarly, a tiny difference in phase can lead to large gain margin differences for frequency-domain comparisons of gain margins.

VanZwieten, Tannen↗

MOOSE-based Tritium Migration Analysis Program, Version 8 (TMAP8) for advanced open-source tritium transport and fuel cycle modeling

Tritium management is critical for the safety, sustainability, and economics of fusion energy systems, and advanced and reliable modeling tools help accelerate the development of tritium technologies. This paper presents the Tritium Migration Analysis Program, Version 8 (TMAP8), an open-source, MOOSE-based application developed to provide state-of-the-art tritium transport and fuel cycle modeling capabilities. TMAP8 aims to expand the capabilities of previous versions (i.e., TMAP4 and TMAP7) by leveraging modern computational techniques, ensuring high software quality assurance standards (key to building trust), and enabling multispecies, multiscale, and multiphysics simulations for integrated tritium transport modeling in complex geometries. This paper outlines TMAP8’s scope and rigorous development practices, emphasizing its transparency, accessibility, modularity, and reliability. We present the current suite of verification and validation cases based on those from TMAP4, demonstrating TMAP8’s accuracy and reliability against analytical solutions and experimental data. Additionally, the paper showcases TMAP8’s integrated fuel cycle modeling capabilities, highlighting its applicability at various scales and levels. The TMAP8 code and documentation are openly available, promoting collaborative development and widespread adoption within the fusion community. Future work will soon expand TMAP8’s verification and validation suite to include those from TMAP7 and other recent experimental studies for validation.

11 NUCLEAR FUEL CYCLE AND FUEL MATERIALS↗

Software Validation via Model Animation

This paper explores a new approach to validating software implementations that have been produced from formally-verified algorithms. Although visual inspection gives some confidence that the implementations faithfully reflect the formal models, it does not provide complete assurance that the software is correct. The proposed approach, which is based on animation of formal specifications, compares the outputs computed by the software implementations on a given suite of input values to the outputs computed by the formal models on the same inputs, and determines if they are equal up to a given tolerance. The approach is illustrated on a prototype air traffic management system that computes simple kinematic trajectories for aircraft. Proofs for the mathematical models of the system's algorithms are carried out in the Prototype Verification System (PVS). The animation tool PVSio is used to evaluate the formal models on a set of randomly generated test cases. Output values computed by PVSio are compared against output values computed by the actual software. This comparison improves the assurance that the translation from formal models to code is faithful and that, for example, floating point errors do not greatly affect correctness and safety properties.

Dutle, Aaron M.↗

Visual Odometry Thinking While Driving for the Curiosity Mars Rover's Three-Year Test Campaign: Impact of Evolving Constraints on Verification and Validation

Over the first 9 years of the Mars Science Laboratory (MSL) Curiosity rover's surface mission, more than 87\% of its driving has been performed using Visual Odometry (VO). The benefits of using VO during driving are that it minimizes rover position uncertainty and can be used to monitor wheel slip, halting a drive if excessive wheel slip is occurring. The VO implementation on board Curiosity acquires and processes VO images in between drive steps while the rover is stationary. A VO Thinking While Driving (VTWD) flight software capability has been developed which enables the processing of VO images during rover driving, increasing the distance Curiosity can drive with VO in a given time period up to as much as 1.75x total distance. Verification and Validation (V\&V) of the capability has been challenging due to impacts from the COVID-19 pandemic and unavailability of the JPL Mars Yard outdoor test site. The VTWD V\&V test procedures were modified to use a small indoor space with Mars-like terrain. This paper describes the 3 year V\&V effort under challenging conditions to approve the VTWD capability for use on the Curiosity rover.

Rankin, Arturo↗

SU-E-J-117: Verification Method for the Detection Accuracy of Automatic Winston Lutz Test

Purpose: Winston Lutz test (WLT) has been a standard QA procedure performed prior to SRS treatment, to verify the mechanical iso-center setup accuracy upon different Gantry/Couch movements. Several detection algorithms exist,for analyzing the ball-radiation field alignment automatically. However, the accuracy of these algorithms have not been fully addressed. Here, we reveal the possible errors arise from each step in WLT, and verify the software detection accuracy with the Rectilinear Phantom Pointer (RLPP), a tool commonly used for aligning treatment plan coordinate with mechanical iso-center. Methods: WLT was performed with the radio-opaque ball mounted on a MIS and irradiated onto EDR2 films. The films were scanned and processed with an in-house Matlab program for automatic iso-center detection. Tests were also performed to identify the errors arise from setup, film development and scanning process. The radioopaque ball was then mounted onto the RLPP, and offset laterally and longitudinally in 7 known positions ( 0, ±0.2, ±0.5, ±0.8 mm) manually for irradiations. The gantry and couch was set to zero degree for all irradiation. The same scanned images were processed repeatedly to check the repeatability of the software. Results: Miminal discrepancies (mean=0.05mm) were detected with 2 films overlapped and irradiated but developed separately. This reveals the error arise from film processor and scanner alone. Maximum setup errors were found to be around 0.2mm, by analyzing data collected from 10 irradiations over 2 months. For the known shift introduced using the RLPP, the results agree with the manual offset, and fit linearly (R{sup 2}>0.99) when plotted relative to the first ball with zero shift. Conclusion: We systematically reveal the possible errors arise from each step in WLT, and introduce a simple method to verify the detection accuracy of our in-house software using a clinically available tool.

60 APPLIED LIFE SCIENCES↗

Verification of Electromagnetic Physics Models for Parallel Computing Architectures in the GeantV Project

An intensive R&D and programming effort is required to accomplish new challenges posed by future experimental high-energy particle physics (HEP) programs. The GeantV project aims to narrow the gap between the performance of the existing HEP detector simulation software and the ideal performance achievable, exploiting latest advances in computing technology. The project has developed a particle detector simulation prototype capable of transporting in parallel particles in complex geometries exploiting instruction level microparallelism (SIMD and SIMT), task-level parallelism (multithreading) and high-level parallelism (MPI), leveraging both the multi-core and the many-core opportunities. We present preliminary verification results concerning the electromagnetic (EM) physics models developed for parallel computing architectures within the GeantV project. In order to exploit the potential of vectorization and accelerators and to make the physics model effectively parallelizable, advanced sampling techniques have been implemented and tested. In this paper we introduce a set of automated statistical tests in order to verify the vectorized models by checking their consistency with the corresponding Geant4 models and to validate them against experimental data.

72 PHYSICS OF ELEMENTARY PARTICLES AND FIELDS↗

Development and Verification of a Numerical Library for Solving Global Terrestrial Multiphysics Problems

Abstract Current generation Land Surface Models (LSMs) routinely simulate many nonlinear multiphysics processes. The complexity of future generation LSMs is expected to increase as critical new biophysical and biogeochemical processes are incorporated. Current generation LSMs have several shortcomings including the lack of robust numerical methods to solve discretized equations, monolithic software design that hinders testing of a process representation in isolation from other components, and absence of a flexible coupling framework to solve tightly coupled multiphysics problems. While the LSMs community vigorously evaluates the accuracy of the conceptual model to represent reality (validation), the accuracy of the numerical implementation of the conceptual model (verification) is seldom examined. Method of Manufactured Solutions is a technique for verifying complex codes when analytical solutions are unavailable. In this work, we present a stand‐alone, open source numerical library for solving global terrestrial multiphysics processes that includes a flexible coupling framework. Robust numerical solution for linear and nonlinear equations is provided via the Portable, Extensible Toolkit for Scientific Computation library. We verify the numerical library using Method of Manufactured Solutions for a range of problems comprising single and multiphysics steady state problems that are applied in one or multiple physical domains. This work provides an example of incorporating code verification as an integral part of Land Surface Model development activities.

58 GEOSCIENCES↗

Time and Frequency-Domain Cross-Verification of SLS 6DOF Trajectory Simulations

The Space Launch System (SLS) Guidance, Navigation, and Control (GNC) team and its partners have developed several time- and frequency-based simulations for development and analysis of the proposed SLS launch vehicle. The simulations differ in fidelity and some have unique functionality that allows them to perform specific analyses. Some examples of the purposes of the various models are: trajectory simulation, multi-body separation, Monte Carlo, hardware in the loop, loads, and frequency domain stability analyses. While no two simulations are identical, many of the models are essentially six degree-of-freedom (6DOF) representations of the SLS plant dynamics, hardware implementation, and flight software. Thus at a high level all of those models should be in agreement. Comparison of outputs from several SLS trajectory and stability analysis tools are ongoing as part of the program's current verification effort. The purpose of these comparisons is to highlight modeling and analysis differences, verify simulation data sources, identify inconsistencies and minor errors, and ultimately to verify output data as being a good representation of the vehicle and subsystem dynamics. This paper will show selected verification work in both the time and frequency domain from the current design analysis cycle of the SLS for several of the design and analysis simulations. In the time domain, the tools that will be compared are MAVERIC, CLVTOPS, SAVANT, STARS, ARTEMIS, and POST 2. For the frequency domain analysis, the tools to be compared are FRACTAL, SAVANT, and STARS. The paper will include discussion of these tools including their capabilities, configurations, and the uses to which they are put in the SLS program. Determination of the criteria by which the simulations are compared (matching criteria) requires thoughtful consideration, and there are several pitfalls that may occur that can severely punish a simulation if not considered carefully. The paper will discuss these considerations and will present a framework for responding to these issues when they arise. For example, small event timing differences can lead to large differences in mass properties if the criteria are to measure those properties at the same time, or large differences in altitude if the criteria are to measure those properties when the simulation experiences a staging event. Similarly, a tiny difference in phase can lead to large gain margin differences for frequency-domain comparisons of gain margins.

Johnson, Matthew↗

Orbiter altitude at ALT interface based on ALSES (Approach and Landing Shuttle Engineering Simulation) analyses. Mission planning, mission analysis and software formulation

The details of the orbiter altitude attainable at the approach and landing tests (ALT) interface determined by the approach and landing shuttle engineering simulation (manned) were documented. The analysis culminated in the verification of the trends observed in a similar analysis performed previously on the space vehicle dynamics simulation (unmanned). Altitude variations attributable to pilot steering variability ranged between 492 ft higher to 383 ft lower. The requirement for this parametric analysis is first elaborated. The specifications, assumptions, and analytical approach used to determine the orbiter altitude at the ALT interface are then presented, followed by the results of the analytical approach and the conclusions and recommendations.

Colwell, C. L.↗

eqtools: Modular, extensible, open-source, cross-machine Python tools for working with magnetic equilibria

As plasma physics research for fusion energy transitions to an increasing emphasis on cross-machine collaboration and numerical simulation, it becomes increasingly important that portable tools be developed to enable data from diverse sources to be analyzed in a consistent manner. This paper presents eqtools, a modular, extensible, open-source toolkit implemented in the Python programming language for handling magnetic equilibria and associated data from tokamaks. eqtools provides a single interface for working with magnetic equilibrium data, both for handling derived quantities and mapping between coordinate systems, extensible to function with data from different experiments, data formats, and magnetic reconstruction codes, replacing the diverse, non-portable solutions currently in use. Moreover, while the open-source Python programming language offers a number of advantages as a scripting language for research purposes, the lack of basic tokamak-specific functionality has impeded the adoption of the language for regular use. Implementing equilibrium-mapping tools in Python removes a substantial barrier to new development in and porting legacy code into Python. In this paper, we introduce the design of the eqtools package and detail the workflow for usage and expansion to additional devices. Additionally, the implementation of a novel three-dimensional spline solution (in two spatial dimensions and in time) is also detailed. Finally, verification and benchmarking for accuracy and speed against existing tools are detailed. Wider deployment of these tools will enable efficient sharing of data and software between institutions and machines as well as self-consistent analysis of the shared data.

70 PLASMA PHYSICS AND FUSION TECHNOLOGY↗

Sci—Fri PM: Dosimetry—06: Commissioning of a 3D patient specific QA system for hypofractionated prostate treatments

The objective of this work is to commission the 6MV-SRS beam model in COMPASS (v2.1, IBA-Dosimetry) and validate its use for patient specific QA of hypofractionated prostate treatments. The COMPASS system consists of a 2D ion chamber array (MatriXX{sup Evolution}), an independent gantry angle sensor and associated software. The system can either directly calculate or reconstruct (using measured detector responses) a 3D dose distribution on the patient CT dataset for plan verification. Beam models are developed and commissioned in the same manner as a beam model is commissioned in a standard treatment planning system. Model validation was initially performed by comparing both COMPASS calculations and reconstructions to measured open field beam data. Next, 10 hypofractionated prostate RapidArc plans were delivered to both the COMPASS system and a phantom with ion chamber and film inserted. COMPASS dose distributions calculated and reconstructed on the phantom CT dataset were compared to the chamber and film measurements. The mean (± standard deviation) difference between COMPASS reconstructed dose and ion chamber measurement was 1.4 ± 1.0%. The maximum discrepancy was 2.6%. Corresponding values for COMPASS calculation were 0.9 ± 0.9% and 2.6%, respectively. The average gamma agreement index (3%/3mm) for COMPAS reconstruction and film was 96.7% and 95.3% when using 70% and 20% dose thresholds, respectively. The corresponding values for COMPASS calculation were 97.1% and 97.1%, respectively. Based on our results, COMPASS can be used for the patient specific QA of hypofractionated prostate treatments delivered with the 6MV-SRS beam.

62 RADIOLOGY AND NUCLEAR MEDICINE↗

Establishing model credibility for process-microstructure-property relationships in additive manufacturing using exascale computing

Additive Manufacturing (AM) of alloys holds significant promise as a disruptive technology in various industries, yet its adoption is often hindered by challenges in achieving consistent part quality. These issues are primarily due to the complex process-microstructure-property (PSP) relationships inherent to AM. Computational models can greatly aid in understanding these relationships, but their widespread impact and adoption has been limited by a lack of validated, open-source, and computationally efficient PSP modeling frameworks and hardware limitations. Here, this study leverages the ExaAM software suite and data from the AMBench-2018 series of laser powder bed fusion (LPBF) benchmark experiments to perform a comprehensive model assessment, including verification, validation, sensitivity analysis, and uncertainty quantification. The RADICAL-EnTK workflow manager was used to perform an ensemble of heat transport, solidification, and mechanical response simulations on the exascale computer Frontier, considering uncertainties in critical model inputs such as laser spot size and nucleation parameters, and consisting of 125 explicit grain structure simulations and 7875 crystal plasticity simulations. For a selected location within the Inconel 625 AMBench-2018 test artifact, sensitivity analysis and uncertainty quantification were performed using the predicted distributions of grain structure and mechanical properties. Qualitative agreement was found between the predicted grain size and texture and the observed AMBench-2018 microstructure, the mean predicted yield stress was within 5% of the experimental measurement mean, and the mean predicted engineering stress at 5% strain was within 10% of the experimental measurement mean. The insights gained from development and validation of the ExaAM PSP modeling framework will help guide future directions for enhancing the credibility and reliability of PSP models in AM, thereby accelerating the adoption of AM technologies in various industries.

Additive manufacturing↗

Contingency Contractor Optimization Phase 3 Sustainment Verification and Validation V&V) Report - Contingency Contractor Optimization Engineering - Prototype

The reports and test plans contained within this document serve as supporting materials to the activities listed within the “Contingency Contractor Optimization Tool – Prototype (CCOT-P) Verification & Validation Plan” [1]. The activities included test development, testing, peer reviews, and expert reviews. The engineering prototype reviews were done for both the software and the mathematical model used in CCOT-P. Section 2 includes the peer and expert review reports, which summarize the findings from each of the reviews and document the resolution of any issues. Section 3 details the test plans that were followed for functional testing of the application through the interface. Section 4 describes the unit tests that were run on the code.

97 MATHEMATICS AND COMPUTING↗

BuildingIQ Technology Field Validation

This document describes a field validation and verification of the BuildingIQ Predictive Energy Optimization (PEO) technology based on a five-site study. BuildingIQ describes its PEO technology as a software-as-a-service (SaaS) platform that optimizes commercial building HVAC control for system efficiency, occupant comfort, and cost. It is targeted for use in large, complex buildings, and integrates with the building automation system (BAS) to conduct supervisory control. The PEO algorithm defines optimal space air temperature setpoints that are automatically implemented at the variable air volume (VAV) terminal units when possible, or through supply air temperature and duct static pressure setpoints at the air handling unit (AHU) level. The optimization is built upon a learned predictive model that provides a 24-hour ahead forecast of the building’s power profile, using weather forecasts and historical operational data; this model is updated every 4 to 6 hours. Demand-responsive load reductions may also be implemented.

32 ENERGY CONSERVATION, CONSUMPTION, AND UTILIZATI↗

AEDAM: Whole Program Adaptive Error Detection and Mitigation—Year 3 Report

In Year 3, MIT was primarily responsible for the effort on Software Characterization, with the following milestones: 1. Completed design of domain-specific language for fault models 2. Completed design of program logic for verification 3. Completed evaluation of relational verification system for jacobi, steepest descent, and conjugate gradient 4. Completed development of relational verification system 5. Completed evaluation of relational verification system

97 MATHEMATICS AND COMPUTING↗

BuildingIQ Technology Field Validation

This document describes a field validation and verification of the BuildingIQ Predictive Energy Optimization (PEO) technology based on a five-site study. BuildingIQ describes its PEO technology as a software-as-a-service (SaaS) platform that optimizes commercial building HVAC control for system efficiency, occupant comfort, and cost. It is targeted for use in large, complex buildings, and integrates with the building automation system (BAS) to conduct supervisory control. The PEO algorithm defines optimal space air temperature setpoints that are automatically implemented at the variable air volume (VAV) terminal units when possible, or through supply air temperature and duct static pressure setpoints at the air handling unit (AHU) level. The optimization is built upon a learned predictive model that provides a 24-hour ahead forecast of the building’s power profile, using weather forecasts and historical operational data; this model is updated every 4 to 6 hours. Demand-responsive load reductions may also be implemented.

29 ENERGY PLANNING, POLICY, AND ECONOMY↗

An innovative design for autonomous backup attitude control of the Gamma Ray Observatory

The Gamma Ray Observatory is a NASA funded three-axis stabilized spacecraft which will carry four scientific instruments to observe gamma ray phenomena. The requirement to protect the scientific mission from system failures led to the attitude control and determination system design described in this paper. The design employs nine control modes with error detection, hardware substitution, and autonomous mode switching. The system architecture evolved to eliminate cross-dependence between the primary on-board computer (OBC) and the backup control processor electronics. Cross strapping of sensors and actuators and separation of the input/output electronics ensure that a reliable set of sensors and actuators will be available for backup mode operation. The OBC software includes failure detection, hardware reconfiguration, and mode switching logic which provide the ability to autonomously transfer, upon anomaly, to a reliable backup mode. Verification of this mode transition design is done in four test programs: at the unit level, by analytical simulation, by a hybrid breadboard electronics-simulation setup, and by a flight hardware-simulation test.

Tai, F.↗