SEARCH · Engineering Papers
Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.
Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.
Prof. Xiaodong Sun and I are Guest Editors and have written a Foreword for the Journal. The Nuclear Science and Engineering Special Issue focuses on thermal hydraulics experiments and modelling, supporting verification and validation needs for advanced reactor development.
The Multi-Fidelity Toolkit (MFTK) is a simulation tool being developed at Sandia National Laboratories for aerodynamic predictions of compressible flows over a range of physics fidelities and computational speeds. These models include the Reynolds-Averaged-Navier-Stokes (RANS) equations, the Euler equations, and modified Newtonian aerodynamics (MNA) equations, and they can be invoked independently or coupled with hierarchical Kriging to interpolate between high-fidelity simulations using lower-fidelity data. However, as with any new simulation capability, verification and validation are necessary to gather credibility evidence. This work describes formal code- and solution-verification activities as well as model validation with uncertainty considerations. Code verification is performed on the MNA model by comparing with an analytical solution for flat-plate and inclined-plate geometries. Solution-verification activities include grid-refinement studies of HIFiRE-1 wind tunnel measurements, which are used for validation, for all model fidelities. A thorough treatment of the validation comparison with prediction error and validation uncertainty is also presented.
The MCNP6® code, version 6.3, has been extensively verified and validated for many applications. The use of the same default capabilities existing in the MCNP6.2 code are also used in the verification and validation (V&V) of the MCNP6.3 code. In this paper, selected new features, code enhancements, and bug fixes in the MCNP6.3 code that impact criticality safety applications are described and investigated. More specifically, the changes within the MCNP6.3 code that are studied in this work include both the new fission matrix and the Doppler broadening resonance correction (DBRC) features. For nuclear criticality safety applications, the V&V benchmark problems within the criticality, extended criticality, and Rossi-α suites are used to study the upgrades within the MCNP6.3 code. Some additional investigations into benchmarks at elevated temperatures are used to showcase the impacts of the temperature-specific capabilities.
Many systems are being considered in which artificial intelligence (AI) will be a key element. Failure of an AI element can lead to system failure, hence the need for AI verification and validation (V&V). This article addresses V&V of such a system, focusing on the issues created by characteristics of AI that make V&V challenging. The element(s) containing AI capabilities is treated as a subsystem and V&V is conducted on that subsystem and its interfaces with other elements of the system under study, just as V&V would be conducted on other subsystems. That is, the high-level definitions of verification and of validation do not change for systems containing one or more AI elements. However, AI V&V challenges require approaches and solutions beyond those for conventional or traditional (those without AI elements) systems. This article provides an overview of how machine learning components/subsystems “fit” in the systems engineering framework (Section 1), identifies characteristics of AI subsystems that create challenges in their V&V (Section 2), describes those challenges (Section 3) and provides some potential solutions (Section 4).
In the HPC area, both hardware and software move quickly. Often new hardware is developed and deployed, the corresponding software stack, including compilers and other tools, are under active development while leading edge software developers are working to port and tune their applications, all at the same time. While the software ecosystem is in flux, one of the key challenges for users is obtaining insight into the state of implementation of key features in the programming languages and models their applications are using – whether they have been implemented, and whether the implementation conforms to the specification, especially for newly implemented features (less tested by widespread use). OpenMP is one of the most prominent shared memory programming models used for on-node programming in HPC. With the shift towards accelerators (such as GPUs and FPGAs) and heterogeneous programming OpenMP features are getting more complex. It is natural to ask whether generative AI approaches, and large language models (LLMs) in particular, can help in producing validation and verification test suites to allow users better and faster insights into the availability and correctness of OpenMP features of interest. In this work, we explore the use of ChatGPT-4 to generate a suite of tests for OpenMP features. We have chosen a set of directives and clauses, a total of 78 combinations, which first appeared in OpenMP 3.0 (released in May 2008) but are also relevant for accelerators. We prompted ChatGPT to generate tests in the C and Fortran languages, for both host (CPU) and device (accelerator). On the Summit super-computer using the GNU implementation, we found that, of the 78 generated tests 67 C tests and 43 Fortran tests compiled successfully and fewer than those executed to completion. On further analysis we show that not all generated tests are valid. We document the process, results, and provide detailed analysis regarding the quality of tests generated. With the aim of providing input to a production quality validation and verification suite, we manually implement the corrections required to make the tests valid according to the current OpenMP specification. We quantify this effort as small, medium, or large, and record the lines of code changed to correct the invalid tests. With the corrected tests we validate recent implementations from HPE, AMD, and GNU on the Frontier supercomputer. Our experiment and subsequent analysis show that although LLMs are capable of producing HPC specific codes, they are limited by their understanding of the deeper semantics and restrictions of programming models such as OpenMP. Unsurprisingly more commonly used features have better support, while some OpenMP 3.0 directives such as sections and tasking are not universally supported on accelerators. We demonstrate that successful compilation and execution to completion are inadequate metrics for evaluating generated code and that, at this time, commodity LLMs require expert intervention for code verification. This points to gaps in the training data that is currently available for HPC. We demonstrate that with "small" effort 37% of generated invalid C tests and 63% of generated invalid Fortran tests could be corrected. This improves productivity of test generation as we circumvent writing from scratch and the common programming errors associated with it.
This project was only active for two months and was terminated due to a re-organization of the sponsoring initiative. The original purpose of the project was to develop and apply verification and validation (V&V) methodologies for the Energy System Co-Design with Multiple Objectives and Power Electronics (E-COMP) Initiative. This report summarizes the project and the work that was completed in the shortened time frame.
This is Volume 1 of a two-volume document that collects the verification and validation (V&V) works done for the PLTEMP/ANL code during the years of its development and improvement. Volume 1 describes the V&V of sixteen capabilities of the PLTEMP/ANL code that were identified by research reactor analysts as frequently used in their thermal-hydraulic analysis. Volume 2 describes the V&V of developments and improvements since the release of PLTEMP/ANL Version 4.3. Each chapter of the document focuses on verifying or validating a specific part of the software that calculates a specific phenomenon, e.g., channel flow calculation, coolant property calculation, heat transfer calculation, and flow instability calculation. Software verification is performed by comparing the code with hand calculation, Microsoft spreadsheet calculation, Mathematica calculation, or MATLAB calculation. The software validation is done by comparing the code with experimental data or a widely tested code like the RELAP5 code. In addition, some PLTEMP/ANL V&V works that are available in the open literature are simply cited in Volume 1 of the document. PLTEMP/ANL has been used in the safety analysis reports of several US and foreign research reactors licensed and converted from highly enriched uranium fuel to low-enriched uranium fuel. A list of such reactors is given in Volume 1 of the document.
This is Volume 2 of a two-volume document that collects the verification and validation (V&V) works done for the PLTEMP/ANL code during the years of its development and improvement. Volume 1 describes the V&V of sixteen capabilities of the PLTEMP/ANL code that were identified by research reactor analysts as frequently used in their thermal-hydraulic analysis. Volume 2 describes the V&V of developments and improvements since the release of PLTEMP/ANL Version 4.3. Each chapter of the document focuses on verifying or validating a specific part of the software that calculates a particular phenomenon, e.g., channel flow calculation, coolant property calculation, heat transfer calculation, and flow instability calculation. Software verification is performed by comparing the code with a hand calculation, Microsoft spreadsheet calculation, Mathematica calculation, or MATLAB calculation. The software validation is achieved by comparing the code with experimental data or a widely tested code like the RELAP5 code. In addition, some PLTEMP/ANL V&V works that are available in the open literature are simply referenced in Volume 1 of the document. PLTEMP/ANL has been used in safety analysis reports of several US and foreign research reactors licensed and converted from highly enriched uranium fuel to low-enriched uranium fuel. A list of such reactors is given in Volume 1 of the document.
The objective of this document is to provide relevant information about the verification and validation (V&V) status of RELAP5-3D for the MARVEL microreactor design and safety analysis and to provide indications about needed V&V activities before MARVEL microreactor operation.
The HYBRID modeling repository is an in-house developed library of models for selected integrated energy systems (IES) modelling. HYBRID models have been developed since 2015 to describe the physical operation of tightly coupled thermal systems including power generators, thermal transport systems, thermal storage, thermal-to-electric conversion systems, and other thermal applications. Here, validation and verification (V&V) capabilities are demonstrated using the Thermal Energy Distribution System (TEDS) at INL. Building upon prior work, the TEDS model has been updated and verified so that it better represents the installed system configuration and the operating control system. The model control system was changed to allow replication of actual experimental procedures. Experimental operations focusing primarily on thermocline tank performance were devised and performed. Several anomalies were found in the operation data of the experiment facility. V&V activities calibrating a selected input parameter are demonstrated on a single component as well as with a single parameter within the thermocline. Calibrating is then demonstrated on multiple components and a multi-parameter metric for the entire system. The validation methodology is successfully applied to validate the model with experimental data. It is also used to confirm a hypothesis behind one of the anomalies in experimental performance.
Zapdos is an open-source finite element plasma fluid solver based on the MOOSE multiphysics framework. This paper outlines Zapdos verification, benchmarking, and validation efforts for 1D and 2D RF capacitively coupled plasma discharge models for mid-range pressures (0.1 - 1 Torr). The verification process involved using the method of manufactured solutions to assess Zapdos spatial and temporal error convergence. L2 errors ranged from 10 -2 to 10 -4 , while the convergence’s slope were in agreement with the predicted slopes for the tested variable and time integration orders. The benchmarking process involved comparisons to previously results from the validated finite element code, LSODI. These works included 1D and 2D simulations for a range of plasma parameters (densities, temperatures, voltage, etc.). For the 1D cases, Zapdos and LSODI results were in very good agreement. In the 2D cases, variable behaviors matched, with slight discrepancies in peak values. The validation process involved comparisons to experimental works including electron density measurements by microwave interferometry and metastable density measurements by planar laser-induced fluorescence imaging. Results shown reasonable agreement at higher pressure, with results starting to diverge at low pressures. Furthermore, probable causes for this diverges are the limitation of the fluid assumption for plasmas at low pressure, or the need for more robust boundary conditions. Overall, Zapdos shown reasonable results for the verification, benchmarking, and validation efforts, and Zapdos can be downloaded at https://github.com/shannon-lab/zapdos.
As the goal of achieving fusion power on the grid comes closer to fruition, fully coupled multiphysics models of fusion devices will be crucial. Currently, there are two main approaches to developing these platforms: (1) loosely coupled, where one couples existing codes and solvers together through input and output parameters and data, and (2) tightly coupled, where one develops the necessary models within a singular, integrated framework. This work focuses on the latter approach for magnetically confined fusion devices by developing a fluid-based plasma-edge model within the Multiphysics Object Oriented Simulation Environment (MOOSE) Framework. This effort is coordinated with other efforts to develop, test, demonstrate, and deploy fusion relevant multiphysics capabilities including electromagnetics, particle-in-cell plasma, tritium transport, and fusion blanket design. This new model is an expansion of the MOOSE-based plasma application, Zapdos, which was originally formulated to model low-temperature, non-magnetized plasma processes. Verification, benchmarking, and validation studies have been conducted. Verification studies involved utilizing the method of manufactured solutions and comparing the convergence slope of a known solution to the theoretical slope. Benchmarking consists of comparisons to existing edge codes, namely BOUT++ and SOLEDGE3X. Validation efforts focused on comparisons against open-source data from the TCV tokamak.
The objective of this Engineering Calculations and Analysis Report (ECAR) is to provide documentation and highlight relevant information regarding the verification and validation (V&V) of the commercial computational fluid dynamics (CFD) code STAR-CCM+ for the thermal and fluids analyses performed for the MARVEL microreactor.
SAS4A/SASSYS-1 is a safety analysis software package already under copyright by Argonne National Laboratory. There is interest from third-party users of the software to gain access to our internal verification and validation test suite. Therefore we are seeking to establish copyright over the test suite, which includes input files and reference results for hundreds of test cases that demonstrate functionality of the software. Making the test suite available to licensed users allows them to utilize the broad range of tests for commercial grade dedication activities needed to qualify the SAS4A/SASSYS-1 software under their own Software Quality Assurance program for use in a regulatory environment (e.g. with the Nuclear Regulatory Commission).
Some of the most promising microreactor designs currently under development utilize heat pipe technology to transfer heat from the reactor core to the secondary side heat exchanger, due to the technology’s compactness, efficiency, passivity, and reliability. Sockeye is an engineering-scale heat pipe tool developed under the Nuclear Energy Advanced Modeling and Simulation Program to be used for the design and safety analysis of microreactors. Sockeye’s core capability lies in a 1D, two-phase, compressible flow model, used to simulate the working fluid inside the heat pipe. Sockeye is built on the Multiphysics Object-Oriented Simulation Environment framework, which allows for seamless multiphysics coupling with other Nuclear Energy Advanced Modeling and Simulation tools and can thus be used in a full-scale simulation of a microreactor assembly, which can include hundreds of heat pipes. This paper presents some initial verification and validation assessments performed for Sockeye. We demonstrate good agreement between Sockeye’s numerical results and steady-state analytic solutions for velocity and pressure drop, with differences attributable to the underlying assumptions made by the analytic solutions. We also show that Sockeye reproduces analytic predictions of key operational limits, such as the capillary limit and the sonic limit. Finally, we compare Sockeye results to experimental data for the SAFE-30 heat pipe module test.
A new Python-based framework has been developed to enable a more consistent layout with automatable setup, execution, and documentation of all verification and validation (V&V) test suites previously established for use with the MCNP code. In this paper, the new general framework for the V&V test suites is discussed, including information on all of the current capabilities and plans for future capabilities. For nuclear criticality safety applications, the existing V&V benchmark problems within the criticality, extended criticality, and analytic k-effective test suites have been ported into this new framework. In addition, the status and updates to the Rossi-alpha and subcritical multiplicationtest suites will be discussed. Some V&V results exercising new MCNP6.3 capabilities will be demonstrated.
This report documents the software testing which has been performed for the PARET/ANL version 7.7 software. The software testing is based on code capabilities identified by research reactor analysts as frequently used in their safety analyses. The verification and validation procedures have been performed and documented to address the steady-state capabilities of the software, as described in Chapter 2, and the transient capabilities, as described in Chapter 3. Testing based on the comparison between PARET/ANL calculations and analytical solutions, hand calculations, or other code calculations of the test cases confirms that all the identified capabilities of the software were implemented correctly. In addition, results from code comparisons against SPERT-I and SPERT-IV experiments for various flow rates are reported for the peak power, energy release and cladding surface temperature. The comparisons showed overall good agreement for the peak power and conservative predictions of the cladding surface temperature
This report documents the software testing which has been performed for the PARET/ANL version 7.7 software. The software testing is based on code capabilities identified by research reactor analysts as frequently used in their safety analyses. The verification and validation procedures have been performed and documented to address the steady-state capabilities of the software, as described in Chapter 2, and the transient capabilities, as described in Chapter 3. Testing based on the comparison between PARET/ANL calculations and analytical solutions, hand calculations, or other code calculations of the test cases confirms that all the identified capabilities of the software were implemented correctly. In addition, results from code comparisons against SPERT-I and SPERT-IV experiments for various flow rates are reported for the peak power, energy release and cladding surface temperature. The comparisons showed overall good agreement for the peak power and conservative predictions of the cladding surface temperature.