Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “verification & validation”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 901 records · Page 50

CTF (Subchannel) Calculations and Validation L3:VVI.H2L.P15.01

The goal of the Verification and Validation Implementation (VVI) High to Low (Hi2Lo) process is utilizing a validated model in a high resolution code to generate synthetic data for improvement of the same model in a lower resolution code. This process is useful in circumstances where experimental data does not exist or it is not sufficient in quantity or resolution. Data from the high-fidelity code is treated as calibration data (with appropriate uncertainties and error bounds) which can be used to train parameters that affect solution accuracy in the lower-fidelity code model, thereby reducing uncertainty. This milestone presents a demonstration of the Hi2Lo process derived in the VVI focus area. The majority of the work performed herein describes the steps of the low-fidelity code used in the process with references to the work detailed in the companion high-fidelity code milestone (Reference 1). The CASL low-fidelity code used to perform this work was Cobra Thermal Fluid (CTF) and the high-fidelity code was STAR-CCM+ (STAR). The master branch version of CTF (pulled May 5, 2017 – Reference 2) was utilized for all CTF analyses performed as part of this milestone. The statistical and VVUQ components of the Hi2Lo framework were performed using Dakota version 6.6 (release date May 15, 2017 – Reference 3). Experimental data from Westinghouse Electric Company (WEC – Reference 4) was used throughout the demonstrated process to compare with the high-fidelity STAR results. A CTF parameter called Beta was chosen as the calibration parameter for this work. By default, Beta is defined as a constant mixing coefficient in CTF and is essentially a tuning parameter for mixing between subchannels. Since CTF does not have turbulence models like STAR, Beta is the parameter that performs the most similar function to the turbulence models in STAR. The purpose of the work performed in this milestone is to tune Beta to an optimal value that brings the CTF results closer to those measured in the WEC experiments.

97 MATHEMATICS AND COMPUTING↗

CYVET: A Cyber-Physical Assurance Framework Based on a Semi-Supervised Vetting Approach

The project is designed to directly address the needs described in the Topic Area 4 (“Cybersecurity Verification and Validation”) of the CEDS’19 Research Call. The proposed CYVET system directly addresses the need to elevate the current industry capabilities to verify and validate OT cybersecurity and associated control system infrastructure. Currently, there is a significant gap in the energy sector’s capability in that regard during infrastructure improvement, equipment procurement, and compliance certification. CYVET provides that needed capability. CYVET is device and architecture agnostic and thus broadly applicable across the energy sector. The goal of this project is to develop and deliver a cybersecurity verification and validation framework testing capability to verify and validate OT equipment, software, and the underlying control system architecture. The primary project objectives are (i) Verification: the synthesis and reconciliation of standards and vendor supplied features, (ii) Validation: the generation, execution, and presentation of testing scripts of verified security features, and (3) Demonstration: apply the developed technology capabilities for verification and validation at a relevant end-user facility in the energy sector.

Lopez Jr, Juan↗

Operational Modal Analysis of the Space Launch System Mobile Launcher on the Crawler Transporter ISVV-010 Rollout

NASA is developing an expendable heavy lift launch vehicle capability, the Space Launch System, to support lunar and deep space exploration. To support this capability, an updated ground infrastructure is required including modifying an existing Mobile Launcher system. The Mobile Launcher is a very large heavy beam/truss steel structure designed to support the Space Launch System during its buildup and integration in the Vehicle Assembly Building, transportation between the Vehicle Assembly Building and launch pad 39B by the Crawler Transporter, and provides the launch platform at the launch pad. As part of the Verification and Validation of the Mobile Launcher and Crawler Transporter, two rollouts of the Mobile Launcher transported by the Crawler Transporter, Integrated System Verification and Validation (ISVV) 005 and 010, have been performed to demonstrate the Crawler Transporter’s ability to transport the Mobile Launcher. ISVV-005 occurred in September 2018 and ISVV-010 occurred in late June 2019. ISVV-005 and ISVV-010 also provided the opportunity to gather data that can be used identify the Mobile Launcher on the Crawler Transporter rollout modal characteristics and refine the estimates of the Artemis I integrated launch vehicle rollout forcing functions. While the rollout environment has historically produced relatively small launch vehicle structural loads for the Saturn/Apollo and Space Shuttle programs in comparison to launch and ascent loads, these relatively small structural loads are inputs to structural fatigue analyses. The same holds true for the Space Launch System. Because the rollout forces acting on the Mobile Launcher and the Crawler Transporter are not directly measurable, Operational Modal Analysis techniques, instead of traditional Experimental Modal Analysis techniques, provide an empirical means to identify the Mobile Launcher on the Crawler Transporter rollout modal characteristics. The ISVV-010 rollout modal characteristics provide important supplemental modal information, which along with the Mobile Launcher modal test that was performed in June 2019 immediately prior to ISVV-010 rollout, combine to reduce uncertainty in the test correlated Mobile Launcher on the Crawler Transporter finite element model. A well test correlated finite element model will play a key role in the Building Block approach the Space Launch System program has implemented as part of its certification process for the Artemis I flight and in refining the Artemis I rollout forcing functions. At the time of the ISVV-005 rollout in September 2018, the Mobile Launcher was still undergoing construction, and therefore its modal characteristics are not directly comparable to those of the Mobile Launcher during the June 2019 Mobile Launcher modal test and subsequent ISVV-010 rollout. Hence the ISVV-005 rollout modal characteristics will not be looked at in this paper. This paper will briefly describe the Mobile Launcher and Crawler Transporter physical characteristics, ISVV-010 rollout data collection, the challenges in implementing Operational Modal Analysis techniques due in part to the Crawler Transporter harmonics, and how these challenges were overcome to obtain the ISVV-010 Mobile Launcher on the Crawler Transporter rollout modal characteristics.

Apollo↗

Operational Modal Analysis of the Space Launch System Mobile Launcher on the Crawler Transporter ISVV-010 Rollout

NASA is developing an expendable heavy lift launch vehicle capability, the Space Launch System, to support lunar and deep space exploration. To support this capability, an updated ground infrastructure is required including modifying an existing Mobile Launcher system. The Mobile Launcher is a very large heavy beam/truss steel structure designed to support the Space Launch System during its buildup and integration in the Vehicle Assembly Building, transportation between the Vehicle Assembly Building and launch pad 39B by the Crawler Transporter, and provides the launch platform at the launch pad. As part of the Verification and Validation of the Mobile Launcher and Crawler Transporter, two rollouts of the Mobile Launcher transported by the Crawler Transporter, Integrated System Verification and Validation (ISVV) 005 and 010, have been performed to demonstrate the Crawler Transporter’s ability to transport the Mobile Launcher. ISVV-005 occurred in September 2018 and ISVV-010 occurred in late June 2019. ISVV-005 and ISVV-010 also provided the opportunity to gather data that can be used identify the Mobile Launcher on the Crawler Transporter rollout modal characteristics and refine the estimates of the Artemis I integrated launch vehicle rollout forcing functions. While the rollout environment has historically produced relatively small launch vehicle structural loads for the Saturn/Apollo and Space Shuttle programs in comparison to launch and ascent loads, these relatively small structural loads are inputs to structural fatigue analyses. The same holds true for the Space Launch System. Because the rollout forces acting on the Mobile Launcher and the Crawler Transporter are not directly measurable, Operational Modal Analysis techniques, instead of traditional Experimental Modal Analysis techniques, provide an empirical means to identify the Mobile Launcher on the Crawler Transporter rollout modal characteristics. The ISVV-010 rollout modal characteristics provide important supplemental modal information, which along with the Mobile Launcher modal test that was performed in June 2019 immediately prior to ISVV-010 rollout, combine to reduce uncertainty in the test correlated Mobile Launcher on the Crawler Transporter finite element model. A well test correlated finite element model will play a key role in the Building Block approach the Space Launch System program has implemented as part of its certification process for the Artemis I flight and in refining the Artemis I rollout forcing functions. At the time of the ISVV-005 rollout in September 2018, the Mobile Launcher was still undergoing construction, and therefore its modal characteristics are not directly comparable to those of the Mobile Launcher during the June 2019 Mobile Launcher modal test and subsequent ISVV-010 rollout. Hence the ISVV-005 rollout modal characteristics will not be looked at in this paper. This paper will briefly describe the Mobile Launcher and Crawler Transporter physical characteristics, ISVV-010 rollout data collection, the challenges in implementing Operational Modal Analysis techniques due in part to the Crawler Transporter harmonics, and how these challenges were overcome to obtain the ISVV-010 Mobile Launcher on the Crawler Transporter rollout modal characteristics.

Apollo↗

A Greenhouse-Gas Information System: Monitoring and Validating Emissions Reporting and Mitigation

This study and report focus on attributes of a greenhouse-gas information system (GHGIS) needed to support MRV&V needs. These needs set the function of such a system apart from scientific/research monitoring of GHGs and carbon-cycle systems, and include (not exclusively): the need for a GHGIS that is operational, as required for decision-support; the need for a system that meets specifications derived from imposed requirements; the need for rigorous calibration, verification, and validation (CV&V) standards, processes, and records for all measurement and modeling/data-inversion data; the need to develop and adopt an uncertainty-quantification (UQ) regimen for all measurement and modeling data; and the requirement that GHGIS products can be subjected to third-party questioning and scientific scrutiny. This report examines and assesses presently available capabilities that could contribute to a future GHGIS. These capabilities include sensors and measurement technologies; data analysis and data uncertainty quantification (UQ) practices and methods; and model-based data-inversion practices, methods, and their associated UQ. The report further examines the need for traceable calibration, verification, and validation processes and attached metadata; differences between present science-/research-oriented needs and those that would be required for an operational GHGIS; the development, operation, and maintenance of a GHGIS missions-operations center (GMOC); and the complex systems engineering and integration that would be required to develop, operate, and evolve a future GHGIS.

54 ENVIRONMENTAL SCIENCES↗

Analysis of Validating and Verifying OpenACC Compilers 3.0 and Above

OpenACC is a high-level directive-based parallel programming model that can manage the sophistication of heterogeneity in architectures and abstract it from the users. The portability of the model across CPUs and accelerators has gained the model a wide variety of users. This means it is also crucial to analyze the reliability of the compilers’ implementations. To address this challenge, the OpenACC Validation and Verification team has proposed a validation testsuite to verify the OpenACC implementations across various compilers with an infrastructure for a more streamlined execution. This paper will cover the following aspects: (a) the new developments since the last publication on the testsuite, (b) outline the use of the infrastructure, (c) discuss tests that highlight our workflow process, (d) analyze the results from executing the testsuite on various systems, and (e) outline future developments.

Jarmusch, Aaron↗

Evaluation of 3D Simulation Capabilities within xRAGE using a Suite of Test Problems

This presentation describes a 3D verification and validation suite for xRAGE. Three verification test problems for hydrodynamics are presented: the Kidder ball problem, the Verney shell problem, and a 5-material compression problem. There is excellent agreement between 2D and 3D XRAGE simulation results for these problems, and between the xRAGE results and the benchmark solutions. In addition, two 3D ICF test problems are presented, based on an OMEGA direct drive capsule experiment, and on a NIF indirect drive capsule experiment. These two ICF test problems are used to evaluate two hydrodynamic methods within xRAGE. In addition to hydrodynamics the simulations also include three temperature plasma physics, thermal conduction and radiation diffusion. When simulating either capsule in 3D the newer unsplit hydrodynamic method in xRAGE produces more vorticity relative to the older default method. For the indirect drive capsule the 3D simulations are in reasonable agreement with the experimental values of ion temperature and neutron production. Unstable 3D vortex rings are present near the DT ice/carbon interface which enhance the confinement of the DT ice.

70 PLASMA PHYSICS AND FUSION TECHNOLOGY↗

Adopting Code Verification Methodology Based on Model Form

Code verification is an essential part of credibility analysis for computational models. It assesses whether the mathematical model is implemented correctly into the code and whether the numerical methods behave consistently, and is done before solution verification and validation. Robust guidance for code verification exists in the literature. However, there is no known, concise guide for selecting the approach based on the model form that also presents an overview of the common elements. This document was written to address this gap as an accessible reference for beginning a code-verification effort.

97 MATHEMATICS AND COMPUTING↗

Adapting Code Verification Methodology to Model Form

Code verification is an essential part of credibility analysis for computational models. It assesses whether the mathematical model is implemented correctly into the code and whether the numerical methods behave consistently, and is done before solution verification and validation. Robust guidance for code verification exists in the literature. However, there is no known, concise guide for selecting the approach based on the model form that also presents an overview of the common elements. This document was written to address this gap as an accessible reference for beginning a code-verification effort.

97 MATHEMATICS AND COMPUTING↗

SLS Navigation Model-Based Design Approach

The SLS Program chose to implement a Model-based Design and Model-based Requirements approach for managing component design information and system requirements. This approach differs from previous large-scale design efforts at Marshall Space Flight Center where design documentation alone conveyed information required for vehicle design and analysis and where extensive requirements sets were used to scope and constrain the design. The SLS Navigation Team has been responsible for the Program-controlled Design Math Models (DMMs) which describe and represent the performance of the Inertial Navigation System (INS) and the Rate Gyro Assemblies (RGAs) used by Guidance, Navigation, and Controls (GN&C). The SLS Navigation Team is also responsible for the navigation algorithms. The navigation algorithms are delivered for implementation on the flight hardware as a DMM. For the SLS Block 1-B design, the additional GPS Receiver hardware is managed as a DMM at the vehicle design level. This paper provides a discussion of the processes and methods used to engineer, design, and coordinate engineering trades and performance assessments using SLS practices as applied to the GN&C system, with a particular focus on the Navigation components. These include composing system requirements, requirements verification, model development, model verification and validation, and modeling and analysis approaches. The Model-based Design and Requirements approach does not reduce the effort associated with the design process versus previous processes used at Marshall Space Flight Center. Instead, the approach takes advantage of overlap between the requirements development and management process, and the design and analysis process by efficiently combining the control (i.e. the requirement) and the design mechanisms. The design mechanism is the representation of the component behavior and performance in design and analysis tools. The focus in the early design process shifts from the development and management of design requirements to the development of usable models, model requirements, and model verification and validation efforts. The models themselves are represented in C/C++ code and accompanying data files. Under the idealized process, potential ambiguity in specification is reduced because the model must be implementable versus a requirement which is not necessarily subject to this constraint. Further, the models are shown to emulate the hardware during validation. For models developed by the Navigation Team, a common interface/standalone environment was developed. The common environment allows for easy implementation in design and analysis tools. Mechanisms such as unit test cases ensure implementation as the developer intended. The model verification and validation process provides a very high level of component design insight. The origin and implementation of the SLS variant of Model-based Design is described from the perspective of the SLS Navigation Team. The format of the models and the requirements are described. The Model-based Design approach has many benefits but is not without potential complications. Key lessons learned associated with the implementation of the Model Based Design approach and process from infancy to verification and certification are discussed

Oliver, T. Emerson↗

Assessment of a hybrid finite element and finite volume code for turbulent incompressible flows

Hydra-TH is a hybrid finite-element/finite-volume incompressible/low-Mach flow simulation code based on the Hydra multiphysics toolkit being developed and used for thermal-hydraulics applications. In the present work, a suite of verification and validation (V&V) test problems for Hydra-TH was defined to meet the design requirements of the Consortium for Advanced Simulation of Light Water Reactors (CASL). The intent for this test problem suite is to provide baseline comparison data that demonstrates the performance of the Hydra-TH solution methods. The simulation problems vary in complexity from laminar to turbulent flows. A set of RANS and LES turbulence models were used in the simulation of four classical test problems. Numerical results obtained by Hydra-TH agreed well with either the available analytical solution or experimental data, indicating the verified and validated implementation of these turbulence models in Hydra-TH. Where possible, some form of solution verification has been attempted to identify sensitivities in the solution methods, and suggest best practices when using the Hydra-TH code. -- Highlights: •We performed a comprehensive study to verify and validate the turbulence models in Hydra-TH. •Hydra-TH delivers 2nd-order grid convergence for the incompressible Navier–Stokes equations. •Hydra-TH can accurately simulate the laminar boundary layers. •Hydra-TH can accurately simulate the turbulent boundary layers with RANS turbulence models. •Hydra-TH delivers high-fidelity LES capability for simulating turbulent flows in confined space.

21 SPECIFIC NUCLEAR REACTORS AND ASSOCIATED PLANTS↗

Experimental Validation of Stratified Flow Phenomena, Graphite Oxidation, and Mitigation Strategies of Air Ingress Accidents

The US Department of Energy is performing research and development (R&D) that focuses on key phenomena that are important during challenging scenarios that may occur in the Next Generation Nuclear Plant (NGNP) Program / GEN-IV Very High Temperature Reactor (VHTR). Phenomena identification and ranking studies (PIRT) to date have identified the air ingress event, following on the heels of a VHTR depressurization, as very important (Schultz et al., 2006). Consequently, the development of advanced air ingress-related models and verification and validation (V&V) are very high priority for the NGNP program. Following a loss of coolant and system depressurization, air will enter the core through the break. Air ingress leads to oxidation of the in-core graphite structure and fuel. The oxidation will accelerate heat-up of the bottom reflector and the reactor core and will cause the release of fission products eventually. The potential collapse of the bottom reflector because of burn-off and the release of CO lead to serious safety problems. For estimation of the proper safety margin we need experimental data and tools, including accurate multi-dimensional thermal-hydraulic and reactor physics models, a burn-off model, and a fracture model. We also need to develop effective strategies to mitigate the effects of oxidation. The results from this research will provide crucial inputs to the INL NGNP/VHTR Methods R&D project. This project is focused on (a) analytical and experimental study of air ingress caused by density-driven, stratified, countercurrent flow, (b) advanced graphite oxidation experiments, (c) experimental study of burn-off in the bottom reflector, (d) structural tests of the burnt-off bottom reflector, (e) implementation of advanced models developed during the previous tasks into the GAMMA code, (f) full air ingress and oxidation mitigation analyses, (g) development of core neutronic models, (h) coupling of the core neutronic and thermal hydraulic models, and (i) verification and validation of the coupled models.

99 GENERAL AND MISCELLANEOUS↗

Approaches to the verification of rule-based expert systems

Expert systems are a highly useful spinoff of artificial intelligence research. One major stumbling block to extended use of expert systems is the lack of well-defined verification and validation (V and V) methodologies. Since expert systems are computer programs, the definitions of verification and validation from conventional software are applicable. The primary difficulty with expert systems is the use of development methodologies which do not support effective V and V. If proper techniques are used to document requirements, V and V of rule-based expert systems is possible, and may be easier than with conventional code. For NASA applications, the flight technique panels used in previous programs should provide an excellent way to verify the rules used in expert systems. There are, however, some inherent differences in expert systems that will affect V and V considerations.

Culbert, Chris↗

A Tool for Automatic Verification of Real-Time Expert Systems

The creation of an automated, user-driven tool for expert system development, validation, and verification is curretly onoging at NASA's Jet Propulsion Laboratory. In the new age of faster, better, cheaper missions, there is an increased willingness to utilize embedded expert systems for encapsulating and preserving mission expertise in systems which combine conventional algorithmic processing and artifical intelligence. The once-questioned role of automation in spacecraft monitoring is now becoming one of increasing importance.

real-time expert systems automated utilities knowl↗

Tradeoffs When Considering Deep Reinforcement Learning for Contingency Management in Advanced Air Mobility

Air transportation is undergoing a rapid evolution globally with the introduction of Advanced Air Mobility (AAM) and with it comes novel challenges and opportunities for transforming aviation. As AAM operations introduce increasing heterogeneity in vehicle capabilities and density, increased levels of automation are likely necessary to achieve operational safety and efficiency goals. This paper focuses on one example where increased automation has been suggested. Autonomous operations will need contingency management systems that can monitor evolving risk across a span of interrelated (or interdependent) hazards and, if necessary, execute appropriate control interventions via supervised or automated decision making. Accommodating this complex environment may require automated functions (autonomy) that apply artificial intelligence (AI) techniques that can adapt and respond to a quickly changing environment. This paper explores the use of Deep Reinforcement Learning (DRL) which has shown promising performance in complex and high-dimensional environments where the objective can be constructed as a sequential decision-making problem. An extension of a prior formulation of the contingency management problem as a Markov Decision Process (MDP) is presented and uses a DRL framework to train agents that mitigate hazards present in the simulation environment. A comparison of these learning-based agents and classical techniques is presented in terms of their performance, verification difficulties, and development process.

machine learningautonomous systems; flight simulat↗

Sierra/SD – Verification Test Manual – 5.22

Verification and validation (V&V) of scientific computing programs are important at Sandia National Labs due to the expanding role of computational simulation in managing the United States nuclear stockpile. The complexities of structural response calculations used to analyze physical problems, the varieties of codes applied to the calculations, and the importance of accurate predictions when assessing field conditions demand confidence in the consistency and accuracy of computer codes. Confidence in the accuracy of the predictions arising from computer simulations must ultimately be gained through verification and validation. The Sierra salinas structural dynamics analysis code, Sierra/SD, is used at the DOE Laboratories, and in several DOD projects. The roles of Sierra/SD in the qualification of weapon systems and components for normal and hostile environments throughout the Stockpile-to-Target Sequence include to, • Redesign weapon components. • Certify weapon components and systems for target environments such as hypersonic vehicles. • Certify that components will survive the thermal mechanical shock loads associated with hostile environments. • Evaluate current stockpile issues, including issues associated with uncertainty quantification. • Address many other problems that are encountered in stockpile management. The Sierra/SD verification plan is described, and an evolving set of key verification tests are described in detail. The verification tests ensure the correctness of the mathematics and numerical algorithms associated with functionality describing engineering phenomena. Development is in accordance with a set of tailored Software Quality Engineering (SQE) practices. SQE practices guide the overall verification and validation effort.

97 MATHEMATICS AND COMPUTING↗

Software verification and testing

General procedures for software verification and validation are provided as a guide for managers, programmers, and analysts involved in software development. The verification and validation procedures described are based primarily on testing techniques. Testing refers to the execution of all or part of a software system for the purpose of detecting errors. Planning, execution, and analysis of tests are outlined in this document. Code reading and static analysis techniques for software verification are also described.

Source record↗

Mars Exploration Rover Entry, Descent, and Landing: A Thermal Perspective

Perhaps the most challenging mission phase for the Mars Exploration Rovers was the Entry, Descent, and Landing (EDL). During this phase, the entry vehicle attached to its cruise stage was transformed into a stowed tetrahedral Lander that was surrounded by inflated airbags through a series of complex events. There was only one opportunity to successfully execute an automated command sequence without any possible ground intervention. The success of EDL was reliant upon the system thermal design: 1) to thermally condition EDL hardware from cruise storage temperatures to operating temperature ranges; 2) to maintain the Rover electronics within operating temperature ranges without the benefit of the cruise single phase cooling loop, which had been evacuated in preparation for EDL; and 3) to maintain the cruise stage propulsion components for the critical turn to entry attitude. Since the EDL architecture was inherited from Mars Pathfinder (MPF), the initial EDL thermal design would be inherited from MPF. However, hardware and implementation differences from MPF ultimately changed the MPF inheritance approach for the EDL thermal design. With the lack of full inheritance, the verification and validation of the EDL thermal design took on increased significance. This paper will summarize the verification and validation approach for the EDL thermal design along with applicable system level thermal testing results as well as appropriate thermal analyses. In addition, the lessons learned during the system-level testing will be discussed. Finally, the in-flight EDL experiences of both MER-A and -B missions (Spirit and Opportunity, respectively) will be presented, demonstrated how lessons learned from Spirit were applied to Opportunity.

thermal↗