Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “verification and validation”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 703 records · Page 39
All sourcesNASA NTRSDOE OSTI
NASA NTRS2017

Security Vulnerability Profiles of NASA Mission Software: Empirical Analysis of Security Related Bug Reports

NASA develops, runs, and maintains software systems for which security is of vital importance. Therefore, it is becoming an imperative to develop secure systems and extend the current software assurance capabilities to cover information assurance and cybersecurity concerns of NASA missions. The results presented in this report are based on the information provided in the issue tracking systems of one ground mission and one flight mission. The extracted data were used to create three datasets: Ground mission IVV issues, Flight mission IVV issues, and Flight mission Developers issues. In each dataset, we identified the software bugs that are security related and classified them in specific security classes. This information was then used to create the security vulnerability profiles (i.e., to determine how, why, where, and when the security vulnerabilities were introduced) and explore the existence of common trends. The main findings of our work include:- Code related security issues dominated both the Ground and Flight mission IVV security issues, with 95 and 92, respectively. Therefore, enforcing secure coding practices and verification and validation focused on coding errors would be cost effective ways to improve mission's security. (Flight mission Developers issues dataset did not contain data in the Issue Category.)- In both the Ground and Flight mission IVV issues datasets, the majority of security issues (i.e., 91 and 85, respectively) were introduced in the Implementation phase. In most cases, the phase in which the issues were found was the same as the phase in which they were introduced. The most security related issues of the Flight mission Developers issues dataset were found during Code Implementation, Build Integration, and Build Verification; the data on the phase in which these issues were introduced were not available for this dataset.- The location of security related issues, as the location of software issues in general, followed the Pareto principle. Specifically, for all three datasets, from 86 to 88 the security related issues were located in two to four subsystems.- The severity levels of most security issues were moderate, in all three datasets.- Out of 21 primary security classes, five dominated: Exception Management, Memory Access, Other, Risky Values, and Unused Entities. Together, these classes contributed from around 80 to 90 of all security issues in each dataset. This again proves the Pareto principle of uneven distribution of security issues, in this case across CWE classes, and supports the fact that addressing these dominant security classes provides the most cost efficient way to improve missions' security. The findings presented in this report uncovered the security vulnerability profiles and identified the common trends and dominant classes of security issues, which in turn can be used to select the most efficient secure design and coding best practices compiled by the part of the SARP project team associated with the NASA's Johnson Space Center. In addition, these findings provide valuable input to the NASA IVV initiative aimed at identification of the two 25 CWEs of ground and flight missions.

vulnerability↗
DOE OSTI2023

FY23 Status Report on MOOSE-Based Approaches to Modeling Core Bowing in Fast Reactors

Under the U.S. Department of Energy Office of Nuclear Energy’s Advanced Modeling and Simulation (NEAMS) Program, an integrated multiphysics approach is being developed to model the core bowing phenomena important to liquid metal-cooled fast reactors. Core bowing is an important passive safety mechanism whereby increased power (which leads to temperature and flux gradients) influences the core to bow into less reactive configurations when the restraint system is properly designed. The phenomenon includes a complex interplay of radiation transport, duct temperature calculations involving fluid flow and heat transfer, and thermo-mechanical responses to the induced temperature and flux gradients. Structural material properties are also important to determining inelastic response to longer term flux gradients which cause irradiation creep and swelling. While core bowing provides a strong negative reactivity feedback when the restraint system is designed properly, it also results in additional forces between assemblies which increase the loads required to extricate them during refueling or control rod movement. Therefore, the restraint system must be designed with these tradeoffs in mind. The first stage of the work, which commenced in FY21 and continues through FY23, assesses thermo-mechanical modeling tools for producing core bowing predictions consistent with conventional tools. The Multiphysics Object Oriented Simulation Environment (MOOSE) Tensor Mechanics and Contact Modules are employed. This status report describes work on additional thermo-mechanical benchmark verification problems with increased complexity from the examples demonstrated in FY21 and FY22. Several benchmark verification examples were selected from the IAEA verification and validation report with increased number of ducts and more complex contact interaction behavior. These examples involve a full symmetric sector with restraint rings at multiple load pad locations to simulate a limited free-bow restraint system concept, as well as irradiation induced swelling and creep effects in a sector. In addition, improvements to the contact module sideset assignment were assessed and compared with previous MOOSE results to verify the contact behavior. The MOOSE-based results were compared to IAEA benchmark participants’ results. In every case, the MOOSE results agreed with other simulations results for estimating bowing behavior, providing additional verification basis of the tools for this particular physics application. Estimation of contact forces were mostly in agreement, with a few outlier results. A plan was suggested for dealing with the discrepancies with estimating contact force values. In addition, a 1-way coupling demonstration was performed using subchannel analysis code Pronghorn-SC and MOOSE on an ABR-1000-design sodium-cooled fast reactor assembly to evaluate coolant and duct wall temperatures.

22 GENERAL STUDIES OF NUCLEAR REACTORS↗
DOE OSTI2024

Development & Experimental Validation of a Generalized Resistance-Capacitance Model for Numerical Simulation of Phase-Change Material Embedded Heat Exchangers

Latent heat thermal energy storage (LHTES) using phase change material (PCM) has attracted increased attention as a viable solution for overcoming the mismatch between energy supply and demand for renewable energy-based systems. PCM-embedded heat exchangers (PCM-HX) have the potential to significantly improve thermal performance due to high storage capacity and low temperature variation during the phase change process. Most models for simulating LHTES heat transfer use Computational Fluid Dynamics (CFD) simulations, which have high computational costs resulting from considering the complex and time-dependent physics relevant to PCM-HXs. In this paper, a Generalized Resistance Capacitance-based Model (GRCM) was developed to predict the thermal performance of arbitrary PCM-HXs in a computationally efficient manner without compromising modeling accuracy. The GRCM is exercised for three case studies: (i) verification for a single-slabbed finned PCM-HX, (ii) verification and validation for a copper foam/paraffin composite PCM-HX, and (iii) validation for a straight tube annular finned PCM-HX. The copper foam PCM-HX uses an electric heater at the top of HX, while the other two configurations utilize water as heat transfer fluid. For the single-slabbed finned PCM-HX melting case, the mean deviation in average PCM temperature predicted by the GRCM compared to the CFD model was between 0.56 – 0.73 K, with maximum temperature deviation of 2.68 K. For the HTF outlet temperature, the validation results showed that GRCM prediction matches very well with experimental data, with mean temperature deviation of 0.24 K during melting case, while for solidification case was 0.34 K. These results showcase the GRCM’s capability for accurately reproducing the thermal characteristics of PCM-HXs with considerably lower computational effort.

42 ENGINEERING↗
NASA NTRSDate not supplied

Adaptive Stress Testing of Collision Avoidance Systems for Small UASs with Deep Reinforcement Learning

The next-generation Airborne Collision Avoidance System for smaller UASs (ACAS sXu) is currently being developed and tested by the Federal Aviation Administration (FAA) to provide detect-and-avoid capability for small unmanned aircraft operating beyond line-of-sight. Due to the complexity and safety-critical nature of the system, safety validation is important not only for the certification of the final system, but also for informing changes during the iterative development process. In this paper, we analyze a prototype of ACAS sXu in simulated aircraft encounters to discover scenarios of small near mid-air collisions (sNMACs), an important safety event in which two aircraft come closer than 50 feet horizontally and 15 feet vertically. Due to the size and complexity of the system as well as rarity of sNMAC events, traditional methods such as Monte Carlo testing often require informed setup and targeting to elicit failures. However, such a dependence on domain knowledge can be incompatible with the independent verification and validation (IV&V) process, the aim of which is to discover unforeseen issues. To address these challenges, we apply an accelerated validation method called adaptive stress testing (AST) to find the most likely sNMAC scenarios without reliance on system introspection. AST uses reinforcement learning to adapt the search towards the most promising areas of the search space as it progresses. We use a state-of-the-art deep reinforcement learning algorithm, proximate policy optimization, to more efficiently search the large and continuous state space. We find that this approach significantly improves the performance of AST compared to a prior approach based on Monte Carlo tree search. We perform experiments using AST to find sNMAC events under various encounter configurations, varying parameters pertaining to dynamics and coordination. Our experiments show AST to be very effective at finding sNMAC scenarios. We summarize our findings, presenting high-level categories of discovered sNMACs and specific examples of encounters in each category.

aircraft collision avoidance↗
DOE OSTI2026

Validation of electron backscattering in the MCNP6.3 code

A new electron backscattering validation suite has been developed for the MCNP® code. The calculations in this suite cover both condensed history and single-event transport methods, seven different elemental materials ranging from beryllium to uranium, and incident electron energies from 200 eV to 14.1 MeV. In general, the accuracy of the MCNP code for electron backscattering calculations is energy-dependent. For incident electron energies below 2 keV, the condensed history method is not applied while the single-event method shows poor agreement with experimental measurements due to poor accuracy of the underlying atomic data. For incident energies between 2 keV and 256 keV, the single-event method typically shows better agreement with experiments, while for incident energies above 256 keV the condensed history method gives better accuracy. Additionally, agreement with experimental data is generally worse for low- Z materials such as beryllium and carbon, which may reflect deficiencies in the data or physics underlying electron transport through light elements in the MCNP code. The new electron backscattering validation suite, along with the previously introduced electron stopping power and Lockwood energy deposition validation suites, provide MCNP users with a quantified understanding of the uncertainties in electron transport calculations. Future improvements to electron transport in the MCNP code are necessary for low energies (≤ 2 keV) and for low- Z materials.

36 MATERIALS SCIENCE↗
DOE OSTI2025

Small-scale validation tests for MFIX-Exa CFD-DEM

This report describes several bench-scale fluidization experiments that can be used to validate the CFD-DEM method as encapsulated in the MFIX-Exa code. The five cases considered are the cold-flow fluidized beds of Müller et al., Link et al. (spout-fluid), and Goldschmidt et al. (bi-disperse), the hot fluidized bed of Patil et al., and the adsorbing fluidized bed of Li et al. and Janssen. In most cases, MFIX-Exa with “standard” or “typical” CFD-DEM settings, the Gidaspow drag model, and the Gunn heat transfer provide a relatively good prediction of the quantities considered: mean void fraction profiles, mean velocity profiles, fluctuating velocity profiles, mean particle temperature and segregation index. These results, with other verification and validation tests reported elsewhere, contribute to a body of work providing confidence and credibility in CFD predictions from the MFIX-Exa code.

97 MATHEMATICS AND COMPUTING↗
NASA NTRSDate not supplied

An Interim Set of TNT Curves for LOX/LNG Explosions

Space launch companies are actively developing, or in some cases have already developed, new vehicles that use large quantities of liquid oxygen (LOX) and liquid natural gas (LNG) propellants. This propellant mixture currently lacks sound/verified explosive safety standards for use in explosive siting and flight safety analysis. In the U.S., multiple government and commercial organizations have conducted limited testing, or are preparing to conduct tests, intended to provide a sound technical basis for LOX/LNG explosive standards applicable to space launch vehicle ground and flight operations. The limited testing and analysis conducted so far indicates the potential to produce extremely energetic explosions due to the miscibility of LOX/LNG (methane), a unique feature relative to previously used propellant mixtures such as hydrogen/oxygen or kerosene/oxygen. MOX is a homogenous mixture of methane and oxygen that is possible because of methane’s 100% miscibility in LOX. Preliminary testing by N2L, Inc. described in this paper shows that it is possible to form MOX with a variety of mixing methods, and that MOX is a sensitive high explosive capable of producing overpressures greater than comparable masses of C-4. The limited amount and quality of large-scale LOX/LNG explosive test data, and the potential formation of high-explosive MOX, create significant unknowns in the determination of hazard areas (cleared of the public) for ground operations (such as a static fire test) and flights of launch vehicles with LOX/LNG propellant. This paper will review the methods used by NASA to develop and validate the LOX/LH2 blast model using large-scale explosive test programs such as Project PYRO, the Hydrogen-Oxygen Vertical Impact (HOVI), and the Large-Scale Hydrogen-Oxygen Explosion (LSHOE) tests. This paper will describe how the same process was used to prepare an interim LOX/LNG blast model that applies to various LV failure scenarios and conservatively accounts for potential MOX formation. This paper also summarizes past and future testing and modeling efforts funded by a consortium of NASA, the US Federal Aviation Administration (FAA), and the US Space Force (USSF). These test programs should be concluded within the next 3 years and are intended to provide empirical data for model verification and validation.

Liquid Oxygen↗
NASA NTRSDate not supplied

An Interim Set of TNT Curves for LOX/LNG Explosions

Space launch companies are actively developing, or in some cases have already developed, new vehicles that use large quantities of liquid oxygen (LOX) and liquid natural gas (LNG) propellants. This propellant mixture currently lacks sound/verified LOX/LNG explosive safety standards for use in explosive siting and flight safety analysis. In the U.S., multiple government and commercial organizations have conducted limited testing, or are preparing to conduct tests, intended to provide a sound technical basis for explosive standards applicable to space launch vehicle ground and flight operations. The limited testing and analysis conducted so far indicates the potential to produce extremely energetic explosions due to the miscibility of LOX/LNG (methane), a unique feature relative to previously used propellant mixtures such as hydrogen/oxygen or kerosene/oxygen. MOX is a homogenous mixture of methane and oxygen that is possible because of methane’s 100% miscibility in LOX. Preliminary testing by N2L, Inc. described in this paper shows that it is possible to form MOX with a variety of mixing methods, and that MOX is a sensitive high explosive capable of producing overpressures greater than comparable masses of C-4. The limited amount and quality of large-scale LOX/LNG explosive test data, and the potential formation of high-explosive MOX, create significant unknowns in the determination of hazard areas (cleared of the public) for ground operations (such as a static fire test) and flights of launch vehicles with LOX/LNG propellant. This paper will review the methods used by NASA to develop and validate the LOX/LH2 blast model using large-scale explosive test programs such as Project PYRO, the Hydrogen-Oxygen Vertical Impact (HOVI), and the Large-Scale Hydrogen-Oxygen Explosion (LSHOE) tests. This paper will describe how the same process was used to prepare an interim LOX/LNG blast model that applies to various LV failure scenarios and conservatively accounts for potential MOX formation. This paper also summarizes past and future testing and modeling efforts funded by a consortium of NASA, the US Federal Aviation Administration (FAA), and the US Space Force (USSF). These test programs should be concluded within the next 3 years and are intended to provide empirical data for model verification and validation.

Liquid Oxygen↗
NASA NTRS2014

Loads and Structural Dynamics Requirements for Spaceflight Hardware

The purpose of this document is to establish requirements relating to the loads and structural dynamics technical discipline for NASA and commercial spaceflight launch vehicle and spacecraft hardware. Requirements are defined for the development of structural design loads and recommendations regarding methodologies and practices for the conduct of load analyses are provided. As such, this document represents an implementation of NASA STD-5002. Requirements are also defined for structural mathematical model development and verification to ensure sufficient accuracy of predicted responses. Finally, requirements for model/data delivery and exchange are specified to facilitate interactions between Launch Vehicle Providers (LVPs), Spacecraft Providers (SCPs), and the NASA Technical Authority (TA) providing insight/oversight and serving in the Independent Verification and Validation role. In addition to the analysis-related requirements described above, a set of requirements are established concerning coupling phenomena or other interaction between structural dynamics and aerodynamic environments or control or propulsion system elements. Such requirements may reasonably be considered structure or control system design criteria, since good engineering practice dictates consideration of and/or elimination of the identified conditions in the development of those subsystems. The requirements are included here, however, to ensure that such considerations are captured in the design space for launch vehicles (LV), spacecraft (SC) and the Launch Abort Vehicle (LAV). The requirements in this document are focused on analyses to be performed to develop data needed to support structural verification. As described in JSC 65828, Structural Design Requirements and Factors of Safety for Spaceflight Hardware, implementation of the structural verification requirements is expected to be described in a Structural Verification Plan (SVP), which should describe the verification of each structural item for the applicable requirements. The requirement for and expected contents of the SVP are defined in JSC 65828. The SVP may also document unique verifications that meet or exceed these requirements with Technical Authority approval.

George James↗
DOE OSTI2024

From Sim to Real: A Pipeline for Training and Deploying Traffic Smoothing Cruise Controllers

Designing and validating controllers for connected and automated vehicles to enhance traffic flow presents significant challenges, from the complexity of replicating real-world stop-and-go traffic dynamics in simulation, to the intricacies involved in transitioning from simulation to actual deployment. In this work, we present a full pipeline from data collection to controller deployment. Specifically, we collect 772 km of driving data from the I-24 in Tennessee, and use it to build a one-lane simulator, placing simulated vehicles behind real-world trajectories. Using policy-gradient methods with an asymmetric critic, we improve fuel efficiency by over 10% when simulating congested scenarios. Our comprehensive approach includes reinforcement learning for controller training, software verification, hardware validation and setup, and navigating various sim-to-real challenges. Furthermore, we analyze the controller's behavior and wave-smoothing properties, and deploy it on four Toyota Rav4’s in a real-world validation experiment on the I-24. Lastly, we release the driving dataset, the simulator and the trained controller, to enable future benchmarking and controller design.

42 ENGINEERING↗
NASA NTRS2018

High Accuracy Coronagraph Flight Model For WFIRST–CGI Raw Contrast Sensitivity Analysis

A high-accuracy high-fidelity flight wavefront control (WFC) model is developed for detailed raw contrast sensitivity analysis of WFIRST-CGI. Built upon features of recently testbed validated model, it is further refined to combine a full Fresnel propagation diffraction model for high accuracy contrast truth evaluation, and an economical compact model for WFC purposes. Extensive individual raw contrast error sensitivities are evaluated systematically, both as known imperfections and as unknown calibration errors, for both spectroscopy mode and wide field-of-view mode with shaped pupil coronagraph. More than 90 distinct error items were identified, including system aberrations, optical misalignment, component manufacturing error, telescope interface related errors, etc. The result forms the basis for raw contrast error budget flow down to a sub-system level, where detailed specifications needed to aid in component design and manufacturing, mechanical alignment and instrument integration, and verification and validation operations. Evaluations are mostly automated, making it relatively easy for repeat runs of revised design or at new desired error quantity. Top error sensitivities and contrast floor contributors are discussed and several observations are noted.

Poberezhskiy, Ilya↗
NASA NTRS2019

SAFE50 Reference Design Study for Large-Scale High-Density Low-Altitude UAS Operations in Urban Areas

Enabling safe, routine, and high-density flight operations of small UAS at low-altitude over heavily populated urban centers presents a difficult challenge for emerging UAS Traffic Management (UTM) system concepts. Urban operations by definition involve flight over people, property, and infrastructure. Low-altitude urban environments - such as urban canyons – are one of the most difficult areas for UTM to consider. Mission concepts require routine operations in a cluttered radio-frequency (RF) environment with degraded or denied Global Positioning System (GPS) reception. Flights with any appreciable distance will be beyond visual and communications line-of-sight from ground operators. Timely detection and response to emergencies and onboard failures, which is critical for safe aircraft operation, will be difficult. This work seeks to establish a feasible reference autonomy architecture for autonomous vehicles in an urban UTM system, then verifying and validating this architecture within a complete UTM concept point-design and systems analysis study. In this paper, we present the results from the NASA SAFE50 conceptual design and systems study that investigates the trade-space of urban UTM operations. This advanced conceptual design study develops a feasible, verified, validated point-design solution. The SAFE50 point-design concept places emphasis on advanced, highly-autonomous, and highly-capable vehicles that favors intelligent onboard autonomy over direct human control with today's technologies and operating in today's urban environments. This paper focuses on an general overview of the design study, highlighting decisions made in the architectural solution. This paper will presents a summary of the study, architectures, and requirements. We present an overview of the architecture designs as derived from the top-level UTM system. The point-design has been implemented in both simulation and through flight testing of hardware design prototypes. The results from simulation and flight testing as part of the verification and validation process of the reference design study.

Ippolito, Corey A.↗
DOE OSTI2022

Validation of NEAMS Tools Using MSRE Data

This report documents the benchmarks developed and performed for the tools and codes developed under the Nuclear Energy Advanced Modeling and Simulation (NEAMS) program utilizing the MSRE experiment data. There are three focus areas explored in this work: fuel evolution, CFD, and system transient analyses. Fuel depletion is important for MSR reactor physics and fuel cycle analysis. Fuel depletion capability is currently being developed in the Griffin reactor physics code. To support this, a benchmark problem is developed using the MSRE data and an existing depletion code with flowing fuel analysis functionality (ADDER) to validate this new capability in Griffin. The ADDER calculation is compared with the TRITON-MSR results as well as the MSRE experimental data. This benchmark can be directly used to verify and validate the Griffin fuel depletion capability in future years. For CFD simulation, the MSRE CAD model is obtained from the Copenhagen Atomics and a mesh is generated for the MSRE inlet plenum. The Nek5000 code will be employed to find the flow distribution in the MSRE inlet plenum based on the mesh generated. For system analysis, a simple benchmark problem for three types of MSR system transients is developed. The SAM and SPECTRA codes are used to solve the simple problem, and the results from the two codes agree well. The SAM code is also employed to obtain the frequency response of the MSRE system at the 5 and 8 MWt power levels. The results agree well with the experiment measurements. These verification and validation efforts gives SAM users more confidence in MSR system transient simulations using SAM. Lastly, the SAM-Griffin Multiphysics coupling work is initiated. A simple one-way coupling of evaluating the precursor drift effect on the MSRE k-eff is demonstrated. This preliminary coupling work serves as the basis for future more complex Multiphysics simulation and modelling.

46 INSTRUMENTATION RELATED TO NUCLEAR SCIENCE AND ↗
NASA NTRS2024

Recommendations on Evidence and Process for Certification of Learning-enabled Components in Aerospace Systems

This report primarily identifies a collection of relevant and necessary evidence for assurance of machine learnt components (MLCs)—also known as learning-enabled components—integrated into aircraft systems, and gives preliminary suggestions on the elements of a certification process that invoke the identified evidence. The main focus is on feedforward neural networks that are static and trained offline through supervised learning. A brief background on the generic elements of the lifecycle of an MLC is given to contextualize the assurance considerations and, consequently, the evidence that is relevant and necessary to support certification. At the level of an MLC, those considerations relate to: (i) the consistency and correctness of MLC contributions to system functions in the context of a validated functional intent; and (ii) the absence of MLC contributions to aircraft-level failure conditions. At an ML model level, confidence in model and data properties contribute to assurance of the containing MLC, in particular: (a) generalizability and robustness of models, in the presence of inputs not previously seen during training, disturbances to inputs, and unexpected inputs; and (b) valid data, i.e., data that are at least representative, relevant, complete, and accurate. Evidence for the above span the elements of the ML lifecycle, and includes, at a minimum, lifecycle artifacts that pertain to: (1) properties of requirements capturing functional intent, safety constraints, and aspects of the intended use and operating environment; (2) model performance, model complexity and design, and algorithm choice; (3) achievement of required performance at the levels of a trained model during model development, a trained model after model development is complete, and a trained model that is transformed into an executable equivalent; (4) model implementation aspects necessary for transforming a trained model into the executable equivalent; (5) integration of the executable trained model into the containing MLC, and eventually the larger system; and, (6) lastly, the verification and validation (V&V) of each of the above. Such V&V lifecycle artifacts themselves include: aspects of coverage, e.g., of various levels of requirements by the input space of the model and the data; traceability (where applicable); application of formal methods for property specification, analysis, and checking. Examples of evidence generation methods and tools further ground the discussion on what constitutes evidence, and the contribution to assurance during certification. The identified assurance considerations and supporting evidence is not a comprehensive set. Additionally, neither what should be considered as sufficient evidence relative to the assigned criticality of an MLC, nor how criticality ought to be determined and adjusted, have been considered in this report. However, suggestions are made for potential activities of the ML lifecycle that are aimed at providing confidence that an MLC can be relied upon when integrated into its containing (aircraft) system. Those activities are proposed as candidate elements of a certification process for MLCs. The main purpose of this report to inform regulatory guidance and consensus standards that may be used to meet the safety intent of the applicable regulations.

Aviation safety↗
NASA NTRS2018

Assurance of Model-Based Fault Diagnosis

Autonomy is an increasingly important technology for robotic scientific and commercial spacecraft. An important motivation for developing onboard autonomy is to enable quick response to dynamic environment and situations, including fault conditions that a spacecraft may encounter. The reliability of such autonomous capabilities depends on the quality of their knowledge of a spacecraft’s health state. Model-based approaches to fault management, i.e. model-based fault diagnosis (MBFD), is one approach to continuously verify correct behavior in addition to diagnosing symptoms to estimate the spacecraft’s health state. The proper functioning of MBFD is dependent on 1) the quality of the model that is analyzed and compared to the outputs of onboard sensors to estimate the system’s health state, and 2) the correct functioning of the diagnosis engine that interrogates the model and compares its analysis to observed system behavior. We are currently developing Verification and Validation (V&V) approaches to provide the necessary confidence that MBFD systems are correctly estimating the health of on-board spacecraft components and systems. Our work is intended to narrow the gap between the rapidly maturing field of ModelBased System Engineering (including MBFD) and the less-well understood area of identifying and applying appropriate V&V techniques to MBFD. Our effort is investigating three areas: 1) developing V&V techniques for the diagnostic model, 2) developing V&V techniques for the diagnostic engine in isolation, and 3) developing V&V techniques for the diagnostic engine and model in combination. This paper describes the work we have completed in the first area. We describe our approach to selecting a system to be represented by a model, the approach to modeling the system, the verification approach we developed for the model, and the results of the verification activity. We conclude with a description of the work remaining in the last two areas, which will be addressed over the next two years.

Chung, Seung↗
DOE OSTI2023

Efforts to enhance reproducibility in a human performance research project

Background: Ensuring the validity of results from funded programs is a critical concern for agencies that sponsor biological research. In recent years, the open science movement has sought to promote reproducibility by encouraging sharing not only of finished manuscripts but also of data and code supporting their findings. While these innovations have lent support to third-party efforts to replicate calculations underlying key results in the scientific literature, fields of inquiry where privacy considerations or other sensitivities preclude the broad distribution of raw data or analysis may require a more targeted approach to promote the quality of research output. Methods: We describe efforts oriented toward this goal that were implemented in one human performance research program, Measuring Biological Aptitude, organized by the Defense Advanced Research Project Agency's Biological Technologies Office. Our team implemented a four-pronged independent verification and validation (IV&V) strategy including 1) a centralized data storage and exchange platform, 2) quality assurance and quality control (QA/QC) of data collection, 3) test and evaluation of performer models, and 4) an archival software and data repository. Results: Our IV&V plan was carried out with assistance from both the funding agency and participating teams of researchers. QA/QC of data acquisition aided in process improvement and the flagging of experimental errors. Holdout validation set tests provided an independent gauge of model performance. Conclusions: In circumstances that do not support a fully open approach to scientific criticism, standing up independent teams to cross-check and validate the results generated by primary investigators can be an important tool to promote reproducibility of results.

59 BASIC BIOLOGICAL SCIENCES↗
DOE OSTI2021

Evaluating Symmetry in 3-dimensional Simulation Data

The increasing use of three-dimensional simulations has led to an increased need for effective verification and validation, or V&V. Unfortunately, the high computational cost associated with three-dimensional simulations makes many standard V&V techniques cost-prohibitive. Given these circumstances, we must pay close attention to V&V metrics that can be evaluated using only a single simulation. One of the simplest of these metrics is the evaluation of physical symmetries. This report details a method of computing the deviation from symmetry in a three-dimensional dataset with an arbitrarily distributed mesh. Examples are presented for spherical and axisymmetric systems using the Paraview post-processing analysis workflow.

97 MATHEMATICS AND COMPUTING↗
NASA NTRS2013

Validating an Air Traffic Management Concept of Operation Using Statistical Modeling

Validating a concept of operation for a complex, safety-critical system (like the National Airspace System) is challenging because of the high dimensionality of the controllable parameters and the infinite number of states of the system. In this paper, we use statistical modeling techniques to explore the behavior of a conflict detection and resolution algorithm designed for the terminal airspace. These techniques predict the robustness of the system simulation to both nominal and off-nominal behaviors within the overall airspace. They also can be used to evaluate the output of the simulation against recorded airspace data. Additionally, the techniques carry with them a mathematical value of the worth of each prediction-a statistical uncertainty for any robustness estimate. Uncertainty Quantification (UQ) is the process of quantitative characterization and ultimately a reduction of uncertainties in complex systems. UQ is important for understanding the influence of uncertainties on the behavior of a system and therefore is valuable for design, analysis, and verification and validation. In this paper, we apply advanced statistical modeling methodologies and techniques on an advanced air traffic management system, namely the Terminal Tactical Separation Assured Flight Environment (T-TSAFE). We show initial results for a parameter analysis and safety boundary (envelope) detection in the high-dimensional parameter space. For our boundary analysis, we developed a new sequential approach based upon the design of computer experiments, allowing us to incorporate knowledge from domain experts into our modeling and to determine the most likely boundary shapes and its parameters. We carried out the analysis on system parameters and describe an initial approach that will allow us to include time-series inputs, such as the radar track data, into the analysis

Statistical emulation↗