Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “Benchmarking Software”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 217 records · Page 12
All sourcesNASA NTRSDOE OSTI
NASA NTRS2021

Modeling of Guided Waves for Aerospace Applications

Advancements in computer hardware has led to new possibilities for rapid modeling and simulation capabilities across many scientific fields. Nondestructive evaluation (NDE) can benefit from increased use of simulation tools to guide optimization of inspection and health monitoring methods, enhance understanding of data, aid in development of defect characterization methods, and generate data sets for use with machine learning and model-assisted probability of detection. Recent work at NASA has entailed development and benchmarking of both custom simulation codes and commercial simulation tools for ultrasonic wave propagation. This paper describes recent work at NASA in modeling of guided waves in composites and other aerospace materials. Results and computational speeds for a composite benchmark case are reported for a custom finite difference Rotated Staggered Grid code and for the commercial finite element software package, Pogo. Recent progress in linking NDE models to parametric analysis tools is also discussed.

Nondestructive evaluation↗
DOE OSTI2022

Accelerating the Inference of the Exa.TrkX Pipeline

Recently, graph neural networks (GNNs) have been successfully used for a variety of particle reconstruction problems in high energy physics, including particle tracking. The Exa.TrkX pipeline based on GNNs demonstrated promising performance in reconstructing particle tracks in dense environments. It includes five discrete steps: data encoding, graph building, edge filtering, GNN, and track labeling. All steps were written in Python and run on both GPUs and CPUs. In this work, we accelerate the Python implementation of the pipeline through customized and commercial GPU-enabled software libraries, and develop a C++ implementation for inferencing the pipeline. The implementation features an improved, CUDA-enabled fixed-radius nearest neighbor search for graph building and a weakly connected component graph algorithm for track labeling. GNNs and other trained deep learning models are converted to ONNX and inferenced via the ONNX Runtime C++ API. The complete C++ implementation of the pipeline allows integration with existing tracking software. We report the memory usage and average event latency tracking performance of our implementation applied to the TrackML benchmark dataset.

46 INSTRUMENTATION RELATED TO NUCLEAR SCIENCE AND ↗
DOE OSTI2023

SABATH: Surrogate AI Benchmarking Applications' Testing Harness

SABATH provides benchmarking infrastructure for evaluating scientific ML/AI models. It contains support for scientific machine learning surrogates from external repositories. The software dependences are explicitly exposed in the surrogate model definition, which allows the use of advanced optimization, communication, and hardware features.

Luszczek, Piotr [Univ. of Tennessee, Knoxville, TN↗
NASA NTRS2012

Development of Benchmark Examples for Quasi-Static Delamination Propagation and Fatigue Growth Predictions

The development of benchmark examples for quasi-static delamination propagation and cyclic delamination onset and growth prediction is presented and demonstrated for Abaqus/Standard. The example is based on a finite element model of a Double-Cantilever Beam specimen. The example is independent of the analysis software used and allows the assessment of the automated delamination propagation, onset and growth prediction capabilities in commercial finite element codes based on the virtual crack closure technique (VCCT). First, a quasi-static benchmark example was created for the specimen. Second, based on the static results, benchmark examples for cyclic delamination growth were created. Third, the load-displacement relationship from a propagation analysis and the benchmark results were compared, and good agreement could be achieved by selecting the appropriate input parameters. Fourth, starting from an initially straight front, the delamination was allowed to grow under cyclic loading. The number of cycles to delamination onset and the number of cycles during delamination growth for each growth increment were obtained from the automated analysis and compared to the benchmark examples. Again, good agreement between the results obtained from the growth analysis and the benchmark results could be achieved by selecting the appropriate input parameters. The benchmarking procedure proved valuable by highlighting the issues associated with choosing the input parameters of the particular implementation. Selecting the appropriate input parameters, however, was not straightforward and often required an iterative procedure. Overall the results are encouraging, but further assessment for mixed-mode delamination is required.

Krueger, Ronald↗
NASA NTRS2023

Space Networking Implementation for Lunar Operations

The High-Rate Delay Tolerant Networking (HDTN) project at NASA has developed a performance optimized and open-source Delay Tolerant Networking (DTN) implementation. The primary goal is to create a scalable networking solution to increase the scientific data return rate of space missions. To reach this goal, HDTN must span multiple edge cases in space networking by including tools and configurations to accommodate a wide range of space systems. Typically, HDTN evaluations are conducted on a laboratory emulation test bed, made up of hardware accelerated x86 based systems capable of data rates over 10 Gbps. HDTN must have an effective implementation process on a wide range of systems to increase the sustainability of the design. One important implementation option is with low-level embedded systems which could be used on small robotic missions. This paper details the implementation process, benchmark testing, and performance results of HDTN in multiple configurations on Raspberry Pi 4 devices. By implementing HDTN on a Raspberry Pi 4, a process for building HDTN onto ARM processors was developed and utilized to conduct benchmark tests in multiple network configurations, achieving a data rate performance exceeding 600 Mbps. Based on these results, HDTN proved to run on small ARM based systems with slight modifications to the build procedure. These results were then extended to evaluating an implementation of the HDTN software parsed across several Raspberry Pi 4 nodes. To test this capability, HDTN was configured in a simplified cut-through setup and distributed among multiple Raspberry Pi 4 processors. This distributed architecture was benchmark tested in a similar fashion to the testing of a singular HDTN implementation. The results from the benchmark testing are used to examine how these implementation options and capabilities can expand the use cases for DTN, and particularly with small robotic missions.

Space Networking↗
DOE OSTI2025

LLM Benchmarking with LLaMA2: Evaluating Code Development Performance Across Multiple Programming Languages

The rapid evolution of large language models (LLMs) has opened new possibilities for automating various tasks in software development. This paper evaluates the capabilities of the LLaMA 2-70B model in automating these tasks for scientific applications written in commonly used programming languages. Using representative test problems, we assess the model's capacity to generate code, documentation, and unit tests, as well as its ability to translate existing code between commonly used programming languages. Our comprehensive analysis evaluates the compilation, runtime behavior, and correctness of the generated and translated code. Additionally, we assess the quality of automatically generated code, documentation, and unit tests. Here, our results indicate that while LLaMA 2-70B frequently generates syntactically correct and functional code for simpler numerical tasks, it encounters substantial difficulties with more complex, parallelized, or distributed computations, requiring considerable manual corrections. We identify key limitations and suggest areas for future improvements to better leverage AI-driven automation in scientific computing workflows.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2025

Cluster Dynamics Modeling Needs for the Advanced Materials and Manufacturing Technologies Program

This milestone report aims to identify and assess the cluster dynamics (CD) modeling requirements within the Department of Energy's Office of Nuclear Energy (DOE-NE) Advanced Materials and Manufacturing Technologies (AMMT) program and to communicate these needs to the DOE-NE Nuclear Energy Advanced Modeling and Simulation (NEAMS) program. The goal is to ensure NEAMS is well-informed about the CD modeling requirements to support AMMT's mission of accelerating the development, qualification, demonstration, and deployment of advanced structural materials and manufacturing for nuclear energy applications. CD modeling is an essential tool for predicting the degradation of structural materials under irradiation, which is a key component of AMMT's accelerated qualification process. The AMMT program focuses on both additively manufactured and wrought structural alloys, such as laser powder-bed fusion 316H austenitic stainless steel, alloy 709, Haynes 244, and alloy 617. These materials require a generalized CD modeling framework to facilitate rapid model development and computational simulation. A flexible, generalized CD software, similar to the Multiphysics Object-Oriented Simulation Environment (MOOSE) finite element framework, would enable modeling of various cluster types, including defect clusters, defect-solute clusters, and multicomponent clusters, incorporating thermodynamics and kinetics parameters. Radiation effects, microstructural feature evolution, and multi-dimensional modeling are critical considerations for the CD model. The usability of the CD code should allow for easy modification and coupling with MOOSE-based simulations. Additionally, the software should adhere to Nuclear Quality Assurance-1 standards, include a testing suite for verification and validation, and be version-controlled within a national laboratory-managed Git repository. Benchmark problems are needed to assess code predictions and performance.

11 - NUCLEAR FUEL CYCLE AND FUEL MATERIALS↗
DOE OSTI2021

High-order algorithmic developments and optimizations for large-scale GPU-accelerated simulations (Milestone CEED-MS36)

The goal of this milestone was to improve the high-order software ecosystem for CEED-enabled ECP applications by making progress on efficient matrix-free kernels targeting forthcoming ECP architectures. These kernels included matrix-free preconditioning and the development of new set of CEED solver bake-off problems. As part of this milestone, we also released the next version of the CEED software stack, CEED-4.0, reported on results from several application collaborations, and documented the efforts of porting to AMD GPUs for Frontier and other modern architectures, such as Fugaku. The specific tasks addressed in this milestone were: (1) Port and run CEED benchmarks/miniapps on Frontier EA systems; (2) Demonstrate performant libCEED integration in MFEM, Nek and applications; (3) Matrix-free preconditioning of high-order operators; (4) Benchmark problems for fast high-order solvers on GPU platforms; and (5) Public release of CEED-4.0. The artifacts delivered include the next version of the CEED software stack, CEED-4.0, the next libCEED release, libCEED-0.8, and a number of developments integrated within applications to improve their GPU and CPU performance and capabilities. See the CEED website, https://ceed.exascaleproject.org and the CEED GitHub organization, https://github.com/ceed for more details.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2021

Modeling fission product diffusion in TRISO fuel particles with BISON

Diffusion of fission products in intact TRISO particles depends on particle geometry, fission product source rates, time, temperature, and temperature-dependent diffusion coefficients. Simulating this diffusion process requires models for source rates and diffusion coefficients, plus computation of the temperature field if not prescribed. In addition, simulation quality depends on discretization of the geometry, appropriate time stepping, and the accuracy of the solution method. In this paper, we explore the simulation of fission product diffusion in TRISO fuel particles using the finite element method via the fuel performance code Bison. Recent material model development has occurred in Bison for each material present in tri-structural isotropic (TRISO) fuel particles: the buffer, inner pyrolytic carbon, silicon carbide, and outer pyrolytic carbon layers, as well as the fuel kernel. Also, new mesh generation and fission product release fraction capabilities have been added. Diffusion capabilities are shown to converge to the correct solution via formal verification tests. A large number of code benchmarking problems are also given, with good results, showing that Bison’s computed release fractions closely match those of other software tools. Finally, a significant validation effort is detailed in which fission product release, measured as part of the AGR-1 capsule experiments, is compared to Bison outputs. Bison outputs compare very well to the experimental data and to PARFUME results.

11 NUCLEAR FUEL CYCLE AND FUEL MATERIALS↗
DOE OSTI2021

PRISMS-Fatigue computational framework for fatigue analysis in polycrystalline metals and alloys

Abstract The PRISMS-Fatigue open-source framework for simulation-based analysis of microstructural influences on fatigue resistance for polycrystalline metals and alloys is presented here. The framework uses the crystal plasticity finite element method as its microstructure analysis tool and provides a highly efficient, scalable, flexible, and easy-to-use ICME community platform. The PRISMS-Fatigue framework is linked to different open-source software to instantiate microstructures, compute the material response, and assess fatigue indicator parameters. The performance of PRISMS-Fatigue is benchmarked against a similar framework implemented using ABAQUS. Results indicate that the multilevel parallelism scheme of PRISMS-Fatigue is more efficient and scalable than ABAQUS for large-scale fatigue simulations. The performance and flexibility of this framework is demonstrated with various examples that assess the driving force for fatigue crack formation of microstructures with different crystallographic textures, grain morphologies, and grain numbers, and under different multiaxial strain states, strain magnitudes, and boundary conditions.

Chemistry↗
NASA NTRS1992

High performance flight simulation at NASA Langley

The use of real-time simulation at the NASA facility is reviewed specifically with regard to hardware, software, and the use of a fiberoptic-based digital simulation network. The network hardware includes supercomputers that support 32- and 64-bit scalar, vector, and parallel processing technologies. The software include drivers, real-time supervisors, and routines for site-configuration management and scheduling. Performance specifications include: (1) benchmark solution at 165 sec for a single CPU; (2) a transfer rate of 24 million bits/s; and (3) time-critical system responsiveness of less than 35 msec. Simulation applications include the Differential Maneuvering Simulator, Transport Systems Research Vehicle simulations, and the Visual Motion Simulator. NASA is shown to be in the final stages of developing a high-performance computing system for the real-time simulation of complex high-performance aircraft.

Cleveland, Jeff I., II↗
NASA NTRS2010

Automatically Finding the Control Variables for Complex System Behavior

Testing large-scale systems is expensive in terms of both time and money. Running simulations early in the process is a proven method of finding the design faults likely to lead to critical system failures, but determining the exact cause of those errors is still time-consuming and requires access to a limited number of domain experts. It is desirable to find an automated method that explores the large number of combinations and is able to isolate likely fault points. Treatment learning is a subset of minimal contrast-set learning that, rather than classifying data into distinct categories, focuses on finding the unique factors that lead to a particular classification. That is, they find the smallest change to the data that causes the largest change in the class distribution. These treatments, when imposed, are able to identify the factors most likely to cause a mission-critical failure. The goal of this research is to comparatively assess treatment learning against state-of-the-art numerical optimization techniques. To achieve this, this paper benchmarks the TAR3 and TAR4.1 treatment learners against optimization techniques across three complex systems, including two projects from the Robust Software Engineering (RSE) group within the National Aeronautics and Space Administration (NASA) Ames Research Center. The results clearly show that treatment learning is both faster and more accurate than traditional optimization methods.

Gay, Gregory↗
NASA NTRS2015

A Vehicle Management End-to-End Testing and Analysis Platform for Validation of Mission and Fault Management Algorithms to Reduce Risk for NASA's Space Launch System

The engineering development of the new Space Launch System (SLS) launch vehicle requires cross discipline teams with extensive knowledge of launch vehicle subsystems, information theory, and autonomous algorithms dealing with all operations from pre-launch through on orbit operations. The characteristics of these spacecraft systems must be matched with the autonomous algorithm monitoring and mitigation capabilities for accurate control and response to abnormal conditions throughout all vehicle mission flight phases, including precipitating safing actions and crew aborts. This presents a large and complex system engineering challenge, which is being addressed in part by focusing on the specific subsystems involved in the handling of off-nominal mission and fault tolerance with response management. Using traditional model based system and software engineering design principles from the Unified Modeling Language (UML) and Systems Modeling Language (SysML), the Mission and Fault Management (M&FM) algorithms for the vehicle are crafted and vetted in specialized Integrated Development Teams (IDTs) composed of multiple development disciplines such as Systems Engineering (SE), Flight Software (FSW), Safety and Mission Assurance (S&MA) and the major subsystems and vehicle elements such as Main Propulsion Systems (MPS), boosters, avionics, Guidance, Navigation, and Control (GNC), Thrust Vector Control (TVC), and liquid engines. These model based algorithms and their development lifecycle from inception through Flight Software certification are an important focus of this development effort to further insure reliable detection and response to off-nominal vehicle states during all phases of vehicle operation from pre-launch through end of flight. NASA formed a dedicated M&FM team for addressing fault management early in the development lifecycle for the SLS initiative. As part of the development of the M&FM capabilities, this team has developed a dedicated testbed that integrates specific M&FM algorithms, specialized nominal and off-nominal test cases, and vendor-supplied physics-based launch vehicle subsystem models. Additionally, the team has developed processes for implementing and validating these algorithms for concept validation and risk reduction for the SLS program. The flexibility of the Vehicle Management End-to-end Testbed (VMET) enables thorough testing of the M&FM algorithms by providing configurable suites of both nominal and off-nominal test cases to validate the developed algorithms utilizing actual subsystem models such as MPS. The intent of VMET is to validate the M&FM algorithms and substantiate them with performance baselines for each of the target vehicle subsystems in an independent platform exterior to the flight software development infrastructure and its related testing entities. In any software development process there is inherent risk in the interpretation and implementation of concepts into software through requirements and test cases into flight software compounded with potential human errors throughout the development lifecycle. Risk reduction is addressed by the M&FM analysis group working with other organizations such as S&MA, Structures and Environments, GNC, Orion, the Crew Office, Flight Operations, and Ground Operations by assessing performance of the M&FM algorithms in terms of their ability to reduce Loss of Mission and Loss of Crew probabilities. In addition, through state machine and diagnostic modeling, analysis efforts investigate a broader suite of failure effects and associated detection and responses that can be tested in VMET to ensure that failures can be detected, and confirm that responses do not create additional risks or cause undesired states through interactive dynamic effects with other algorithms and systems. VMET further contributes to risk reduction by prototyping and exercising the M&FM algorithms early in their implementation and without any inherent hindrances such as meeting FSW processor scheduling constraints due to their target platform - ARINC 653 partitioned OS, resource limitations, and other factors related to integration with other subsystems not directly involved with M&FM such as telemetry packing and processing. The baseline plan for use of VMET encompasses testing the original M&FM algorithms coded in the same C++ language and state machine architectural concepts as that used by Flight Software. This enables the development of performance standards and test cases to characterize the M&FM algorithms and sets a benchmark from which to measure the effectiveness of M&FM algorithms performance in the FSW development and test processes.

Trevino, Luis↗
DOE OSTI2025

Monte Carlo N-Particle Transport Performance of Predicting Digital Radiographic IQI Inspection

The identification of porosity, geometric noncompliance, and other defect types are critical to the qualification of materials and components. X-ray radiographic nondestructive testing is a common industrial inspection method for process quality control and component qualification and certification. Digital radiography provides a quick and efficient alternative when compared to traditional film-based inspection. The quality of radiographic inspection is dependent on equipment specifications, such as the source spot size and detector pixel size, and the specific parameters selected for use for the radiographic technique. To evaluate if an x-ray system and technique is sufficient for a given requirement, a radiographic image quality indicator (IQI) can be used. Radiographic IQIs in hard to machine materials or hard to manufacture defects can be time consuming and expensive to manufacture. This study was conducted to evaluate current Savannah River National Laboratory (SRNL) x-ray imaging systems with a custom tantalum IQI and using Monte Carlo simulations to predict the performance of future systems. The tantalum IQI was tested using a Siefert Isovolt 420 keV x-ray tube with a Perkin Elmer XRD 1611 flat panel with 100-micron pixels. Using the Monte Carlo N-Particle transport software, the radiographic tally was used to simulate the photon flux through an identical tantalum IQI. These simulations provided a benchmark as to the best theoretical identification on a given system using our tantalum IQI. The simulations were refined to match SRNL’s current systems’ noise levels, leading to confidence in their ability to predict the performance of other systems that may be purchased and deployed in the future at the Savannah River Site. Future studies will be conducted to prove this research can be extended to artificially evaluate the ability for systems to identify critical defect sizes through x-ray radiographic inspection, drastically reducing the cost and time burdens of producing high-fidelity radiographic test articles.

digital X-ray radiography↗
NASA NTRS2020

DEVELOPMENT AND APPLICATION OF BENCHMARK EXAMPLES FOR MIXED-MODE I/II FATIGUE GROWTH PREDICTIONS

The development of benchmark examples for assessing growth prediction capabilities is presented and demonstrated for a commercial code. The examples are based on finite element models of Mixed-Mode Bending (MMB) specimens for three different mode ratios. The examples are independent of the analysis software used and allow the assessment of the automated delamination propagation, onset and growth prediction capabilities in commercial finite element codes based on the virtual crack closure technique (VCCT). First, based on existing results for delamination propagation under static loading, new benchmark examples for delamination growth under fatigue loading were created. Second, starting from an initially straight front, the delamination was allowed to grow under fatigue loading. The number of cycles to delamination onset and the number of cycles during delamination growth for each growth increment were obtained from the automated analysis and compared to the benchmark examples. Good agreement between the results obtained from the growth analysis and the benchmark results could be achieved by selecting the appropriate input parameters. The benchmarking procedure proved valuable by highlighting the issues associated with choosing the input parameters of the particular implementation. Overall the results from automatic propagation are encouraging, however, selecting the appropriate input parameters was not straightforward and often required an iterative procedure.

Composites↗
DOE OSTI2020

Turbo FRMAC Implemetation of IAEA Radiological Assessment Methodologies for Nuclear and Radiological Emergencies.

This report documents the findings of an assessment of the Turbo FRMAC software's ability to implement International Atomic Energy Agency (IAEA) guidance for calculating Operational Intervention Levels (OIL) 1 & 2 for nuclear and radiological emergencies. The IAEA OIL and U.S. Federal Radiological Monitoring and Assessment Center (FRMAC) Derived Response Level methodology and implementation in respective tools were compared, as demonstrated through benchmarking activities for a nuclear power plant source term and potential radionuclides of concern for radiological dispersal devices. This comparison revealed some shortcomings in Turbo FRMACs ability to perform IAEA OIL calculations and resulted in recommended software modifications to be considered for future development.

61 RADIATION PROTECTION AND DOSIMETRY↗
DOE OSTI2022

Simulated 5g Network Traffic Dataset

This is a dataset of 5G network traffic for use with machine learning tools to benchmark attack detection capabilities for multiple different models. The dataset contains simulated normal and attack 5G network traffic. There is no software in this dataset, only simulated network traffic data.

Anderson, MatthewW↗
DOE OSTI2024

Computer Vision on Edge Devices for the Short Term Prediction of Cloud Cover

Edge Computing and IoT are important pieces of today's technological landscape. Here, we build a low-cost IoT sensor for sky imaging and program it using AWS GreenGrass, one of the leading IoT platforms. We demonstrate remote reprogramming of this device to load software that predicts sun shading events through the linear advection method, which is a baseline algorithm that can be used to benchmark algorithmic improvements in future work. Some future directions for sky imaging research are enumerated.

14 SOLAR ENERGY↗