Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “Benchmarking Software”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 73 records · Page 4
All sourcesNASA NTRSDOE OSTI
DOE OSTI2020

Flexible Reinforcement Learning Framework for Building Control using EnergyPlus-Modelica Energy Models

In recent years, reinforcement learning (RL) methods have been greatly enhanced by leveraging deep learning approaches. RL methods applied to building control have shown potential in many applications due to their ability to complement or replace conventional methods such as model-based or rule-based controls. However, RL-based building control software is likely tailored either to one target building system or to a specific RL method so that significant additional effort would be required to customize the RL-based controller for use in other building systems or with other RL approaches. Also, RL-based building controls usually depend on building energy simulations to train controllers, so emulating building dynamics (i.e., thermal dynamics and control dynamics) and capturing sub-hourly dynamic profiles are crucial to further the development of effective RL-based building control methods. To address these challenges, we present an open source RL-based control software employing a high-fidelity hybrid EnergyPlus-Modelica building energy model which emulates building dynamics at 1-minute resolution. This software consists of decoupled components (environment, building emulator, control agent, and RL algorithm), which allows for quick prototyping and benchmarking of standard RL algorithms in different systems; for example, a single component can be replaced without revising all of the software. To demonstrate this software framework, we conducted a benchmark study using an EnergyPlus-Modelica building energy model for a Chicago office building with an RL-based controller to dynamically control the chilled water temperature setpoint and the air handling unit supply air temperature setpoint on selected floors.

Lee, Joon-Yong↗
DOE OSTI2025

Fundamental limit of jet tagging

Identifying the origin of high-energy hadronic jets (jet tagging) has been a critical benchmark problem for machine learning in particle physics. Jets are ubiquitous at colliders and are complex objects that serve as prototypical examples of collections of particles to be categorized. Over the last decade, machine learning-based classifiers have replaced classical observables as the state of the art in jet tagging. Increasingly complex machine learning models are leading to increasingly more effective tagger performance. Our goal is to address the question of convergence—are we getting close to the fundamental limit on jet tagging or is there still potential for computational, statistical, and physical insights for further improvements? We address this question using state-of-the-art generative models to create a realistic, synthetic dataset with a known jet tagging optimum. Various state-of-the-art taggers are deployed on this dataset, showing that there is a significant gap between their performance and the optimum. Our dataset and software are made public to provide a benchmark task for future developments in jet tagging and other areas of particle physics.

Artificial intelligence↗
DOE OSTI2021

Benchmarking the Performance of Neuromorphic and Spiking Neural Network Simulators

Software simulators play a critical role in the development of new algorithms and system architectures in any field of engineering. Neuromorphic computing, which has shown potential in building brain-inspired energy-efficient hardware, suffers a slow-down in the development cycle due to a lack of flexible and easy-to-use simulators of either neuromorphic hardware itself or of spiking neural networks (SNNs), the type of neural network computation executed on most neuromorphic systems. While there are several openly available neuromorphic or SNN simulation packages developed by a variety of research groups, they have mostly targeted computational neuroscience simulations, and only a few have targeted small-scale machine learning tasks with SNNs. Evaluations or comparisons of these simulators have often targeted computational neuroscience-style workloads. In this work, we seek to evaluate the performance of several publicly available SNN simulators with respect to non-computational neuroscience workloads, in terms of speed, flexibility, and scalability. We evaluate the performance of the NEST, Brian2, Brian2GeNN, BindsNET and Nengo packages under a common front-end neuromorphic framework. Our evaluation tasks include a variety of different network architectures and workload types to mimic the computation common in different algorithms, including feed-forward network inference, genetic algorithms, and reservoir computing. We also study the scalability of each of these simulators when running on different computing hardware, from single core CPU workstations to multi-node supercomputers. Our results show that the BindsNET simulator has the best speed and scalability for most of the SNN workloads (sparse, dense, and layered SNN architectures) on a single core CPU. However, when comparing the simulators leveraging the GPU capabilities, Brian2GeNN outperforms the others for these workloads in terms of scalability. NEST performs the best for small sparse networks and is also the most flexible simulator in terms of reconfiguration capability NEST shows a speedup of at least 2x compared to the other packages when running evolutionary algorithms for SNNs. The multi-node and multi-thread capabilities of NEST show at least 2x speedup compared to the rest of the simulators (single core CPU or GPU based simulators) for large and sparse networks. We conclude our work by providing a set of recommendations on the suitability of employing these simulators for different tasks and scales of operations. We also present the characteristics for a future generic ideal SNN simulator for different neuromorphic computing workloads.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2026

Systematic Benchmarking of Climate Models: Methodologies, Applications, and New Directions

As climate models become increasingly complex, there is a growing need to comprehensively and systematically assess model performance with respect to observations. Given the increasing number and diversity of climate model simulations in use, the community has moved beyond simple model intercomparison and toward developing methods capable of benchmarking a large number of simulations against a suite of climate metrics. Here, we present a detailed review of evaluation and benchmarking methods and approaches developed in the last decade, focusing primarily on scientific implications for Coupled Model Intercomparison Project (CMIP) simulations and CMIP6 results that contributed to the Intergovernmental Panel on Climate Change (IPCC) Sixth Assessment Report (AR6). Based on this review, we explain the resulting contemporary philosophy of model benchmarking, and provide clear distinctions and definitions of the terms model verification, process validation, evaluation, and benchmarking. While significant progress has been made in model development based on systematic evaluation and benchmarking efforts, some climate system biases still remain. The development of open‐source community software packages has played a fundamental role in identifying areas of significant model improvement and bias reduction. We review the key features of several software packages that have been commonly used over the past decade to evaluate and benchmark global and regional climate models. Additionally, we discuss best practices for the selection of evaluation and benchmarking metrics and for interpreting the obtained results, the importance of selecting suitable sources of reference data and accurate uncertainty quantification.

Environmental sciences↗
DOE OSTI2022

Near real-time streaming analysis of big fusion data

Experiments on fusion plasmas produce high-dimensional data time series with ever-increasing magnitude and velocity, but turn-around times for analysis of this data have not kept up. For example, many data analysis tasks are often performed in a manual, ad-hoc manner some time after an experiment. In this article, we introduce the Delta framework that facilitates near real-time streaming analysis of big and fast fusion data. By streaming measurement data from fusion experiments to a high-performance compute center, Delta allows computationally expensive data analysis tasks to be performed in between plasma pulses. This article describes the modular and expandable software architecture of Delta and presents performance benchmarks of individual components as well as of an example workflow. Focusing on a streaming analysis workflow where electron cyclotron emission imaging (ECEi) data is measured at KSTAR on the National Energy Research Scientific Computing Center's (NERSC's) supercomputer we routinely observe data transfer rates of about 4 Gigabit per second. In NERSC, a demanding turbulence analysis workflow effectively utilizes multiple nodes and graphical processing units and executes them in under 5 min. We further discuss how Delta uses modern database systems and container orchestration services to provide web-based real-time data visualization. For the case of ECEi data we demonstrate how data visualizations can be augmented with outputs from machine learning models. Here, by providing session leaders and physics operators, results of higher-order data analysis using live visualizations may make more informed decisions on how to configure the machine for the next shot.

70 PLASMA PHYSICS AND FUSION TECHNOLOGY↗
DOE OSTI2022

NISQ Benchmarking

Test suite of quantum algorithms for Noisy Intermediate Scale Quantum (NISQ) computers. The test suite includes benchmark-style code for quantum volume circuits (QV), fairness sampling circuits, quantum telecloning circuits, and other NISQ benchmark style algorithms on small problems (i.e., up to 100 qubits), such as Variational Quantum Eigensolver (VQE), Hamiltonian Simulation, and Grover unstructured search example circuits. These benchmark-style applications are implemented in quantum software packages, mostly IBM's QISKIT, but may include vendor-specific frameworks, such as PyQuil (for Rigetti) or Q\# for Microsoft, or CirQ (for Google) as the test suite grows with the vendor sample. The test suite also includes numerical simulation code for Quantum Alternating Operator Ansatz (QAOA) algorithms, VQE, Hamiltonian Simulation and search examples. Numerical simulation code simulates quantum computers on classical computers, which is only possible for small problem instances; the implementation framework of choice is typically within Python, using the numpy/scipy libraries as well as extensions to the Julia language.

Pelofske, Elijah↗
DOE OSTI2025

Towards exact finite temperature electronic structure in solids and molecules (Final Technical Report)

This report describes the University of Iowa portion of a project that is now continuing at Michigan State University. We are developing novel methods, algorithms, and software to enable simulations of molecules and materials at high temperature. This is a key challenge in chemistry and materials science. By refining an approach called Density Matrix Quantum Monte Carlo (DMQMC), we developed faster and more accurate ways to conduct these simulations. These advances will help us understand how temperature affects the behavior of electrons, chemical bonds, and phase transitions in solids and molecules. These breakthroughs are especially important for applications where light and heat drive chemical reactions, superconductivity, and materials used in energy and sensing. In addition, this project involved the development of the open-source HANDE-QMC software package, supporting the broader community in benchmarking and developing finite-temperature electronic structure methods.

36 MATERIALS SCIENCE↗
DOE OSTI2022

SLIP (Surrogate Launching and Integration Platform)

SLIP (Surrogate Launching and Integration Platform) is a software ecosystem for downloading and running ML/AI benchmarks. SLIP automatically downloads and sets up the code and data running a benchmark. The large data sets and model code are cached locally with a specific ID within a designated cache directory.

Brown, Cade↗
DOE OSTI2022

Evaluation of China Experimental Fast Reactor Start-up Tests (Final Report, Revision 1)

This report documents the results and observations on the Coordinated Research Project (CRP) of the International Atomic Energy Agency (IAEA) on “Neutronics Benchmark of CEFR Start-Up Tests.” The China Experimental Fast Reactor (CEFR) is a 65MWt sodium-cooled fast reactor with highly enriched uranium oxide fuel. The reactor achieved first criticality in 2010, and a series of start-up tests were conducted to measure various reactor physics parameters. In 2018, the IAEA launched the CRP for validation and qualification of member states' computation capabilities in the field of fast reactor simulation by utilizing the measured data in the CEFR start-up test. Twenty-nine international organizations from eighteen member countries, including Argonne National Laboratory, have participated in the CRP. The CEFR start-up tests offer a rare opportunity to validate U.S. nuclear engineering software because it is a well-specified benchmark with corresponding measurements for a recently built fast reactor starting up with a known fuel composition (fresh fuel). Previous fast reactor validation benchmarks frequently involve reactors that have already been started up and contain irradiated fuel that is difficult to characterize with high certainty.

21 SPECIFIC NUCLEAR REACTORS AND ASSOCIATED PLANTS↗
DOE OSTI2023

TRINIDI (Time-of-Flight Resonance Imaging with Neutrons for Isotopic Density Inference)

This software is an open-source Python library that provides tools for processing hyperspectral neutron time-of-flight radiography data. This type of data allows material decomposed reconstructions to be generated with the use of material characteristic spectral responses and the algorithms provided in this code library. The software library will contain tools for pre-processing the neutron measurement data, estimating measurement system parameters, reconstructing material decomposed radiographs, and computing material decomposed computed tomography (CT). Furthermore, it will have capability to generate and process simulated neutron time-of-flight data with the goal of benchmarking and demonstrating the tools that are provided. The software will include thorough documentation and application examples.

Balke, Thilo↗
DOE OSTI2024

LibERI—A portable and performant multi-GPU accelerated library for electron repulsion integrals via OpenMP offloading and standard language parallelism

A portable and performant graphics processing unit (GPU)-accelerated library for electron repulsion integral (ERI) evaluation, named LibERI, has been developed and implemented via directive-based (e.g., OpenMP and OpenACC) and standard language parallelism (e.g., Fortran DO CONCURRENT). Offloaded ERIs consist of integrals over low and high contraction s, p, and d functions using the rotated-axis and Rys quadrature methods. GPU codes are factorized based on previous developments with two layers of integral screening and quartet presorting. In this work, the density screening is moved to the GPU to enhance the computational efficacy for large molecular systems. Here, the L-shells in the Pople basis set are also separated into pure S and P shells to increase the ERI homogeneity and reduce atomic operations and the memory footprint. LibERI is compatible with any quantum chemistry drivers supporting the MolSSI Driver Interface. Benchmark calculations of LibERI interfaced with the GAMESS software package were carried out on various GPU architectures and molecular systems. The results show that the LibERI performance is comparable to other state-of-the-art GPU-accelerated codes (e.g., TeraChem and GMSHPC) and, in some cases, outperforms conventionally developed ERI CUDA kernels (e.g., QUICK) while fully maintaining portability.

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗
DOE OSTI2022

Evaluation of an in vitro three-dimensional HepaRG spheroid model for genotoxicity testing using the high-throughput CometChip platform

Three-dimensional (3D) culture systems are increasingly being used for genotoxicity studies due to improved cell-to-cell interactions and tissue-like structures that are limited or lacking in 2D cultures. The present study optimized a 3D culture system using metabolically competent HepaRG cells for in vitro genotoxicity testing. 3D HepaRG spheroids, formed in 96- or 384-well ultra-low attachment plates, were exposed to various concentrations of 34 test articles, including 8 direct-acting and 11 indirect-acting genotoxicants/carcinogens as well as 15 compounds that show different genotoxic responses in vitro and in vivo. DNA damage was evaluated using the high-throughput CometChip assay with concurrent cytotoxicity assessment by the ATP assay in both 2D and 3D cultures. 3D HepaRG spheroids maintained a stable phenotype for up to 30 days with higher levels of albumin secretion, cytochrome P450 gene expression, and enzyme activities compared to 2D cultures. 3D spheroids also demonstrated a higher sensitivity than 2D cultures for detecting both direct- and indirect-acting genotoxicants/carcinogens, indicating a better prediction of in vivo genotoxicity responses. When DNA damage dose-response data were quantified using PROAST software, 3D spheroids generally had lower or similar benchmark dose values compared to 2D HepaRG cells and were more comparable with primary human hepatocytes. These results demonstrate that 3D models can be adapted to the CometChip technology for high-throughput genotoxicity testing and that 3D HepaRG spheroids may be used as a reliable and pragmatic in vitro approach to better support the hazard identification and risk assessment of potential human genotoxic carcinogens.

60 APPLIED LIFE SCIENCES↗
DOE OSTI2024

Evaluation of δ-Phase ZrH1.4 to ZrH1.7 Thermal Neutron Scattering Laws Using Ab Initio Molecular Dynamics Simulations

Zirconium hydride is commonly used for next-generation reactor designs due to its excellent hydrogen retention capacity at temperatures below 1000 K. These types of reactors operate at thermal neutron energies and require accurate representation of thermal scattering laws (TSLs) to optimize moderator performance and evaluate the safety indicators for reactor design. In this work, we present an atomic-scale representation of sub-stoichiometric ZrH2−x(0.3≤x≤0.6), which relies on ab initio molecular dynamics (AIMD) in tandem with velocity auto-correlation (VAC) analysis to generate phonon density of states (DOS) for TSL development. The novel NJOY+NCrystal tool, developed by the European Spallation Source community, was utilized to generate the TSL formulations in the A Compact ENDF (ACE) format for its utility in neutron transport software. First, stoichiometric zirconium hydride cross sections were benchmarked with experiments. Then sub-stoichiometric zirconium hydride TSLs were developed. Significant deviations were observed between the new δ-phase ZrH2−x TSLs and the TSLs in the current ENDF release. It was also observed that varying the hydrogen vacancy defect concentration and sites did not cause as significant a change in the TSLs (e.g., ZrH1.4 vs. ZrH1.7) as was caused by the lattice transformation from ϵ- to δ-phase.

42 ENGINEERING↗
DOE OSTI2020

CosTuuM: Polarized Thermal Dust Emission by Magnetically Oriented Spheroidal Grains

We present the new open-source C++-based Python library CosTuuM that can be used to generate infrared absorption and emission coefficients for arbitrary mixtures of spheroidal dust grains that are (partially) aligned with a magnetic field. We outline the algorithms underlying the software, demonstrate the accuracy of our results using benchmarks from literature, and use our tool to investigate some commonly used approximative recipes. We find that the linear polarization fraction for a partially aligned dust grain mixture can be accurately represented by an appropriate linear combination of perfectly aligned grains and grains that are randomly oriented, but that the commonly used picket fence alignment breaks down for short wavelengths. We also find that for a fixed dust grain size, the absorption coefficients and linear polarization fraction for a realistic mixture of grains with various shapes cannot both be accurately represented by a single representative grain with a fixed shape, but that instead an average over an appropriate shape distribution should be used. Insufficient knowledge of an appropriate shape distribution is the main obstacle in obtaining accurate optical properties. CosTuuM is available as a standalone Python library and can be used to generate optical properties to be used in radiative transfer applications.

79 ASTRONOMY AND ASTROPHYSICS↗
DOE OSTI2024

A Systematic Comparison for Consistent Scenario Development Using Microscopic Simulation Software

This study aims to explore a methodology that enables the development of consistent traffic micro simulation for emerging traffic and vehicle control technologies for improved mobility and energy efficiency across different modeling platforms. Researchers might study the same application on different platforms and have the need to benchmark across platforms. However, there lacks a systematic study on simulation software comparison, especially for emerging mobility and energy efficiency applications. For this, a systematic scenario development and evaluation approach is presented and demonstrated to compare scenarios generated in different traffic microsimulation platforms. Network-level and vehicle-level trip performance results of the traffic scenario are evaluated in three microscopic simulation platforms - VISSIM, AIMSUN, and SUMO. The results indicate that the network-level performance is consistent among the three software suites except when the demand is high, where the energy consumption performance varies.

Saroj, Abhilasha [ORNL] (ORCID:0000000191178063)↗
DOE OSTI2023

Fiscal Year 2023 Software Quality Assurance Activities for the ARC Software

The Argonne Reactor Code (ARC) software suite [1-17] has been developed by Argonne researchers for fast reactor design and analysis since the 1970s. With the ARC software suite, a user can quickly build a model of a proposed or existing fast spectrum reactor and carry out fuel cycle, nominal thermal analysis and flow requirements, and assess, as is appropriate, whether the core design and constraint system yield an acceptable mechanical behavior. For transient reactor analysis with SAS4A [18], the ARC software suite can be used to generate reactivity coefficients and kinetics parameters at any modeled fuel cycle time point which forms part of the input to SAS4A. The ARC suite was consistently being developed until the 1990s and followed a software QA program which was an appropriate standard for the time. In the 1990s, the DOE funding to fast reactor research and development was all but eliminated and the ARC software was put into maintenance mode. In the early 2000s, the software quality assurance (SQA) program for ARC was still in place to define an official version, but by 2005 it all but was abandoned as there were insufficient staff to fill the work roles. Since 2005, there has been a considerable increase in research and design work on fast spectrum reactors. The ARC software as a whole has since been exported to many universities and commercial companies and ANL support has been given to the various projects over the years [19-23]. Further, MC 2 -3, PERSENT, and DASSH were all developed after 2005 without any adherence to a software standard. In recent time, the DOE VTR project [22] paid for verification work to be done on the ARC software as part of the goal of making it NQA-1 complaint. The VTR project was not considered the appropriate pathway to fund and maintain a SQA program for the ARC software and while software developments (DASSH) were made and several manuals were updated and software verification work was carried out, the ARC software is not NQA-1 compliant. More recently the Advanced Reactor Development Program (ARDP [23]) has funded the creation of manuals for some ARC utility programs and funded additional software verification work on DIF3D [6, 7] and MC 2 -3 [2-5] for the purpose of commercial grade dedication. Because of the VTR and ARDP projects, software verification work was completed on MC 2 -3 and DIF3D, and detailed reports were created for each piece of software, which discuss the inputs and outputs from the codes that are covered by the verification work and link various analytic, code-to-code, and hand calculation based verification work presented in the report with verification test problems provided with the software. This is a key part of the commercial grade dedication work and constitutes the bulk of the cost to get the ARC software to commercial grade. The ARC software suite is a valuable asset as a fast reactor design and analysis tool set that has been reasonably well verified and validated with various fast reactor benchmark problems and experiments over decades. Some or all of the ARC software suite has been utilized for designing the IFR [20], PGSFR [21], VTR [22], and Natrium [23] reactors and we can expect it to continue to be used for advanced fast reactor design and/or confirmatory calculation purposes in the future. Due to increased interest by commercial companies and regulatory bodies, it is becoming more important to make the ARC software suite complete and ready-to-use in terms of its SQA pedigree and commercial grade dedication needs. This report discusses the achievements made towards building a new SQA program for the ARC software and dealing with outstanding identified QA gaps.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2023

Towards operational atmospheric correction of airborne hyperspectral imaging spectroscopy: Algorithm evaluation, key parameter analysis, and machine learning emulators

Atmospheric correction of airborne hyperspectral imaging spectroscopy (AHIS) to obtain high-quality surface reflectance is the prerequisite for remote sensing applications. Over the last decades, different atmospheric correction methods have been developed based on radiative transfer models (RTMs), however, the relative performances of different algorithms are unclear. Automated operational atmospheric correction methods to process large-volume AHIS data in a high-accurate and high-throughput manner are still lacking. Therefore, this study proposed an operational atmospheric correction pipeline for deriving surface reflectance from AHIS data. To ensure the accuracy and efficiency of the pipeline, we focused on three specific aspects: (1) selecting a suitable RTM for the development of atmospheric lookup tables (LUTs) by comparing the commercial MODerate resolution atmospheric TRANsmission (MODTRAN) and open-sourced Library for Radiative TRANsfer (LibRadTRAN) models, where the widely-used software, Atmospheric/Topographic Correction for Airborne Imagery (ATCOR), was used as benchmarks; (2) identifying key atmospheric correction parameters and determining suitable sources for parameter retrievals including AHIS, Moderate Resolution Imaging Spectroradiometer (MODIS), and AErosol RObotic NETwork (AERONET); and (3) testing the performance of using machine learning emulators to speed up the RTM-based atmospheric correction. Results indicate that (1) atmospheric correction based on MODTRAN LUTs can produce surface reflectance accurately with mean absolute errors < 0.05 and cosine similarities > 0.98 compared to field measurements, which is comparable to the software ATCOR and slightly outperforms the LibRadTRAN LUTs; (2) sobol global sensitivity analysis demonstrates that in the atmospheric correction, visibility and water vapor are two key parameters that can be accurately derived from AHIS in contrast to MODIS or AERONET data; and (3) Random Forest emulators can produce accurate estimations of surface reflectance with mean absolute errors < 0.03 and cosine similarities > 0.98 for higher processing efficiency and determine a suitable set of wavelengths for retrieving atmospheric visibility and water vapor. In conclusion, the proposed atmospheric correction pipeline also improved the four-stream radiative transfer theory for airborne applications by considering adjacent effects from airborne surrounding pixels and can also be applied for atmospheric correction of hyperspectral data from spaceborne missions.

47 OTHER INSTRUMENTATION↗
DOE OSTI2023

Thermomechanical analysis and modeling of involute-shaped fuel plates using the Cheverton–Kelley experiments for the High Flux Isotope Reactor

Three research reactors with involute-shaped fuel plates are pursuing conversion from highly enriched uranium to low-enriched uranium fuel. Various core design and safety evaluation studies are essential to assess the feasibility of the conversion. The use of 3D computational multiphysics codes is being explored in these analyses and therefore they must undergo a thorough evaluation and quality assurance process due to their potential impact on nuclear safety. Here, the Cheverton and Kelley physical tests performed in the late 1960s to investigate the deflections of HFIR’s outer plate under uniform pressure and temperature fields are simulated by employing commercially available computational codes, with the goals to (1) verify and validate the models and numerical solvers implemented in the codes for thermomechanical analysis of involute reactor plates and (2) to develop a benchmark computational test to evaluate future versions of existing software or newly developed computational codes. The results of the simulations showed good agreement with each other as well as against the Cheverton–Kelley experimental data. Some minor deviations were observed for a few multiphysics cases and their potential origins and impact on the analysis results is investigated in the paper. The validated models increase the confidence in using multiphysics codes to evaluate existing or new LEU designs.

21 SPECIFIC NUCLEAR REACTORS AND ASSOCIATED PLANTS↗