SEARCH · Engineering Papers

Results for “Benchmarking Software”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

SourceResearch connectionFrom yearThrough year

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 307 records · Page 17

All sources NASA NTRS DOE OSTI

DOE OSTI2023

BuildingsBench: A Benchmark for Universal Building Load Forecasting [SWR-23-51]

The residential and commercial building stock in the United States is responsible for a significant percentage of energy consumption and greenhouse gas emissions. Electrification of end-uses, as well as decarbonizing the electrical grid through renewable energy sources such as solar and wind, constitutes the pathway to zero-emission buildings. Forecasting day-ahead building energy consumption is an integral part of this solution. Currently, specialized forecasting models are hand-made for each individual building, which is time-consuming, expensive, and leads to duplicated efforts. BuildingsBench is a Python software framework for training and comparing generalized machine learning models for universal building load forecasting. This challenge tasks a single foundational model to generalize its forecasts for a wide variety of buildings, across geographic regions, building types, weather patterns, and more. This software provide code for pre-training such models and subsequently evaluating their performance on a suite of hundreds of diverse real and synthetic buildings. BuildingsBench is a platform for: - Large-scale pretraining with the synthetic Buildings-900K dataset for short-term load forecasting (STLF). Buildings-900K is statistically representative of the entire U.S. building stock and is extracted from the NREL End-Use Load Profiles database. - Benchmarking on two tasks evaluating generalization: zero-shot STLF and transfer learning for STLF. We provide an index-based PyTorch Dataset for large-scale pretraining, easy data loading for multiple real building energy consumption datasets as PyTorch Tensors or Pandas DataFrames, simple (persistence) to advanced (transformer) baselines, metrics management, and more.

Emami, Patrick↗

DOE OSTI2026

Ground and excited state gradients with end-to-end differentiable semiempirical quantum chemistry

Accurate and efficient gradients of molecular energy with respect to nuclear degrees of freedom are essential for geometry optimization and molecular dynamics, including simulations that go beyond the Born–Oppenheimer regime. A common approach involves deriving analytical formulas for new electronic structure methods, which is often conceptually difficult and requires tedious coding. Here, we implement analytical, semi-numerical, and automatic differentiation (AD)-based gradient pathways for semiempirical Hamiltonian models in the PYSEQM software package, leveraging both graphics processing unit (GPU) and central processing unit (CPU) architectures. We further extend these capabilities to excited states calculated using the configuration interaction singles and time-dependent Hartree–Fock ansätze. We benchmark wall time, peak memory usage, and accuracy across three molecular families of varying chemical complexity, including systems of up to a thousand atoms. For ground-state simulations, analytical and AD gradients achieve near-identical GPU runtimes, while semi-numerical gradients are slower on GPU but remain competitive on CPU. For excited states, both analytical and custom AD approaches using implicit differentiation show similar performance and low memory requirements, whereas gradients with full AD are memory-limited. AD gradients match analytical ones in accuracy across all tested systems, aided by a quaternion-based diatomic frame rotation for two-center quantities that ensures smooth energy surfaces. Overall, automatic differentiation emerges as a practical alternative to analytical gradients in semiempirical quantum chemistry, offering high accuracy while allowing seamless integration in AI-driven workflows and popular packages, such as PyTorch and JAX. Our results provide actionable guidance for selecting optimal gradient strategies in large-scale ground- and excited-state molecular dynamics simulations.

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗

DOE OSTI2020

Accelerating Radiation Computations for Dynamical Models With Targeted Machine Learning and Code Optimization

Abstract Atmospheric radiation is the main driver of weather and climate, yet due to a complicated absorption spectrum, the precise treatment of radiative transfer in numerical weather and climate models is computationally unfeasible. Radiation parameterizations need to maximize computational efficiency as well as accuracy, and for predicting the future climate many greenhouse gases need to be included. In this work, neural networks (NNs) were developed to replace the gas optics computations in a modern radiation scheme (RTE+RRTMGP) by using carefully constructed models and training data. The NNs, implemented in Fortran and utilizing BLAS for batched inference, are faster by a factor of 1–6, depending on the software and hardware platforms. We combined the accelerated gas optics with a refactored radiative transfer solver, resulting in clear‐sky longwave (shortwave) fluxes being 3.5 (1.8) faster to compute on an Intel platform. The accuracy, evaluated with benchmark line‐by‐line computations across a large range of atmospheric conditions, is very similar to the original scheme with errors in heating rates and top‐of‐atmosphere radiative forcings typically below 0.1 K day −1 and 0.5 W m −2 , respectively. These results show that targeted machine learning, code restructuring techniques, and the use of numerical libraries can yield material gains in efficiency while retaining accuracy.

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗

DOE OSTI2026

Data from: "Towards CONUS-Wide ML-Augmented Conceptually-Interpretable Modeling of Catchment-Scale Precipitation-Storage-Runoff Dynamics"

This data package was generated to support the manuscript “Towards CONUS-Wide Machine Learning-Augmented Conceptually Interpretable Modeling of Catchment-Scale Precipitation-Storage-Runoff Dynamics.” It provides input files, model outputs, plotting data, scripts, notebooks, and documentation used to develop, evaluate, and reproduce Mass-Conserving Perceptron (MCP)-based hydrologic modeling experiments across 513 selected Catchment Attributes and Meteorology for Large-sample Studies in the United States (CAMELS-US) basins. The files are organized by modeling component and analysis purpose, including rainfall–runoff experiments, snow module experiments, coupled hydrologic-snow experiments, Long Short-Term Memory (LSTM) benchmark results, model skill metrics, initialization and epoch records, cell-state normalization files, Akaike Information Criterion (AIC)-based model comparison files, and data used to generate manuscript figures. Tabular files can be opened using standard spreadsheet software or Python/R data-analysis tools. Python scripts, Jupyter notebooks, and selected MATLAB scripts are included for model execution, postprocessing, plotting, and statistical analysis. Quality assurance and quality control were conducted through the source-data selection and modeling workflow. Meteorological forcing, streamflow, and static catchment attributes were derived from the CAMELS-US dataset, and snow water equivalent data were derived from the University of Arizona (UA) Snow Water Equivalent dataset. Selected basins and time periods were screened during the associated research workflow to avoid missing observations or poor-quality cases. Static geospatial features were processed primarily using Quantum Geographic Information System (QGIS) and Geospatial Data Abstraction Library (GDAL) workflows. Additional details are provided in the associated manuscript and documentation.

ESS-DIVE CSV File Formatting Guidelines Reporting ↗

NASA NTRS2010

SensorWeb 3G: Extending On-Orbit Sensor Capabilities to Enable Near Realtime User Configurability

This research effort prototypes an implementation of a standard interface, Web Coverage Processing Service (WCPS), which is an Open Geospatial Consortium(OGC) standard, to enable users to define, test, upload and execute algorithms for on-orbit sensor systems. The user is able to customize on-orbit data products that result from raw data streaming from an instrument. This extends the SensorWeb 2.0 concept that was developed under a previous Advanced Information System Technology (AIST) effort in which web services wrap sensors and a standardized Extensible Markup Language (XML) based scripting workflow language orchestrates processing steps across multiple domains. SensorWeb 3G extends the concept by providing the user controls into the flight software modules associated with on-orbit sensor and thus provides a degree of flexibility which does not presently exist. The successful demonstrations to date will be presented, which includes a realistic HyspIRI decadal mission testbed. Furthermore, benchmarks that were run will also be presented along with future demonstration and benchmark tests planned. Finally, we conclude with implications for the future and how this concept dovetails into efforts to develop "cloud computing" methods and standards.

Mandl, Daniel↗

DOE OSTI2020

LA17-V-InverseRadTransportProblem-PD2Kb. Final Project Report

In the inverse radiation transport problem, a radiation signal emanating from an unknown object is used to reconstruct geometric and material parameters of the object , e.g., the mass of special nuclear material (SNM), the isotopic composition of the SNM (e.g., the enrichment of 235U), the thickness of shielding, etc. The inverse radiation transport problem is one of the most fundamental and important in the emergency response mission, and has been addressed in previous NA-22 and NA-42/84 investments by using derivative-free optimization methods. Our project was aimed at (i) consolidating previous work by applying polynomial chaos expansion (PCE) for reducing significantly the number of forward model simulations required by derivative-free optimization methods; (ii) developing a general-purpose framework and stand-alone software for solving the inverse radiation transport problem by applying a new “predictive modeling” methodology (predictive modeling of coupled multi-physics systems, PM_CMPS) which enables the fusing of experimental and computational information; and (iii) developing several benchmarks (for unscattered gamma rays) for demonstrating the quantification of non-Gaussian features (e.g., skewness and kurtosis, which characterize asymmetries and long tails) of response distributions by deterministically and efficiently propagating parameter uncertainties (covariances and higher-order moments, where available) using high-order sensitivities computed by the recently developed comprehensive adjoint sensitivity analysis methodology (C-ASAM). With regard to (i), application of PCE, the most significant accomplishment of this project was an implementation of PCE into INVERSE that significantly reduced computation time for the DiffeRential Evolution Adaptive Metropolis (DREAM) and Differential Evolution (DE) methods. For inverse transport problems with measurements of neutron leakage, neutron multiplication, and neutron-induced gamma-rays, the solution times for the DREAM and DE methods were reduced by factors of 30-300, reducing run times from hours to minutes while maintaining the same accuracy as the standard (non-PCE) DREAM and DE. Speedups were further improved by exploring sparse grid quadrature sets using the Toolkit for Adaptive Stochastic Modeling and Non-Intrusive ApproximatioN (Tasmanian) toolset. An initial exploration of sparse grids indicated that further speedups factors of 5-10 are obtainable. The implementation of PCE in INVERSE is not in a form that users can easily access. With regard to (ii), solving the inverse problem using a predictive modeling paradigm, the most significant accomplishment of this project were the derivation and implementation of adjoint-based second derivatives in neutron and photon transport theory. There are two codes that compute these derivatives. SENSPG computes them for uncollided gamma rays and SENSMG computes them for neutron leakage. SENSMG is open-source code available on github. Other significant accomplishments included a stand-alone code, MULTI-PRED, that implements the predictive modeling paradigm, along with a detailed manual. The code was implemented in the existing INVERSE code for inverse problems, but a head-to-head comparison of inverse methods was not achieved. We also derived and implemented the capability to compute sensitivities for multiplicity measurements into the SENSMG code, and this capability was used in a data assimilation application, but it was not used for solving inverse problems. With regard to (iii), developing benchmarks to demonstrate the importance of high-order sensitivity and uncertainty analysis, the most significant accomplishment of this project were two test problems that showed how important second-order sensitivities are.

07 ISOTOPE AND RADIATION SOURCES↗

DOE OSTI2024

Using Likwid and Byfl to Benchmark Hardware Performance [Poster]

Benchmark Study conducted focusing on CPU and program performance analysis. Performance data gathered using 2 different programs and comparisons made based on performance. After comparisons are made, conclusions can be drawn and improvements are made upon hardware and software.

97 MATHEMATICS AND COMPUTING↗

NASA NTRS1989

Flexible missile autopilot design studies with PC-MATLAB/386

Development of a responsive, high-bandwidth missile autopilot for airframes which have structural modes of unusually low frequency presents a challenging design task. Such systems are viable candidates for modern, state-space control design methods. The PC-MATLAB interactive software package provides an environment well-suited to the development of candidate linear control laws for flexible missile autopilots. The strengths of MATLAB include: (1) exceptionally high speed (MATLAB's version for 80386-based PC's offers benchmarks approaching minicomputer and mainframe performance); (2) ability to handle large design models of several hundred degrees of freedom, if necessary; and (3) broad extensibility through user-defined functions. To characterize MATLAB capabilities, a simplified design example is presented. This involves interactive definition of an observer-based state-space compensator for a flexible missile autopilot design task. MATLAB capabilities and limitations, in the context of this design task, are then summarized.

Ruth, Michael J.↗

NASA NTRS2004

The Wallops Flight Facility Rapid Response Range Operations Initiative

While the dominant focus on short response missions has appropriately centered on the launch vehicle and spacecraft, often overlooked or afterthought phases of these missions have been launch site operations and the activities of launch range organizations. Throughout the history of organized spaceflight, launch ranges have been the bane of flight programs as the source of expense, schedule delays, and seemingly endless requirements. Launch Ranges provide three basic functions: (1) provide an appropriate geographical location to meet orbital other mission trajectory requirements, (2) provide project services such as processing facilities, launch complexes, tracking and data services, and expendable products, and (3) assure safety and property protection to participating personnel and third-parties. The challenge with which launch site authorities continuously struggle, is the inherent conflict arising from projects whose singular concern is execution of their mission, and the range s need to support numerous simultaneous customers. So, while tasks carried out by a launch range committed to a single mission pale in comparison to efforts of a launch vehicle or spacecraft provider and could normally be carried out in a matter of weeks, major launch sites have dozens of active projects separate sponsoring organizations. Accommodating the numerous tasks associated with each mission, when hardware failures, weather, maintenance requirements, and other factors constantly conspire against the range resource schedulers, make the launch range as significant an impediment to responsive missions as launch vehicles and their cargo. The obvious solution to the launch site challenge was implemented years ago when the Department of Defense simply established dedicated infrastructure and personnel to dedicated missions, namely an Inter Continental Ballistic Missile. This however proves to be prohibitively expensive for all but the most urgent of applications. So the challenge becomes how can a launch site provide acceptably responsive mission services to a particular customer without dedicating extensive resources and while continuing to serve other projects? NASA's Wallops Flight Facility (WFF) is pursuing solutions to exactly this challenge. NASA, in partnership with the Virginia Commercial Space Flight Authority, has initiated the Rapid Response Range Operations Initiative (R3Ops). R3Ops is a multi-phased effort to incrementally establish and demonstrate increasingly responsive launch operations, with an ultimate goal of providing ELV-class services in a maximum of 7-10 days from initial notification routinely, and shorter schedules possible with committed resources. This target will be pursued within the reality of simultaneous concurrent programs, and ideally, largely independent of specialized flight system configurations. WFF has recently completed Phase 1 of R3Ops, an in-depth collection (through extensive expert interviews) and software modeling of individual steps by various range disciplines. This modeling is now being used to identify existing inefficiencies in current procedures, to identify bottlenecks, and show interdependencies. Existing practices are being tracked to provide a baseline to benchmark against as new procedures are implemented. This paper will describe in detail the philosophies behind WFF's R3Ops, the data collected and modeled in Phase 1, and strategies for meeting responsive launch requirements in a multi-user range environment planned for subsequent phases of this initiative.

Underwood, Bruce E.↗

DOE OSTI2025

Quantum Electrodynamics Coupled-Cluster at Scale: High-Performance Implementation for Complex Systems

Coupled-cluster theory (CC) is a highly accurate and versatile method for simulating complex interactions within quantum systems. The extension of CC theory to model mixed electron-photon processes with quantum electrodynamics (QED) has improved our capability to predict cavity-modified chemistry, a field where photons are used as cost-effective and eco-friendly alternatives to catalyze/inhibit chemical reactions. However, calculations with CC methods, even without incorporating QED effects, are often prohibitively expensive. Simulations of larger systems require scalable infrastructures that exist for traditional CC methods but not for QED-CC methods. As such, we present a GPU-enabled, high-performance, open-source implementation of the quantum electrodynamics coupled-cluster method with single and double excitations (QED-CCSD) within the ExaChem quantum chemistry software package. ExaChem relies on the Tensor Algebra for Many-body Methods (TAMM) infrastructure: a parallel heterogeneous tensor library designed to achieve scalable performance on modern heterogeneous supercomputing platforms. Furthermore, we discuss theoretical foundations, algorithmic details, and numerical benchmarks to showcase the larger systems that ExaChem can simulate and how the integration of photonic degrees-of-freedom alters their ground-state properties.

Basis sets↗

DOE OSTI2026

Did You Win the GPU Cloud Lottery? Benchmarking from TFLOPS to Tokens/$

Cloud GPUs are commonly assumed to deliver consistent performance for a given GPU model. This assumption does not always hold: cloud providers employ diverse system configurations and virtualization mechanisms, and GPUs themselves exhibit non-negligible manufacturing variability (the silicon lottery). In this work, we present a large-scale measurement study of GPU performance variability across 11 cloud providers, covering over 3,500 physical GPUs and 6,800 benchmark runs. Our hierarchical analysis shows that while execution-level variation stays below 9%, performance varies by up to 38% across devices and providers for the same GPU model. Regression analysis indicates that driver- and OS-related software factors contribute less than 1% of the variance; instead, silicon lottery effects dominate observed performance variation, and cloud providers further amplify them through persistent, systematic second-order effects.

Slynko, Platon [Silicon Data, New York, USA] (ORCI↗

NASA NTRSDate not supplied

Scalability of GlennICE in a Parallel Environment

The Glenn Icing Computational Environment (GlennICE) is a computational tool designed to calculate ice growth on complex three-dimensional geometries using the input from a user-supplied computational fluid dynamics (CFD) solution for the geometry of interest. The most significant developments in the advancement of GlennICE have been investigating the convergence of the collection efficiency, efficiently finding trajectories, and improving the refinement methodology. Such developments have increased the efficiency of GlennICE for practical engineering application. With the increasing demand for applying GlennICE for more memory-intensive problems, the scalability of GlennICE has yet to be investigated. This paper is aimed at presenting a method to benchmark the scalability of GlennICE utilizing a relevant engineering problem within a parallel environment. This leads to the final goal of investigating whether an increase in the number of processors utilized results in a linear speedup of the GlennICE software.

Computational Icing, Icing, CFD, MPI↗

DOE OSTI2022

Particle Swarm Optimization Algorithm for Critical Experiment Design

Nuclear criticality experiments are used to validate nuclear cross section data used by simulation software. This is typically achieved by designing a critical system with a high sensitivity to a certain material’s cross section. Once the experiment has been carried out, a high fidelity model of the system is developed into a benchmark. When this benchmark model is simulated by a transport code, some of the difference between the experimental and computational effective neutron multiplication factor can be attributed to inaccurate nuclear data. Nuclear data evaluators then can make adjustments accordingly to improve cross section data.

73 NUCLEAR PHYSICS AND RADIATION PHYSICS↗

DOE OSTI2026

NCCS High Performance GMRES Mixed Precision

HPG-MxP is a software package that performs a fixed number of multigrid preconditioned (using a Gauss-Seidel smoother) Generalized minimal residual (PGMRES) iterations in order to solve a possibly nonsymmetric large sparse linear system of equations. It is designed to be a benchmark to measure a computer's performance for sparse linear algebra workloads typical in scientific computing while allowing the use of mixed precision methods. The solution is required to have convergence characteristics and accuracy similar to double precision GMRES. It is based on the High Performance Conjugate Gradient Benchmark (HPCG) which restricts all implementations to use only the IEEE double precision format (FP64). The original implementation (https://github.com/hpg-mxp/hpg-mxp) was written by Ichitaro Yamazaki, Jennifer Loe, Christian Glusa, Sivasankaran Rajamanickam, Piotr Luszczek, and Jack Dongarra. Please refer to that repository for documentation on the original implementation. This version is maintained by the National Center for Computational Sciences at Oak Ridge National Laboratory. It is highly scalable and optimized for Oak Ridge Leadership Computing Facility (OLCF) systems, particularly Frontier.

Kashi, Aditya [Oak Ridge National Laboratory (ORNL↗

NASA NTRS1987

Multitasking and microtasking experience on the NA S Cray-2 and ACF Cray X-MP

The fast Fourier transform (FFT) kernel of the NAS benchmark program has been utilized to experiment with the multitasking library on the Cray-2 and Cray X-MP/48, and microtasking directives on the Cray X-MP. Some performance figures are shown, and the state of multitasking software is described.

Raiszadeh, Farhad↗

NASA NTRS2018

High-resolution Wave Propagation Method for Stratified Flows

The implementation of the multidimensional f-waves Riemann solver for the time-dependent, three-dimensional, nonhydrostatic, meso- and microscale atmospheric flows is described in detail. The Riemann solver employs flux-based wave decomposition (f-waves) for the calculation of Godunov fluxes in which the flux differences are written directly as the linear combination of the right eigenvectors of the hyperbolic system. The scheme incorporates the source term due to gravity without introducing discretization errors which is an important property in the context of atmospheric flows. The resulting flow solver is conservative, accurate, stable, and well-balanced. The implementation of the solver is evaluated using benchmark test cases for atmospheric dynamics.

Riemann problem↗

DOE OSTI2024

Stereo-DIC Challenge 1.0 – Rigid Body Motion of a Complex Shape

Background Stereo-DIC is a widely used optical measurement technique that provides a dense full-field 3D measurement of the shape, displacement, and strain of a solid sample. When compared with 2D-DIC, Stereo-DIC provides greater flexibility and expands its use beyond flat, planar specimens. Furthermore, the widespread availability of commercial systems has led to the adoption of the technique throughout industry, academia, and government research labs. Objective Even though some research has been done to understand the effects of different experimental and stereo-DIC parameters, no reference is available to benchmark and compare the performance of current stereo-DIC algorithms to each other. Methods This paper provides the description and analysis of a carefully controlled 3D experiment and associated images used to compare the results from five subset based DIC software packages. Both the images and analysis codes used in this paper to compare the results are described here and are available for download and use for continued research. Results We show that over a very large range of motion, the 3D errors are very small, less than 80μm over a travel of ±20 mm out-of-plane and ±20 mm in-plane. While all codes performed similarly, there are important differences noted in the paper. Conclusion The image sets and results comparison software are hosted by the International DIC Society (www.iDICs.org) and are freely available for download and analysis for comparison with results in this paper. Furthermore, it is hoped that this set of images can be used for future research in improving stereo-DIC by future authors.

Algorithms comparison↗

DOE OSTI2026

AmeriFlux FLUXNET-1F US-ADR Amargosa Desert Research Site (ADRS)

This is the AmeriFlux Management Project (AMP) created FLUXNET-1F version of the carbon flux data for the site US-ADR Amargosa Desert Research Site (ADRS). This is the FLUXNET version of the carbon flux data for the site US-ADR Amargosa Desert Research Site (ADRS) produced by applying the standard ONEFlux (1F) software. Site Description - This tower is located at the Amargosa Desert Research Site (ADRS). The U.S. Geological Survey (USGS) began studies of unsaturated zone hydrology at ADRS in 1976. Over the years, USGS investigations at ADRS have provided long-term "benchmark" information about the hydraulic characteristics and soil-water movement for both natural-site conditions and simulated waste-site conditions in an arid environment. The ADRS is located in a creosote-bush community adjacent to disposal trenches for low-level radioactive waste.

Moreo, Michael [U.S. Geological Survey]↗