SEARCH · Engineering Papers
Results for “BENCHMARKS”
Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.
Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.
Benchmarking a Proof-of-Concept Performance Portable SYCL-based Fast Fourier Transformation Library
ABSTRACT In this paper, we present an early version of a SYCL-based FFT library, capable of running on all major vendor hardware, including CPUs and GPUs from AMD, ARM, Intel and NVIDIA. The current limitations of our library is it supports single-dimension FFTs up to 211 in length and base-2 input sequences. Although preliminary, the aim of this work is to seed further developments for a rich set of features for calculating FFTs. The library has the advantage over existing portable FFT libraries in that it is single-source, and there- fore removes the complexities that arise due to abundant use of pre-processor macros and auto-generated kernels to target different architectures. We exercise two SYCL-enabled compilers, Codeplay ComputeCpp and Intel's open-source LLVM project, to evaluate performance portability of our SYCL-based FFT on various hetero- geneous architectures.We provide studies comparing our portable library with highly optimized vendor-specific FFT libraries, and discuss potential sources hindering performance.
Exploring Benchmarks for Self-Driving Labs using Color Matching
Self Driving Labs (SDLs) that combine automation of experimental procedures with autonomous decision making are gaining popularity as a means of increasing the throughput of scientific workflows. The task of identifying quantities of supplied colored pigments that match a target color, the color matching problem, provides a simple and flexible SDL test case, as it requires experiment proposal, sample creation, and sample analysis, three common components in autonomous discovery applications. We present a robotic solution to the color matching problem that allows for fully autonomous execution of a color matching protocol. Our solution leverages the WEI science factory platform to enable portability across different robotic hardware, the use of alternative optimization methods for continuous refinement, and automated publication of results for experiment tracking and post-hoc analysis.
CommBench: Micro-Benchmarking Hierarchical Networks with Multi-GPU, Multi-NIC Nodes
Modern high-performance computing systems have multiple GPUs and network interface cards (NICs) per node. The resulting network architectures have multilevel hierarchies of subnetworks with different interconnect and software technologies. These systems offer multiple vendor-provided communication capabilities and library implementations (IPC, MPI, NCCL, RCCL, OneCCL) with APIs providing varying levels of performance across the different levels. Understanding this performance is currently difficult because of the wide range of architectures and programming models (CUDA, HIP, OneAPI). We present CommBench, a library with cross-system portability and a high-level API that enables developers to easily build microbenchmarks relevant to their use cases and gain insight into the performance (bandwidth & latency) of multiple implementation libraries on different networks. We demonstrate CommBench with three sets of microbenchmarks that profile the performance of six systems. Our experimental results reveal the effect of multiple NICs on optimizing the bandwidth across nodes and also present the performance characteristics of four available communication libraries within and across nodes of NVIDIA, AMD, and Intel GPU networks.
Experimental Benchmarking for High-Reproducibility, Cross-Institutional Evaluation of Iron Redox Electrochemistry
We present a practical case study standardizing experimental protocols between collaborators with the goal of understanding ferrous iron (Fe 2+ ) chemistry and improving the iron deposition reaction for energy-efficient, electrochemical iron production. The study of iron reactions can be difficult, as aqueous iron electrolytes exhibit complex behaviors that can lead to differing interpretation of ostensibly similar experiments. The question we want to answer: are we studying the same chemistry? Our protocols address inherent challenges such as the tendency for Fe 2+ to spontaneously oxidize to ferric iron (Fe 3+ ) and the production of hydrogen at the potentials of interest. Our standardized protocol, executed by four collaborators in different labs and institutions, yields high-reproducibility results, and identified glassy carbon electrode surface quality and Fe 3+ impurities in the salt as key factors with outsized effects on cyclic voltammetry measurements. The process of developing the protocols helped to troubleshoot underlying issues that created poorer reversibility and reproducibility. This study highlights the fact that even nominally straightforward electrochemical systems can yield vastly different outcomes due to small differences in experimental preparation and serves as a useful example for creating transparent and achievable standards for the generation of reliable datasets that can be widely used and shared.
CSBCOO SPMM: Compressed Sparse Block Coordinate format sparse matrix dense vector multiply benchmark (CSBCOO SPMM) v1.0
The purpose of this microbenchmark is to provide a means to explore different programming methodologies using a simple, but not trivial, mathematical kernel. The kernel is based on the matrix vector product in the MFDn nuclear configuration interaction code's LOBPCG eigensolver. This particular mathematical operation is a key element in the solution of systems of equations and block eigensolvers.
sycl-reduction benchmark v1.0.0
Some different SYCL implementations of the reduction operation. The parallel reduction kernel aggregates vector elements into a single result. It consists of four ways to implement the reduction operation using two different data management strategies, which we could not find elsewhere as a single package.
Quantum Application Specifications and Benchmarks
This software describes computational tasks for quantum computers that are derived from LANL basic science research applications such as the modeling of materials, chemicals and compounds at atomic scales. The computations focus on quantum simulation tasks, such as quantum dynamics, thermal state preparation and ground state estimation. The primary function of the software is to develop estimates of the requirements for large-scale fault-tolerant quantum computers to solve these scientific computations. The secondary focus of the software are codes for assessing the limitations of conducting quantum computations on classical computers.
InverseBench: Inverse design benchmark suite that contains inverse problems from science and engineering (InverseBench) v0.0.1
A software package that contains three inverse design blackbox problems to investigate the efficiency and accuracy of inverse design machine learning models. The software contains highly accurate forward machine learning models that can be used to assess the inverse predictions. The package also contains separate test data for each problem. The inverse design problems that are in the package are: airfoil inverse design, scalar boundary reconstruction and photonic surfaces inverse design.
smol_model_benchmark
Machine learning models for protein–small molecule binding prediction using graph (GAT), image-based (ResNet), and transformer-based SMILES approaches (ChemBERTa and MIST). The code includes training pipelines and experiments on the dataset from the BELKA Kaggle competition.
Benchmarking Simulated Precipitation in Earth System Models
Earth system models (ESMs) bridge observationally based and theoretical understanding of the Earth system. They are among the most often used tools to study a variety of questions related to variability and changes in Earth’s climate. For many applications, ESMs must realistically simulate observed large-scale precipitation patterns and seasonal cycles that have a multitude of societal and national security implications. Despite steady improvement in the simulation of precipitation, model errors in many aspects of precipitation characteristics limit the use of ESMs both in understanding Earth system variability and change and for decision-making.
Benchmarking Near-Surface Winds in the HRRR Analyses Using Multisource Observations over Complex Terrain in the Southeastern United States
Wind energy plays a crucial role in sustainable power generation, yet its full potential in the southeast United States (SEUS) remains underexplored. This study advances wind resource assessment in the SEUS using existing regional modeling and multisource observations. We find that terrain complexity, influenced by orography and forest canopy, significantly impacts the accuracy of modeled wind speed. While the High-Resolution Rapid Refresh (HRRR) model effectively simulates wind profiles over flat, nonforested terrain, larger errors are produced above forest canopies, and particularly for those stands in hilly and mountainous terrain where there are no observations aloft that could be used by HRRR’s data assimilation software. Seasonal variations, e.g., changes in the leaf area index, further complicate the relationship between terrain complexity and simulation errors. Here, our findings emphasize the critical need for comprehensive wind profile measurements, extending from below the canopy to height above the canopy and through the lower planetary boundary layer, to fully quantify model performance over and near forests, and for improving wind resource assessment, planning, and management. This study provides valuable insights not only for further model development but also for future field campaign deployment with the goal of further improving our understanding of wind resources in the region.
Benchmarking performance changes in the simulation of extratropical modes of variability across CMIP generations
In this work, we evaluate extratropical modes of variability in the three most recent phases of the Coupled Model Intercomparison Project (CMIP3, 5, and 6) to gauge improvement of climate models over time. A suite of high-level metrics is employed to objectively evaluate how well climate models simulate the qobserved Northern Annular Mode (NAM), North Atlantic Oscillation (NAO), Pacific North America pattern (PNA), Southern Annular Mode (SAM), Pacific Decadal Oscillation (PDO), North Pacific Oscillation (NPO), and North Pacific Gyre Oscillation (NPGO). We apply a Common Basis Function (CBF) approach that projects model anomalies onto observed empirical orthogonal functions (EOFs), together with the traditional EOF approach, to CMIP Historical and AMIP models. We find simulated spatial patterns of those modes have been significantly improved in the newer models, although the skill improvement is sensitive to the mode and season considered. We identify some potential contributions to the pattern improvement of certain modes (e.g., the Southern Hemisphere jet and High-top vertical coordinate), however the performance changes are likely attributed to gradual improvement of the base climate and multiple relevant processes. Less performance improvement is evident in the mode amplitude of these modes and systematic overestimation of the mode amplitude in spring remains in the newer climate models. We find that the post-dominant season amplitude errors in atmospheric modes are not limited to coupled runs but are often already evident in AMIP simulations. This suggests that rectifying the egregious post-dominant season amplitude errors found in many models can be addressed in an atmospheric-only framework making it more tractable to address in the model development process.
Benchmarking Simulated Precipitation Variability Amplitude across Time Scales
Here, objective performance metrics that measure precipitation variability across time scales from subdaily to interannual are presented and applied to Historical simulations of Coupled Model Intercomparison Project phase 5 and 6 (CMIP5 and CMIP6) models. Three satellite-based precipitation estimates (IMERG, TRMM, and CMORPH) are used as reference data. We apply two independent methods to estimate temporal variability of precipitation and compare the consistency in their results. The first method is derived from power spectra analysis of 3-hourly precipitation, measuring forced variability by solar insolation (diurnal and annual cycles) and internal variability at different time scales (subdaily, synoptic, subseasonal, seasonal, and interannual). The second method is based on time averaging and facilitates estimating the seasonality of subdaily variability. Supporting the robustness of our metric, we find a near equivalence between the results obtained from the two methods when examining simulated-to-observed ratios over large domains (global, tropics, extratropics, land, or ocean). Additionally, we demonstrate that our model evaluation is not very sensitive to the discrepancies between observations. Our results reveal that CMIP5 and CMIP6 models in general overestimate the forced variability while they underestimate the internal variability, especially in the tropical ocean and higher-frequency variability. The underestimation of subdaily variability is consistent across different seasons. The internal variability is overall improved in CMIP6, but remains underestimated, and there is little evidence of improvement in forced variability. Increased horizontal resolution results in some improvement of internal variability at subdaily and synoptic time scales, but not at longer time scales.
Towards a standard benchmark for phenotype-driven variant and gene prioritisation algorithms: PhEval - Phenotypic inference Evaluation framework
Not provided.
Quantification of Integral Benchmark Sensitivities to 1D and 2D Thermal Scattering Cross Sections
Explore the source record for details and available documents.
Nuclide Inventory Validation: Radiochemical Assay Data Quality and Modeling Challenges in Benchmark Models Development
Explore the source record for details and available documents.
Neutronic Benchmark on the Westinghouse Lead Fast Reactor
Explore the source record for details and available documents.