Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “BENCHMARKS”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 613 records · Page 34
All sourcesNASA NTRSDOE OSTI
DOE OSTI2024

CI/CD Efforts for Validation, Verification and Benchmarking OpenMP Implementations

Software developers must adapt to keep up with the changing capabilities of platforms so that they can utilize the power of High-Performance Computers (HPC), including exascale systems. OpenMP, a directive-based parallel programming model, allows developers to include directives to existing C, C++, or Fortran code to allow node level parallelism without compromising performance. This paper describes our CI/CD efforts to provide easy evaluation of the support of OpenMP across different compilers using existing testsuites and benchmark suites on HPC platforms. Our main contributions include (1) the set of a Continuous Integration (CI) and Continuous Development (CD) workflow that captures bugs and provides faster feedback to compiler developers, (2) an evaluation of OpenMP (offloading) implementations supported by AMD, HPE, GNU, LLVM, and Intel, and (3) evaluation of the quality of compilers across different heterogeneous HPC platforms. With the comprehensive testing through the CI/CD workflow, we aim to provide a comprehensive understanding of the current state of OpenMP (offloading) support in different compilers and heterogeneous platforms consisting of CPUs and GPUs from NVIDIA, AMD, and Intel.

Jarmusch, Aaron↗
DOE OSTI2020

Stable and radioactive carbon isotope partitioning in soils and saturated systems: a reactive transport modeling benchmark study

This benchmark provides the first rigorous test of a three-isotope system [ 12 C, 13 C, and 14 C] subject to the combined effects of radioactive decay and both stable equilibrium and kinetic fractionation. We present a series of problems building in complexity based on the cycling of carbon in both organic and inorganic forms. The key components implement (1) equilibrium fractionation between multiple coexisting carbon species as a function of pH, (2) radioactive decay of radiocarbon with associated mass-dependent speciation demonstrating appropriate correction of the Δ 14 C value in agreement with reporting convention, and (3) kinetic stable isotope fractionation due to the oxidation of organic carbon to inorganic forms as a function of time and space in an open, through-flowing system. Participating RTM codes are CrunchTope, ToughReact, Hytec, and The Geochemist’s Workbench. Across all problem levels, simulation results from all RTMs demonstrate good agreement.

58 GEOSCIENCES↗
DOE OSTI2025

The Surface-Topography Challenge: A Multi-Laboratory Benchmark Study to Advance the Characterization of Topography

Surface performance is critically influenced by topography in virtually all real-world applications. The current standard practice is to describe topography using one of a few industry-standard parameters. The most commonly reported number is Ra, the average absolute deviation of the height from the mean line (at some, not necessarily known or specified, lateral length scale). However, other parameters, particularly those that are scale-dependent, influence surface and interfacial properties; for example the local surface slope is critical for visual appearance, friction, and wear. The present Surface-Topography Challenge was launched to raise awareness for the need of a multi-scale description, but also to assess the reliability of different metrology techniques. In the resulting international collaborative effort, 153 scientists and engineers from 64 research groups and companies across 20 countries characterized statistically equivalent samples from two different surfaces: a “rough” and a “smooth” surface. The results of the 2088 measurements constitute the most comprehensive surface description ever compiled. We find wide disagreement across measurements and techniques when the lateral scale of the measurement is ignored. Consensus is established through scale-dependent parameters while removing data that violates an established resolution criterion and deviates from the majority measurements at each length scale. Our findings suggest best practices for characterizing and specifying topography. The public release of the accumulated data and presented analyses enables global reuse for further scientific investigation and benchmarking.

42 ENGINEERING↗
DOE OSTI2026

Validation Data for Benchmarking Wire Arc Additive Manufacturing Process Simulations

Residual stresses cause geometric distortion and affect mechanical performance of additively manufactured structures, yet they are notoriously difficult to assess and predict. Distortion (warpage) can drive parts outside dimensional tolerance limits, leading to part rejection or rework. For parts that meet tolerance, locked-in residual stress fields can affect structural integrity during operation, particularly subcritical cracking by fatigue, creep, or corrosion. This work develops benchmark data for a common additive manufacturing process (Wire Arc Additive Manufacturing) that can be applied for calibration and validation of physical process models that predict residual stress fields. The work includes design of two different samples of differing geometry, detailed manufacturing records for a set of physical samples, and an extensive set of residual stress measurement data developed using two diverse techniques (the contour method and neutron diffraction). An initial application of the work is also reported, where a modeling challenge was issued to secure residual stress model predictions from two independent laboratories that were blind to residual stress measurement data. These initial blind residual stress predictions show significant discrepancies relative to the measurement data, illustrating the potential value of the underlying validation data. An open repository for this work, including the sample designs, manufacturing process records, and the residual stress data, is also provided for future application in non-blind validation efforts.

36 MATERIALS SCIENCE↗
DOE OSTI2025

Benchmark microgravity experiments and computations for 3D dendritic-array stability in directional solidification

In this study, we present a comprehensive quantitative analysis of stability bands for dendritic arrays during directional solidification of a transparent succinonitrile-0.46 wt % camphor alloy, spanning a broad range of pulling velocities. Taking advantage of the microgravity environment aboard the International Space Station where most convection effects are suppressed, we obtain unique measurements that quantify the stable primary spacing range of spatially extended three-dimensional dendritic array structures under purely diffusive growth conditions. Through carefully designed velocity jump experiments and detailed examination of sub-grain boundary dynamics, we characterize key instabilities, including elimination and tertiary branching, shedding new light on the mechanisms governing dynamic dendritic spacing selection in extended 3D arrays. Phase field simulations are performed to characterize the stability limits of dendritic array structures for quantitative comparison with the flight experiments. Although the simulations capture general trends, significant deviations are noted at the upper stability boundary, indicating the influence of additional, unexplored factors. These findings contribute to a deeper understanding of dendritic growth dynamics and offer valuable benchmark data that could aid in refining predictive models and improving control of dendritic microstructures in metallurgical applications.

36 MATERIALS SCIENCE↗
DOE OSTI2022

Benchmarking Cu/BEA and HBEA catalysts for high-octane gasoline synthesis

We distinguish rates at which carbon deposition occurs during initiation, rates at which catalytic centers are lost during deactivation, and paraffin-to-olefin ratio during propagation as benchmarks that distinguish 5 wt% Cu/H-BEA and H-BEA (Si/Al = 13.5) catalysts during dimethyl ether (DME) homologation in the presence of hydrogen. Studies that systematically vary initial DME contact time (210, 94, and 45 mol H+, initial s (mol C ) -1 ), DME pressure (4 and 22 kPa), and H 2 pressure (1, 24, and 48 kPa) reveal that Cu enables lower carbon deposition rates (on a per proton basis) in the induction period, increases the effluent paraffin-to-olefin ratio during propagation, and decreases the instantaneous site-loss yields by a factor of ~ 1.5-2x (moles of active sites lost per mole of DME) during termination sequences thus affecting the degree of product saturation and catalyst stability during DME homologation. Furthermore, these results provide mechanistic insights revealing the critical role of Cu in facilitating DME homologation to high value, high-octane gasoline-range hydrocarbons with higher cumulative turnovers than proton form H-BEA.

09 BIOMASS FUELS↗
DOE OSTI2025

Benchmark of numerical modeling approaches on the systematic performance evaluation of wave energy converters

Different numerical modeling methods have been developed and applied to evaluate a variety of performance indicators of wave energy converters (WECs), including the power performance, structural loads, levelized cost of energy, etc. Based on the modeling fidelity, the commonly used numerical modeling approaches can be classified as linear modeling, weakly nonlinear modeling and fully nonlinear modeling approaches. Each method differs in accuracy and computational efficiency, making them suitable for different stages of WEC design. However, the selection of modeling approach could significantly impact evaluation outcomes. For instance, simplified linear models may underestimate structural loads or overestimate energy production in some operational conditions, potentially leading to less cost-effective designs. Given the widespread utilization of these models, it is essential to understand the uncertainties brought by them in performance evaluations. This work is dedicated to benchmarking different linear-potential-flow-based numerical models for evaluating the systematic performance of WECs. Three representative numerical modeling approaches are considered in this work, including linear frequency-domain modeling, statistically linearized spectral-domain modeling and Cummins equation-based nonlinear time-domain modeling. A generic point absorber WEC is considered as the research reference in this work, and different sea sites are taken into account. The numerical models are utilized to predict critical performance indicators, including power performance, the annual energy production, the capacity factor, the levelized cost of energy and the PTO fatigue loads. By comparing the results, this work identifies the uncertainties associated with different modeling approaches in evaluating WEC performance.

Fatigue↗
DOE OSTI2025

Benchmarking image processing techniques for porosity measurement in polymer additive manufacturing: Review and experimental analysis

An image processing workflow is proposed for porosity measurement in polymer additive manufacturing. Various techniques, including global and local thresholding, region growing, and K-means clustering, were applied to microscopic images of carbon fiber reinforced acrylonitrile butadiene styrene (CF-ABS) and benchmarked for their ability to accurately measure porosity. Global methods included Otsu, minimum error, iterative, and entropy-based thresholding, while local methods included Niblack, Bernsen, Sauvola, and Bradley-Roth algorithms. Artificial uneven illumination was introduced to test local adaptive thresholds. Results showed significant differences in porosity values across methods. Otsu, region growing, and K-means clustering excelled under uniform illumination, while Sauvola and Bradley-Roth performed better with uneven illumination. Comparison with X-ray computed tomography (XCT) revealed slightly lower porosity values (2.55 %) than optimized methods (2.73–2.79 %) due to XCT's lower resolution excluding smaller pores. While XCT offers finer pore detection, it limits sample volume and underestimates porosity due to spatial variation. Validation using artificial grayscale images with 5 % porosity confirmed that Otsu, Bradley-Roth, region growing, and Sauvola algorithms produced accurate results. Although tested on a single material system, these methods can be adapted to others with optimization. In conclusion, given XCT's high computational and time costs, this study highlights suitable image processing techniques as cost-effective alternatives for porosity analysis in polymer composites.

Additive manufacturing↗
DOE OSTI2020

Benchmark of the KGMf with a coupled Boltzmann equation solver

The Kinetic Global Model framework (KGMf) is an open-source general-purpose global model (spatially averaged) simulation code developed to explore the reaction kinetics and pathways in plasma discharge systems. It contains species continuity and electron energy balance equations, with a time-dependent evaluated electron energy distribution function (EEDF) for electron impact reactions. The EEDF is utilized to determine the rate coefficients for electron impact reactions, which can have profound impact on the temporal evolutions of plasma parameters. Previously, the EEDF was commonly assumed as Maxwellian or an analytical function of the effective electron temperature. In this work, the KGMf is coupled with a Boltzmann equation (BE) solver to self-consistently compute the EEDF. The EEDF evolution frequency is determined based on relative changes of the reduced electric field. The KGMf is benchmarked with the ZDPlasKin code based on high-pressure low-temperature argon plasma discharge cases. Additionally, the temporal evolutions of reduced electric field, electron temperature, EEDF, reaction rates, and species densities, are obtained and compared under different discharge conditions, showing good agreement between the KGMf and the ZDPlasKin simulations. The application of the KGMf for predicting breakdown times in high power microwave discharges is also presented, which shows qualitative agreement with particle-in-cell simulations. The KGMf can be further applied for more complicated plasma discharge systems, where the reaction kinetics are more intricate (e.g., plasma-assisted combustion systems).

71 CLASSICAL AND QUANTUM MECHANICS, GENERAL PHYSIC↗
DOE OSTI2022

Benchmarking collective effects of electron interactions in a wiggler with OPAL-FEL

OPAL-FEL is a recently developed tool for the modeling of particle accelerators containing wigglers or undulators. It extends the well established 3D electrostatic particle-tracking code OPAL, by merging it with the finite-difference time-domain electromagnetic solver MITHRA. We present results of two benchmark cases where OPAL-FEL simulations are compared to experimental results. Both experiments concern electron beamlines where the longitudinal phase space is modulated with a short magnetic wiggler. Good agreement was found in both the space charge and radiation dominated regimes.

43 PARTICLE ACCELERATORS↗
DOE OSTI2023

Verification and benchmarking relativistic electron beam transport through a background gas

It is necessary to establish confidence in high-consequence codes containing an extensive suite of physics algorithms in the regimes of interest. Verification problems allow code developers to assess numerical accuracy and increase confidence that specific sets of model physics were implemented correctly in the code. The two main verification techniques are code verification and solution verification. In this work, we present verification problems that can be used in other codes to increase confidence in simulations of relativistic beam transport. Specifically, we use the general plasma code EMPIRE to model and compare with the analytical solution to the evolution of the outer radial envelope of a relativistic charged particle beam. Additionally, we also outline a benchmark test of a relativistic beam propagating through a vacuum and pressurized gas cell, and present the results between EMPIRE and the hybrid code GAZEL. Further, we discuss the subtle errors that were caught with these problems and detail lessons learned.

71 CLASSICAL AND QUANTUM MECHANICS, GENERAL PHYSIC↗
DOE OSTI2024

Decomprolute is a benchmarking platform designed for multiomics-based tumor deconvolution

Tumor deconvolution is a reliable way to disentangle the diverse cell types that comprise solid tumors. To date, however, both the algorithms developed to deconvolve tumor samples, and the gold standard datasets used to assess the algorithms are geared toward the analysis of gene expression (e.g., RNA-seq) rather than protein levels in tumor cells. While gene expression is less expensive to measure, protein levels provide a more accurate view of immune markers. To facilitate the development as well as improve the reproducibility and reusability of multi-omic deconvolution algorithms, we introduce Decomprolute, a Common Workflow Language framework that leverages containerization to compare tumor deconvolution algorithms across multiomic data sets. Decomprolute incorporates the large-scale multiomic data sets produced by the Clinical Proteomic Tumor Analysis Consortium (CPTAC), which include matched mRNA expression and proteomic data from thousands of tumors across multiple cancer types to build a fully open-source, containerized proteogenomic tumor deconvolution benchmarking platform. The platform consists of modular architecture and it comes with well-defined input and output formats at each module. As a result, it is robust and extendable easily with additional algorithms or analyses. The platform is available for access and use at http://pnnl-compbio.github.io/decomprolute.

60 APPLIED LIFE SCIENCES↗
DOE OSTI2026

Physics vs structure: A systematic benchmark of learning strategies for multi-zone building thermal dynamics

Recent advances in physics-informed and data-driven machine learning promise improved thermal models for advanced building control, yet there is limited quantitative evidence on when added physics structure and architectural complexity are beneficial. Here, this work presents a systematic benchmark of five representative system identification methods for modeling multi-zone building thermal dynamics: linear state-space models, multi-layer perceptrons, neural state-space models, neural ordinary differential equations, and physically-consistent neural networks. The methods are evaluated across multiple data regimes and zone coupling strategies. Using a high-fidelity multi-zone commercial building emulator, we examine short-term and long-term prediction accuracy, computational efficiency, and ease of development. Our results reveal critical trade-offs between prediction performance, model complexity, and physical consistency. We demonstrate that decoupled, nonlinear black-box models consistently outperform coupled physics-constrained architectures in both predictive accuracy and out-of-distribution robustness in majority of the test cases for the building type considered in the study. Our findings quantify the cost of complexity in building thermal modeling and provide concrete, actionable, scenario-based guidelines for selecting model classes for control-oriented applications.

Building thermal modeling↗
DOE OSTI2023

Benchmarking plasma and electrolysis decomposition technologies for ammonia to power generation

Ammonia is a promising energy and hydrogen carrier due to its ease of liquification, high hydrogen content, and potential to be synthesized without carbon feedstocks. Despite its widespread use in agriculture, the utilization of ammonia to deliver hydrogen or for direct power generation is still under development. Sometimes referred to as “cracking”, the standard approach for recovering hydrogen from ammonia relies on harsh reaction conditions that limit its suitability for low-carbon transitions. Here, this work investigates the technology development status and potential cost of plasma and electrolysis-based ammonia decomposition, benchmarks them with thermal-chemical decomposition, and further compares them with direct ammonia-to-power solid oxide fuel cell systems. Results suggest that in order to reach cost parity with cracking technologies, plasma-based decomposition must achieve one order of magnitude improvement in energy efficiency (to 10 kWh/kg or lower), while electrolysis decomposition must achieve enhanced durability using cheap electrolytes.

30 DIRECT ENERGY CONVERSION↗
DOE OSTI2025

Benchmarking CO₂ storage simulations: Results from the 11 th Society of Petroleum Engineers Comparative Solution Project

The 11 th Society of Petroleum Engineers Comparative Solution Project (shortened SPE11 herein) benchmarked simulation tools for geological carbon dioxide (CO 2 ) storage. A total of 45 groups from leading research institutions and industry across the globe signed up to participate, with 18 ultimately contributing valid results that were included in the comparative study reported here. This paper summarizes the SPE11 results. A comprehensive introduction and qualitative discussion of the submitted data are provided, together with an overview of online resources for accessing the full depth of data. A global metric for analyzing the relative distance between submissions is proposed and used to conduct a quantitative analysis of the submissions. This analysis attempts to statistically resolve the key aspects influencing the variability between submissions. The study shows that the major qualitative variation between the submitted results is related to thermal effects, dissolution-driven convective mixing, and resolution of facies discontinuities. Moreover, a strong dependence on grid resolution is observed across all three versions of the SPE11. However, our quantitative analysis suggests that the observed variations are predominantly influenced by factors not documented in the technical responses provided by the participants. We therefore identify that unreported variations due to human choices within the process of setting up, conducting, and reporting on the simulations underlying each SPE11 submission are at least as impactful as the computational choices reported.

Nordbotten, Jan M. [Univ. of Bergen (Norway); Norw↗
DOE OSTI2021

Benchmarking blockchain-based gene-drug interaction data sharing methods: A case study from the iDASH 2019 secure genome analysis competition blockchain track

Blockchain distributed ledger technology is just starting to be adopted in genomics and healthcare applications. Despite its increased prevalence in biomedical research applications, skepticism regarding the practicality of blockchain technology for real-world problems is still strong and there are few implementations beyond proof-of-concept. We focus on benchmarking blockchain strategies applied to distributed methods for sharing records of gene-drug interactions. We expect this type of sharing will expedite personalized medicine. We generated gene-drug interaction test datasets using the Clinical Pharmacogenetics Implementation Consortium (CPIC) resource. We developed three blockchain-based methods to share patient records on gene-drug interactions: Query Index, Index Everything, and Dual-Scenario Indexing. We achieved a runtime of about 60 s for importing 4,000 gene-drug interaction records from four sites, and about 0.5 s for a data retrieval query. Our results demonstrated that it is feasible to leverage blockchain as a new platform to share data among institutions.

60 APPLIED LIFE SCIENCES↗
DOE OSTI2021

Interdisciplinary Clinical Target Volume Generation for Cardiac Radioablation: Multicenter Benchmarking for the RAdiosurgery for VENtricular TAchycardia (RAVENTA) Trial

Cardiac radioablation is a novel treatment option for therapy-refractory ventricular tachycardia (VT) ineligible for catheter ablation. Three-dimensional clinical target volume (CTV) definition is a key step, and this complex interdisciplinary procedure includes VT-substrate identification based on electroanatomical mapping (EAM) and its transfer to the planning computed tomography (PCT). Benchmarking of this process is necessary for multicenter clinical studies such as the RAVENTA trial.

62 RADIOLOGY AND NUCLEAR MEDICINE↗
DOE OSTI2021

Comprehensive review of photophysical parameters (ε, Φ f , τ s ) of tetraphenylporphyrin (H 2 TPP) and zinc tetraphenylporphyrin (ZnTPP) – Critical benchmark molecules in photochemistry and photosynthesis

Tetraphenylporphyrin (H 2 TPP) and zinc tetraphenylporphyrin (ZnTPP) are widely used benchmark molecules in diverse photochemical studies given facile synthetic access, rich visible-region spectra, and broad structural analogy to chlorophylls. Yet the literature values for each key photophysical parameter – the molar absorption coefficient (ε), fluorescence quantum yield (Φ f ), and also singlet excited-state lifetime (τ S ) – vary over an astonishing range. Here, a comprehensive literature review (~1940–September 2020) encompassing 871 publications is reported for these essential parameters. Each parameter is determined by measurement with distinct instrumentation and suffers idiosyncratic sources of error. The best values for H 2 TPP are ε = 460,000 cm -1 ·M -1 , Φ f = 0.090, and τ S = 12.8 ns in Ar- purged toluene (Φf = 0.070, τS = 9.9 ns in toluene in air); the best values for ZnTPP are ε = 560,000 cm -1 ·M -1 , Φf = 0.030, and τ S = 2.1 ns in Ar-purged toluene (Φ f = 0.029, τ S = 2.0 ns in toluene in air). The choice of values for such parameters has far-reaching consequences in photochemistry ranging from fluorescence (or Förster) resonance energy transfer (FRET) processes to assessments of molecular brightness.

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗