Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “application benchmarks”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 145 records · Page 8

Results of a Geant4 benchmarking study for bio‐medical applications, performed with the G4‐Med system

Geant4, a Monte Carlo Simulation Toolkit extensively used in bio-medical physics, is in continuous evolution to include newest research findings to improve its accuracy and to respond to the evolving needs of a very diverse user community. In 2014, the G4-Med benchmarking system was born from the effort of the Geant4 Medical Simulation Benchmarking Group, to benchmark and monitor the evolution of Geant4 for medical physics applications. The G4-Med system was first described in our Medical Physics Special Report published in 2021. Results of the tests were reported for Geant4 10.5. Purpose In this work, we describe the evolution of the G4-Med benchmarking system. Methods The G4-Med benchmarking suite currently includes 23 tests, which benchmark Geant4 from the calculation of basic physical quantities to the simulation of more clinically relevant set-ups. New tests concern the benchmarking of Geant4-DNA physics and chemistry components for regression testing purposes, dosimetry for brachytherapy with a 125 I source, dosimetry for external x-ray and electron FLASH radiotherapy, experimental microdosimetry for proton therapy, and in vivo PET for carbon and oxygen beams. Regression testing has been performed between Geant4 10.5 and 11.1. Finally, a simple Geant4 simulation has been developed and used to compare Geant4 EM physics constructors and physics lists in terms of execution times. Results In summary, our EM tests show that the parameters of the multiple scattering in the Geant4 EM constructor G4EmStandardPhysics_option3 in Geant4 11.1, while improving the modeling of the electron backscattering in high atomic number targets, are not adequate for dosimetry for clinical x-ray and electron beams. Therefore, these parameters have been reverted back to those of Geant4 10.5 in Geant4 11.2.1. The x-ray radiotherapy test shows significant differences in the modeling of the bremsstrahlung process, especially between G4EmPenelopePhysics and the other constructors under study (G4EmLivermorePhysics, G4EmStandardPhysics_option3, and G4EmStandardPhysics_option4). These differences will be studied in an in-depth investigation within our Group. Improvement in Geant4 11.1 has been observed for the modeling of the proton and carbon ion Bragg peak with energies of clinical interest, thanks to the adoption of ICRU90 to calculate the low energy proton stopping powers in water and of the Linhard–Sorensen ion model, available in Geant4 since version 11.0. Nuclear fragmentation tests of interest for carbon ion therapy show differences between Geant4 10.5 and 11.1 in terms of fragment yields. In particular, a higher production of boron fragments is observed with Geant4 11.1, leading to a better agreement with reference data for this fragment. Conclusions Based on the overall results of our tests, we recommend to use G4EmStandardPhysics_option4 as EM constructor and QGSP_BIC_HP with G4EmStandardPhysics_option4, for hadrontherapy applications. The Geant4-DNA physics lists report differences in modeling electron interactions in water, however, the tests have a pure regression testing purpose so no recommendation can be formulated.

62 RADIOLOGY AND NUCLEAR MEDICINE↗

Constrained variational optimization of counting-time allocation in sequential scattering measurements: Application to Bonse–Hart USANS

Sequential scattering measurements are often performed under a fixed experimental-time budget, even though the expected count rate varies strongly across the measured coordinate. When the dwell time at each measurement position can be controlled independently, this variation creates a general resource-allocation problem: how should the available time be distributed to minimize the uncertainty of the reconstructed profile? We formulate this problem as a constrained variational optimization for measurements governed by Poisson counting statistics. When each measurement is treated independently, minimizing the averaged squared relative uncertainty yields an inverse-square-root intensity allocation. The formulation is then generalized to include correlations between neighboring measurements and an instrumental resolution operator, leading to an allocation criterion that equalizes the marginal reduction in posterior uncertainty per unit measurement time. Bonse–Hart ultra-small-angle neutron scattering (USANS), in which reciprocal space is sampled sequentially through analyzer-angle stepping, provides an experimentally grounded application. Computational benchmarking shows that the optimized allocation outperforms uniform-time and constant-relative-error strategies, while application to an experimentally measured graphite USANS profile from the Spallation Neutron Source, using Poisson resampling under alternative schedules, demonstrates how counting time should be redistributed toward weak-intensity regions under an identical total duration. The resulting framework applies to sequential scattering and related scanning measurements whenever local dwell times are adjustable and directly determine the measurement uncertainties, and when the relevant correlation and instrumental-response models are available.

Tung, Chi-Huan [ORNL] (ORCID:0000000221972074)↗

Static analysis techniques for semiautomatic synthesis of message passing software skeletons

The design of high-performance computing architectures demands performance analysis of large-scale parallel applications to derive various parameters concerning hardware design and software development. The process of performance analysis and benchmarking an application can be done in several ways with varying degrees of fidelity. One of the most cost-effective ways is to do a coarse-grained study of large-scale parallel applications through the use of program skeletons. The concept of a “program skeleton” that we discuss in this article is an abstracted program that is derived from a larger program where source code that is determined to be irrelevant is removed for the purposes of the skeleton. In this work, we develop a semiautomatic approach for extracting program skeletons based on compiler program analysis. Finally, we demonstrate correctness of our skeleton extraction process by comparing details from communication traces, as well as show the performance speedup of using skeletons by running simulations in the SST/macro simulator.

97 MATHEMATICS AND COMPUTING↗

Algorithm and Architecture Independent Benchmarking with SEAK

Many applications of high performance embedded computing are limited by performance or power bottlenecks. We have designed the Suite for Embedded Applications & Kernels (SEAK), a new benchmark suite, (a) to capture these bottlenecks in a way that encourages creative solutions; and (b) to facilitate rigorous, objective, end-user evaluation for their solutions. To avoid biasing solutions toward existing algorithms, SEAK benchmarks use a mission-centric (abstracted from a particular algorithm) and goal-oriented (functional) specification. To encourage solutions that are any combination of software or hardware, we use an end-user black-box evaluation that can capture tradeoffs between performance, power, accuracy, size, and weight. The tradeoffs are especially informative for procurement decisions. We call our benchmarks future proof because each mission-centric interface and evaluation remains useful despite shifting algorithmic preferences. It is challenging to create both concise and precise goal-oriented specifications for mission-centric problems. This paper describes the SEAK benchmark suite and presents an evaluation of sample solutions that highlights power and performance tradeoffs.

algorithms and kernels↗

Nuclear data covariances are critical input to determine upper sub-critical limits and to design experiments to increase it [Slides]

This presentation discusses how Upper Subcritical Limits (USL) are key parameters to determine operational limits in nuclear criticality safety evaluations. It also discusses an example of plutonium casting operation using tantalum at LANL PF-4. The Whisper tool at Los Alamos relies on many inputs, including covariance data, leading the presentation to ask if an existing benchmark data be used in Whisper to adjust nuclear data and covariances to justify a higher USL. If not, Whisper can be used to help design an optimal new benchmark experiment. The presentation also seeks to determine what the possible impacts are on USL and operational limits for plutonium casting. In conclusion, nuclear data covariances are used for by Whisper for: GSSL adjustment of nuclear data and covariances, identification of most similar existing benchmark experiments to application, simulation of Upper Subcritical Limit of application, and input to optimization techniques for designing most appropriate new benchmark experiment(s) to meet requirements. This requires a complete set of nuclear data covariances, benchmarks and k-effective sensitivity profiles (for both benchmarks and applications). The presentation concludes by asking if end users should trust results that depend on current covariance data.

73 NUCLEAR PHYSICS AND RADIATION PHYSICS↗

Validation of continuous energy sensitivity coefficients calculations in the MORET 5 code - 14269

A continuous-energy sensitivity coefficients calculation capability has been developed in the MORET 5 Monte Carlo code recently. As discussed in this paper, it relies on the differential operator method, an adjoint weighting model (to avoid calculating the fission source derivatives) and a constrained sensitivity coefficient model for sensitivity profiles to distributions such as, for instance, the fission spectrum. For now, the coefficients are provided per medium, per isotope, per reaction on an energy binning defined by the user. The sensitivity coefficients are currently available for the following quantities: total, elastic scattering, total inelastic scattering, fission, total capture and radiative capture cross sections, average neutron produced by fission and fission spectrum. This paper deals with the validation of this new feature in the MORET code on several benchmarks and applications with various energy spectra and geometrical configurations. This validation is performed through the UACSA expert group and the Jezebel ICSBEP benchmarks, but also thanks to calculations performed in the frame of experimental program design in the STACY facility. Comparisons are made using the MCNP6.1 code and the SCALE6.1 package as references. Preliminary results show a pretty good agreement between the different software. (authors)

42 ENGINEERING↗

Performance and energy impact of OpenMP runtime configurations on power constrained systems

Energy consumption of an application has attained the same vitality as execution time in current HPC systems, primarily due to the massive energy consumption of these systems and consequently substantial energy bills. This fact forced software developers as well as hardware vendors to put an extensive focus on energy consumption. This focus becomes even more crucial in power constrained systems, where the amount of power dedicated to applications is limited. In such systems, finding the optimal balance between execution time and energy consumption has a significant impact on utilizing these systems efficiently. A programming model is a necessary layer where optimizations can be applied to find the optimal balance. OpenMP is a standard programming model for intra-node parallelism. In this paper, we give a detailed analysis of how to optimize the execution time and energy consumption of an OpenMP application using the OpenMP runtime environment at different power levels. We studied twenty-eight OpenMP parallel regions from five NAS Parallel Benchmark (NPB) applications. Based on the study, we show that for a given power level, a suitable selection of OpenMP runtime parameters can improve the execution time and energy consumption of a parallel region up to 67% and 72%, respectively. We also show that these fine-grained improvements resulted in up to 26% of execution time and 38% of energy consumption improvement for a given OpenMP application. We finally observed that optimizing the energy consumption behavior alone across all power levels regardless of execution time behavior can lead to an additional energy consumption improvement of up to 55%. However, this improvement comes at the expense of execution time.

97 MATHEMATICS AND COMPUTING↗

Correlation Calculations for the Russian Pu Metal Fast Experiments

Nuclear criticality experiments are often conducted in campaigns with multiple variations. These experiments reuse the same basic components, like the fuel, moderator, or positioning machines. The components have uncertainties in their geometry and composition that propagate to models of the experiments. Shared components create shared uncertainty between the $k_{eff}$ of benchmarks. The shared uncertainty is commonly quantified with a covariance, or correlation coefficient. These covariances can impact criticality safety and nuclear data validation applications. While benchmark evaluations tabulate an experiment’s uncertainty, they often lack a detailed calculation of correlations between experiments. Even some very commonly used benchmarks, like the Russian Pu Metal Fast (PMF) experiments, have missing correlations. This paper presents our approach to calculate the correlations for five of the Russian PMF experiments. The experiments share hemispherical Pu shells that induce a correlation between modeled $k_{eff}$ values. We estimated the correlations with simplified and detailed models of the experiments through linear-perturbation theory. The correlations between the experiments vary significantly between the detailed vs. simplified models. We also investigate how the correlations affect validation metrics of the experiments.

73 NUCLEAR PHYSICS AND RADIATION PHYSICS↗

Fisk-based criteria to support validation of detection methods for drinking water and air.

This report was prepared to support the validation of analytical methods for threat contaminants under the U.S. Environmental Protection Agency (EPA) National Homeland Security Research Center (NHSRC) program. It is designed to serve as a resource for certain applications of benchmark and fate information for homeland security threat contaminants. The report identifies risk-based criteria from existing health benchmarks for drinking water and air for potential use as validation targets. The focus is on benchmarks for chronic public exposures. The priority sources are standard EPA concentration limits for drinking water and air, along with oral and inhalation toxicity values. Many contaminants identified as homeland security threats to drinking water or air would convert to other chemicals within minutes to hours of being released. For this reason, a fate analysis has been performed to identify potential transformation products and removal half-lives in air and water so appropriate forms can be targeted for detection over time. The risk-based criteria presented in this report to frame method validation are expected to be lower than actual operational targets based on realistic exposures following a release. Note that many target criteria provided in this report are taken from available benchmarks without assessing the underlying toxicological details. That is, although the relevance of the chemical form and analogues are evaluated, the toxicological interpretations and extrapolations conducted by the authoring organizations are not. It is also important to emphasize that such targets in the current analysis are not health-based advisory levels to guide homeland security responses. This integrated evaluation of chronic public benchmarks and contaminant fate has identified more than 200 risk-based criteria as method validation targets across numerous contaminants and fate products in drinking water and air combined. The gap in directly applicable values is considerable across the full set of threat contaminants, so preliminary indicators were developed from other well-documented benchmarks to serve as a starting point for validation efforts. By this approach, at least preliminary context is available for water or air, and sometimes both, for all chemicals on the NHSRC list that was provided for this evaluation. This means that a number of concentrations presented in this report represent indirect measures derived from related benchmarks or surrogate chemicals, as described within the many results tables provided in this report.

54 ENVIRONMENTAL SCIENCES↗

Parallel application-level behavioral attributes for performance and energy management of high-performance computing systems

Run time variability of parallel applications continues to present significant challenges to their performance and energy efficiency in high-performance computing (HPC) systems. When run times are extended and unpredictable, application developers perceive this as a degradation of system (or subsystem) performance. Extended run times directly contribute to proportionally higher energy consumption, potentially negating efforts by applications, or the HPC system, to optimize energy consumption using low-level control techniques, such as dynamic voltage and frequency scaling (DVFS). Therefore, successful systemic management of application run time performance can result in less wasted energy, or even energy savings. We have been studying run time variability in terms of communication time, from the perspective of the application, focusing on the interconnection network. More recently, our focus has shifted to developing a more complete understanding of the effects of HPC subsystem interactions on parallel applications. In this context, the set of executing applications on the HPC system is treated as a subsystem, along with more traditional subsystems like the communication subsystem, storage subsystem, etc. To gain insight into the run time variability problem, our earlier work developed a framework to emulate parallel applications (PACE) that stresses the communication subsystem. Evaluation of run time sensitivity to network performance of real applications is performed with a tool called PARSE, which uses PACE. In this paper, we propose a model defining application-level behavioral attributes, that collectively describes how applications behave in terms of their run time performance, as functions of their process distribution on the system (spacial locality), and subsystem interactions (communication subsystem degradation). These subsystem interactions are produced when multiple applications execute concurrently on the same HPC system. We also revisit our evaluation framework and tools to demonstrate the flexibility of our application characterization techniques, and the ease with which attributes can be quantified. The validity of the model is demonstrated using our tools with several parallel benchmarks and application fragments. Results suggest that it is possible to articulate application-level behavioral attributes as a tuple of numeric values that describe course-grained performance behavior.

Computer Science↗

Using Whisper-1.1 to Guide Improvements to Nuclear Data Evaluations [Slides]

This presentation covers the use of Whisper-1.1 to guide improvements to nuclear data evaluations. This presentation provides a brief overview on what Whisper-1.1 is. Additionally covered is the benchmark selection: using benchmark and application correlations (ck) computed using nuclear data sensitivity profiles and covariance data. Next the presentation touches on compute bias and bias uncertainty based on selected benchmarks using Extreme Value Theory Finally the presentation shows the additional margin for nuclear data uncertainty estimated by GLLS.

97 MATHEMATICS AND COMPUTING↗

A Task Based Approach for Co-Scheduling Ensemble Workloads on Heterogeneous Nodes

Scientific workflows consist of multiple, connected applications, with data and results flowing from one to another in a pipeline. Traditionally, such workflows are executed in sequential order, storing intermediate data in storage disks. Co-scheduling application workflows concurrently on the same compute nodes would greatly reduce the cost of moving data to/from storage and allow real-time analysis of intermediate results. Nevertheless, most parallel programming runtimes do not allow seamless integration of various applications in a scientific workflow, in part due to the complexity of managing data and resources. The situation is even more complicated for heterogeneous systems. In this work we extend the Minos Computing Library (MCL) runtime to accelerate pipe-lined and parallel workloads where multiple applications are running in the same system. MCL’s asynchronous task library and runtime dynamically manages resources to allow co-scheduling of multiple processes sharing heterogeneous resources. In addition, we design a custom ex- tension of the Open Compute Language (OpenCL) to enable multiple processes to share device memory. We enable MCL to coordinate these shared buffers to allow for easy, fast data sharing between applications. Using malleable micro-benchmarks and two application workflows that combine scientific simulation and AI-based analysis, we show that our method outperforms traditional approaches.

Index Terms—Parallel systems, Scheduling and Task ↗

AI Benchmark Democratization and Carpentry

Benchmarks are a cornerstone of modern machine learning, enabling reproducibility, comparison, and scientific progress. However, AI benchmarks are increasingly complex, requiring dynamic, AI-focused workflows. Rapid evolution in model architectures, scale, datasets, and deployment contexts makes evaluation a moving target. Large language models often memorize static benchmarks, causing a gap between benchmark results and real-world performance. Beyond traditional static benchmarks, continuous adaptive benchmarking frameworks are needed to align scientific assessment with deployment risks. This calls for skills and education in AI Benchmark Carpentry. From our experience with MLCommons, educational initiatives, and programs like the DOE's Trillion Parameter Consortium, key barriers include high resource demands, limited access to specialized hardware, lack of benchmark design expertise, and uncertainty in relating results to application domains. Current benchmarks often emphasize peak performance on top-tier hardware, offering limited guidance for diverse, real-world scenarios. Benchmarking must become dynamic, incorporating evolving models, updated data, and heterogeneous platforms while maintaining transparency, reproducibility, and interpretability. Democratization requires both technical innovation and systematic education across levels, building sustained expertise in benchmark design and use. Benchmarks should support application-relevant comparisons, enabling informed, context-sensitive decisions. Dynamic, inclusive benchmarking will ensure evaluation keeps pace with AI evolution and supports responsible, reproducible, and accessible AI deployment. Community efforts can provide a foundation for AI Benchmark Carpentry.

von Laszewski, Gregor [Virginia U.]↗

Small Hydropower Interconnections: Analysis of Interconnection Processes

Small hydropower projects have faced the challenge of navigating the process to interconnect their generation source to electricity distribution and transmission grids. Small hydropower developers have found interconnection procedures to be opaque and ultimately result in unexpected cost surprises and long timelines. Noting these challenges, the U.S. Department of Energy Water Power Technologies Office enlisted Pacific Northwest National Laboratory (PNNL) and Oak Ridge National Laboratory (ORNL) to investigate the small hydropower interconnection landscape across the United States. After reviewing the status of small hydropower (“Small Hydropower Interconnections: Small Hydropower in the United States”) and the interconnection procedures across the United States (“Small Hydropower Interconnections: State Interconnection Processes”) in the first two white papers of this series, this paper uses recent data from small hydropower interconnection applications to benchmark the efficacy of the process. Using data from interconnection queues hosted by utilities, balancing authorities, independent system operators (ISOs), and regional transmission organizations (RTOs), this paper provides context for the costs, timelines, and types of upgrades required for small hydropower projects. Interconnection applications and study reports for small hydropower projects were analyzed to collect key pieces of information about the interconnection process, timeline, costs, and type of upgrades required for interconnection. Information sourced from the reports was entered into an Interconnection Benchmarking database (IBdb), which may be found in Appendix A.1. Information from this database was used to evaluate the performance and challenges associated with interconnecting small hydropower projects. This white paper presents a description of the sources contained in the interconnection database (Section 2.0), an analysis of the interconnection timeline (Section 3.0), an evaluation the cost of interconnection upgrades (Section 4.0), and a description of the types of infrastructure upgrades (Section 5.0). The final paper in this series (“Small Hydropower Interconnections: Best Practices”) will use the analysis described here to outline best practices for interconnection processes that will help overcome barriers to future small hydropower development.

13 HYDRO ENERGY↗

Systematic Benchmarking of Climate Models: Methodologies, Applications, and New Directions

As climate models become increasingly complex, there is a growing need to comprehensively and systematically assess model performance with respect to observations. Given the increasing number and diversity of climate model simulations in use, the community has moved beyond simple model intercomparison and toward developing methods capable of benchmarking a large number of simulations against a suite of climate metrics. Here, we present a detailed review of evaluation and benchmarking methods and approaches developed in the last decade, focusing primarily on scientific implications for Coupled Model Intercomparison Project (CMIP) simulations and CMIP6 results that contributed to the Intergovernmental Panel on Climate Change (IPCC) Sixth Assessment Report (AR6). Based on this review, we explain the resulting contemporary philosophy of model benchmarking, and provide clear distinctions and definitions of the terms model verification, process validation, evaluation, and benchmarking. While significant progress has been made in model development based on systematic evaluation and benchmarking efforts, some climate system biases still remain. The development of open‐source community software packages has played a fundamental role in identifying areas of significant model improvement and bias reduction. We review the key features of several software packages that have been commonly used over the past decade to evaluate and benchmark global and regional climate models. Additionally, we discuss best practices for the selection of evaluation and benchmarking metrics and for interpreting the obtained results, the importance of selecting suitable sources of reference data and accurate uncertainty quantification.

Environmental sciences↗

Are we witnessing the spectre of an HPC meltdown?: Are We Witnessing the Spectre of an HPC Meltdown?

In this study, we measure and analyze the performance observed when running applications and benchmarks before and after the Meltdown and Spectre fixes have been applied to the Cray supercomputers and supporting systems at the Oak Ridge Leadership Computing Facility (OLCF). Of particular interest is the effect of these fixes on applications selected from the OLCF portfolio when running at scale. This comprehensive study presents results from experiments run on Titan, Eos, Cumulus, and Percival supercomputers at the OLCF. Lastly, the results from this study are useful for HPC users running on Cray supercomputers and serve to better understand the impact that these two vulnerabilities have on diverse HPC workloads at scale.

97 MATHEMATICS AND COMPUTING↗