Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “Benchmarking Software”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 91 records · Page 5
All sourcesNASA NTRSDOE OSTI
DOE OSTI2025

Joint Experimental and Computational Characterization of Sum-Frequency Generation between a Continuous Wave Laser and an Ultrafast Frequency Comb Laser for Tunable Laser Development

Ultrafast optical frequency combs allow for both high spectral and temporal resolution in molecular spectroscopy and have become a powerful tool in many areas of chemistry and physics. Ultrafast lasers and frequency combs generated from ultrafast mode-locked lasers often need to be converted to other wavelengths. Commonly used wavelength conversions are optical parametric oscillators, which require an external optical cavity, and supercontinuum generation combined with optical parametric amplifiers. Whether commercial or home-built, these systems are complex and costly. Here, we investigate an alternative, simple, and easy-to-implement approach to tunable frequency comb ultrafast lasers enabled by new continuous-wave laser technology. Sum-frequency generation between an Nd:YAG continuous-wave laser and a Yb:fiber femtosecond frequency comb in a beta-barium borate (BBO) crystal is explored. The resulting sum-frequency beam is a pulsed frequency comb with the same repetition rate as the Yb:fiber source. SNLO simulation software is used to simulate the results and provide benchmarks for designing future systems to achieve wavelength conversion and tunability in otherwise difficult-to-reach spectral regions.

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗
DOE OSTI2025

pnnl/soda-benchmarks

The SODA-Benchmarks repository (currently hosted at: https://gitlab.pnnl.gov/sodalite/soda-benchmarks) provides a comprehensive benchmark suite for evaluating tools within the SODA (Software-Defined Accelerators) toolchain, which focuses on hardware/software co-design and accelerator generation for FPGAs and ASICs

Agostini, Nicolas [Pacific Northwest National Labo↗
DOE OSTI2025

InverseBench: Inverse design benchmark suite that contains inverse problems from science and engineering (InverseBench) v0.0.1

A software package that contains three inverse design blackbox problems to investigate the efficiency and accuracy of inverse design machine learning models. The software contains highly accurate forward machine learning models that can be used to assess the inverse predictions. The package also contains separate test data for each problem. The inverse design problems that are in the package are: airfoil inverse design, scalar boundary reconstruction and photonic surfaces inverse design.

Grbcic, Luka [Lawrence Berkeley National Laborator↗
DOE OSTI2020

Using Neural Architecture Search for Improving Software Flaw Detection in Multimodal Deep Learning Models

Software flaw detection using multimodal deep learning models has been demonstrated as a very competitive approach on benchmark problems. In this work, we demonstrate that even better performance can be achieved using neural architecture search (NAS) combined with multimodal learning models. We adapt a NAS framework aimed at investigating image classification to the problem of software flaw detection and demonstrate improved results on the Juliet Test Suite, a popular benchmarking data set for measuring performance of machine learning models in this problem domain.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2023

HamLib: A Library of Hamiltonians for Benchmarking Quantum Algorithms and Hardware

For a considerable time, large datasets containing problem instances have proven valuable for analyzing computer hardware, software, and algorithms. One notable example of the value of large datasets is ImageNet [1], a vast repository of images that has been instrumental in testing numerous deep learning packages. Similarly, in the domain of computational chemistry and materials science, the availability of extensive datasets such as the Protein Data Bank [2], the Materials Project [3], and QM9 [4] has greatly facilitated the evaluation of new algorithms and software approaches, while also promoting standardization within the field. These well-defined datasets and problem instances, in turn, serve as the foundation for creating benchmarking suites like MLPerf [5] and LINPACK [6], [7]. These suites enable fair and rigorous comparisons of different methodologies and solutions, fostering continuous advancements in various areas of computer science and beyond.

Sawaya, Nicolas PD↗
DOE OSTI2020

Automated Integration of Continental-Scale Observations in Near-Real Time for Simulation and Analysis of Biosphere–Atmosphere Interactions

The National Ecological Observatory Network (NEON) is a continental-scale observatory with sites across the US collecting standardized ecological observations that will operate for multiple decades. To maximize the utility of NEON data, we envision edge computing systems that gather, calibrate, aggregate, and ingest measurements in an integrated fashion. Edge systems will employ machine learning methods to cross-calibrate, gap-fill and provision data in near-real time to the NEON Data Portal and to High Performance Computing (HPC) systems, running ensembles of Earth system models (ESMs) that assimilate the data. For the first time gridded EC data products and response functions promise to offset pervasive observational biases through evaluating, benchmarking, optimizing parameters, and training new machine learning parameterizations within ESMs all at the same model-grid scale. Leveraging open-source software for EC data analysis, we are already building software infrastructure for integration of near-real time data streams into the International Land Model Benchmarking (ILAMB) package for use by the wider research community. We will present a perspective on the design and integration of end-to-end infrastructure for data acquisition, edge computing, HPC simulation, analysis, and validation, where Artificial Intelligence (AI) approaches are used throughout the distributed workflow to improve accuracy and computational performance.

Durden, David J.↗
DOE OSTI2023

Editorial: Neuroscience, computing, performance, and benchmarks: Why it matters to neuroscience how fast we can compute

At the turn of the millennium the computational neuroscience community realized that neuroscience was in a software crisis: software development was no longer progressing as expected and reproducibility declined. The International Neuroinformatics Coordinating Facility (INCF) was inaugurated in 2007 as an initiative to improve this situation. The INCF has since pursued its mission to help the development of standards and best practices. In a community paper published this very same year, Brette et al. tried to assess the state of the field and to establish a scientific approach to simulation technology, addressing foundational topics, such as which simulation schemes are best suited for the types of models we see in neuroscience. In 2015, a Frontiers Research Topic “Python in neuroscience” by Muller et al. triggered and documented a revolution in the neuroscience community, namely in the usage of the scripting language Python as a common language for interfacing with simulation codes and connecting between applications. The review by Einevoll et al. documented that simulation tools have since further matured and become reliable research instruments used by many scientific groups for their respective questions. Open source and community standard simulators today allow research groups to focus on their scientific questions and leave the details of the computational work to the community of simulator developers. A parallel development has occurred, which has been barely visible in neuroscientific circles beyond the community of simulator developers: Supercomputers used for large and complex scientific calculations have increased their performance from ~10 TeraFLOPS (10 13 floating point operations per second) in the early 2000s to above 1 ExaFLOPS (10 18 floating point operations per second) in the year 2022. This represents a 100,000-fold increase in our computational capabilities, or almost 17 doublings of computational capability in 22 years. Moore's law (the observation that it is economically viable to double the number of transistors in an integrated circuit every other 18–24 months) explains a part of this; our ability and willingness to build and operate physically larger computers, explains another part. It should be clear, however, that such a technological advancement requires software adaptations and under the hood, simulators had to reinvent themselves and change substantially to embrace this technological opportunity. It actually is quite remarkable that—apart from the change in semantics for the parallelization—this has mostly happened without the users knowing. The current Research Topic was motivated by the wish to assemble an update on the state of neuroscientific software (mostly simulators) in 2022, to assess whether we can see more clearly which scientific questions can (or cannot) be asked due to our increased capability of simulation, and also to anticipate whether and for how long we can expect this increase of computational capabilities to continue.

biophysically detailed models↗
DOE OSTI2020

Using Neural Architecture Search for Improving Software Flaw Detection in Multimodal Deep Learning Models

Software flaw detection using multimodal deep learning models has been demonstrated as a very competitive approach on benchmark problems. In this work, we demonstrate that even better performance can be achieved using neural architecture search (NAS) combined with multimodal learning models. We adapt a NAS framework aimed at investigating image classification to the problem of software flaw detection and demonstrate improved results on the Juliet Test Suite, a popular benchmarking data set for measuring performance of machine learning models in this problem domain.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2023

SwitchX : Gmin-Gmax Switching for Energy-efficient and Robust Implementation of Binarized Neural Networks on ReRAM Xbars

Memristive crossbars can efficiently implement Binarized Neural Networks (BNNs) wherein the weights are stored in high-resistance states (HRS) and low-resistance states (LRS) of the synapses. We propose SwitchX mapping of BNN weights onto ReRAM crossbars such that the impact of crossbar non-idealities, that lead to degradation in computational accuracy, are minimized. Essentially, SwitchX maps the binary weights in such a manner that a crossbar instance comprises of more HRS than LRS synapses. We find BNNs mapped onto crossbars with SwitchX to exhibit better robustness against adversarial attacks than the standard crossbar mapped BNNs, the baseline. Finally, we combine SwitchX with state-aware training (that further increases the feasibility of HRS states during weight mapping) to boost the robustness of a BNN on hardware. We find that this approach yields stronger defense against adversarial attacks than adversarial training, a state-of the-art software defense. We perform experiments on a VGG16 BNN with benchmark datasets (CIFAR-10, CIFAR-100 and TinyImagenet) and use Fast Gradient Sign Method (ϵ = 0.05 to 0.3) and Projected Gradient Descent (ϵ = $\frac{2}{255}$ to $\frac{32}{255}$, α = $\frac{2}{255}$) adversarial attacks. We show that SwitchX combined with state-aware training can yield upto ~35% improvements in clean accuracy and ~6–16% in adversarial accuracies against conventional BNNs. Furthermore, an important by-product of SwitchX mapping is increased crossbar power savings, owing to an increased proportion of HRS synapses, which is furthered with state-aware training. We obtain upto ~21–22% savings in crossbar power consumption for state-aware trained BNN mapped via SwitchX on 16 × 16 and 32 × 32 crossbars using the CIFAR-10 and CIFAR-100 datasets.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2024

Flow reversal benchmark of a one-sided heated narrow rectangular channel with CATHARE and RELAP5

Flow reversal in narrow coolant channels can be a crucial phenomenon for the safety of research reactors with a downward nominal flow direction. During a loss of forced flow accident, the downward flow stagnates briefly before transitioning into an upward natural circulation flow. The fuel may be damaged if dryout occurs and threshold fuel and/or cladding temperatures are exceeded. A comprehensive study is provided for flow reversal in narrow rectangular channels by examining experimental data and conducting software model analyses. The literature on flow reversal was reviewed, and selected experimental datasets were used to benchmark against CATHARE and RELAP5 models and also compare the code calculations with each other. The experimental data comes from flow reversal tests conducted with a narrow rectangular channel with one-sided heating. The results were compared with experimental data for successful flow reversal tests and predicted dryout power for dryout conditions. Also, the study examined the effects of the pump coastdown period, inlet liquid temperature, system pressure, and localized pressure drops. The experimental results showed that shorter coastdown periods, reduced pressure drops, and lower coolant inlet temperatures increased the dryout power. However, the system pressure did not noticeably affect the results. The simulation results showed that both CATHARE and RELAP5 agreed with experimental data, capturing the trends of the experimental results. Slight differences between each code calculation, as well as the predicted and measured dryout powers, were attributed to experimental uncertainties and the modeling of physical phenomena such as wall nucleation, interfacial heat transfer, drag coefficients, and critical heat flux. Overall, this study provides an understanding of flow reversal and the prediction capabilities of thermal-hydraulics software models. In conclusion, a future study of the flow reversal benchmark of a narrow rectangular channel with two-sided heating may provide additional valuable insights.

CATHARE↗
DOE OSTI2022

Hybrid Solid Oxide Fuel Cell/Gas Turbine Model Development for Electric Aviation

A thermodynamic model was developed and validated to analyze a high-performance solid oxide fuel cell and gas turbine (SOFC-GT) hybrid power system for electric aviation. This study used a process simulation software package (ProMax) to study the role of SOFC design and operation on the feasibility and performance of the hybrid system. Standard modules, including compressor, turbine, heat exchanger, reforming reactor, and combustor were used from the ProMax tool suite while a custom module was created to simulate the SOFC stack. The model used an SOFC test data set as an input. Additional SOFC stack performance effects, such as pressure, temperature, and utilization of air and fuel, were added from open source data. System performance predictors were SOFC specific power, fuel-to-electricity conversion efficiency, and hybrid system efficiency. Using these input data and predictors, a static thermodynamic performance model was created that can be modified for different system configurations and operating conditions. Prior to creating the final aircraft performance model, initial demonstration models were developed to validate output results. We used the NASA SOFC model as a benchmark, which was created with their Numerical Propulsion System Simulator (NPSS) software framework. Our output results matched within 1% of both the NASA model and open source SOFC performance data. With confidence gained in the accuracy of this model, a 1-MW SOFC-GT hybrid power system was constructed for an aircraft propulsion concept. Overall hybrid system efficiencies of > 75% FTE were observed during standard 36,000 feet cruise flight conditions.

30 DIRECT ENERGY CONVERSION↗
DOE OSTI2022

GriddingMachine, a database and software for Earth system modeling at global and regional scales

Land and Earth system modeling is moving towards more explicit biophysical representations, requiring increasing variety of datasets for initialization and benchmarking. However, researchers often have difficulties in identifying and integrating non-standardized datasets from various sources. We aim towards a standardized database and one-stop distribution method of global datasets. Here, we present the GriddingMachine as (1) a database of global-scale datasets commonly used to parameterize or benchmark the models, from plant traits to vegetation indices and geophysical information and (2) a cross-platform open source software to download and request a subset of datasets with only a few lines of code. The GriddingMachine datasets can be accessed either manually through traditional HTTP, or automatically using modern programming languages including Julia, Matlab, Octave, Python, and R. The GriddingMachine collections can be used for any land and Earth modeling framework and ecological research at the regional and global scales, and the number of datasets will continue to grow to meet the increasing needs of research communities.

58 GEOSCIENCES↗
DOE OSTI2022

[Presentation Slides] Code-to-Code Benchmark Study for Thermal Stress Modeling and Preliminary Analysis of the High-temperature Single Heat-Pipe Experiment

In the heat-pipe-cooled microreactor, heat pipes remove heat from the reactor core as a passive heat-transfer device, so the fluid circulation is not required for cooling, which can substantially simplify the overall reactor design. However, given the extremely high temperatures in the core region and potentially large temperature gradients across the structure materials, thermal stresses need to be well-analyzed to ensure structural integrity during normal operations and accident scenarios. This presentation slides discuss finite element method-based thermal-stress analysis for the high-temperature single heat-pipe test article in the Single Primary Heat Extraction and Removal Emulator (SPHERE) facility at Idaho National Laboratory (INL), using two commercial software packages, Abaqus and Star-CCM+. A code-to-code benchmark study was performed to crosscheck the model setup and capability of each code and to gain preliminary insights into the potential thermal stress concerns from the current experimental setup. It is observed the significant thermal stresses happen at the inner surface of the heat pipe hole surrounded by electric heaters where the largest temperature gradients appear. The temperature fields have good agreements between Abaqus and Star-CCM+, while the induced thermal stresses show modest deviations probably due to differences of meshing engines used in these two codes. It is found that the local maximum thermal stresses may reach close to the ultimate tensile strength and yield strength of structural material depending on the heater power. Ultimately, the coupled thermal-structural analysis will help guide the current experimental plan and ensure the facility safety for future experimental study.

22 GENERAL STUDIES OF NUCLEAR REACTORS↗
DOE OSTI2022

CODE-TO-CODE BENCHMARK STUDY FOR THERMAL STRESS MODELING AND PRELIMINARY ANALYSIS OF THE HIGH-TEMPERATURE SINGLE HEAT PIPE EXPERIMENT

Microreactors are very small nuclear reactors with typical thermal-energy output of up to 20 MWth. The microreactor concept is gaining more and more attention for safe, robust, and reliable supply of electricity for remote locations and to meet industrial process heat needs. Heat-pipe-cooled microreactor is one of the microreactors being investigated at Idaho National Laboratory. In the heat-pipe-cooled microreactor, heat pipes remove heat from the reactor core as a passive heat-transfer device, so the fluid circulation is not required for cooling, which can substantially simplify the overall reactor design. However, given the extremely high temperatures in the core region and potentially large temperature gradients across the structure materials, thermal stresses need to be well-analyzed to ensure structural integrity during normal operations and accident scenarios for the heat-pipe-cooled microreactor designs. This paper discusses finite element method-based thermal-stress analysis for the high-temperature single heat-pipe test article in the Single Primary Heat Extraction and Removal Emulator (SPHERE) facility at Idaho National Laboratory, using two commercial software packages, Abaqus and STAR-CCM+. A code-to-code benchmark study was performed to crosscheck the model setup and capability of each code and to gain insights into the potential thermal stress concerns from the current experimental setup. It is observed the significant thermal stresses happen at the locations where the largest temperature gradients appeared between heat pipe and electric heater. The temperature fields have good agreements between Abaqus and STAR-CCM+, while the induced thermal stresses show modest deviations probably due to differences of meshing engines used in these two codes.

22 GENERAL STUDIES OF NUCLEAR REACTORS↗
DOE OSTI2024

hypredrive: high-level interface for solving linear systems with hypre

This software introduces a high-level interface designed to simplify solving linear systems using hypre, a renowned library for such computational challenges. It is crafted to be accessible and user-friendly, making the powerful capabilities of hypre available to a broader audience without requiring in-depth technical knowledge. The interface is characterized by its use of YAML for input, a format celebrated for its structured yet straightforward readability. This choice ensures that users can easily configure the software to meet their specific needs. Additionally, the software boasts an intuitive API that encapsulates hypre's functionalities, making it easier for users to interact with the process of solving linear systems. It is particularly beneficial for prototyping, offering a quick and efficient means to test various solver and preconditioner configurations. Furthermore, the software allows for the creation of an offline testing framework in which predefined linear systems are read from files and benchmarked with user-defined solution strategies. This makes it an invaluable tool for developers and researchers exploring and validating their computational models. Overall, the software serves as a bridge, bringing the advanced computational capabilities of hypre closer to users who may need more specialized technical expertise, thereby facilitating innovation and exploration in the field of numerical linear algebra.

Paludetto Magri, Victor↗
DOE OSTI2020

Gaussian Process Optimization of Sensitivity-Based Similarity Metrics between New Nuclear Applications and New/Existing Benchmarks [Slides]

This presentation discusses Nuclear Criticality Safety (NCS) and how designing safe, new nuclear criticality experiments requires expert judgement, which could take years of experience. Sensitivity/uncertainty (S/U) analysis can be utilized by less experienced individuals to conservatively estimate uncertainties in important parameters, such as k eff , in newly proposed nuclear experiments. The presentation poses the question of how this analysis can be performed and states that the answer lies in matching new nuclear experiments with existing benchmark experiments using similarity metrics. By increasing the criticality safety of the application in this work, higher mass limits could be used in PF-4 operations. Additionally, the presentation discusses MCNP6.2®, Whisper-1.1, the software that can be used in this analysis. Also discussed is the fact that International Criticality Safety Benchmark Evaluation Project (ICSBEP) benchmarks rarely match new nuclear applications and that there are significant differences in given set of materials and/or geometry. If there are no benchmarks that match the application, the presentation discusses the possibility of creating new benchmarks. In conclusion, this work presents a Gaussian process (GP) optimization scheme that was used to generate new benchmarks with the highest sensitivity-based similarity metrics to user-defined nuclear applications. The Gaussian process optimization successfully designed 3 new experimental benchmarks that were highly correlated to the application of interest and had k eff values near critical. Optimization over c k,i-r has shown that investigating specific isotope reactions for different applications is crucial to designing benchmark experiments. Partial contribution from Pu dominates c k similarity metric. Future work includes testing new stand-alone similarity metrics or new combinations of similarity metrics as the design criterion of this optimization – design criterion is application dependent.

73 NUCLEAR PHYSICS AND RADIATION PHYSICS↗
DOE OSTI2023

AMG2023

The AMG2023 benchmark solves two diffusion problems with a linear solver preconditioned with algebraic multigrid. The code only contains a driver, a Makefile, and a documentation file. It requires an installation of the open source software library hypre that needs to be downloaded elsewhere and is not included here. Its purpose is to benchmark linear solver performance on high performance computers.

Li, Ruipeng↗
DOE OSTI2024

The ATLAS experiment software on ARM

With an increased dataset obtained during the Run 3 of the LHC at CERN and the even larger expected increase of the dataset by more than one order of magnitude for the HL-LHC, the ATLAS experiment is reaching the limits of the current data processing model in terms of traditional CPU resources based on x86_64 architectures and an extensive program for software upgrades towards the HL-LHC has been set up. The ARM architecture is becoming a competitive and energy efficient alternative. Some surveys indicate its increased presence in HPCs and commercial clouds, and some WLCG sites have expressed their interest. Chip makers are also developing their next generation solutions on ARM architectures, sometimes combining ARM and GPU processors in the same chip. Consequently it is important that the ATLAS software embraces the change and is able to successfully exploit this architecture. We report on the successful porting to ARM of the Athena software framework, which is used by ATLAS for both online and offline computing operations. Furthermore we report on the successful validation of simulation workflows running on ARM resources. For this we have set up an ATLAS Grid site using ARM compatible middleware and containers on Amazon Web Services (AWS) ARM resources. The ARM version of Athena is fully integrated in the regular software build system and distributed in the same way as other software releases. In addition, the workflows have been integrated into the HEPscore benchmark suite which is the planned WLCG wide replacement of the HepSpec06 benchmark used for Grid site pledges. In the overall porting process we have used resources on AWS, Google Cloud Platform (GCP) and CERN. A performance comparison of different architectures and resources will be discussed.

72 PHYSICS OF ELEMENTARY PARTICLES AND FIELDS↗