Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “model benchmark”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 1,423 records · Page 79
All sourcesNASA NTRSDOE OSTI
NASA NTRS1997

Performance Comparison of HPF and MPI Based NAS Parallel Benchmarks

Compilers supporting High Performance Form (HPF) features first appeared in late 1994 and early 1995 from Applied Parallel Research (APR), Digital Equipment Corporation, and The Portland Group (PGI). IBM introduced an HPF compiler for the IBM RS/6000 SP2 in April of 1996. Over the past two years, these implementations have shown steady improvement in terms of both features and performance. The performance of various hardware/ programming model (HPF and MPI) combinations will be compared, based on latest NAS Parallel Benchmark results, thus providing a cross-machine and cross-model comparison. Specifically, HPF based NPB results will be compared with MPI based NPB results to provide perspective on performance currently obtainable using HPF versus MPI or versus hand-tuned implementations such as those supplied by the hardware vendors. In addition, we would also present NPB, (Version 1.0) performance results for the following systems: DEC Alpha Server 8400 5/440, Fujitsu CAPP Series (VX, VPP300, and VPP700), HP/Convex Exemplar SPP2000, IBM RS/6000 SP P2SC node (120 MHz), NEC SX-4/32, SGI/CRAY T3E, and SGI Origin2000. We would also present sustained performance per dollar for Class B LU, SP and BT benchmarks.

Saini, Subhash↗
NASA NTRS2026

Benchmarking Bayesian Optimization Frameworks and Acquisition Strategies for Materials Discovery and Autonomous Laboratories

Bayesian optimization (BO) can accelerate materials discovery by guiding expensive experiments toward the most promising processing conditions. We systematically compare five BO surrogate and framework combinations (Gaussian processes in Ax, Gaussian processes and Monte-Carlo neural networks in BayBE, random forests in Lolopy, and tree-structured Parzen (TPE) estimators in Hyperopt) on three benchmarks that mimic common materials design tasks (a discrete solid-electrolyte composition space, a hybrid discrete/continuous laminate-composite design problem solved with micromechanics modeling, and the continuous Ishigami analytic function which is a standard optimization benchmark). Each BO surrogate is paired with posterior mean, probability of improvement, and expected improvement acquisition functions and run for 100 trials from randomized initial samples with uniform random search providing a control. Across five random seeds per setting, BayBE’s Gaussian-process surrogate with expected improvement consistently reached ≥95 % of the known optimum in the fewest evaluations, while Lolopy’s random forest matched or exceeded GP performance on purely categorical or mixed spaces at a higher computational cost. Posterior mean alone often stagnated at local optima, underscoring the need for exploration, whereas probability and expected improvement balanced exploration and exploitation leading to better optimization in fewer trials. Execution times ranged from milliseconds for TPE to minutes for neural-network and random-forest surrogates. These results establish baseline expectations for BO in automated materials laboratories and highlight expected improvement with Gaussian processes as a reliable first choice, with random forests offering a strong alternative when categorical variables dominate. The benchmark suite and code are released to facilitate future surrogate, acquisition, and constraint-handling research in data-driven materials optimization.

Bayesian optimization↗
DOE OSTI2022

$t\overline{t}t\overline{t}$ signatures through the lens of color-octet scalars

We reinterpret two recent LHC searches for events containing four top quarks ($t\overline{t}t\overline{t}$) in the context of supersymmetric models with Dirac gauginos and color-octet scalars (sgluons). We explore whether sgluon contributions to the four-top production cross section σ ($pp$ → $t\overline{t}t\overline{t}$) can accommodate an excess of four-top events recently reported by the ATLAS collaboration. We also study constraints on these models from an ATLAS search for new phenomena with high jet multiplicity and significant missing transverse energy ($E^{miss}_{T}$) sensitive to signals with four top quarks. We find that these two analyses provide complementary constraints, with the jets + $E^{miss}_{T}$ search exceeding the four-top cross section measurement in sensitivity for sgluons heavier than about 800 GeV. We ultimately find that either a scalar or a pseudoscalar sgluon can currently fit the ATLAS excess in a range of reasonable benchmark scenarios, though a pseudoscalar in minimal Dirac gaugino models is ruled out. We finally offer sensitivity projections for these analyses at the HL-LHC, mapping the 5σ discovery potential in sgluon parameter space and computing exclusion limits at 95% CL in scenarios where no excess is found.

71 CLASSICAL AND QUANTUM MECHANICS, GENERAL PHYSIC↗
DOE OSTI2023

Quantifying uncertainty for deep learning based forecasting and flow-reconstruction using neural architecture search ensembles

Classical problems in computational physics such as data-driven forecasting and signal reconstruction from sparse sensors have recently seen an explosion in deep neural network (DNN) based algorithmic approaches. However, most DNN models do not provide uncertainty estimates, which are crucial for establishing the trustworthiness of these techniques in downstream decision making tasks and scenarios. In recent years, ensemble-based methods have achieved significant success for the uncertainty quantification in DNNs on a number of benchmark problems. However, their performance on real-world applications remains under-explored. In this work, we present an automated approach to DNN discovery and demonstrate how this may also be utilized for ensemble-based uncertainty quantification. Specifically, we propose the use of a scalable neural and hyperparameter architecture search for discovering an ensemble of DNN models for complex dynamical systems. We highlight how the proposed method not only discovers high-performing neural network ensembles for our tasks, but also quantifies uncertainty seamlessly. This is achieved by using genetic algorithms and Bayesian optimization for sampling the search space of neural network architectures and hyperparameters. Subsequently, a model selection approach is used to identify candidate models for an ensemble set construction. Afterwards, a variance decomposition approach is used to estimate the uncertainty of the predictions from the ensemble. We demonstrate the feasibility of this framework for two tasks — forecasting from historical data and flow reconstruction from sparse sensors for the sea-surface temperature. In conclusion, we demonstrate superior performance from the ensemble in contrast with individual high-performing models and other benchmarks.

Deep ensembles↗
DOE OSTI2026

Physics vs structure: A systematic benchmark of learning strategies for multi-zone building thermal dynamics

Recent advances in physics-informed and data-driven machine learning promise improved thermal models for advanced building control, yet there is limited quantitative evidence on when added physics structure and architectural complexity are beneficial. Here, this work presents a systematic benchmark of five representative system identification methods for modeling multi-zone building thermal dynamics: linear state-space models, multi-layer perceptrons, neural state-space models, neural ordinary differential equations, and physically-consistent neural networks. The methods are evaluated across multiple data regimes and zone coupling strategies. Using a high-fidelity multi-zone commercial building emulator, we examine short-term and long-term prediction accuracy, computational efficiency, and ease of development. Our results reveal critical trade-offs between prediction performance, model complexity, and physical consistency. We demonstrate that decoupled, nonlinear black-box models consistently outperform coupled physics-constrained architectures in both predictive accuracy and out-of-distribution robustness in majority of the test cases for the building type considered in the study. Our findings quantify the cost of complexity in building thermal modeling and provide concrete, actionable, scenario-based guidelines for selecting model classes for control-oriented applications.

Building thermal modeling↗
DOE OSTI2025

Extending quantum-mechanical benchmark accuracy to biological ligand-pocket interactions

Predicting the binding affinity of ligands to protein pockets is key in the drug design pipeline. The flexibility of ligand-pocket motifs arises from a range of attractive and repulsive electronic interactions during binding. Accurately accounting for all interactions requires robust quantum-mechanical (QM) benchmarks, which are scarce for ligand-pocket systems. Additionally, disagreement between “gold standard” Coupled Cluster (CC) and Quantum Monte Carlo (QMC) methods casts doubt on many benchmarks for larger non-covalent systems. We introduce the “QUantum Interacting Dimer” (QUID) benchmark framework containing 170 non-covalent (non-)equilibrium systems modeling chemically and structurally diverse ligand-pocket motifs. Symmetry-adapted perturbation theory shows that QUID broadly covers non-covalent binding motifs and energetic contributions. Robust binding energies are obtained using complementary CC and QMC methods, achieving agreement of 0.5 kcal/mol. The benchmark data analysis reveals that several dispersion-inclusive density functional approximations provide accurate energy predictions, though their atomic van der Waals forces differ in magnitude and orientation. Contrarily, semiempirical methods and empirical force fields require improvements in capturing non-covalent interactions (NCIs) for out-of-equilibrium geometries. The wide span of NCIs, highly accurate interaction energies, and analysis of molecular properties take QUID beyond the “gold standard” for QM benchmarks of ligand-protein systems.

Puleva, Mirela [University of Luxembourg, Luxembou↗
NASA NTRS2016

Bumper 3 Update for IADC Protection Manual

The Bumper code has been the standard in use by NASA and contractors to perform meteoroid/debris risk assessments since 1990. It has undergone extensive revisions and updates [NASA JSC HITF website; Christiansen et al., 1992, 1997]. NASA Johnson Space Center (JSC) has applied BUMPER to risk assessments for Space Station, Shuttle, Mir, Extravehicular Mobility Units (EMU) space suits, and other spacecraft (e.g., LDEF, Iridium, TDRS, and Hubble Space Telescope). Bumper continues to be updated with changes in the ballistic limit equations describing failure threshold of various spacecraft components, as well as changes in the meteoroid and debris environment models. Significant efforts are expended to validate Bumper and benchmark it to other meteoroid/debris risk assessment codes. Bumper 3 is a refactored version of Bumper II. The structure of the code was extensively modified to improve maintenance, performance and flexibility. The architecture was changed to separate the frequently updated ballistic limit equations from the relatively stable common core functions of the program. These updates allow NASA to produce specific editions of the Bumper 3 that are tailored for specific customer requirements. The core consists of common code necessary to process the Micrometeoroid and Orbital Debris (MMOD) environment models, assess shadowing and calculate MMOD risk. The library of target response subroutines includes a board range of different types of MMOD shield ballistic limit equations as well as equations describing damage to various spacecraft subsystems or hardware (thermal protection materials, windows, radiators, solar arrays, cables, etc.). The core and library of ballistic response subroutines are maintained under configuration control. A change in the core will affect all editions of the code, whereas a change in one or more of the response subroutines will affect all editions of the code that contain the particular response subroutines which are modified. Note that the Bumper II program is no longer maintained or distributed by NASA.

Christiansen, Eric L.↗
DOE OSTI2022

Dynamic signatures of electronically nonadiabatic coupling in sodium hydride: a rigorous test for the symmetric quasi-classical model applied to realistic, ab initio electronic states in the adiabatic representation

Sodium hydride (NaH) in the gas phase presents a seemingly simple electronic structure making it a potentially tractable system for the detailed investigation of nonadiabatic molecular dynamics from both computational and experimental standpoints. The single vibrational degree of freedom, as well as the strong nonadiabatic coupling that arises from the excited electronic states taking on considerable ionic character, provides a realistic chemical system to test the accuracy of quasi-classical methods to model population dynamics where the results are directly comparable against quantum mechanical benchmarks. Here, using a simulated pump–probe type experiment, this work presents computational predictions of population transfer through the avoided crossings of NaH via symmetric quasi-classical Meyer–Miller (SQC/MM), Ehrenfest, and exact quantum dynamics on realistic, ab initio potential energy surfaces. The main driving force for population transfer arises from the ground vibrational level of the D 1 Σ + adiabatic state that is embedded in the manifold of near-dissociation C 1 Σ + vibrational states. When coupled through a sharply localized first-order derivative coupling most of the population transfers between t = 15 and t = 30 fs depending on the initially excited vibronic wavepacket. While quantum mechanical effects are expected due to the reduced mass of NaH, predictions of the population dynamics from both the SQC/MM and Ehrenfest models perform remarkably well against the quantum dynamics benchmark. Additionally, an analysis of the vibronic structure in the nonadiabatically coupled regime is presented using a variational eigensolver methodology.

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗
DOE OSTI2025

Intelligent Manufacturing Support: Specialized LLMs for Composite Material Processing and Equipment Operation

Engineering educational curriculum and standards cover many material and manufacturing options. However, engineers and designers are often unfamiliar with certain composite materials or manufacturing techniques. Large language models (LLMs) could potentially bridge the gap. Their capacity to store and retrieve data from large databases provides them with a breadth of knowledge across disciplines. However, their generalized knowledge base can lack targeted, industry-specific knowledge. To this end, we present two LLM-based applications based on the GPT-4 architecture: (1) The Composites Guide: a system that provides expert knowledge on composites material and connects users with research and industry professionals who can provide additional support and (2) The Equipment Assistant: a system that provides guidance for manufacturing tool operation and material characterization. By combining the knowledge of general AI models with industry-specific knowledge, both applications are intended to provide more meaningful information for engineers. In this paper, we discuss the development of the applications and evaluate it through a benchmark and two informal user studies. The benchmark analysis uses the Rouge and Bertscore metrics to evaluate our models’ performance against GPT-4o. The results show that GPT-4o and the proposed models perform similarly or better on the ROUGE and BERTScore metrics. The two user studies supplement this quantitative evaluation by asking experts to provide qualitative and open-ended feedback about our model’s performance on a set of domain-specific questions. The results of both studies highlight a potential for more detailed and specific responses with the Composites Guide and the Equipment Assistant.

Kapoor, Gunnika [Oak Ridge National Laboratory (OR↗
DOE OSTI2023

MRT 7365 Power flow physics and key physics phenomena: EMPIRE verification suite

This milestone work baselines electromagnetic particle-in-cell capability of the EMPIRE plasma simulation code to model key processes germane to the physics of electrode plasmas arising in magnetically-insulated transmission lines operating at or near 20 MA. This evaluation is done so through the provision of benchmark verification problems designed to exercise the individual and combined physics models on a small-scale surrogate geometry for the final-feed-to-load region of the Z accelerator under representative operating conditions. In this report, we overview our test designs, and present a portfolio of simulation results along with performance assessments which altogether establish state-of-the-art. In particular, two main verification categories are covered this report: (1) Z-relevant desorption physics (Temkin isotherm), and (2) two approaches to simulate electrode plasma creation and dynamics (automatic creation versus self-consistent creation through direct simulation Monte Carlo collisions).

43 PARTICLE ACCELERATORS↗
DOE OSTI2024

SRBench++: Principled Benchmarking of Symbolic Regression With Domain-Expert Interpretation

Symbolic regression searches for analytic expressions that accurately describe studied phenomena. The main promise of this approach is that it may return an interpretable model that can be insightful to users, while maintaining high accuracy. The current standard for benchmarking these algorithms is SRBench, which evaluates methods on hundreds of datasets that are a mix of real-world and simulated processes spanning multiple domains. At present, the ability of SRBench to evaluate interpretability is limited to measuring the size of expressions on real-world data, and the exactness of model forms on synthetic data. In practice, model size is only one of many factors used by subject experts to determine how interpretable a model truly is. Furthermore, SRBench does not characterize algorithm performance on specific, challenging sub-tasks of regression such as feature selection and evasion of local minima. In this work, we propose and evaluate an approach to benchmarking SR algorithms that addresses these limitations of SRBench by 1) incorporating expert evaluations of interpretability on a domain-specific task, and 2) evaluating algorithms over distinct properties of data science tasks. We evaluate 12 modern symbolic regression algorithms on these benchmarks and present an in-depth analysis of the results, discuss current challenges of symbolic regression algorithms and highlight possible improvements for the benchmark itself.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2022

Convolutional neural networks for intra-hour solar forecasting based on sky image sequences

Accurate and timely solar forecasts play an increasingly critical role in power systems. Compared to longer forecasting timescales, very short-term solar forecasting has lagged behind in both research and practice. In this paper, we propose deep convolutional neural networks (CNNs) to provide operational intra-hour (10-minute-ahead to 60-minute-ahead) solar forecasts. We develop two CNN structures inspired by a widely-used CNN architecture. The CNNs are tailored to our solar forecasting regression tasks and rely solely on sky image sequences. Case studies based on six years of data (over 150,000 data points) demonstrate that the best CNN model has forecast skill scores of 20%-39% over the naive persistence of cloudiness benchmark, even at these very short timescales. The CNNs also have consistently superior performance when compared to shallow machine learning models with meteorological predictors, where the improvement averages around 7%. The sensitivity analyses show that the sky image length, resolution, and weather conditions have impacts on the deep learning model accuracy. In our intra-hour problem with specific setups, two sky images with a 10-minute 128 x 128 resolution yield the most accurate forecasts. Current limitations, future work, and deployment challenges and solutions are also discussed.

14 SOLAR ENERGY↗
NASA NTRS2002

Memory-Intensive Benchmarks: IRAM vs. Cache-Based Machines

The increasing gap between processor and memory performance has lead to new architectural models for memory-intensive applications. In this paper, we explore the performance of a set of memory-intensive benchmarks and use them to compare the performance of conventional cache-based microprocessors to a mixed logic and DRAM processor called VIRAM. The benchmarks are based on problem statements, rather than specific implementations, and in each case we explore the fundamental hardware requirements of the problem, as well as alternative algorithms and data structures that can help expose fine-grained parallelism or simplify memory access patterns. The benchmarks are characterized by their memory access patterns, their basic control structures, and the ratio of computation to memory operation.

Biswas, Rupak↗
DOE OSTI2022

Evaluation of Oak Ridge National Laboratory Health Physics Research Reactor Operation Data for Critical Benchmark Creation

The Oak Ridge National Laboratory (ORNL) Health Physics Research Reactor (HPRR) was a research reactor designed and built at ORNL in 1961. The critical assembly used a highly enriched uranium and molybdenum alloy as the fuel and could be operated in steady-state or burst modes. The HPRR has recently been the object of an investigation to create a criticality benchmark. Such benchmarks are very important, as they are used primarily to show the accuracy of newly developed modeling codes and to help experimental validation and reactor licensing. The evaluated experiments considered in this paper were carried out between 1974 and 1986 from various HPRR activities such as steady-state subcritical, steady-state critical, and burst prompt super-critical operations of the reactor for dosimetry, irradiation, or training purposes. By using the HPRR experimental logbook information and the as-built drawings of the critical assembly, a highly detailed model of the HPRR was created with SCALE 6.2.4/KENO-VI, and a first version of a critical benchmark of the HPRR was developed following the International Criticality Safety Benchmark Evaluation Project (ICSBEP) guidelines for thorough description and uncertainty/sensitivity quantification. Unfortunately, in most of the evaluated experiments, the obtained difference between calculated and experimental k eff is around 1,000 pcm, corresponding to a relative error of approximately 1%, beyond the quality standards of the ICSBEP recommending a relative error below 0.1%. Moreover, the derived experimental uncertainty is high, around 4% relative, mainly due to the U-Mo fuel density uncertainty, but also from numerous other factors. For these reasons, the creation of a valuable critical benchmark from HPRR operation data is thus far compromised. In this paper, the different steps of the experiments’ evaluation are summarized, and the reasons for the experimental/calculation discrepancies and potential ways to solve them are explored. This paper also aims to remind us always to exercise considerable care when performing experimental work, and to record all the data possible for potential future uses.

21 SPECIFIC NUCLEAR REACTORS AND ASSOCIATED PLANTS↗
NASA NTRS2022

A Benchmark Example for Delamination Propagation Predictions Based on the Calibrated End-Loaded Split Specimen

A benchmark example based on the Calibrated End-Loaded Split (C-ELS) specimen is developed and used to assess the performance of recently developed delamination propagation capabilities in the Abaqus/Standard finite element code. The C-ELS specimen has the advantage of a longer region of stable delamination propagation compared to the existing mode II benchmark case. The new benchmark example may therefore provide a better assessment tool by enabling more stable crack growth in regions further away from the boundary conditions or load application. First, a benchmark result is created manually using two-dimensional finite element models of the C-ELS specimen with different delamination lengths. Second, the performance of the delamination propagation capabilities in the Abaqus/Standard finite element code are assessed by comparing the results to the benchmark case. Two examples with different starter delamination lengths are studied. A shorter starter length is chosen to create a scenario with unstable delamination propagation. A longer delamination causes stable delamination propagation. Detailed results from three-dimensional analyses with aligned and mis-aligned meshes and two levels of mesh refinement are provided for several permutations of numerical input parameters. In general, good agreement can be achieved between the results obtained from the quasi-static propagation analysis and the benchmark analysis. However, particular non-default settings are found to be most reliable, accurate, and numerically efficient. Numerical artifacts including anomalous unreleased nodes in the crack wake and zig-zag crack fronts occur, and further development of the Abaqus/Standard VCCT propagation may be required. Use of the benchmark case to assess the continuous improvements in one finite element code illustrates the value of establishing benchmark solutions.

delamination↗
NASA NTRS1982

Benchmark notch test for life prediction

The laser Interferometric Strain Displacement Gage (ISDG) was used to measure local strains in notched Inconel 718 test bars subjected to six different load histories at 649 C (1200 F) and including effects of tensile and compressive hold periods. The measurements were compared to simplified Neuber notch analysis predictions of notch root stress and strain. The actual strains incurred at the root of a discontinuity in cyclically loaded test samples subjected to inelastic deformation at high temperature where creep deformations readily occur were determined. The steady state cyclic, stress-strain response at the root of the discontinuity was analyzed. Flat, double notched uniaxially loaded fatigue specimens manufactured from the nickel base, superalloy Inconel 718 were used. The ISDG was used to obtain cycle by cycle recordings of notch root strain during continuous and hold time cycling at 649 C. Comparisons to Neuber and finite element model analyses were made. The results obtained provide a benchmark data set in high technology design where notch fatigue life is the predominant component service life limitation.

Domas, P. A.↗
DOE OSTI2020

An Uncertainty Management Framework for Integrated Gas-Electric Energy Systems

In many parts of the world, electric power systems have seen a significant shift toward generation from renewable energy and natural gas. Because of their ability to flexibly adjust power generation in real time, gas-fired power plants are frequently seen as the perfect partner for variable renewable generation. However, this reliance on gas generation increases interdependence and propagates uncertainty between power grids and gas pipelines and brings coordination and uncertainty management challenges. To address these issues, we propose an uncertainty management framework for uncertain, but bounded gas consumption by gas-fired power plants. The admissible ranges are computed based on a joint optimization problem for the combined gas and electricity networks, which involves chance-constrained scheduling for the electric grid and a novel robust optimization formulation for the natural-gas network. This formulation ensures feasibility of the integrated system with a high probability, while providing a tractable numerical formulation. A key advance with respect to existing methods is that our method is based on a physically accurate, validated model for transient gas pipeline flows. Our case study benchmarks our proposed formulation against methods that ignore how reserve activation impacts the fuel use of gas power plants and only consider predetermined gas consumption. Here, the results demonstrate the importance of considering uncertainty to avoid operating constraint violations and curtailment of gas to the generators.

24 POWER TRANSMISSION AND DISTRIBUTION↗
NASA NTRSDate not supplied

Spacecraft Observatory Benchmark Problem for Optical Disturbance Rejection

Since mid-1980s NASA has launched several first-generation interplanetary observatory missions. A salient feature of these missions has been the mounting of the optical payload on top of a large flexible space structure. This imposes challenging control-structure interaction problems (CSI) that have been studied over several decades. The Dec. 2021 launch of the James Webb Telescope marks the latest example of such a mission. One way to encourage development of novel control methods is to provide a benchmark problem for researchers. By the mid-2000s, there were a few well-known benchmark problems developed by the control community with a focus on CSI applications. However, in the past 15 years, the control challenges in optical observation missions have shifted from CSI to line-of-sight (LOS) precision pointing. This is because popular designs of the second-generation (Gen II) space earth missions have eliminated the large structure interface by mounting the optical payload directly on top of a rigid bus platform. The focus now is to achieve accurate pointing with maximum rejection of the surrounding disturbances. The NASA Safety and Engineering Center (NESC) has tasked T he Aerospace Corporation to develop a Benchmark Problem to study the challenging control issues that Gen II observatory missions encounter. This software tool is to be released as a public domain platform aimed at government, academic, and industry researchers. The focus of this benchmark problem is for researchers to design a set of innovative payload control laws with the maximum Optical Disturbance Rejection (ODR) capability to meet a set of pre-defined μrad-level of LOS jitter requirements. This paper describes the development of the benchmark problem. Aerospace/NASA will provide an integrated LOS plant model and disturbance/command profiles for users to run their design and simulation with as well as a user’s guide describing the required interfaces. The Aerospace Corp is also working to develop a hardware fast steering mirror testbed for users to demonstrate their innovative design in real hardware, if so desired.

Spacecraft Observatory Benchmark Problem↗