Engineering PapersSearch

SEARCH · Engineering Papers

Results for “Massively Parallel Simulations”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 19 records

4.0 MOOSE: Enabling massively parallel Multiphysics simulation

Approaching 18 years of existence, MOOSE—the Multiphysics Object-Oriented Simulation Environment—is being developed at a higher pace than ever before. With significant support from four research institutions across the globe, and dozens of new contributors, the capabilities of the framework are being expanded to meet modeling challenges in a wide variety of fields from nuclear system design, to geomechanics, to material science. This includes new development in equation discretization techniques, solver methods, meshing capabilities, application deployment, and user interface improvements. Applications built on MOOSE benefit from all these improvements.

22 - GENERAL STUDIES OF NUCLEAR REACTORS

Intelligently Partitioned Phasor-EMT Hybrid Simulations of Large-Scale, High-IBR Power Systems

As the penetration level of power electronics-interfaced renewables such as photovoltaics (PV) and wind has surged in modern electric grids, new operational risks caused by the dynamics of those inverter-based resources (IBRs) are emerging in parallel. Lessons learned from various grid events include that the impact of IBRs on system-level grid stability will become prominent along with the increase of renewables and that the short-timescale dynamic impacts of IBRs on grid stability are not fully captured by current commercial dynamic simulation tools [1] [2]. For example, IBRs can be controlled to mitigate those destabilizing interactions, but conventional phasor-domain tools (e.g. PSS/E, PSLF) often cannot capture that; likewise, the existing electromagnetic transient (EMT) simulation tools (e.g. PSCAD, EMTP) can simulate detailed IBR controls, but for large power systems with many IBRs, slow simulation speeds severely impede the ability to study dynamic events [3] [4]. Massively paralleling simulations using high-performance computing (HPC) can help address this, especially now that cloud-based HPC capability is widely available, but today s EMT tools are not HPC-compatible, and parallelization of dynamic simulation solvers is not trivial because each region can dynamically affect the others. Thus, dynamic simulation of grids with very large numbers of IBRs potentially poses a barrier to the ongoing energy transition.

24 POWER TRANSMISSION AND DISTRIBUTION

Comparison of DeePMD, MTP, GAP, ACE and MACE Machine‐Learned Potentials for Radiation‐Damage Simulations: A User Perspective

Accurate and efficient interatomic potentials are essential for molecular dynamics (MD) simulations of radiation damage, gas diffusion, and phase stability in complex ceramics such as LiAlO 2 , especially under extreme conditions relevant to tritium production. Here, we evaluate the performance of six machine-learned interatomic potentials (MLIPs), moment tensor potential (MTP), Gaussian approximation potential, deep potential (DeePMD), atomic cluster expansion (ACE), message-passing ACE (multilayer atomic cluster expansion (MACE) pretrained) and MACE (trained from-scratch), all trained on the same density functional theory dataset with inclusion of tritium. The MLIPs are benchmarked against traditional Buckingham and ReaxFF potentials in terms of energy accuracy, density predictions, thermal equilibration behavior, threshold displacement energy (E d ), tritium diffusivity, and computational cost. Among the models, MTP shows the best overall balance between efficiency and accuracy, with low force and energy errors and realistic E d values for Li and Al. The ACE and MACE (pretrained and trained from scratch) models exhibit high E d (>200 eV) and unphysical pair interactions. DeePMD underestimates Ed due to overly repulsive behavior even at equilibrium distances. All models over-estimate tritium diffusion but the pretrained MACE model behaves well during tritium-diffusion simulations up to 500 K, maintaining diffusivities in the physically consistent 10 −11 m 2 /s range. Finally, we quantify the computational cost of each potential in large-scale atomic/molecular massively parallel simulator, finding that only MTP is more efficient than traditional empirical potentials, while others are significantly more expensive. These findings explain the trade-offs between accuracy and computational cost in MLIP development and provide essential guidance for use in high-throughput radiation damage and gas diffusion simulations in nuclear ceramics.

74 ATOMIC AND MOLECULAR PHYSICS

In situ multi-tier auto-ignition detection applied to dual-fuel combustion simulations

Here we use an anomaly detection methodology that is centered on analyzing fourth-order joint moments (co-kurtosis), particularly focusing on its application in auto-ignition of combustion problems with large numbers of species. Unsupervised anomaly detection is challenging to generalize across problem types and domains. A recent technique, centered on analyzing information in the fourth-order joint moment co-kurtosis, has shown promise, especially for high-dimensional scientific data. In this work we present developments to the co-kurtosis based anomaly detection method needed to make it effective and scalable for large-scale distributed scientific data, such as those generated by massively parallel simulations. An in situ co-kurtosis algorithm is employed as the anomaly detection method for identifying ignition kernels in simulations of turbulent combustion. Here, we extend an existing methodology which identifies regions of the domain where anomalies are present, and add another tier of anomaly detection where the individual samples contributing to the anomaly are identified. We apply this algorithm on-the-fly to a variety of turbulent reacting flow problems and compare it to the widely used (but significantly more expensive) chemical explosive mode analysis (CEMA). We demonstrate the ability of the method to detect and identify the onset of low and high temperature ignition which can be used for computational steering, as chemical and combustion anomalies occur intermittently at spatio-temporal locations unknown a priori. Finally, we apply our lightweight in situ algorithm to an exascale high-fidelity simulation with a total of 2.4 Trillion degrees of freedom, performed using an adaptive mesh refinement solver. Furthermore, through a scalability analysis, we show that the relative computational cost of this in-situ anomaly detection algorithm compared to an iteration of the reacting flow solver is negligible.

97 MATHEMATICS AND COMPUTING

Developing reliable machine learning interatomic potential for Fe–Cr–Ni austenitic alloys

Gaining atomistic understanding of mechanical behavior of heat-resistant structural materials such as Fe–Cr–Ni-based alloys requires an approach with an accuracy close to density functional theory (DFT) that considers the intrinsic properties of the bulk lattice and important defects such as stacking faults, grain boundaries, and surfaces. This work aims to develop reliable machine learning interatomic potential (MLIAP) at cross-scale for Fe–Cr–Ni ternary alloys with a focus on the face-centered-cubic (fcc) solid solution structure. Leveraging the advantages of moment tensor potentials, which typically necessitate a relatively small training dataset and enable rapid calculations using the large-scale atomic/molecular massively parallel simulator package, we ensure the stability and accuracy of the trained potentials. Important defects such as stacking faults, grain boundaries, and surfaces for wide-range compositions are investigated. Structural, thermal, elastic, and defect properties are determined from molecular dynamics simulations comprising several thousand atoms, generated via canonical Monte Carlo simulations guided by the trained potential. The trained potential allows efficient atomic simulations of structural, thermal, and mechanical properties of fcc Fe–Cr–Ni solid solution alloys as a function of composition and temperature. Therefore, the MLIAP approach represents a major advancement from DFT calculations that are limited to small simulation sizes and traditional molecular dynamics simulations using relatively low accuracy potentials. Furthermore, this work outlines a practical foundation for further investigating the structural evolution and mechanical behavior of austenitic stainless steel and nickel-based alloys in a wide array of applications in extreme environments.

Crystal structure

Optimizing Grain Boundary Structures with LAMMPS Using Evolutionary Algorithms

Grain boundary structure optimization is an important part of materials modeling. Current methods for grain boundary structure optimization involve inefficient, time-consuming processes that do not fully explore the interface parameter space. Evolutionary algorithms have recently been demonstrated to be effective at determining both stable and metastable grain boundary interface structures. In this work, we demonstrate the use of GBOpt, a grain boundary structure optimization software designed to use the Large-scale Atomic/Molecular Massively Parallel Simulation (LAMMPS) software to efficiently determine grain boundary structures. We demonstrate that a only a few manipulations, namely atom insertion, atom removal, and relative grain displacement, are sufficient to explore much of the grain boundary structure parameter space. The efficacy of this approach is demonstrated on an FCC Ni system, and a BCC Fe system. The computational cost is compared against the gamma-surface sampling approach to demonstrate performance improvement.

Evolutionary algorithms

Massively parallel phase-field simulations targeting exascale

The interface thickness in the phase-field (PF) method limits its simulation scales. Consequently, large-scale PF simulations become prohibitively expensive for resolving the extremely fine microstructures that typically form during rapid solidification processing. This challenge is significant in predicting microstructure evolution in metal additive manufacturing and has been identified by the United States Department of Energy’s Exascale Computing Project. Here, to address this, we develop a multi-GPU and MPI-based massively parallel simulation code, utilizing state-of-the-art algorithms, software, and libraries, for large-scale three-dimensional (3D) PF simulations. We report the first GPU-parallel PF simulations on Frontier (currently the second TOP500 exascale cluster) and Summit machines, taking dendritic growth as an example problem. We evaluate the parallel performance of our implementation using scaling studies with more than 24 000 GPUs (among the largest known computations to date) and the acceleration performance using large-scale simulations of dendritic growth in 3D. Finally, massively parallel GPUs in these supercomputers enabled the first coupled multiscale simulations of laser melting and subsequent dendritic solidification on the scale of a full melt-pool, demonstrating the feasibility of performing PF simulations with a point total over 2 billion grid points within an acceptable time.

Exascale

OpenSn: A massively parallel, open-source simulation environment for discrete ordinates radiation transport

OpenSn is an open-source, massively parallel deterministic radiation transport code for solving the discrete-ordinates ( S N ) form of the Boltzmann transport equation on unstructured, arbitrary polyhedral meshes. It supports high-fidelity simulations involving steady-state, eigenvalue, and adjoint problems for neutral particles (e.g., neutrons, photons, multi-particles), using the multigroup approximation in energy. OpenSn combines angular discretization via discrete ordinates with a discontinuous Galerkin finite element method (DGFEM) in space, enabling accurate resolution of transport physics on arbitrary polyhedral cells, included locally refined spatial grids. It includes multiple angular quadrature types, including locally refined angular quadratures. Written in modern C++ with a Python API, OpenSn runs efficiently on platforms ranging from laptops to supercomputers. The transport sweep algorithm is implemented using a task-based, directed-acyclic-graph (DAG) approach for each angle and supports asynchronous parallelism across thousands of MPI ranks. Group-set aggregation improves compute intensity, and synthetic acceleration techniques (e.g., diffusion synthetic acceleration, second-moment method) enhance solver convergence. OpenSn has been verified on reactor physics problems and demonstrated excellent weak and strong scaling performance on more than 32,768 processes, making it a versatile and robust platform for large-scale transport simulations in complex geometries.

73 NUCLEAR PHYSICS AND RADIATION PHYSICS

KRAS4a and KRAS4b show distinct lipid-dependent regulation of RAS-RAF membrane dynamics

KRAS4a and KRAS4b are important regulators of signaling, and their interactions with the plasma membrane are dynamic and influenced by lipid composition. KRAS 4a and 4b have nearly identical globular domains but differ in their membrane-associated hyper variable region (HVR). The functional distinctions between these isoforms remain unclear, particularly with regards to their dependence on specific lipids and the membrane environment. Previous work showed that the membrane orientation of KRAS4b affects its ability to bind to RAF kinase RBDCRD and that the KRAS–RBDCRD complex adopts different poses on the membrane as well as influences the size and composition of the lipid environment. To model differences between KRAS 4a and 4b protein–lipid interactions, we extended the Multiscale Machine-Learned Modeling Infrastructure (MuMMI) to incorporate continuum simulations in the grand canonical ensemble, enabling sampling across macroscopic, coarse-grained, and all-atom resolutions. Using this framework, we systematically altered PIP2 concentrations, KRAS 4a versus 4b, and RAF RBDCRD complexation to assess impacts on membrane–protein interactions and dynamics. Our results reveal that reducing PIP2 shifts and broadens the membrane orientational preference of both KRAS 4b and 4a, with stronger effects on 4b HVR localization versus 4a. We demonstrate that with depletion of the strong negatively charged PIP2 lipid, the less charged phosphatidylserine replaces PIP2. Our findings highlight similarities and distinctions in the dynamics and lipid dependency of KRAS isoforms and suggest that ordering of the local lipid composition by HVRs is a shared property and key modulator of RAS-mediated signaling at the plasma membrane.

Biological and medical sciences

Effects of Galactic Irradiation on Thermal and Electronic Transport in Tungsten

The impact of irradiation on the thermal and electronic properties of materials is a persistent puzzle, particularly defect formation at the atomic and nanoscales. This work examines the nanoscale effects of low-energy irradiation on tungsten (W), focusing on defect-induced modifications to thermal and electronic transport. Using the site-projected thermal conductivity (SPTC) method [A. Gautam et al. PSS-RRL, 2400306, 2024], bulk and twin-grain boundary W with vacancy defects based on the Norgett–Robinson–Torrens displacements per atom model is analyzed. SPTC provides a detailed prediction of postcascade spatial thermal conductivity distribution. Electronic conductivity activity is estimated using the “N 2 method” [K. Nepal et al. Carbon, 119711, 2025] to explore the consequences of vacancies and grain boundaries, highlighting the defect-dependent nature of charge transport behavior. These findings offer high-resolution insights into irradiation-driven transport phenomena, with implications for space-exposed materials and nanoscale thermal/electronic management.

75 CONDENSED MATTER PHYSICS, SUPERCONDUCTIVITY AND

HPC for Optimizing Process Parameters to Control Material Evolution in Seamless Induction Hardening of Wind Turbine Main Shaft Bearings

Work proposed in this project focused on understanding the effect of martensitic transformation in the steel on the potential for cracking during seamless induction hardening (SIH) as a function of process conditions to allow the process to optimally scale up. Large-scale, three-dimensional phase-field simulations of martensitic transformation were performed using MEUMAPPS-SS (Microstructure Evolution Using Massively Parallel Phase-field Simulations – Solid State) code developed at Oak Ridge National Laboratory. The simulations were guided by location-specific thermal history generated by experimental measurements of time-temperature history generated at The Timken Company. The simulations were able to capture the morphological evolution of the martensite variants in an Fe-1.0C-1.5Cr steel based on the Nishiyama-Wasserman (NW) orientation relationship. The simulations were also able to quantify the stress-state at the interface between impinging martensite variants. The simulations indicated that the magnitude of the various stress and strain components were dependent on the sizes of the impinging plates with a reduction in these quantities with reduced plate size in agreement with experimental findings. The results obtained from the simulations will be used to guide the optimization of the alloy thermal conditions to eliminate quench cracking during SIH of bearing steels.

99 GENERAL AND MISCELLANEOUS

Additive Manufactured Composite Phase-Change Material for Thermal Energy Storage Applications

Phase-change materials play a critical role in industrial energy storage applications to drive efficiency improvements, thermal energy management, and carbon emissions reductions. Recently, it has been shown that rapid solidification of alloys with metastable immiscibility in the liquid phase has the potential to form unique microstructures in which a low-melting phase is uniformly distributed in a high-melting matrix. This feature can be exploited using additive manufacturing to produce components with complex geometries containing such unique phase-change microstructures. Phase-field simulations utilizing high-performance computing were used to provide a detailed description of the evolution of the active phase during service in terms of their morphology and composition in different polycrystalline matrix grain morphologies that are typically produced during additive manufacturing. Phase field simulations were performed using, MEUMAPPS-SL (Microstructure Evolution Using Massively Parallel Phase-field Simulations – Solid Liquid) code that was developed in-house by the Oak Ridge National Laboratory. The simulations utilized the capabilities of the Kestrel supercomputer at the National Renewable Energy Laboratory. The simulation results were compared with experimental results generated at Siemens Energy, Inc. The results indicate that the kinetics of liquid spreading along grain boundaries is largely determined by the mobility of the triple line along the intersection of the grain boundary liquid and the grain boundary plane.

25 ENERGY STORAGE

Simulation of 24,000 Electron Dynamics: Real-Time Time-Dependent Density Functional Theory (TDDFT) with the Real-Space Multigrids (RMG)

Here, we present the theory, implementation, and benchmarking of a real-time time-dependent density functional theory (RT-TDDFT) module within the RMG code, designed to simulate the electronic response of molecular systems to external perturbations. Our method offers insights into nonequilibrium dynamics and excited states across a diverse range of systems, from small organic molecules to large metallic nanoparticles. Benchmarking results demonstrate excellent agreement with established TDDFT implementations and showcase the superior stability of our time integration algorithm, enabling long-term simulations with minimal energy drift. The scalability and efficiency of RMG on massively parallel architectures allow for simulations of complex systems, such as plasmonic nanoparticles with thousands of atoms. Future extensions, including nuclear and spin dynamics, will broaden the applicability of this RT-TDDFT implementation, providing a powerful toolset for studies of photoactive materials, nanoscale devices, and other systems where real-time electronic dynamics is essential.

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH

MOOSE ProbML: Parallelizable Probabilistic Machine Learning and Uncertainty Quantification Capabilities

The Multiphysics Object Oriented Simulation Environment (MOOSE) is a widely used open- source finite element software for performing multiphysics multiscale simulations in a massively parallel fashion. Recently, the computational team at Idaho National Laboratory (INL) has implemented Probabilistic Machine Learning (ProbML) capabilities in MOOSE—in a parallelized fashion—and enable active learning with large-scale computational models for tasks such as surrogate model development, scale bridging, forward/inverse uncertainty quantification (UQ), Bayesian optimization, etc. This presentation summarizes these developments in MOOSE along with demonstrations on several real applications relevant to nuclear energy. At the fundamental level, samplers like Monte Carlo/Latin Hypercube, variance reduction, parallelized Markov Chain Monte Carlo (MCMC) support uncertainty propagation in both forward and inverse settings. These samplers can be integrated with the Gaussian processes (GP) suite in MOOSE, which offer several variants like scalar GPs, multi-output GPs, and deep GPs, to enable active learning. These GPs can be tuned using gradient-based optimization methods like Adam and its variants or gradient-free methods like the elliptical slice sampler (a variant of MCMC adept under Gaussian settings) for more complex covariance kernels or likelihoods whose gradient computations can be cumbersome. A variety of batch acquisition functions permit parallelized evaluation of the computational model and support different learning objectives with high efficiency like Bayesian inference, global surrogate development, optimization, etc. Furthermore, libtorch integration supports training, evaluation, and re-training of neural networks and other complex machine learning models in active learning settings. The impacts of these developments are shown on several real applications: (1) nuclear fuel inverse UQ and model inadequacy assessment using the Kennedy O’Hagan framework; (2) uncertainty aware surrogate modeling for additive manufacturing to predict field quantities; (3) nuclear reactor rare events analysis; and (4) complex fluid flow prediction using a global surrogate with quantified prediction uncertainty. Finally, the outlook of MOOSE ProbML is discussed for both outer-loop and inner-loop computations in the broad view to accelerate fuels and materials qualification, address gaps in knowledge and data, and assess new reactor/fuel systems.

11 - NUCLEAR FUEL CYCLE AND FUEL MATERIALS

HPC for optimizing process parameters to control material evolution in seamless induction hardening of wind turbine main shaft bearings

Work proposed in this project focused on understanding the effect of martensitic transformation in the steel on the potential for cracking during seamless induction hardening (SIH) as a function of process conditions to allow the process to optimally scale up. Large-scale, three-dimensional phase-field simulations of martensitic transformation were performed using MEUMAPPS-SS (Microstructure Evolution Using Massively Parallel Phase-field Simulations – Solid State) code developed at Oak Ridge National Laboratory. The simulations were guided by location-specific thermal history generated by experimental measurements of time-temperature history generated at The Timken Company. The simulations were able to capture the morphological evolution of the martensite variants in an Fe-1.0C-1.5Cr steel based on the Nishiyama-Wasserman (NW) orientation relationship. The simulations were also able to quantify the stress-state at the interface between impinging martensite variants. The simulations indicated that the magnitude of the various stress and strain components were dependent on the sizes of the impinging plates with a reduction in these quantities with reduced plate size in agreement with experimental findings. The results obtained from the simulations will be used to guide the optimization of the alloy thermal conditions to eliminate quench cracking during SIH of bearing steels.

17 WIND ENERGY

Kinetic Plasma Simulation Capabilities in the MOOSE Framework: Verification of Particle-Particle Collisions

High-fidelity simulations of complex plasma systems allow researchers to gain key insights into and understanding of these systems. To facilitate massively parallel high-fidelity plasma simulations, finite-element-based particle-in-cell capabilities are being developed within the open-source Multiphysics Object-Oriented Simulation Environment (MOOSE) based framework called Software for Advanced Large-scale Analysis of MAgnetic confinement for Numerical Design, Engineering & Research (SALAMANDER). While SALAMANDER’s primary objective is modeling edge plasmas and plasma-facing components in fusion devices, the particle-in-cell capabilities being developed are general and will support modeling low-temperature plasmas as well. Previously, collisionless magnetostatic simulation capabilities have been verified with the two-stream and Dorey-Guest-Harris instabilities, and single particle motion. Collisions were implemented using the direct simulation Monte Carlo method, and verification of this capability will be presented here several verification problems: relaxation of a randomly initialized gas to a Maxwellian distribution, Fourier heat flow, and comparison of reaction rates to both analytic calculations and those calculated using a multi-term Boltzmann solver.

70 - PLASMA PHYSICS AND FUSION TECHNOLOGY

Asynchronous-many-task systems: Challenges and opportunities - Scaling an AMR astrophysics code on exascale machines using Kokkos and HPX

Dynamic and adaptive mesh refinement is pivotal in high-resolution, multi-physics, multi-model simulations, necessitating precise physics resolution in localized areas across expansive domains. Today’s supercomputers’ extreme heterogeneity presents a significant challenge for dynamically adaptive codes, highlighting the importance of achieving performance portability at scale. Our research focuses on astrophysical simulations, particularly stellar mergers, to elucidate early universe dynamics. Here, we present Octo-Tiger, leveraging Kokkos, HPX, and SIMD for portable performance at scale in complex, massively parallel adaptive multi-physics simulations. Octo-Tiger supports diverse processors, accelerators, and network backends. Experiments demonstrate exceptional scalability across several heterogeneous supercomputers including Perlmutter, Frontier, and Fugaku, encompassing major GPU architectures and x86, ARM, and RISC-V CPUs. Parallel efficiency of 47.59% (110,080 cores and 6880 hybrid A100 GPUs) on a full-system run on Perlmutter (26% HPCG peak performance) and 51.37% (using 32,768 cores and 2048 MI250X) on Frontier are achieved.

97 MATHEMATICS AND COMPUTING

Self-Consistent Relativistic Electron Scattering using the Sherlock Scattering Model for X-ray Diagnostics

We present on a new, self-consistent, arbitrary-temperature Romberg integration scheme for modeling electron scattering in materials in a LANL Lagrangian Shock Hydro (LSH) code. Electron beam-target interactions are fundamental to a wide range of scientific and technological applications. When high-energy electron beams hit their target, they may scatter, deposit energy, or ionize the source. These processes govern the behavior and outcomes in nanotechnology manufacturing, electron microscopy, and modern X-ray diagnostics. Simulating these interactions is essential for interpreting experimental results, predicting material responses, and designing efficient tools and experiments. At Los Alamos, this is done using a LSH code, which is a multi-dimension, multi-material, massively parallel, multi-physics code used to simulate applications from asteroid impacts to electron beam interactions. By effectively and efficiently modeling the way that electrons scatter from the beam we can bolster these simulations and more accurately predict experimental outcomes. The model currently implemented in the LSH of interest is based on work by Papp and does not self-consistently preserve momentum in the slightly relativistic regime; here we adopt a model proposed by Braams and Karney and implement a Romberg integration scheme to compute the diffusion tensor. In this paper we will provide background on the Braams-Karney diffusion tensor as well as the Romberg integration scheme we employed to numerically solve for it. We will show that our integration scheme is accurate in solving for the set of scalar potentials used to re-express the diffusion tensor in differential form, and in solving for the diffusion coefficients in the larger LSH code. By using this diffusion tensor rather than the existing Papp one, and numerically integrating it with a Romberg method, we produce much more accurate, self-consistent results.

70 PLASMA PHYSICS AND FUSION TECHNOLOGY