Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “Parallel application”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 901 records · Page 50
All sourcesNASA NTRSDOE OSTI
DOE OSTI2020

Making ultrastrong steel tough by grain-boundary delamination

Developing ultrahigh-strength steels that are ductile, fracture resistant, and cost effective would be attractive for a variety of structural applications. We show that improved fracture resistance in a steel with an ultrahigh yield strength of nearly 2 gigapascals can be achieved by activating delamination toughening coupled with transformation-induced plasticity. Delamination toughening associated with intensive but controlled cracking at manganese-enriched prior-austenite grain boundaries normal to the primary fracture surface dramatically improves the overall fracture resistance. As a result, fracture under plane-strain conditions is automatically transformed into a series of fracture processes in “parallel” plane-stress conditions through the thickness. The present “high-strength induced multidelamination” strategy offers a different pathway to develop engineering materials with ultrahigh strength and superior toughness at economical materials cost.

36 MATERIALS SCIENCE↗
DOE OSTI2020

kynema-sgf [SWR-20-85]

Kynema-SGF (formerly AMR-Wind), wherein SGF stands for structured-grid fluid dynamics, is a massively parallel, block-structured adaptive-mesh, incompressible flow solver. The codebase was initiated in 2019 from incflo. The solver is built on top of the AMReX library. AMReX library provides the mesh data structures, mesh adaptivity, as well as the linear solvers used for solving the governing equations. Kynema-SGF is actively developed and maintained by a dedicated multi-institutional team from Lawrence Berkeley National Laboratory, National Laboratory of the Rockies, and Sandia National Laboratories. The primary applications for Kynema-SGF are: performing large-eddy simulations (LES) of atmospheric boundary layer (ABL) flows, simulating wind farm turbine-wake interactions using actuator disk or actuator line models for turbines, and as a background solver when coupled with a near-body solver (e.g., Kynema-UGF) with overset methodology to perform blade-resolved simulations of multiple wind turbines within a wind farm. For offshore applications, the ability to model the air-sea interaction effects and its impact on the ABL characteristics is another focus for the code development effort. As with other codes in the Kynema ecosystem, Kynema-SGF shares the following objectives: *an open, well-documented implementation of the state-of-the-art computational models for modeling wind farm flow physics at various fidelities that are backed by a comprehensive verification and validation (V&V) process; *be capable of performing the highest-fidelity simulations of flow fields within wind farms; and *be able to leverage the high-performance leadership class computing facilities available at DOE national laboratories.

Ananthan, Shreyas↗
DOE OSTI2020

US Department of Energy, Office of Science High Performance Computing Facility Operational Assessment 2019 Oak Ridge Leadership Computing Facility

Oak Ridge National Laboratory's (ORNL's) Leadership Computing Facility (OLCF) continues to surpass its operational target goals: supporting users; delivering fast, reliable computational ecosystems; creating innovative solutions for high performance computing (HPC) needs; and managing risks, safety, and security associated with operating some of the most powerful computers in the world. The results can be seen in the cutting-edge science conducted by users and the praise from the research community. Calendar year (CY) 2019 was a big year as OLCF staff ran five world-class resources (the leadershipclass computers Titan and Summit, the large analysis cluster called Eos, and the massive parallel filesystems called Atlas and Alpine)) and also began power and cooling upgrades for a 2021 exascale system called Frontier. While continuing exceptional operation of Titan, Eos, and Rhea, the OLCF released the Summit supercomputer for production on January 1, 2019. Summit debuted as the most capable and efficient system in its class and has been recognized as the most powerful system in the world for its performance on both the high performance linpack (HPL) and conjugate gradient (HPCG) benchmark applications since June 2018 according to TOP500. Summit represents the culmination of a multiyear effort between the OLCF, IBM, NVIDIA, and Mellanox to deliver a system that is unmatched for modeling, simulation, data analysis, and learning. To hit the ground running with science-ready applications on day one, application teams worked closely with the OLCF through the Center for Accelerated Application Readiness (CAAR) program for years in advance of the Summit deployment. CY 2019 was filled with outstanding results and accomplishments: a very high rating from users on overall satisfaction for the sixth year in a row; a tremendous amount of core-hours delivered to researchers from two leadership-class systems; and success in delivering on the allocation split of roughly 60%, 30%, and 10% of core-hours offered for the Innovative and Novel Computational Impact on Theory and Experiment (INCITE), Advanced Scientific Computing Research Leadership Computing Challenge (ALCC), and Director's Discretionary (DD) programs, respectively (see Operational Performance section). These accomplishments, coupled with the high utilization rates (overall and capability usage), represent the fulfillment of the promise of both leadership-class machines: efficient facilitation of leadership-class computational applications. Table ES.1 presents a summary of the 2019 OLCF metric targets and the associated results. More information can be found in the Operational Performance section for each OLCF resource. The scientific accomplishments of OLCF users are a strong indication of long-term operational success, with publications this year in such notable journals and publications as Nature, Nature Physics, Nature Plants, Physical Review X, Journal of the American Physical Society, Cell, Nano Letters, and Trends in Biotechnology. Crucial domain-specific discoveries facilitated by resources at the OLCF are described in the High Performance Computing Facility Operational Assessment 2019 Oak Ridge Leadership Computing Facility (OAR) Strategic Results section. For example, researchers used Summit to pinpoint and understand the production of proteins from genetic information, including mutations and the functional expression of disease (Section 8.2).

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2022

Efficient reconstruction and validation of heterogeneous microstructures for energy applications

The digital reconstruction of microstructures is necessary for simulations in fields ranging from geology to electrochemistry, but the state-of-the-art digital reconstruction techniques often compromise between resolution and field of view. It is challenging to retain detailed microstructure information in large-scale reconstructions. Here, this study investigates different aspects of the Yeong-Torquato algorithm based on correlation functions to make it more efficient. We achieve this goal by reducing the computational complexity of the chord-length distribution function and the two-point correlation function, applying the random sphere-packing method as the initial condition, and restricting potential voxel swaps to interfaces. In addition, a novel superposition parallel scheme is introduced to aid in searching for potential voxel swaps. The algorithm proposed is validated by comparing the pore-size distributions of reconstructed 3D custom battery electrodes from a sample dataset obtained from transmission X-ray microscopy. From a sample image with 200 x 200 pixels, the code can reconstruct a 300 x 300 x 300 structure in under 22 h and reconstruct a 400 x 400 x 400 structure in 43 h with eight cores.

42 ENGINEERING↗
DOE OSTI2020

Experimental apparatus and methodology to test and quantify thermal performance of micro and macro-encapsulated phase change materials in building envelope applications

Thermal energy storage (TES) is used as a viable technology to shift peak electricity demand caused by the space cooling requirements in buildings. Passive TES is implemented in building envelope via micro and macroencapsulation methods. This study describes the use of a state-of-the-art laboratory to test different PCM inclusions in simplified building walls. A microencapsulated PCM and two macroencapsulated PCMs are tested in a controlled environment to gather data for validation purposes of PCM modelling algorithms in building energy modelling programs. Data indicates that the chamber environment and the heating and cooling system can conduct full-cycle tests of wall panels with PCM inclusions. This study also generates data from parallel tests on 4 wall panels which can be used in building energy modelling programs to validate the PCM modelling algorithms. The cyclic tests also capture the thermal effects of PCMs and complex PCM behaviors like sub-cooling in PCM hydrate-salts.

25 ENERGY STORAGE↗
DOE OSTI2022

Visualization of defect induced in-gap states in monolayer MoS 2

Atomic-scale intrinsic defects play a key role in controlling functional electronic properties of two-dimensional (2D) materials. Here, we present a low-temperature scanning–tunneling microscopy and spectroscopy investigation of a common point-defect in monolayer molybdenum disulfide (MoS 2 ). We employ a sample preparation method in which the film surface is never exposed to air so that the native dangling bonds surrounding the defects in the film are preserved. Molybdenum vacancies are identified by their three characteristic in-gap resonances by combining scanning–tunneling measurements with parallel Green’s function-based theoretical modeling. The relative energy shifts between the various in-gap states allow us to identify a relative charge difference between two of the observed vacancies. The role of the substrate on the band structure of the defective MoS 2 monolayer is unveiled. Our study highlights the effects of the substrate on the in-gap states of common defects found in MoS 2 providing a pathway in designing and optimizing 2D materials for electronic applications.

36 MATERIALS SCIENCE↗
DOE OSTI2020

A fast and large bandwidth superconducting variable coupler

Variable microwave-frequency couplers are highly useful components in classical communication systems and likely will play an important role in quantum communication applications. Conventional semiconductor-based microwave couplers have been used with superconducting quantum circuits, enabling, for example, the in situ measurements of multiple devices via a common readout chain. However, the semiconducting elements are lossy and furthermore dissipate energy when switched, making them unsuitable for cryogenic applications requiring rapid, repeated switching. Superconducting Josephson junction-based couplers can be designed for dissipation-free operation with fast switching and are easily integrated with superconducting quantum circuits. These enable on-chip, quantum-coherent routing of microwave photons, providing an appealing alternative to semiconductor switches. Here, we present and characterize a chip-based broadband microwave variable coupler, tunable over 4-8GHz with over 1.5GHz instantaneous bandwidth, based on the superconducting quantum interference device with two parallel Josephson junctions. The coupler is dissipation-free and features large on-off ratios in excess of 40dB, and the coupling can be changed in about 10ns. The simple design presented here can be readily integrated with superconducting qubit circuits and can be easily generalized to realize a four- or more port device.

75 CONDENSED MATTER PHYSICS, SUPERCONDUCTIVITY AND↗
DOE OSTI2023

SERGHEI (SERGHEI-SWE) v1.0: a performance-portable high-performance parallel-computing shallow-water solver for hydrology and environmental hydraulics

The Simulation EnviRonment for Geomorphology, Hydrodynamics, and Ecohydrology in Integrated form (SERGHEI) is a multi-dimensional, multi-domain, and multi-physics model framework for environmental and landscape simulation, designed with an outlook towards Earth system modelling. At the core of SERGHEI's innovation is its performance-portable high-performance parallel-computing (HPC) implementation, built from scratch on the Kokkos portability layer, allowing SERGHEI to be deployed, in a performance-portable fashion, in graphics processing unit (GPU)-based heterogeneous systems. In this work, we explore combinations of MPI and Kokkos using OpenMP and CUDA backends. In this contribution, we introduce the SERGHEI model framework and present with detail its first operational module for solving shallow-water equations (SERGHEI-SWE) and its HPC implementation. This module is designed to be applicable to hydrological and environmental problems including flooding and runoff generation, with an outlook towards Earth system modelling. Its applicability is demonstrated by testing several well-known benchmarks and large-scale problems, for which SERGHEI-SWE achieves excellent results for the different types of shallow-water problems. Finally, SERGHEI-SWE scalability and performance portability is demonstrated and evaluated on several TOP500 HPC systems, with very good scaling in the range of over 20 000 CPUs and up to 256 state-of-the art GPUs.

58 GEOSCIENCES↗
DOE OSTI2022

MOOSE framework enhancements for meshing reactor geometries

MOOSE is an open-source, parallel finite element framework designed to permit rapid development of robust multi-physics modeling capabilities. Under the DOE-NEAMS program, numerous solvers have been developed utilizing the open-source MOOSE framework for multiphysics reactor analysis. These solvers require input finite element meshes representing the discretized geometry. Typically, reactor analysts turn to licensed external tools for creation of reactor geometry meshes. Recently, enhancements have been added to the MOOSE framework to mesh common reactor geometries and improve MOOSE-based application user workflows. Support for hexagonal pins, assemblies, and cores has been added, and Cartesian support has been extended. Options for modeling static and rotating control drums within a hexagonal assembly are now available. Pin, assembly, and plane regions can be identified through automatically applied tags on the mesh called 'reporting IDs' for easier post- processing of physics results. An external open-source triangle routine has been leveraged within MOOSE to mesh core periphery zones. A set of reactor geometry builder routines further streamline the construction of hexagonal and Cartesian cores and include the ability to assign materials to regions during mesh generation. The new meshing routines are available through the MOOSE framework in the open-source 'Reactor' module, and the resulting directly within MOOSE-based applications or exported as Exodus II files for use in other finite element solvers. (authors)

22 GENERAL STUDIES OF NUCLEAR REACTORS↗
DOE OSTI2021

Resiliency in numerical algorithm design for extreme scale simulations

Here this work is based on the seminar titled ‘Resiliency in Numerical Algorithm Design for Extreme Scale Simulations’ held March 1–6, 2020, at Schloss Dagstuhl, that was attended by all the authors. Advanced supercomputing is characterized by very high computation speeds at the cost of involving an enormous amount of resources and costs. A typical large-scale computation running for 48 h on a system consuming 20 MW, as predicted for exascale systems, would consume a million kWh, corresponding to about 100k Euro in energy cost for executing 10 23 floating-point operations. It is clearly unacceptable to lose the whole computation if any of the several million parallel processes fails during the execution. Moreover, if a single operation suffers from a bit-flip error, should the whole computation be declared invalid? What about the notion of reproducibility itself: should this core paradigm of science be revised and refined for results that are obtained by large-scale simulation? Naive versions of conventional resilience techniques will not scale to the exascale regime: with a main memory footprint of tens of Petabytes, synchronously writing checkpoint data all the way to background storage at frequent intervals will create intolerable overheads in runtime and energy consumption. Forecasts show that the mean time between failures could be lower than the time to recover from such a checkpoint, so that large calculations at scale might not make any progress if robust alternatives are not investigated. More advanced resilience techniques must be devised. The key may lie in exploiting both advanced system features as well as specific application knowledge. Research will face two essential questions: (1) what are the reliability requirements for a particular computation and (2) how do we best design the algorithms and software to meet these requirements? While the analysis of use cases can help understand the particular reliability requirements, the construction of remedies is currently wide open. One avenue would be to refine and improve on system- or application-level checkpointing and rollback strategies in the case an error is detected. Developers might use fault notification interfaces and flexible runtime systems to respond to node failures in an application-dependent fashion. Novel numerical algorithms or more stochastic computational approaches may be required to meet accuracy requirements in the face of undetectable soft errors. These ideas constituted an essential topic of the seminar. The goal of this Dagstuhl Seminar was to bring together a diverse group of scientists with expertise in exascale computing to discuss novel ways to make applications resilient against detected and undetected faults. In particular, participants explored the role that algorithms and applications play in the holistic approach needed to tackle this challenge. This article gathers a broad range of perspectives on the role of algorithms, applications and systems in achieving resilience for extreme scale simulations. The ultimate goal is to spark novel ideas and encourage the development of concrete solutions for achieving such resilience holistically.

79 ASTRONOMY AND ASTROPHYSICS↗
DOE OSTI2022

Assessing MP2 frozen natural orbitals in relativistic correlated electronic structure calculations

The high computational scaling with the basis set size and the number of correlated electrons is a bottleneck limiting applications of coupled cluster algorithms, in particular for calculations based on two- or four-component relativistic Hamiltonians, which often employ uncontracted basis sets. This problem may be alleviated by replacing canonical Hartree–Fock virtual orbitals by natural orbitals (NOs). Here, in this paper, we describe the implementation of a module for generating NOs for correlated wavefunctions and, in particular, second order Møller–Plesset perturbation frozen natural orbitals (MP2FNOs) as a component of our novel implementation of relativistic coupled cluster theory for massively parallel architectures [Pototschnig et al. J. Chem. Theory Comput. 17, 5509, (2021)]. Our implementation can manipulate complex or quaternion density matrices, thus allowing for the generation of both Kramers-restricted and Kramers-unrestricted MP2FNOs. Furthermore, NOs are re-expressed in the parent atomic orbital (AO) basis, allowing for generating coupled cluster singles and doubles NOs in the AO basis for further analysis. By investigating the truncation errors of MP2FNOs for both the correlation energy and molecular properties—electric field gradients at the nuclei, electric dipole and quadrupole moments for hydrogen halides HX (X = F–Ts), and parity-violating energy differences for H 2 Z 2 (Z = O–Se)—we find MP2FNOs accelerate the convergence of the correlation energy in a roughly uniform manner across the Periodic Table. It is possible to obtain reliable estimates for both energies and the molecular properties considered with virtual molecular orbital spaces truncated to about half the size of the full spaces.

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗
DOE OSTI2022

Confinement-Driven Heterogeneous Benzene Crystallization in Silica Nanopores

Nanoconfinement alters the thermodynamics, dynamics, and kinetics of fluids hosted in nanoscale solid nanopores to an extent that depends on the characteristics of the confining space and the chemistry of the confined fluids. Confinement-induced alterations in the phase behavior of confined energetic fluids under high pressure or low temperature are highly relevant to subsurface and subsea phenomena such as fluid flow in porous media, hydrate formation and dissociation, and gas storage capacity. Although extensive efforts have been directed toward understanding the phase behavior of confined fluids, the role of solid-liquid interfaces in the phase transitions of organic liquids has not been resolved yet. Here, we explore the onset and growth of benzene crystallization confined in 6 nm sized SBA-15 silica nanopores in the temperature range 300-200 K using in situ extended range small-angle and wide-angle X-ray scattering (SAXS/WAXS) measurements and atomistic classical molecular dynamics (MD) simulations. The crystallization onset of confined benzene depresses to 265 K compared to the freezing point of bulk benzene ( ~278 K), followed by the continuous growth of the emerged crystals in the pore space with complete crystallization at 200 K. The orientation of the emerged benzene crystals is dominated by parallel (π-π stacking) and perpendicular (T-shape stacking) orientations along the cylindrical pore radius and pore length, respectively. The onset of benzene crystals occurs heterogeneously on the pore surface and grows continuously toward the pore center. Further, confined benzene undergoes a dynamical crossover from fragile to strong dynamics behavior, inferred from the rotational and translational diffusion. The insights provided by this study have significant implications for the phase transitions of confined organic liquids that are relevant to a wide range of applications in the biological, geological, environmental, and chemical fields.

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗
DOE OSTI2024

Electron mobility of SnO 2 from first principles

The transparent conducting oxide SnO 2 is a wide bandgap semiconductor that is easily n-type doped and widely used in various electronic and optoelectronic applications. Experimental reports of the electron mobility of this material vary widely depending on the growth conditions and doping concentrations. In this work, we calculate the electron mobility of SnO 2 from first principles to examine the temperature and doping concentration dependence and to elucidate the scattering mechanisms that limit transport. We include both electron–phonon scattering and electron-ionized impurity scattering to accurately model scattering in a doped semiconductor. Furthermore, we find a strongly anisotropic mobility that favors transport in the direction parallel to the c-axis. At room temperature and intrinsic carrier concentrations, the low-energy polar-optical phonon modes dominate scattering, while ionized-impurity scattering dominates above 10 18 cm −3 .

36 MATERIALS SCIENCE↗
DOE OSTI2022

Characterization of 128 × 128 MM-PAD-2.1 ASIC: a fast framing hard x-ray detector with high dynamic range

Here, we characterize a new x-ray Mixed-Mode Pixel Array Detector (MM-PAD-2.1) Application Specific Integrated Circuit (ASIC). Using an integrating pixel front-end with dynamic charge removal architecture, the MM-PAD-2.1 ASIC extends the maximum measurable x-ray signal (in 20 keV photon units) to > 10 7 x-rays/pixel/frame while maintaining a low read noise across the full dynamic range, all while imaging continuously at a frame rate of up to 10 kHz. The in-pixel dynamic charge removal mechanism prevents saturation of the input amplifier and proceeds in parallel with signal integration to achieve deadtime-less measurements with incident x-ray rates of > 10 10 x-rays/pixel/s. The ASIC format consists of 128 × 128 square pixels each 150 μm on a side and is designed to be 3-side buttable so large arrays can be effectively tiled. Here we use both laboratory x-ray sources and the Cornell High Energy Synchrotron Source (CHESS) to characterize two single ASIC prototype detectors for both low (single x-ray) and high incident flux detection. In the first detector the ASIC was solder bump-bonded to a 500 μm thick Si sensor for efficient detection of x-rays below 20 keV, whereas the second detector used a 750 μm thick CdTe sensor for x-rays above ~20 keV.

36 MATERIALS SCIENCE↗
DOE OSTI2021

Noncollinear ferromagnetic Weyl semimetal with anisotropic anomalous Hall effect

An emerging frontier in condensed matter physics involves novel electromagnetic responses, such as the anomalous Hall effect (AHE), in ferromagnetic Weyl semimetals (FM-WSMs). Candidate FM-WSMs have been limited to materials that preserve inversion symmetry and generate Weyl crossings by breaking the time-reversal symmetry. These materials share three common features: a centrosymmetric lattice, a collinear FM ordering, and a large AHE observed when the field is parallel to the magnetic easy axis. Here, we present CeAlSi as a new type of FM-WSM in which the Weyl nodes are stabilized by breaking the inversion symmetry, but their positions are tuned by breaking the time-reversal symmetry. Unlike the other FM-WSMs, CeAlSi has a noncentrosymmetric lattice, a noncollinear FM ordering, and a novel AHE that is anisotropic between the easy and hard magnetic axes. It also exhibits large FM domains that are promising for exploring both device applications and the interplay between the Weyl nodes and FM domain walls.

75 CONDENSED MATTER PHYSICS, SUPERCONDUCTIVITY AND↗
DOE OSTI2024

Towards exascale for wind energy simulations

We examine large-eddy-simulation modeling approaches and computational performance of two open-source computational fluid dynamics codes for the simulation of atmospheric boundary layer flows that are of direct relevance to wind energy production. The first code, NekRS, is a high-order, unstructured-grid, spectral element code. The second code, AMR-Wind, is a second-order, block-structured, finite-volume code with adaptive mesh refinement capabilities. The objective of this study is to co-develop these codes in order to improve model fidelity and performance for each. These features will be critical for running ABL-based applications such as wind farm analysis on advanced computing architectures. To this end, we investigate the performance of NekRS and AMR-Wind on the Oak Ridge Leadership Facility supercomputers Summit, using 4 to 800 nodes (24 to 4,800 NVIDIA V100 GPUs), and Crusher, the testbed for the Frontier exascale system, using 18 to 384 Graphics Compute Dies on AMD MI250X GPUs. We compare strong- and weak-scaling capabilities, linear solver performance, and time to solution. We also identify leading inhibitors to parallel scaling.

17 WIND ENERGY↗
DOE OSTI2023

S4PST: Sustainability for Programming Systems and Tools: May Workshop Report

The US Department of Energy (DOE) Exascale Computing Project (ECP) has fostered and strengthened the use of modern software engineering practices for developing applications and libraries, and this effort has resulted in the coordinated and interoperable E4S1 and xSDK2 ecosystems. Although this approach is cost-effective, it relies on robust programming systems and tools (PST) as the underlying foundation for our HPC software. At present, our primary PST stack consists of traditional high-performance computing (HPC) languages, namely Fortran, C, C++, and the popular Python language for data analysis and AI workflows. These languages support various programming frameworks and run-time abstractions that enable parallelism and concurrency across multiple node architectures and thousands of nodes through a variety of interconnect systems. However, to accommodate users’ diverse needs, certain aspects of the HPC ecosystem are delegated to vendor-specific or third-party implementations that extend beyond a particular scientific domain. This broader scope results in a multitude of specifications and variations, which leads to a complex orchestration of many-ecosystems. Unfortunately, this complexity in the ecosystem imposes additional overhead costs on consumers during the latter stages of the development cycle. In addition to the software ecosystem challenge, the upcoming conclusion of the ECP by December 2023 has raised significant concerns within the HPC programming systems community, from both the economic and social perspectives. The ECP has implemented a management structure for software development and funding decisions across all ECP participants by following a conventional hierarchical and centralized approach. However, this structure has prompted certain considerations within the community, particularly in anticipation of the Software Sustainability initiative by the DOE’s Advanced Scientific Computing Research Program (ASCR). For the success of this new initiative, it is of utmost importance to secure consistent funding and foster close engagement with researchers and core developers of existing programming-system products. This collaboration is vital to maintaining the critical capabilities of the current software during the transition phase while proactively adapting to future technology and workforce trends. The community recognizes the significance of adapting to emerging trends and is aware of the inherent fragility of the HPC software ecosystem, particularly in relation to programming systems that cater to all users. The ability to adapt and evolve is essential to staying relevant and effectively addressing these technical, economic, and social challenges. The S4PST team, which represents one of the six ASCR Software Sustainability seedling projects, is dedicated to tackling these challenges through community-based approaches that go beyond the scope of the DOE. This involves collaboration between national laboratories with academia, non-DOE institutions, hardware and system vendors, and international partners. By fostering these partnerships, we aim to create a robust and sustainable HPC software ecosystem that can effectively meet the needs of the community. This new community effort, driven by the eight DOE labs, will take on the responsibility of guiding funding decisions for programming-systems development and maintenance with transparency and consistency across all decisions. Additionally, the team will offer common technical services to the programming systems community, irrespective of their funding situations, and facilitate community-wide incubation to proactively nurture the software ecosystem. By actively engaging with stakeholders and employing a collaborative approach, we can collectively shape the future of programming systems and ensure a robust and thriving HPC software landscape. On May 11–12, 2023, the S4PST team conducted its inaugural kick-off workshop at the Innovative Computing Laboratory (ICL) in the University of Tennessee, Knoxville, hosted by Hartwig Anzt. The workshop encompassed various sessions dedicated to presentations and discussions, with the aim of comprehending the team members’ perspectives on the vision of software sustainability. Additionally, the workshop aimed to identify the technical, economic, and social requirements for sustaining the programming-systems community in the field of HPC. This report provides a summary of the S4PST effort by highlighting five major thrust areas discussed during the workshop: (i) community, (ii) technical support, (iii) training and diversity, (iv) verification, validation and correctness, and (v) emerging technologies. It also encompasses an overview of the presentations and discussions held throughout the event, our views and potential synergies with other seedling efforts, along with the outcomes and key takeaways from our initial discussions.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2024

Optimizing Distributed Training on Frontier for Large Language Models

Large language models (LLMs) have demonstrated remarkable success as foundational models, benefiting various downstream applications through fine-tuning. Loss scaling studies have demonstrated the superior performance of larger LLMs compared to their smaller counterparts. Nevertheless, training LLMs with billions of parameters poses significant challenges and requires considerable computational resources. For example, training a one trillion parameter GPT-style model on 20 trillion tokens requires a staggering 120 million exaflops. This research explores efficient distributed training strategies to extract this computation from Frontier, the world's first exascale supercomputer. We enable and investigate various model and data parallel training techniques, such as tensor parallelism, pipeline parallelism, and sharded data parallelism, to facilitate training a trillion-parameter model on Frontier. We empirically assess these techniques and their associated parameters to determine their impact on memory footprint, communication latency, and GPU's computational efficiency. We analyze the complex interplay among these techniques and find a strategy to combine them to achieve high throughput through hyperparameter tuning. We have identified efficient strategies for training large LLMs of varying sizes through empirical analysis and hyperparameter tuning. For 22 Billion, 175 Billion, and 1 Trillion parameters, we achieved GPU throughputs of 38.38%, 36.14%, and 31.96%, respectively. For the training of the 175 Billion parameter model and the 1 Trillion parameter model, we achieved 100% weak scaling efficiency on 1024 and 3072 Mi250X GPUs, respectively. We also achieved strong scaling efficiencies of 89% and 87% for these two models. We trained these models only tens of iterations instead of training till completion.

Yin, Junqi↗