Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “Distributed training”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 1,135 records · Page 63
All sourcesNASA NTRSDOE OSTI
DOE OSTI2024

Automating the Analysis of Large Language Models Responses through Zero-Shot Question Answering

Recent advancements in Large Language Models (LLMs) have shown significant potential in various applications, yet their evaluation, particularly in zero-shot question answering scenarios, remains a challenging task. In this study, our objective was to explore precision metrics for Large Language Models (LLM) and design and implement a software pipeline to automatically evaluate LLMs' outputs under zero-shot question answering. Zero-shot question answering involves a model providing answers to questions about topics it hasn't seen during training. It leverages the principles of zero-shot learning by relying on semantic understanding and generalization from related knowledge. The data used was metadata from medical databases on congenital heart disease. We explored eleven LLM metrics and selected three for our evaluation: BLEU, BERTScore, and MoverScore. BLEU calculates a score based on the overlap of n-grams (contiguous sequences of n items, typically words) between the machine-generated translation and the reference translations. Higher BLEU scores indicate better correspondence between the machine-generated and human-generated translations. BERTScore is a metric used to evaluate the quality of machine-generated text by measuring the similarity of token embeddings produced by BERT (Bidirectional Encoder Representations from Transformers) between the generated text and reference text. MoverScore is a metric that quantifies the dissimilarity between the distributions of word embeddings from machine-generated text and reference text, emphasizing semantic similarity over exact token overlap. We also introduced HBKI, a composite metric summarizing these approaches. We tested five models —GPT-3, Llama-2, Gemini 1.5 Pro, Solar 10.7B, and Mixtral-8x7b. Our software pipeline, designed and implemented using Object-Oriented Programming principles, allows users to customize the selection and extraction of features for topics of interest in their own research. Our results show that MoverScore delivered the most precise evaluation of the LLM's outputs, while Mixtral-8x7b achieved the best overall performance in extracting metadata from the databases.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2020

SuperCDMS Event Reconstruction Using Convolutional Neural Networks [Thesis]

The SuperCDMS experiment uses cryogenic silicon and germanium detectors to search for dark matter candidates such as WIMPs (Weakly Interacting Massive Particles) streaming through the Earth. Collisions in the silicon and germanium crystals are expected to produce phonons whose thermal signatures can be measured. This thesis first describes the integration of a new Signal Distribution Unit (SDU) to the SuperCDMS data acquisition system, which allows for synchronization of multiple detectors and electronic/mechanical noise characterization via accelerometer, antenna, and AC phase measurements. From SuperCDMS detector data it is necessary to reconstruct the energies of the particle events. This thesis explores the use of Convolutional Neural Networks (CNNs) to perform this reconstruction and finds that, although they perform well, changing the noise model breaks the model and requires the neural network to be retrained. In order to mitigate this issue, a new CNN model is proposed which includes the noise Power Spectral Density (PSD) of the data as an additional input to the CNN. While it proves to be effective as a denoising algorithm, it still fails for data with a different noise model. However, including data from multiple PSDs in the neural network training sample allows it to handle data with different types of noise while still maintaining the quality of the reconstruction. Nevertheless, neural networks trained even on multiple PSDs do not robustly handle data taken with PSDs dissimilar to those in the training sample, suggesting that CNNs may need to be retrained whenever the noise environment changes in a significant way.

46 INSTRUMENTATION RELATED TO NUCLEAR SCIENCE AND ↗
DOE OSTI2025

Signal-preserving CMB component separation with machine learning

Analysis of microwave sky signals, such as the cosmic microwave background, often requires component separation using multifrequency methods, whereby different signals are isolated according to their different frequency behaviors. Many so-called blind methods, such as the internal linear combination (ILC), make minimal assumptions about the spatial distribution of the signal or contaminants, and only assume knowledge of the frequency dependence of the signal. The ILC produces a minimum-variance linear combination of the measured frequency maps. In the case of Gaussian, statistically isotropic fields, this is the optimal linear combination, as the variance is the only statistic of interest. However, in many cases the signal we wish to isolate, or the foregrounds we wish to remove, are non-Gaussian and/or statistically anisotropic (in particular for the case of Galactic foregrounds). In such cases, it is possible that machine learning (ML) techniques can be used to exploit the non-Gaussian features of the foregrounds and thereby improve component separation. However, many ML techniques require the use of complex, difficult-to-interpret operations on the data. We propose a hybrid method whereby we train an ML model using only combinations of the data that , and combine the resulting ML-predicted foreground estimate with the ILC solution to reduce the error from the ILC. We demonstrate our methods on simulations of extragalactic temperature and Galactic polarization foregrounds and show that our ML model can exploit non-Gaussian features, such as point sources and spatially varying spectral indices, to produce lower-variance maps than ILC—e.g., reducing the variance of the B-mode residual by factors of up to 5—while preserving the signal of interest in an unbiased manner. Moreover, we often find improved performance even when applying our ML technique to foreground models on which it was not trained. Published by the American Physical Society 2025

McCarthy, Fiona (ORCID:0000000253893565)↗
DOE OSTI2022

Wavelet and Deep-Learning-Based Approach for Generation System Problematic Parameters Identification and Calibration

Accurate models of generation systems are critical for maintaining reliable and secure grid operations. In this paper, a novel and systematic approach is proposed to identify and calibrate the generation system problematic parameters using continuous wavelet transform (CWT) and advanced deep-learning technology. The phasor measurement unit (PMU) data are used through “event playback” to check whether the parameter calibration is required, and if yes, a group of suspicious parameters will be identified as the primary problematic parameter candidates (PPCs). These primary PPCs are randomly perturbed to generate the event playback simulation data, which are used by the CWT and convolutional neural networks (CNNs) to further narrow down the primary PPCs into a smaller set of candidates. Then, the identified candidates are perturbed again to generate massive event playback simulation data for training a parameter calibration neural network. Here, we designed a multi-output neural network structure to find the mappings between the perturbed parameters and the simulation data using both CNN and long short-term memory (LSTM) models. Finally, the well-trained and tested CNN-LSTM model is used to estimate the accurate value of the suspicious parameters with actual PMU measurements. The proposed CNN-LSTM network can accurately and reliably estimate the generation-system problematic parameters, and has better performance when compared to other machine-learning methods, such as the multilayer perceptron network and the conditional variational autoencoder method. The accuracy and effectiveness of the proposed approach have been validated through simulation and real-world data.

24 POWER TRANSMISSION AND DISTRIBUTION↗
DOE OSTI2022

Seeking Repeating Anthropogenic Seismic Sources: Implications for Seismic Velocity Monitoring at Fault Zones

Abstract Seismic velocities in rocks are highly sensitive to changes in permanent deformation and fluid content. The temporal variation of seismic velocity during the preparation phase of earthquakes has been well documented in laboratories but rarely observed in nature. It has been recently found that some anthropogenic, high‐frequency (>1 Hz) seismic sources are powerful enough to generate body waves that travel down to a few kilometers and can be used to monitor fault zones at seismogenic depth. Anthropogenic seismic sources typically have fixed spatial distribution and provide new perspectives for velocity monitoring. In this work, we propose a systematic workflow to seek such powerful seismic sources in a rapid and straightforward manner. We tackle the problem from a statistical point of view, considering that persistent, powerful seismic sources yield highly coherent correlation functions (CFs) between pairs of seismic sensors. The algorithm is tested in California and Japan. Multiple sites close to fault zones show high‐frequency CFs stable for an extended period of time. These findings have great potential for monitoring fault zones, including the San Jacinto Fault and the Ridgecrest area in Southern California, Napa in Northern California, and faults in central Japan. However, extra steps, such as beamforming or polarization analysis, are required to determine the dominant seismic sources and study the source characteristics, which are crucial to interpreting the velocity monitoring results. Train tremors identified by the present approach have been successfully used for seismic velocity monitoring of the San Jacinto Fault in previous studies.

58 GEOSCIENCES↗
DOE OSTI2020

Automatic log analysis with NLP for the CMS workflow handling

The central Monte-Carlo production of the CMS experiment utilizes the WLCG infrastructure and manages daily thousands of tasks, each up to thousands of jobs. The distributed computing system is bound to sustain a certain rate of failures of various types, which are currently handled by computing operators a posteriori. Within the context of computing operations, and operation intelligence, we propose a Machine Learning technique to learn from the operators with a view to reduce the operational workload and delays. This work is in continuation of CMS work on operation intelligence to try and reach accurate predictions with Machine Learning. We present an approach to consider the log files of the workflows as regular text to leverage modern techniques from Natural Language Processing (NLP). In general, log files contain a substantial amount of text that is not human language. Therefore, different log parsing approaches are studied in order to map the log files’ words to high dimensional vectors. These vectors are then exploited as feature space to train a model that predicts the action that the operator has to take. This approach has the advantage that the information of the log files is extracted automatically and the format of the logs can be arbitrary. In this work the performance of the log file analysis with NLP is presented and compared to previous approaches.

Layer, Lukas↗
DOE OSTI2025

Generative learning for slow manifolds and bifurcation diagrams

In dynamical systems characterized by separation of time scales, the approximation of so called “slow manifolds”, on which the long term dynamics lie, is a useful step for model reduction. Initializing on such slow manifolds is a useful step in modeling, since it circumvents fast transients, and is crucial in multiscale algorithms (like the equation-free approach) alternating between fine scale (fast) and coarser scale (slow) simulations. In a similar spirit, when one studies the infinite time dynamics of systems depending on parameters, the system attractors (e.g., its steady states) lie on bifurcation diagrams (curves for one-parameter continuation, and more generally, on manifolds in state parameter space. Sampling these manifolds gives us representative attractors (here, steady states of ODEs or PDEs) at different parameter values. Algorithms for the systematic construction of these manifolds (slow manifolds, bifurcation diagrams) are required parts of the “traditional” numerical nonlinear dynamics toolkit. In more recent years, as the field of Machine Learning develops, conditional score-based generative models (cSGMs) have been demonstrated to exhibit remarkable capabilities in generating plausible data from target distributions that are conditioned on some given label. It is tempting to exploit such generative models to produce samples of data distributions (points on a slow manifold, steady states on a bifurcation surface) conditioned on (consistent with) some quantity of interest (QoI, observable). In this work, we present a framework for using cSGMs to quickly (a) initialize on a low-dimensional (reduced-order) slow manifold of a multi-time-scale system consistent with desired value(s) of a QoI (a “label”) on the manifold, and (b) approximate steady states in a bifurcation diagram consistent with a (new, out-of-sample) parameter value. This conditional sampling can help uncover the geometry of the reduced slow-manifold and/or approximately “fill in” missing segments of steady states in a bifurcation diagram. Finally, the quantity of interest, which determines how the sampling is conditioned, is either known a priori or identified using manifold learning-based dimensionality reduction techniques applied to the training data.

Dynamical systems↗
DOE OSTI2021

AENET–LAMMPS and AENET–TINKER : Interfaces for accurate and efficient molecular dynamics simulations with machine learning potentials

Machine-learning potentials (MLPs) trained on data from quantum-mechanics based first-principles methods can approach the accuracy of the reference method at a fraction of the computational cost. To facilitate efficient MLP-based molecular dynamics and Monte Carlo simulations, an integration of the MLPs with sampling software is needed. Here, we develop two interfaces that link the atomic energy network (ænet) MLP package with the popular sampling packages TINKER and LAMMPS. The three packages, ænet, TINKER, and LAMMPS, are free and open-source software that enable, in combination, accurate simulations of large and complex systems with low computational cost that scales linearly with the number of atoms. Scaling tests show that the parallel efficiency of the ænet–TINKER interface is nearly optimal but is limited to shared-memory systems. The ænet–LAMMPS interface achieves excellent parallel efficiency on highly parallel distributed memory systems and benefits from the highly optimized neighbor list implemented in LAMMPS. We demonstrate the utility of the two MLP interfaces for two relevant example applications: the investigation of diffusion phenomena in liquid water and the equilibration of nanostructured amorphous battery materials.

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗
DOE OSTI2023

Regional Medium-Term Hourly Electricity Demand Forecasting Based on LSTM

This paper aims to forecast high-resolution (hourly) aggregated load for a certain region in the medium term (a few days to over a year). One region is defined as some places with similar climate characteristics because the climate influences people's daily lifestyles and hence the electric usage. We decom- pose the electric usage records into two parts: base load and seasonal load. Considering both temperature and time factors, different deep learning methods are adopted to characterize them. The first goal of our approach is to predict the peak load which is critical for power system planning. Furthermore, our proposed forecast method can provide the depiction of the hourly load profile to provide customized load curves for high- level real-time applications. The proposed method is tested on real-world historical data collected by CAISO, BPA, and PACW. The experimental results show that trained by three years of data, our method could reduce the prediction error for one-year lead hourly load below 5% MAPE, and predict the occurrence of the peak load for next year in CAISO with an error within three days. Furthermore, as a byproduct, an interesting observation on the impact of COVID-19 on human life was made and discussed based on these case studies.

deep learning↗
DOE OSTI2024

Design and optimization of a modular hydrogen-based integrated energy system to maximize revenue via nuclear-renewable sources

Here, this paper demonstrates a novel modular distributed framework that uses optimal energy-dispatching strategies to enable greater flexibility and profitability in nuclear-renewable integrated energy systems (NR-IES). Hydrogen is used as a commodity in this framework since its production can improve grid stability and system operational flexibility, decarbonize heavy industry, and create an additional revenue stream for electricity generators, particularly nuclear power plants with high operational expenses. The proposed solution addresses the challenges associated with merging multiple software and services from various domains by using functional mock-up units (FMU) to co-simulate diverse subsystems designed in various platforms. The tightly coupled integrated energy system (IES) is optimized to maximize revenue by utilizing the deep reinforcement learning (DRL) technique to make smart dispatching decisions based on variable electricity prices and the availability of renewable energy. Proximal policy optimization (PPO) algorithm is used in training and testing the DRL agent. Over a period of 120 days, the proposed hydrogen-based IES framework showed about 10% revenue boost compared to a non-hydrogen generating baseline IES while also providing an easily-adoptable framework which can help to improve the flexibility of future generation nuclear power plants.

29 ENERGY PLANNING, POLICY, AND ECONOMY↗
DOE OSTI2024

Gray-Box Modeling for Distribution Systems With Inverter-Based Resources: Integrating Physics-Based and Data-Driven Approaches

Here, in this paper, we develop a novel gray-box modeling approach for distribution systems with inverter-based resources (IBRs). The proposed gray-box modeling method aims to improve estimation accuracy by taking advantages of both physics-based (white-box) and data-driven (black-box) modeling approaches. To this end, we utilize partial physical knowledge of the system, including the inverters’ structures and control diagrams, as well as the equivalent network model simplified through Kron reduction. The white-box model containing unknown parameters is then constructed with mathematical equations and an optimization-based method is subsequently employed to identify these unknown parameters within the white-box model. Next, the graybox modeling framework is then constructed by embedding the output variables of the white-box model into the input vector of a black-box model (represented using a neural network). Finally, the black-box section is trained using the collected input-output datasets and the gray-box model is then obtained. Furthermore, case studies demonstrate that our gray-box modeling approach effectively improves estimation accuracy compared to purely physics-based or data-driven methods.

42 ENGINEERING↗
DOE OSTI2023

Regional Medium-Term Hourly Electricity Demand Forecasting Based on LSTM: Preprint

This paper aims to forecast high-resolution (hourly) aggregated load for a certain region in the medium term (a few days to over a year). One region is defined as some places with similar climate characteristics because the climate influences people's daily lifestyles and hence the electric usage. We decompose the electric usage records into two parts: base load and seasonal load. Considering both temperature and time factors, different deep-learning methods are adopted to characterize them. The first goal of our approach is to predict the peak load which is critical for power system planning. Furthermore, our proposed forecast method can provide the depiction of the hourly load profile to provide customized load curves for high-level real-time applications. The proposed method is tested on real-world historical data collected by CAISO, BPA, and PACW. The experimental results show that trained by three years of data, our method could reduce the prediction error for a one-year lead hourly load below 5% MAPE, and predict the occurrence of the peak load for next year in CAISO with an error within three days. Furthermore, as a byproduct, an interesting observation on the impact of COVID-19 on human life was made and discussed based on these case studies.

deep learning↗
DOE OSTI2025

EMPOWERED Distributed Energy Resources Permit Accelerator Pilot

This report summarizes the EMPOWERED Distributed Energy Resources (DER) Permit Accelerator Pilot project. Its primary objective was to streamline the design, installation, permitting, and inspections of targeted Distributed Energy Resource (DER) solutions. Streamlining these processes is critical to strengthening and accelerating the adoption of DER solutions nationwide. Moreover, streamlined processes help expedite the clean energy transition and expand energy resilience by improving understanding and capability among key stakeholders like property owners, code officials, and the general workforce. The project team accomplished this by developing and deploying a set of design and permitting guides for simple DER solutions, which they rolled out via a permitting pilot program. The project team developed nine permitting and inspection guides for simple DER solutions, as follows: 1. Single-Family and Duplex: Electric Vehicle Service Equipment (EVSE), and Storage, Solar + Storage (each in two code cycles) 2. Multi-family and Office: EVSE, Storage, Solar + Storage (each in two code cycles) 3. A permitting process guide to support the implementation of the permitting guides. The guides cover key code requirements via plan review and field inspection checklists for code officials and building owners to follow to streamline the permitting and inspections process for DER solutions. Following the development of the initial set of guides, the project team formed two cohorts, each made up of four jurisdictions, to participate in the Permit Accelerator Pilot. Each cohort received training and technical assistance to support them in incorporating the guides into use. The project successfully engaged jurisdictions in two disparate US regions: Chelsea, Somerville, Natick, and Norwood in metro Boston; and Pima County, Town of Gilbert, Flagstaff, and Sedona in Arizona. Feedback gathered from the project was valuable in refining the guides to better address jurisdictional needs. After the pilot period, the project team developed final versions of the guides based on extensive feedback from the cohorts and from external technical peer review. Feedback included needs and barriers related to implementation, as well as detailed technical improvements from external peer review. The final guides were presented in a series of webinars with a national reach, with 192 attendees from 35 states in the live session of the final webinar, and the final guides are posted online and available for use nationwide. Key findings: • The most receptive jurisdictions were those adjacent to other jurisdictions that had already adopted similar initiatives. The least receptive jurisdictions were those in the midst of adopting new code cycles. • The most common barrier to jurisdictional adoption of new guidelines were internal administrative and process delays, more so than technical barriers. • The main technical barrier to adoption proved to be the concern of fire hazard risks posed by energy storage systems. There is some apprehension in the code enforcement community about the safety of the batteries. Education coordinated with fire and safety services will help alleviate concerns and increase confidence in the acceptance of innovative technologies. • Opportunities to further leverage and advance the guides and related resources include outreach, education, and jurisdictional support; updates to the resources to align with code cycle changes and local requirements; and advancements supporting emerging technologies.

14 SOLAR ENERGY↗
DOE OSTI2021

Deep Learning-Based Adaptive Remedial Action Scheme with Security Margin for Renewable-Dominated Power Grids

The Remedial Action Scheme (RAS) is designed to take corrective actions after detecting predetermined conditions to maintain system transient stability in large interconnected power grids. However, since RAS is usually designed based on a few selected typical operating conditions, it is not optimal in operating conditions that are not considered in the offline design, especially under frequently and dramatically varying operating conditions due to the increasing integration of intermittent renewables. The deep learning-based RAS is proposed to enhance the adaptivity of RAS to varying operating conditions. During the training, a customized loss function is developed to penalize the negative loss and suggest corrective actions with a security margin to avoid triggering under-frequency and over-frequency relays. Simulation results of the reduced United States Western Interconnection system model demonstrate that the proposed deep learning–based RAS can provide optimal corrective actions for unseen operating conditions while maintaining a sufficient security margin.

24 POWER TRANSMISSION AND DISTRIBUTION↗
DOE OSTI2020

Cybersecurity for Distance Relay Protection

This project is a DOE follow-up effort on the CREDC workshop held on September 13, 2018 in Cambridge, MA to discuss cybersecurity of distance relays, which considered the benefits, vulnerabilities and risk mitigations for the use of communication systems in power system protection. The objectives of this project are to define the taxonomy of relay protection and associated communications; define use cases describing approaches to reduce the cyber-attack surface on those protective relays; and evaluate the loss of operational functional capability from changes to communication coverage. Mitigating controls will also be evaluated to understand if there are other approaches to reduce attack surfaces while maintaining communications or partial communications. Distance relays are used to protect transmission lines of approximately 10 to 300 miles in length, by detecting short circuits (i.e., faults) on the lines and then tripping circuit breakers in the substation. Such protection systems are a subset of the power system and they incorporate sensing, logic and communication functions. Protection system exposure to cyberattack could be drastically limited by disconnecting relays from all vulnerable communication systems, but this may adversely impact overall power system performance in the absence of cyberattack. This project began with a use case analysis of protection systems with communications, as summarized in this report. It continued with modeling, testing and evaluation in a miniature power system (MPS), located in the Western Area Power Administration (WAPA) Electric Power Training Center (EPTC). The project also incorporated feedback from two industry meetings held in February and September 2019. The suggested next steps account for and complement the work already underway with DOE/CESER funding: 1. Study the performance of LCD and PC vs. PUTT, which is less reliant on communication system performance and GPS timing references. The PUTT scheme could prove to be more resilient to cyberattack or communications-related disruption. It could also be more tolerant of message re-routing with SDN/SDR communication systems. On the other hand, it will be more vulnerable to false tripping during dynamic events or to loss of the voltage signal. The optimum choice of scheme may depend on the specific power system and risk assessment. This study could provide a new template for evaluation based on business functions. 2. Research and develop new methods to detect and monitor distributed physical attacks, possibly using drones, video sensors, thermal sensors, machine learning and other advanced techniques. This will help mitigate the impact of cyberattack on the protection system, and will also help mitigate the impact of wild fires. 3. Implement a scalable PKI for use in electric utility protection systems. This will encourage widespread adoption of secure authentication methods that are already available, but not widely used at present. This will help secure engineering access to the relays. 4. Investigate the use of SDN in combination with SDR to achieve better cybersecurity and electromagnetic security of the network, incorporating path variability. This would help secure both engineering access and peer-to-peer GOOSE messaging. 5. Perform additional testing, with operator evaluation of “red button” scenarios, PUTT vs. LCD, relay mis-operations, and other cyberattacks in the EPTC. This is an important advantage of testing in the EPTC rather than by computer simulation or even hardware-in-the-loop simulation; the EPTC is already dedicated to managing the situational awareness, operator response times and other human impacts. One of the project objectives was to settle on a common nomenclature for this problem space. We have concluded that the OSI layer model, supplemented by ANSI device numbers and other IEEE standards, is already well-accepted by the industry. The IEEE PSRC knowledge base provides a great deal of public information

24 POWER TRANSMISSION AND DISTRIBUTION↗
DOE OSTI2021

Data-Driven Specification of Earth's Magnetic Field

As our society is increasingly dependent on technologies in space, predicting space weather becomes ever more important. We have come to rely on technologies such as cellphones, GPS, the Internet, and other commercial and military assets. In order to protect these systems a high priority is to develop better understanding of the harmful conditions of space weather encountered in the near-Earth radiation environment which are mostly dictated by the dynamics of energetic charged particles. The main objective of our proposal was to improve the specification and forecast of the particle distribution in the Earth’s inner magnetosphere using data assimilation together with machine learning (ML) algorithms in our ring current-atmosphere interactions model with self-consistent magnetic and electric fields (RAM-SCBE). Specifically, we explored the application of machine learning algorithms to emulate and improve, through infusion of observational data, the magnetic field (SCBE) of our ring current model. Our work advanced towards the characterization of the RAM-SCBE particle flux field and its potentially different simulation scenarios. We explored how the model behaves in a number of simulation scenarios in order to better understand how the model reacts to perturbations in the solar wind. We were able to capture this behavior through the simulation of an ensemble of RAM-SCB model solutions. The ensemble showed the important correlations between the particle fluxes and the magnetic field. We were able to start the training of a neural network (Autoencoder) to identify the dominant features of the magnetic field, and the main correlations (linear and non-linear) between the particle fluxes and the magnetic field. Our next steps consists of training a neural network to capture the time evolution of the dominant features and obtain a consistent surrogate model for the RAM-SCB.

42 ENGINEERING↗
DOE OSTI2023

Can the United States Maintain Its Leadership in High-Performance Computing? - A report from the ASCAC Subcommittee on American Competitiveness and Innovation to the ASCR Office

The United States (U.S.) is no longer the unambiguous leader in the vitally important field of high performance computing (HPC). Japan, the European Union (EU), and China have fielded systems that are on par with our fastest supercomputers. The supply chain for everything from semiconductors to scientific software is globally distributed. Yet our economic future and security depend critically on our ability to innovate faster than our competitors, and the speed of innovation depends increasingly on large-scale computational science and engineering and thus HPC. How should the United States respond to this challenge? This report seeks to initiate a new and potentially transformative national discussion on this vital question. The Department of Energy’s (DOE) Advanced Scientific Computing Research (ASCR) program is well-positioned to make informed, targeted decisions about where the United States should cooperate and where it should compete in the global market for scientific exploration and discovery. By setting its sights on problems critical to our nation and the world, by establishing productive new collaborations, and by making strategic investments, ASCR can restore and maintain U.S. scientific leadership in the critical areas described in this report while strengthening our research infrastructure and training a large, diverse cohort of scientists. In doing so, ASCR and its scientists will pave the way for a secure and prosperous future for America. For more than 30 years, the ASCR program has provided the HPC and networking capabilities and expertise needed to support DOE’s mission to advance the national, economic, and energy security of the United States. The program now faces the challenge of developing and deploying the next generation of HPC systems and technologies, as well as supporting the application of HPC and artificial intelligence (AI) technologies to a wide range of scientific and engineering research problems. Through its research and development efforts, the ASCR program must also advance the state of the art in HPC and accelerate the pace of scientific discovery and technological innovation. Fulfilling this promise will require significantly increased investments, as well as innovative policies and programs. This subcommittee is aware that we are making recommendations and calls for action at a time when federal resources are limited. We understand that a wide range of competing priorities must be balanced by the nation’s leaders and that there is a need to leverage resources in new ways and seek efficiencies in facilities and operations. However, we must not let these realities limit our imagination or silence our advocacy. The ASCR program is a key part of the U.S. research infrastructure and an important component of economic growth and U.S. competitiveness. ASCR has a responsibility to pursue its mission, including advanced scientific computing, applications of AI technologies, and the required advanced research facilities, with determination and enthusiasm. To fulfill the scientific enterprise’s responsibility to the nation, the ASCR program must not only develop and publish a clear vision with an associated list of goals, priorities, and recommendations but also demonstrate scientific leadership by consistently securing long-term funding. This will allow the program to build on its achievements to date, to realize its ambitious vision, and to make lasting contributions to the field.

29 ENERGY PLANNING, POLICY, AND ECONOMY↗
DOE OSTI2025

TorchBraid: High-Performance Layer-Parallel Training of Deep Neural Networks with MPI and GPU Acceleration

TorchBraid is a high-performance implementation of layer-parallel training for deep neural networks (DNNs) supporting MPI-based parallelism and GPU acceleration. Layer-parallel training has been developed to overcome the serialization inherent in forward and backward propagation of DNNs that limits utilization of computational resources in the strong scaling limit. To achieve this, TorchBraid integrates the PyTorch neural network framework with the state-of-the-art XBraid time-parallel library. Furthermore, this article presents the use and performance of TorchBraid, in addition to solutions for overcoming the algorithmic challenges inherent in combining automatic differentiation with layer-parallel. Results are presented with and without GPU acceleration for the Tiny ImageNet and MNIST image classification data sets, as well as recurrent neural networks. Overall, TorchBraid enables fast training of DNNs, both in a strong and weak scaling context. In addition to the TorchBraid software, several new advances in applying layer-parallel algorithms are detailed. Integration of layer-parallel with data-parallel algorithms is presented for the first time, showing the computational advantages of the combination. Standard deep learning techniques, like batch-normalization, are developed for layer-parallel training. Finally, a new approach combining layer-parallel with spatial coarsening in order to accelerate training for 3D image classification shows roughly a 10× speedup over serial execution.

Layer-parallel↗