Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “Trains”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 145 records · Page 8
All sourcesNASA NTRSDOE OSTI
DOE OSTI2020

Layer-Parallel Training of Deep Residual Neural Networks

Residual neural networks (ResNets) are a promising class of deep neural networks that have shown excellent performance for a number of learning tasks, e.g., image classification and recognition. Mathematically, ResNet architectures can be interpreted as forward Euler discretizations of a nonlinear initial value problem whose time-dependent control variables represent the weights of the neural network. Hence, training a ResNet can be cast as an optimal control problem of the associated dynamical system. For similar time-dependent optimal control problems arising in engineering applications, parallel-in-time methods have shown notable improvements in scalability. This paper demonstrates the use of those techniques for efficient and effective training of ResNets. The proposed algorithms replace the classical (sequential) forward and backward propagation through the network layers with a parallel nonlinear multigrid iteration applied to the layer domain. This adds a new dimension of parallelism across layers that is attractive when training very deep networks. From this basic idea, we derive multiple layer-parallel methods. The most efficient version employs a simultaneous optimization approach where updates to the network parameters are based on inexact gradient information in order to speed up the training process. Finally, using numerical examples from supervised classification, we demonstrate that the new approach achieves a training performance similar to that of traditional methods, but enables layer-parallelism and thus provides speedup over layer-serial methods through greater concurrency.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2020

3D-ReG: A 3D ReRAM-based Heterogeneous Architecture for Training Deep Neural Networks

Deep neural network (DNN) models are being expanded to a broader range of applications. The computational capability of traditional hardware platforms cannot accommodate the growth of model complexity. Among recent technologies to accelerate DNN, resistive memory (ReRAM)-based processing-in-memory (PIM) emerged as a promising solution for DNN inference due to its high efficiency for matrix-based computation. We face two major technical challenges in extending the use of ReRAM-based accelerators for training: (1) full-precision data is essential in back-propagation; (2) the need to support both feed-forward and back-propagation aggravates the data-movement burden. We propose a heterogeneous architecture named as 3D-ReG, which leverages full-precision GPU to ensure training accuracy and low-overhead 3D integration to provide low-cost data movements. Moreover, we introduce conservative and aggressive task-mapping schemes, which partition the computation phases in different ways to balance execution efficiency and training accuracy. We evaluate 3D-ReG implemented with two 3D integration technologies, through-silicon vias (TSVs) and monolithic inter-tier vias (MIVs), and compare them with GPU-only and PIM-only counterparts. Various GPU-only platforms using two main-memory technologies (DRAM, ReRAM) and three interconnect technologies (2D, TSV, MIV) are evaluated as well. Experimental results show that 3D-ReG can achieve on average 5.64× training speedup and 3.56× higher energy efficiency compared with the GPU with DRAM as main memory, at the cost of 0.05%–3.39% accuracy drop. We define a new metric, gain-loss ratio (GLR), which quantitatively evaluates the capability of a DNN training hardware in terms of the model accuracy and hardware efficiency. The results of our comparison show that the aggressive task-mapping scheme on MIV-based 3D-ReG outperforms the other methods.

Computer Science↗
DOE OSTI2021

D2U: Data Driven User Emulation for the Enhancement of Cyber Testing, Training, and Data Set Generation

Whether testing intrusion detection systems, conducting training exercises, or creating data sets to be used by the broader cybersecurity community, realistic user behavior is a critical component of a cyber range. Existing methods either rely on network level data or replay recorded user actions to approximate real users in a network. Our work is the first to produce generative models trained on actual user data (sequences of application usage) collected on endpoints. Once trained to the user's behavioral data, these models can generate novel sequences of actions %that appear to come from the same distribution as the training data. These sequences of actions are then fed to our custom software via configuration files, which replicate those behaviors on end devices. Notably, our models are platform agnostic and could generate behavior data for any emulation software package. In this paper we present our model generation process, software architecture, and an initial evaluation of the fidelity of our models. Our software is currently deployed in a cyber range to help evaluate the efficacy of defensive cyber technologies. We suggest additional ways that the cyber community as a whole can benefit from more realistic user behavior emulation. The data used to train our model, as well as sample configuration files produced by the model, are available at [redacted].

Oesch, T↗
DOE OSTI2023

DDStore: Distributed Data Store for Scalable Training of Graph Neural Networks on Large Atomistic Modeling Datasets

Graph neural networks (GNNs) are a class of Deep Learning models used in designing atomistic materials for effective screening of large chemical spaces. To ensure robust prediction, GNN models must be trained on large volumes of atomistic data on leadership class supercomputers. Even with the advent of modern architectures that consist of multiple storage layers that include node-local NVMe devices in addition to device memory for caching large datasets, extreme-scale model training faces I/O challenges at scale.We present DDStore, an in-memory distributed data store designed for GNN training on large-scale graph data. DDStore provides a hierarchical, distributed, data caching technique that combines data chunking, replication, low-latency random access, and high throughput communication. DDStore achieves near-linear scaling for training a GNN model using up to 1000 GPUs on the Summit and Perlmutter supercomputers, and reaches up to a 6.15x reduction in GNN training time compared to state-of-the-art methodologies.

Choi, Jong Youl↗
DOE OSTI2024

Towards a High Fidelity Training Environment for Autonomous Cyber Defense Agents

Cyber defenders are overwhelmed by the frequency and scale of attacks against their networks. This problem will only be exacerbated as attackers leverage AI to automate their workflows. Autonomous cyber defense capabilities could aid defenders by automating operations and adapting dynamically to novel threats. However, existing training environments fall short in areas such as generalization, explainability, scalability, and transferability, making it intractable to train agents that will be effective in real networks. In this paper we take an important step towards creating autonomous cyber defense agents — we present a high fidelity training environment called Cyberwheel that includes both simulation and emulation capabilities. Cyberwheel simplifies customization of the training network and easily allows redefining the agent’s reward function, observation space, and action space to support rapid experimentation of novel approaches to agent design. It also provides visibility into agent behaviors necessary for agent evaluation and sufficient documentation / examples to lower the barrier to entry. As an example use case of Cyberwheel, we present initial results training an autonomous agent to deploy cyber deception strategies in simulation.

Oesch, T↗
DOE OSTI2025

ATTNChecker: Highly-Optimized Fault Tolerant Attention for Large Language Model Training

Large Language Models (LLMs) have demonstrated remarkable performance in various natural language processing tasks. However, the training of these models is computationally intensive and susceptible to faults, particularly in the attention mechanism, which is a critical component of transformer-based LLMs. In this paper, we investigate the impact of faults on LLM training, focusing on INF, NaN, and near-INF values in the computation results with systematic fault injection experiments. We observe the propagation patterns of these errors, which can trigger non-trainable states in the model and disrupt training, forcing the procedure to load from checkpoints. To mitigate the impact of these faults, we propose ATTNChecker, the first Algorithm-Based Fault Tolerance (ABFT) technique tailored for the attention mechanism in LLMs. ATTNChecker is designed based on fault propagation patterns of LLM and incorporates performance optimization to adapt to both system reliability and model vulnerability while providing lightweight protection for fast LLM training. Evaluations on four LLMs show that ATTNChecker on average incurs on average 7% overhead on training while detecting and correcting all extreme errors. Compared with the state-of-the-art checkpoint/restore approach, ATTNChecker reduces recovery overhead by up to 49×.

Liang, Yuhang [University of Alabama - Birmingham]↗
DOE OSTI2025

CAFE AU LAIT: Compute-Aware Federated Augmented Low-Rank AI Training

Federated finetuning is crucial for unlocking the knowledge embedded in pretrained Large Language Models (LLMs) when data are geographically distributed across clients. Unlike finetuning with data from a single institution, federated finetuning allows collaboration across multiple institutions, enabling the utilization of diverse and decentralized datasets while preserving data privacy. Given the high computing costs of LLM training and the emphasis on energy efficiency in Federated Learning (FL), Low-Rank Adaptation (LoRA) has emerged as a widely adopted algorithm due to its significantly reduced number of trainable parameters. However, this assumes that all data silos have the necessary computing resources to compute local updates of LLMs. Nevertheless, in practice, the computing resources across clients are highly heterogeneous: while some may have access to hundreds of GPUs, others might have limited or no GPU access. Recently, federated finetuning using synthetic data has been proposed, allowing clients to participate in a collaborative training run without training LLMs locally. However, our experimental results reveal a performance gap between models trained using synthetic data and those trained using local updates. Motivated by the observed heterogeneity in computing resources and the performance gap, we propose a novel two-stage algorithm that leverages the storage and computing capabilities of a strong server. In the first stage, under the coordination of the strong server, clients with limited computing resources collaborate to generate synthetic data, which is transferred to and stored on the strong server. In the second stage, the strong server uses this synthetic data on behalf of the resource-constrained clients to perform federated LoRA finetuning alongside clients with sufficient computing resources. This approach ensures that all clients can participate in the finetuning process. Experimental results demonstrate that incorporating local updates from even a small fraction of clients improves performance compared to using synthetic data for all clients. Furthermore, we incorporate the Gaussian mechanism in both stages to guarantee client-level differential privacy.

Wang, Jiayi [ORNL]↗
DOE OSTI2025

TorchBraid: High-Performance Layer-Parallel Training of Deep Neural Networks with MPI and GPU Acceleration

TorchBraid is a high-performance implementation of layer-parallel training for deep neural networks (DNNs) supporting MPI-based parallelism and GPU acceleration. Layer-parallel training has been developed to overcome the serialization inherent in forward and backward propagation of DNNs that limits utilization of computational resources in the strong scaling limit. To achieve this, TorchBraid integrates the PyTorch neural network framework with the state-of-the-art XBraid time-parallel library. Furthermore, this article presents the use and performance of TorchBraid, in addition to solutions for overcoming the algorithmic challenges inherent in combining automatic differentiation with layer-parallel. Results are presented with and without GPU acceleration for the Tiny ImageNet and MNIST image classification data sets, as well as recurrent neural networks. Overall, TorchBraid enables fast training of DNNs, both in a strong and weak scaling context. In addition to the TorchBraid software, several new advances in applying layer-parallel algorithms are detailed. Integration of layer-parallel with data-parallel algorithms is presented for the first time, showing the computational advantages of the combination. Standard deep learning techniques, like batch-normalization, are developed for layer-parallel training. Finally, a new approach combining layer-parallel with spatial coarsening in order to accelerate training for 3D image classification shows roughly a 10× speedup over serial execution.

Layer-parallel↗
DOE OSTI2024

Hyperparameter Studies for Vision Transformers Trained on High-Fidelity Simulations

This library is a collection of python modules that define, train, and analyze vision-transformer (ViT) machine learning models. The code implements, with mild modifications, ViT models that have been made publicly available through publication and GitHub code. The training data for these models is hydrodynamic simulation output in the form of numpy arrays. This library contains code to train these ViT models on the hydrodynamic simulation output with a variety of hyperparameters, and to compare the results of such models. Furthermore, the library contains definitions of simple convolutional neural network (CNN) machine learning architectures which can be trained on the same hydrodynamic simulation output. These are included as a reference point to compare the ViT models to. Additionally, the library includes trained ViT and CNN models and example input data for demonstration purposes. The code is based on the PyTorch python library.

Callis, Skylar↗
DOE OSTI2023

Machine learning-based inversion for acoustic impedance with large synthetic training data: Workflow and data characterization

Where wells are sparse or training data are difficult to label with high-quality wireline-derived impedance logs, machine learning (ML)-based inversion of acoustic impedance typically depends on small training data sets, leading to biased prediction. We have advanced a novel workflow that applies large synthetic seismic training data to reduce facies-related bias. Using a geologically realistic model as the truth model, we randomly select sparse seed wells to perform sequential Gaussian simulation (SGS) for impedance models of the same geometry and simulate facies variability. We implement random forest regression on 30 features extracted from the synthetic volume. We observe that more seed wells tend to reduce facies-induced bias by sampling more types of facies, resulting in a better prediction. We then focus on the responses of SGS models to facies changes, the number of seed wells necessary for a useful synthetic model, and how much a synthetic model can help ML-based inversion. Here, we observe that the SGS synthetic training model outperforms well-direct training in general. For modeled clastic shore-zone systems in Miocene Gulf of Mexico, two or more seed wells are necessary for a significant reduction of root-mean-square error and outliners, and improvement of facies imaging. In a field-data test, we apply a similar workflow to quantitatively predict acoustic impedance, which is then converted to a sand-volume map at a high-frequency sequence (10–100 m), revealing detailed facies and sandstone patterns. Such results are valuable in many geologic and engineering applications, such as hydrocarbon and CO 2 reservoir prospecting, reserve estimation, simulation, etc.

3D seismic↗
DOE OSTI2020

Gen IV Education and Training Working Group Webinars’ Initiative

Collaboration and support among national laboratories, industry, universities, and research and development organizations are vital to not only maintain a skilled and competent nuclear workforce but also to avert the risk of human resource shortages. As stated in a recent IAEA paper [1], the world nuclear electrical generating capacity is projected to increase to 554 GW(e) by 2030 and up to 874 GW(e) by 2050 (Fig. 1). This represents a 42% increase over current levels by 2030 and a doubling of the current capacity by 2050. To promote Education and Training (E&T) on Gen IV reactor systems and other nuclear related topics of interest, the Gen IV Education and Training Working Group (ETWG) identifies and advertises training courses; engages collaboration with other international education and training organizations; delivers webinars dedicated to Gen IV systems; and maintains a modern social medium platform to exchange information and ideas on Gen IV R&D topics, as well as on related GIF education and training activities. At the end of June 2020, the GIF ETWG has produced, podcasted and posted forty-one webinars covering the six Gen IV systems and various subjects addressing e.g. the economics of the nuclear fuel cycle, sustainability aspects of Gen IV systems, nuclear fuels and materials challenges, the thorium fuel cycle, energy conversion systems, and lessons learned for knowledge management and preservation. This paper describes the development of these webinars from the initial concept to its full realization, with future webinars planned in 2021, addressing topics such as materials and fuels performance for advanced reactors, small modular reactor.

Paviet, Patricia D.↗
DOE OSTI2022

Maximizing Strength: The Stimuli and Mediators of Strength Gains and Their Application to Training and Rehabilitation

Abstract Spiering, BA, Clark, BC, Schoenfeld, BJ, Foulis, SA, and Pasiakos, SM. Maximizing strength: the stimuli and mediators of strength gains and their application to training and rehabilitation. J Strength Cond Res 37(4): 919–929, 2023—Traditional heavy resistance exercise (RE) training increases maximal strength, a valuable adaptation in many situations. That stated, some populations seek new opportunities for pushing the upper limits of strength gains (e.g., athletes and military personnel). Alternatively, other populations strive to increase or maintain strength but cannot perform heavy RE (e.g., during at-home exercise, during deployment, or after injury or illness). Therefore, the purpose of this narrative review is to (a) identify the known stimuli that trigger gains in strength; (b) identify the known factors that mediate the long-term effectiveness of these stimuli; (c) discuss (and in some cases, speculate on) potential opportunities for maximizing strength gains beyond current limits; and (d) discuss practical applications for increasing or maintaining strength when traditional heavy RE cannot be performed. First, by conceptually deconstructing traditional heavy RE, we identify that strength gains are stimulated through a sequence of events, namely: giving maximal mental effort, leading to maximal neural activation of muscle to produce forceful contractions, involving lifting and lowering movements, training through a full range of motion, and (potentially) inducing muscular metabolic stress. Second, we identify factors that mediate the long-term effectiveness of these RE stimuli, namely: optimizing the dose of RE within a session, beginning each set of RE in a minimally fatigued state, optimizing recovery between training sessions, and (potentially) periodizing the training stimulus over time. Equipped with these insights, we identify potential opportunities for further maximizing strength gains. Finally, we identify opportunities for increasing or maintaining strength when traditional heavy RE cannot be performed.

Sport Sciences↗
DOE OSTI2020

Training Population Optimization for Genomic Selection in Miscanthus

Miscanthus is a perennial grass with potential for lignocellulosic ethanol production. To ensure its utility for this purpose, breeding efforts should focus on increasing genetic diversity of the nothospecies Miscanthus × giganteus (M×g) beyond the single clone used in many programs. Germplasm from the corresponding parental species M. sinensis (Msi) and M. sacchariflorus (Msa) could theoretically be used as training sets for genomic prediction of M×g clones with optimal genomic estimated breeding values for biofuel traits. To this end, we first showed that subpopulation structure makes a substantial contribution to the genomic selection (GS) prediction accuracies within a 538-member diversity panel of predominately Msi individuals and a 598-member diversity panels of Msa individuals. We then assessed the ability of these two diversity panels to train GS models that predict breeding values in an interspecific diploid 216-member M×g F2 panel. Low and negative prediction accuracies were observed when various subsets of the two diversity panels were used to train these GS models. To overcome the drawback of having only one interspecific M×g F2 panel available, we also evaluated prediction accuracies for traits simulated in 50 simulated interspecific M×g F2 panels derived from different sets of Msi and diploid Msa parents. The results revealed that genetic architectures with common causal mutations across Msi and Msa yielded the highest prediction accuracies. Ultimately, these results suggest that the ideal training set should contain the same causal mutations segregating within interspecific M×g populations, and thus efforts should be undertaken to ensure that individuals in the training and validation sets are as closely related as possible.

59 BASIC BIOLOGICAL SCIENCES↗
DOE OSTI2021

Attribute-Guided Adversarial Training for Robustness to Natural Perturbations

We report while existing work in robust deep learning has focused on small pixel-level norm-based perturbations, this may not account for perturbations encountered in several real world settings. In many such cases although test data might not be available, broad specifications about the types of perturbations (such as an unknown degree of rotation) may be known. We consider a setup where robustness is expected over an unseen test domain that is not i.i.d. but deviates from the training domain. While this deviation may not be exactly known, its broad characterization is specified a priori, in terms of attributes. We propose an adversarial training approach which learns to generate new samples so as to maximize exposure of the classifier to the attributes-space, without having access to the data from the test domain. Our adversarial training solves a min-max optimization problem, with the inner maximization generating adversarial perturbations, and the outer minimization finding model parameters by optimizing the loss on adversarial perturbations generated from the inner maximization. We demonstrate the applicability of our approach on three types of naturally occurring perturbations --- object-related shifts, geometric transformations, and common image corruptions. Our approach enables deep neural networks to be robust against a wide range of naturally occurring perturbations. We demonstrate the usefulness of the proposed approach by showing the robustness gains of deep neural networks trained using our adversarial training on MNIST, CIFAR-10, and a new variant of the CLEVR dataset.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2025

Fiats: Functional inference and training for surrogates

Fiats provides a platform for research on the training and deployment of neural-network surrogate models for computational science. Fiats also supports exploring, advancing, and combining functional, object-oriented, and parallel programming patterns in Fortran 2023. As such, the Fiats name has dual expansions: “Functional Inference And Training for Surrogates” or “Fortran Inference And Training for Science.” Fiats inference and training procedures are pure and therefore satisfy a language constraint imposed on procedure invocations inside Fortran’s parallel loop construct: do concurrent. Furthermore, the Fiats training procedures are built around a do concurrent parallel reduction. Several compilers can automatically parallelize do concurrent on Central Processing Units (CPUs) or Graphics Processing Units (GPUs). Fiats thus aims to achieve performance portability through standard language mechanisms.

Rouson, Damian [Lawrence Berkeley National Laborat↗
DOE OSTI2020

UTrain Manual for Training Administrators

This Institutional Training Administrative Procedure provides the business rules that must be followed when working in UTrain. As commercial off-the-shelf software with customizations, UTrain cannot address all the implementation business requirements, and thus, administrative business rules and processes were defined and documented. Many business rules were incorporated into the UTrain training materials. UTrain is the Laboratory’s official learning management system. UTrain is the system of record for all Laboratory training, qualification, certification, and worker authorization. All training at LANL must be documented in UTrain. The detailed steps of how to operate UTrain are contained in the UTrain courses and training manuals and in the UTrain job aids, not in this document.

96 KNOWLEDGE MANAGEMENT AND PRESERVATION↗
DOE OSTI2020

Metaevaluation Final Report: Sandia National Laboratories Recorded Information Management Training Program Evaluation

For my internship project, I chose to evaluate the training program for the Sandia National Laboratories (SNL) Recorded Information Management (RIM) Program. The purpose of the evaluation was to identify: Where the current training meets federal and corporate requirements and if not, what gaps exist; If the training is accessible in a consistent location; If the training contains a consistent message about the responsibility of managing documents; If the training is organized by records management subject areas.

96 KNOWLEDGE MANAGEMENT AND PRESERVATION↗
DOE OSTI2020

TA-55 Forensic Support Operations Cross-Training Exercise

As part of the annual FBI/LANL training schedule, a joint training exercise was conducted during the week of 10FEB2020 with FBI HEAT members and LANL Fissile Material Handlers (FMHs). The joint training exercise was conducted in a non-radiological area to reduce cost and ensure the test object would not be contaminated. The exercise test object was fabricated by LANL using surrogate materials to represent an object containing Special Nuclear Material (SNM). Working with Jim Blankenship and Kevin Swearingen, functional requirements of the test object included the following: unclassified, disassembly (to include screws), and moderately heavy. To meet the objectives, the test object was fabricated by welding a custom aluminum box (8” x 8” x 12”) with a lid which was secured by four (4) screws. The custom box housed a track and field shot put (16 lbs.) that was anchored by a hose clamp. The intent of the training was to disassemble the test object and conduct traditional forensic determinations on the parts. This lessons learned report documents all of the associated aspects related to the joint training effort.

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗