Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “Transformer model”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 235 records · Page 13
All sourcesNASA NTRSDOE OSTI
DOE OSTI2024

Adaptive Patching for High-resolution Image Segmentation with Transformers

Attention-based models are proliferating in the space of image analytics, including segmentation. The standard method of feeding images to transformer encoders is to divide the images into patches and then feed the patches to the model as a linear sequence of tokens. For high-resolution images, e.g. microscopic pathology images, the quadratic compute and memory cost prohibits the use of an attention-based model, if we are to use smaller patch sizes that are favorable in segmentation. The solution is to either use custom complex multi-resolution models or approximate attention schemes. We take inspiration from Adapative Mesh Refinement (AMR) methods in HPC by adaptively patching the images, as a pre-processing step, based on the image details to reduce the number of patches being fed to the model, by orders of magnitude. This method has a negligible overhead, and works seamlessly with any attention-based model, i.e. it is a pre-processing step that can be adopted by any attention-based model without friction. We demonstrate superior segmentation quality over SoTA segmentation models for realworld pathology datasets while gaining a geomean speedup of 6.9× for resolutions up to 64K2, on up to 2, 048 GPUs.

Zhang, Enzhi↗
DOE OSTI2023

Topological structure of complex predictions

Abstract Current complex prediction models are the result of fitting deep neural networks, graph convolutional networks or transducers to a set of training data. A key challenge with these models is that they are highly parameterized, which makes describing and interpreting the prediction strategies difficult. We use topological data analysis to transform these complex prediction models into a simplified topological view of the prediction landscape. The result is a map of the predictions that enables inspection of the model results with more specificity than dimensionality-reduction methods such as tSNE and UMAP. The methods scale up to large datasets across different domains. We present a case study of a transformer-based model previously designed to predict expression levels of a piece of DNA in thousands of genomic tracks. When the model is used to study mutations in the BRCA1 gene, our topological analysis shows that it is sensitive to the location of a mutation and the exon structure of BRCA1 in ways that cannot be found with tools based on dimensionality reduction. Moreover, the topological framework offers multiple ways to inspect results, including an error estimate that is more accurate than model uncertainty. Further studies show how these ideas produce useful results in graph-based learning and image classification.

Computer Science↗
DOE OSTI2023

Protein model quality assessment using rotation–equivariant transformations on point clouds

Machine learning research concerning protein structure has seen a surge in popularity over the last years with promising advances for basic science and drug discovery. Working with macromolecular structure in a machine learning context requires an adequate numerical representation, and researchers have extensively studied representations such as graphs, discretized 3D grids, and distance maps. As part of CASP14, we explored a new and conceptually simple representation in a blind experiment: atoms as points in 3D, each with associated features. These features—initially just the basic element type of each atom—are updated through a series of neural network layers featuring rotation-equivariant convolutions. Starting from all atoms, we further aggregate information at the level of alpha carbons before making a prediction at the level of the entire protein structure. We find that this approach yields competitive results in protein model quality assessment despite its simplicity and despite the fact that it incorporates minimal prior information and is trained on relatively little data. As a result, its performance and generality are particularly noteworthy in an era where highly complex, customized machine learning methods such as AlphaFold 2 have come to dominate protein structure prediction.

59 BASIC BIOLOGICAL SCIENCES↗
DOE OSTI2020

Peer-to-Peer Energy Trading and Grid Impact Studies in Smart Communities

The rise of peer-to-peer (P2P) marketplace paradigms has transformed existing marketplace models, but the extent to which this approach can be applied to the energy marketplace has yet to be considered. In this paper, we examine existing approaches taken in the application of a P2P paradigm to the energy marketplace, further presenting an approach towards facilitating an online P2P energy marketplace, implementing a prototype P2P web application named SolTrade. Furthermore, we submit initial statistics based on simulated transactions facilitated through the platform, which illustrate the physical impact of marketplace transactions on the energy grid. In particular, these results show that, as the number of users rises, the chance of overloading the grid rises, but the chance of the grid being unable to sustain itself without an external source of energy falls.

peer-to-peer economy, web development, energy shar↗
DOE OSTI2020

Friction stir gradient alloying: A high-throughput method to explore the influence of V in enabling HCP to BCC transformation in a γ-FCC dominated high entropy alloy

The compositional possibilities in high entropy alloys (HEAs) is very vast and effective strategies are needed to establish potential alloy chemistries. In this study, friction stir gradient alloying (FSGA), a recently-introduced high-throughput (HT) technique incorporating compositional and microstructural gradients, was used to explore the possibility of introducing a bcc transformation domain in γ-fcc dominated TRIP HEA Fe 38.5 Mn 20 Cr 15 Co 20 Si 5 Cu 1.5 (at.%) by vanadium addition. FSGA has accelerated the process of attaining the composition-microstructure library and the results suggest that vanadium addition of ~1 at.% results in nucleation of α-bcc. Additionally, this is the first observation of ε-hcp to α-bcc transformation in a TRIP HEA and supports the Olson-Cohen model of martensitic transformation (γ-fcc → ε-hcp → α-bcc). ε-hcp is nucleated with a pyramidal orientation by the formation of planar faults on the {111} close-packed planes of γ-fcc, and α-bcc is nucleated in the vicinity of ε-hcp along the {111} γ-fcc trace. There are indications for the formation of basal orientated ε-hcp from intersecting ε-hcp planes orientated for the pyramidal slip. Thus, the study shows the nucleation and growth of the α-bcc phase attributed to the dual effects of chemistry and strain and potential co-existence of all the three phases.

36 MATERIALS SCIENCE↗
DOE OSTI2020

Vulnerable window of yield strength for swelling-driven fracture of phase-transforming battery materials

Abstract Despite numerous experimental and theoretical investigations of the mechanical behavior of high-capacity Si and Ge Li-ion battery anodes, our basic understanding of swelling-driven fracture in these materials remains limited. Existing theoretical studies have provided insights into elasto-plastic deformations caused by large volume change phase transformations, but have not modeled fracture explicitly beyond Griffith’s criterion. Here, we use a multi-physics phase-field approach to model self-consistently anisotropic phase transformation, elasto-plastic deformation, and crack initiation and propagation during lithiation of Si nanopillars. Our computational results reveal that fracture occurs within a “vulnerable window” inside the two-dimensional parameter space of yield strength and fracture energy and highlight the importance of taking into account the surface localization of plastic deformation to accurately predict the magnitude of tensile stresses at the onset of fracture. They further demonstrate how the increased robustness of hollow nanopillars can be understood as a direct effect of anode geometry on the size of this vulnerable window. Those insights provide an improved theoretical basis for designing next-generation mechanically stable phase-transforming battery materials undergoing large volume changes.

25 ENERGY STORAGE↗
DOE OSTI2025

Observer-Based Nonlinear Control Scheme to Reduce Oscillations and Zero Crossing in Skid-Steer Vehicles

Motion sickness is a common condition experienced by drivers of skid-steer vehicles, primarily caused by zero crossing and oscillations in undamped systems. This study proposes an observer-based nonlinear control scheme to reduce transient oscillations and zero-crossing phenomena in skid-steer vehicles, thereby potentially alleviating motion sickness. Reducing transient oscillations and zero crossing in the transient response may alleviate motion sickness. A nonlinear damping controller is designed to improve transient response by reducing oscillations and zero-crossing. To design the controller, a reduced-order kinematic model based on coordinate transformation is developed. This transformation not only converts the system modeling into a controllable form but also enhances control performance. Modeling error is addressed by considering the distance between the center of the vehicle and the sensor location. Despite these improvements, model uncertainties and external disturbances remain, which may degrade control performance. To ensure robustness and estimate such disturbances, a high-order sliding mode observer (HOSMO) is incorporated. The effectiveness of the proposed method is validated through MATLAB/Simulink and TruckMaker simulations. From the simulation results, it was shown that the proposed method reduced the mean squared error of the tracking error to within 10 % compared to the state feedback controller with the HOSMO.

Seo, Jiwon [Chung-Ang University, Seoul (Korea, Re↗
DOE OSTI2020

Calibration and modification of the constitutive model for the mechanical response of P91

The objective of this first milestone was to calibrate and if necessary, modify a constitutive model predicting the primary and secondary creep response of P91. The overall path to be followed consists of first calibrating this constitutive model against experimental data available from the literature (this milestone). In parallel, data acquired by indentation tests will be used to calibrate the model (this involves transforming the indentation data into equivalent uniaxial tests). An overall comparison of model parameters will allow assessing to what extent these rapid tests can be used to calibrate models predicting the overall mechanical response of the bulk. Given that COVID naturally hindered the gathering of new experimental data, the choice was made to ensure that LANL would have sufficient funds in year two to perform the bulk of the new calibration work. The following report details (i) how and which of the bulk experimental data was selected, (ii) constitutive model formulation, (iii) predictions of the constitutive model: effects of imposed stress, temperature and precipitate content, (iv) future work.

11 NUCLEAR FUEL CYCLE AND FUEL MATERIALS↗
DOE OSTI2025

Dual‐Transformer Deep Learning Framework for Seasonal Forecasting of Great Lakes Water Levels

Abstract The Great Lakes of North America form one of the largest freshwater systems on Earth, and their lake‐wide average water levels (lake levels) can fluctuate by more than 0.5 m on a seasonal scale. These fluctuations pose substantial challenges for coastal resilience, flood risk management, and navigation planning. Accurate seasonal forecasting of lake levels using traditional mechanistic models is challenging due to the complex physical mechanisms and coupled hydroclimatic processes involved. Recently, deep learning has gained prominence in geoscience applications for its ability to recognize intricate patterns within multiphysical data sets. Here, we introduce a novel Dual‐Transformer deep learning framework, tested on the Great Lakes. This architecture integrates two modified Transformer models: the Prophet, which predicts underlying trends, and the Critic, which refines the Prophet's predictions. The final lake level prediction is derived by weighting the outputs of both models through a multi‐layer perceptron, jointly trained with the Prophet and Critic to enhance overall accuracy. Our results demonstrate that the innovative learning framework achieves the highest prediction accuracy compared to established deep learning models when using identical input features. It attains a root mean square error of 4–7 cm in predicting lake levels up to 6 months in advance across the lakes. Additionally, the Dual‐Transformer model runs six orders of magnitude faster than conventional mechanistic models, producing results in less than one second on a typical personal computer. These findings suggest that our deep learning framework has strong potential to advance lake level prediction and carries important implications for water management and disaster mitigation, thereby enhancing the quality of life in coastal regions.

Chen, Yi [Great Lakes Research Center Michigan Tec↗
DOE OSTI2025

Electron-Proton Scattering Event Generation using Structured Tokenization

Recent work such as Omnijet-$\alpha$ has demonstrated that effective tokenization combined with transformer-based architectures can produce effective foundation models for jet physics. While tokenization may help models capture generalizable event characteristics, it also introduces discretization errors that may compromise the precision required for downstream physics analyses. As the number and complexity of the particle features grow, these errors are likely to grow proportionally. In this study, we investigate new tokenization strategies to improve the application of generative transformer models to \textsc{Pythia8} simulations of electron-proton scattering at the Electron-Ion Collider. Specifically, we propose a feature-based structured tokenization approach that utilizes multiple tokens per particle, improving expressivity, while reducing the total number of unique tokens needed. We evaluate this method against grid-based binning, K-means clustering, and vector-quantized variational auto-encoders on the event simulations. Our results show that feature-based structured tokenization reduces discretization error, leading to more accurate generative modeling of particle-level events.

Goldenberg, Steven [Thomas Jefferson National Acce↗
DOE OSTI2025

Electron-Proton Scattering Event Generation using Structured Tokenization

Recent work such as Omnijet-$\alpha$ has demonstrated that effective tokenization combined with transformer-based architectures can produce effective foundation models for jet physics. While tokenization may help models capture generalizable event characteristics, it also introduces discretization errors that may compromise the precision required for downstream physics analyses. As the number and complexity of the particle features grow, these errors are likely to grow proportionally. In this study, we investigate new tokenization strategies to improve the application of generative transformer models to \textsc{Pythia8} simulations of electron-proton scattering at the Electron-Ion Collider. Specifically, we propose a feature-based structured tokenization approach that utilizes multiple tokens per particle, improving expressivity, while reducing the total number of unique tokens needed. We evaluate this method against grid-based binning, K-means clustering, and vector-quantized variational auto-encoders on the event simulations. Our results show that feature-based structured tokenization reduces discretization error, leading to more accurate generative modeling of particle-level events.

Goldenberg, Steven [Thomas Jefferson National Acce↗
DOE OSTI2025

Seismic Contingency Auto Generator

This code takes in premade earthquake scenario XML files from USGS, power grid data, and converts them into a contingency file (.con file) that can be used by power grid solvers. Within the .con file are a number (Specified by the user) of contingencies that have randomly failed power transformers based on their likelihood of failure and peak ground acceleration (PGA) value around the transformer. The transformers' likelihood of failure was calculated based on a variety of finite element modeling on various transformer designed for specific transformer voltage classes. Parameters from these FEM were used to create generic fragility curves for transformers within a specific voltage class, which correspond with earthquake PGA values to produced a probability of failure for a given earthquake scenario. More refined versions of this process, such as specifying specific transformer design categories within a voltage class, could also be applied in future iterations of the software.

Vaagensmith, Bjorn [Idaho National Laboratory (INL↗
DOE OSTI2020

Engineering transformation pathways in an Al 0.3 CoFeNi complex concentrated alloy leads to excellent strength–ductility combination

Guided by thermodynamic modeling, engineering phase transformation pathways via thermo-mechanical processing, in a complex concentrated alloy/high entropy alloy (HEA) of composition Al 0.3 CoFeNi, lead to a novel multi-scale microstructure consisting of fine-scale FCC + L1 2 grains mixed with B2 + BCC grains. The two-step pathway comprises initial decomposition of the parent single-phase FCC to form a fine-grained FCC + B2 microstructure, which further decomposes in the second step into the complex four-phase mixture, exhibiting an excellent combination of tensile yield stress of ~1490 MPa, ultimate tensile strength of ~1663 MPa, with a good ductility of ~12% at room temperature. This paper reports a novel multi-phase microstructure in a HEA/CCA similar to the microstructure observed in dual-phase stainless steel. This report exemplifies the novelty of HEA/CCA compositional space.

36 MATERIALS SCIENCE↗