Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “Transformer model”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 523 records · Page 29
All sourcesNASA NTRSDOE OSTI
DOE OSTI2025

Manipulating Aromaticity to Redirect Topochemical Polymerization Pathways

Topochemical polymerization (TCP) represents an essential route to create regio- and stereoregular polymers through solid-state transformations. Herein, we present an innovative strategy for controlling topochemical polymerization pathways by tailoring the terminal group aromaticity in the para-azaquinodimethane (AQM) ring system. Substituting phenyl groups with less aromatic furyl units extends significant spin density delocalization across the conjugated core upon thermal activation, inducing significant diradicaloid characters at furyl positions and enabling unconventional reactivities in both solution and solid states. Thermal treatment in toluene yields a unique cyclophane dimer formed via furyl-methine C-C coupling, confirmed by X-ray crystallography, while solid-state reactions produce polymers formed via both intercolumnar furyl-methine coupling and intracolumnar methine-methine coupling. The spin-center-directed mechanism underlying these transformations is validated through theoretical modeling and isotopic labeling experiments. This study highlights the prowess of aromaticity modulation in functional pro-aromatic systems, which enables the synthesis of polymers with main chain structures that are otherwise difficult to access.

Zhang, Qingsong↗
DOE OSTI2024

Role of dislocations on martensitic transformation temperatures and microstructure: A molecular dynamics study

Microstructure and defects strongly affect martensitic transformations in metallic alloys. Significant progress has been made in understanding the atomic-level processes that control the role of grain boundaries and precipitates in these solid-to-solid phase transformations. Yet, the role of dislocations and their structures on martensitic transformation temperature and the resulting microstructure remains unclear. Therefore, we used large-scale molecular dynamics simulations to study the forward and reverse transformation of a martensitic material modeled after Ni63Al37 under cyclic thermal loading. The simulations reveal that dislocations in the austenite phase act as one-dimensional seeds for the martensite phase, which is present at temperatures significantly above the martensite start value. We find a reduction in the dislocation density during cyclic thermal loading, which results in the increase in martensite and austenite transition temperatures, in agreement with experiments. Importantly, we extracted a critical martensitic nuclei size for developing stable domains and found that relatively low dislocation densities are needed to grow independent martensitic variants resulting in a multi-domain structure.

Physics↗
DOE OSTI2026

Zero-day Attack Detection in Digital Substations Using In-Context Learning

In this paper, we address the critical challenge of detecting zero-day attacks in digital substations that employ the IEC-61850 communication protocol to ensure the security and reliability of modern power systems. While many heuristic and machine learning (ML)-based methods have been proposed for attack detection in IEC-61850 digital substations, generalization to unknown or zero-day attacks remains a challenge. We propose an approach that leverages the in-context learning ability of transformer architecture, which enables the model to learn from a few examples of a new task without explicit retraining. Our experiments on the IEC-61850 dataset demonstrate that the proposed method achieves more than 87% detection accuracy on zero-day attacks while the existing baselines fail. We believe this work has the potential to enhance the security of digital substations by enabling the effective detection of zero-day attacks.

LIu, Chen-Ching [Virginia Tech] (ORCID:00000002894↗
DOE OSTI2023

Development of a Machine-Learned Cruise Guide Indicator for Rotorcraft

This paper presents a machine-learned virtual cruise guide indicator (vCGI) for Chinook helicopters. Two temporal neural networks were trained and evaluated on measured data from 55 flight tests, one for the fore rotor and another for the aft rotor, to predict a vCGI value, which protects 23 components from fatigue damage during steady-state conditions. Three different classes of machine learning architectures were evaluated for prediction of the vCGI from time sequences: a temporal convolutional neural network with 1D dilated causal convolutions, a long short-term memory recurrent neural network, and an attention-based transformer architecture. The final average model accuracy on unseen flight data is currently greater than 93% for CGI values which could result in fatigue damage and 90% for normal operation CGI values. Model accuracy was improved through a series of advancements in:(1) selection of optimal training data using temporal collective variables and unsupervised learning, (2) dataset augmentation with maximum-entropy temporal collective variables, and (3) implementation of a mixture-of-experts classification- regression approach using an adversarial classification approach to assign maneuver labels. The results are presented for each advancement in model development along with lessons learned in training machine learning models on real- world, time-dependent rotorcraft data.

Boyer, Mathew↗
DOE OSTI2022

A new large-scale suspended sediment model and its application over the United States

Abstract. Suspended sediment plays a vital role in the regional and global cycling of carbon and nutrients by carrying carbon and nutrients from headwaters into lowland rivers and the oceans. Sediment transport through river systems is often fundamentally modified by human activities such as reservoir management. However, a physically based representation of sediment transport is still missing in most existing Earth system models (ESMs), which are essential tools for modeling and predicting Earth system changes. Here, we introduce a multi-process river sediment module for ESMs, which includes (1) hillslope soil erosion and sediment discharge into streams, (2) sediment transport processes through river networks, (3) reservoir operation based on the inflows from upstream areas and water demand from downstream areas, and (4) sediment trapping by reservoirs. All model parameters are estimated a priori without calibration. We apply this new sediment modeling framework to the contiguous United States and validate it against historical observations of monthly streamflow and sediment discharges at 35 river gauges. The model reasonably well captures the long-term balance and seasonal variations of suspended sediment in large river systems. Furthermore, our model results show that suspended sediment discharge in managed rivers is affected more by reservoirs' direct trapping of sediment particles than by their flow regulation. This new sediment module enables future modeling of the transportation and transformation of carbon and nutrients carried by the fine sediment along the river–ocean continuum to close the global carbon and nutrient cycles.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2025

A Risk-Informed Approach to Trustworthiness Assessment in Digital Twins-Based Autonomous Control

In autonomous control systems, digital twins (DTs) are used to perform diagnostic and prognostic functions. The trustworthiness of these DTs is dependent on quality and coverage of the training data, model accuracy and integrity of sensor data. This work introduces a methodology to determine the trustworthiness of a DT system given faulty sensor data using a risk informed approach. Bayesian Belief Networks (BBNs) are used to propagate uncertainties and determine the probability of trustable recommendations. The decision to trust the control action provided by the DT is based on the DT output, expert opinion, and severity of problems. The performance of DTs is reliant on the data they are trained on. When they encounter out of distribution data, the trustworthiness of the recommendations decreases. To address this issue, we include an expert component that provides input on sensor degradation. For this, we utilize a generative artificial intelligence (AI) model, such as Generative Pretrained Transformer (GPT). The GPT functions as an expert with broad knowledge. The GPT is fine-tuned to understand and discriminate sensor degradation scenarios using manufactured data. This methodology is demonstrated through a case study on a Nearly Autonomous Management and Control System (NAMAC) during a steady state scenario. Various sensor degradation types with different severity levels are considered. Degraded sensor data is processed by the DT system and the fine-tuned GPT. Finally, using the BBN, we combine the GPT information and the DT output with its sources of uncertainty. This provides an output regarding the trustworthiness of the DT recommendation.

22 - GENERAL STUDIES OF NUCLEAR REACTORS↗
DOE OSTI2025

Orbital-Radar v1.0.0: a tool to transform suborbital radar observations to synthetic EarthCARE cloud radar data

The Earth Cloud, Aerosol and Radiation Explorer (EarthCARE) satellite developed by the European Space Agency (ESA) and the Japan Aerospace Exploration Agency (JAXA) launched in May 2024 carries a novel 94 GHz cloud profiling radar (CPR) with Doppler capability. This work describes the open-source instrument simulator Orbital-Radar, which transforms high-resolution radar data from field observations or forward simulations of numerical models to CPR primary measurements and uncertainties. The transformation accounts for sampling geometry and surface effects. We demonstrate Orbital-Radar's ability to provide realistic CPR views of typical cloud and precipitation scenes. The presented case studies show small-scale convection, marine stratus clouds, and Arctic mixed-phase cloud cases. These results provide valuable insights into the capabilities and challenges of the EarthCARE CPR mission and its advantages over the CloudSat CPR. Finally, Orbital-Radar allows for evaluating kilometre-scale numerical weather prediction models with EarthCARE CPR observations. So, Orbital-Radar can generate calibration and validation (Cal/Val) data sets already pre-launch. Nevertheless, an evaluation of synthetic CPR output data to accurate EarthCARE CPR data is missing.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2024

Part-scale microstructure prediction for laser powder bed fusion Ti-6Al-4V using a hybrid mechanistic and machine learning model

Laser powder bed fusion (LPBF) Ti-6Al-4V is widely studied for use in structural applications in aerospace and medical industries, but mechanical anisotropy and microstructural inhomogeneity prohibits its wider adoption. Although successful microstructure prediction models have been developed, a remaining challenge is their limited integration across length/time scales and validation by experimental studies. Here, this work proposes a physics-augmented machine learning surrogate model to unite predictions of LPBF temperature, β phase morphology and texture, and α/α’ formation into a single framework that is calibrated and validated with experiments. First, a phase field (PF) model of the martensitic β→α’ transformation is developed and calibrated using data from in-situ synchrotron cyclic heating/cooling studies quantifying the variation of α phase fraction with time. In parallel, an established finite difference-Monte Carlo (FDMC) model predicts the part-scale temperature profile and β grain formation during solidification. A dataset is developed using LPBF cyclic temperature descriptors from the FDMC model as inputs and corresponding α/α’ phase fraction and width from the PF model as outputs. Five machine learning (ML) regression models are tested and optimized, having mean absolute error in testing ≤ 4 %, and the k-nearest neighbors (KNN) model is selected as the best performing. The KNN model is called at the nodal level during post-processing of the FDMC model to replace and downscale the response of the PF model. The combined agility and accuracy of the hybrid FDMC-ML model enables part-scale microstructure predictions that can be further used for property predictions to accelerate AM process optimization.

36 MATERIALS SCIENCE↗
DOE OSTI2025

Computational investigation of water glasses using machine-learning potentials

The molecular origins of water’s anomalous properties have long been a subject of scientific inquiry. The liquid–liquid phase transition hypothesis, which posits the existence of distinct low-density and high-density liquid states separated by a first-order phase transition terminating at a critical point, has gained increasing experimental and computational support and offers a thermodynamically consistent framework for many of water’s anomalies. However, experimental challenges in avoiding crystallization near the postulated liquid–liquid critical point have focused attention to water’s canonical glassy states: low-density and high-density amorphous ice. Here, we use two Deep Potential machine-learning models, trained on the Strongly Constrained and Appropriately Normed density functional and the highly accurate Many-Body Polarizable potential, to conduct an investigation of water’s glassy phenomenology based on quantum mechanical calculations. Despite not being explicitly trained on amorphous ices, both models accurately capture the structure and transformation of the water glasses, including their interconversion along different thermodynamic paths. Isobaric quenching of liquid water at various pressures generates a continuum of intermediate amorphous ices and density fluctuations increase near the liquid–liquid critical pressure. The glass transition temperatures of the amorphous ices produced at different pressures exhibit two distinct branches, corresponding to low-density and high-density amorphous ice behaviors, consistent with experiment and the liquid–liquid transition hypothesis. Extrapolating transformation pressures from isothermal compressions to experimental compression rates brings our simulations into excellent agreement with data. Our findings demonstrate that machine-learning potentials trained on equilibrium phases can effectively model nonequilibrium glassy behavior and pave the way for studying long-timescale, out-of-equilibrium processes with quantum mechanical accuracy.

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗
DOE OSTI2024

Development of Transformational Solvents for CO2 Separations

The primary objective of this CRADA activity is to use a combined molecular modeling and experimental validation approach to refine and develop transformational solvents for carbon capture. PNNL’s role on this project is currently funded by the Department of Energy’s (DOE) Office of Fossil Energy (OFE). PNNL is developing advanced molecular modeling based on their CO2BOLs solvent platform as a demonstration solvent for the activity; the model was developed and compared against measured data for CO2BOL derivatives. Here, a CRADA with PNNL and GE will leverage their current molecular models and apply them to solvent classes that operate on carbamate chemistry, specifically GE’s aminosilicone solvent class. The molecular models will be used to predict physical and thermodynamic properties, such as viscosity, as a means to predict advanced formulations with reduced viscosity compared to current aminosilicone derivatives, enabling optimized thermodynamic and kinetic metrics for economical carbon capture for this class of materials. Together, PNNL and GE will develop a comprehensive means of linking molecular modeling parameters to intermediate physical properties as a means to improve solvent performance.

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗
DOE OSTI2021

Predicting Nugget Size of Resistance Spot Welds Using Infrared Thermal Videos With Image Segmentation and Convolutional Neural Network

Resistance spot welding (RSW) is a widely adopted joining technique in automotive industry. Recent advancement in sensing technology makes it possible to collect thermal videos of the weld nugget during RSW using an infrared (IR) camera. The effective and timely analysis of such thermal videos has the potential of enabling in situ nondestructive evaluation (NDE) of the weld nugget by predicting nugget thickness and diameter. Deep learning (DL) has demonstrated to be effective in analyzing imaging data in many applications. However, the thermal videos in RSW present unique data-level challenges that compromise the effectiveness of most pre-trained DL models. We propose a novel image segmentation method for handling the RSW thermal videos to improve the prediction performance of DL models in RSW. The proposed method transforms raw thermal videos into spatial-temporal instances in four steps: video-wise normalization, removal of uninformative images, watershed segmentation, and spatial-temporal instance construction. The extracted spatial-temporal instances serve as the input data for training a DL-based NDE model. The proposed method is able to extract high-quality data with spatial-temporal correlations in the thermal videos, while being robust to the impact of unknown surface emissivity. Overall, our case studies demonstrate that the proposed method achieves better prediction of nugget thickness and diameter than predicting without the transformation.

42 ENGINEERING↗
DOE OSTI2020

SBML Level 3: an extensible format for the exchange and reuse of biological models

Systems biology has experienced dramatic growth in the number, size, and complexity of computational models. To reproduce simulation results and reuse models, researchers must exchange unambiguous model descriptions. We review the latest edition of the Systems Biology Markup Language (SBML), a format designed for this purpose. A community of modelers and software authors developed SBML Level 3 over the past decade. Its modular form consists of a core suited to representing reaction-based models and packages that extend the core with features suited to other model types including constraint-based models, reaction-diffusion models, logical network models, and rule-based models. The format leverages two decades of SBML and a rich software ecosystem that transformed how systems biologists build and interact with models. More recently, the rise of multiscale models of whole cells and organs, and new data sources such as single-cell measurements and live imaging, has precipitated new ways of integrating data with models. We provide our perspectives on the challenges presented by these developments and how SBML Level 3 provides the foundation needed to support this evolution.

59 BASIC BIOLOGICAL SCIENCES↗
DOE OSTI2025

Masked Symbol Modeling for Demodulation of Oversampled Baseband Communication Signals in Impulsive Noise-Dominated Channels

Recent breakthroughs in natural language processing show that attention mech- anism in Transformer networks, trained via masked-token prediction, enables models to capture the semantic context of the tokens and internalize the grammar of language. While the application of Transformers to communication systems is a burgeoning field, the notion of context within physical waveforms remains under-explored. This paper addresses that gap by re-examining inter-symbol con- tribution (ISC) caused by pulse-shaping overlap. Rather than treating ISC as a nuisance, we view it as a deterministic source of contextual information embedded in oversampled complex baseband signals. We propose Masked Symbol Model- ing (MSM), a framework for the physical (PHY) layer inspired by Bidirectional Encoder Representations from Transformers methodology. In MSM, a subset of symbol-aligned samples is randomly masked, and a Transformer predicts the missing symbol identifiers using the surrounding “in-between” samples. Through this objective, the model learns the latent syntax of complex baseband waveforms. We illustrate MSM’s potential by applying it to the task of demodulating sig- nals corrupted by impulsive noise, where the model infers corrupted segments by leveraging the learned context. Our results suggest a path toward receivers that interpret, rather than merely detect communication signals, opening new avenues for context-aware PHY layer design.

Bedir, Oguz↗
DOE OSTI2024

Interpreting and Accelerating Transformers for Jet Tagging

Attention-based transformers are ubiquitous in machine learning applications from natural language processing to computer vision. In high energy physics, one central application is to classify collimated particle showers in colliders based on the particle of origin, known as jet tagging. In this work, we study the interpretatbility and prospects for acceleration of Particle Transformer (ParT), a state-of-the-art model, leverages particle-level attention to improve jet-tagging performance. We analyzing ParT's attention maps and particle-pair correlations in the eta-phi plane, revealing intriguing features, such as a binary attention pattern that identifies critical substructure in jets. These insights enhance our understanding of the model's internal workings and learning process and hint at ways to improve its efficiency. Along these lines, we also explore low-rank attention, attention alternatives, and dynamic quantization to accelerate transformers for jet tagging. With quantization, we achieve a 50% reduction in model size and a 10% increase in inference speed without compromising accuracy. These combined efforts enhance both the performance and the interpretability of transformers in high-energy physics, opening avenues for more efficient and physics-driven model designs.

72 PHYSICS OF ELEMENTARY PARTICLES AND FIELDS↗
DOE OSTI2025

Compressing Vision Transformers in Geospatial Transfer Learning with Manifold-Constrained Optimization

Deploying geospatial foundation models on resource-constrained edge devices demands compact architectures that maintain high downstream performance. However, their large parameter counts and the accuracy loss often induced by compression limit practical adoption.In this work, we leverage manifold-constrained optimization framework DLRT to compress large vision transformer–based geospatial foundation models during transfer learning. By enforcing structured low-dimensional parameterizations aligned with downstream objectives, this approach achieves strong compression while preserving task-specific accuracy. We show that the method outperforms of-the-shelf low-rank methods as LoRA. Experiments on diverse geospatial benchmarks confirm substantial parameter reduction with minimal accuracy loss, enabling high-performing, on-device geospatial models.

Snyder, Thomas [Yale University]↗
DOE OSTI2022

An Overview of Nano Multilayers as Model Systems for Developing Nanoscale Microstructures

The microstructural transformations of binary nanometallic multilayers (NMMs) to equiaxed nanostructured materials were explored by characterizing a variety of nanoscale multilayer films. Four material systems of multilayer films, Hf-Ti, Ta-Hf, W-Cr, and Mo-Au, were synthesized by magnetron sputtering, heat treated at 1000 °C, and subsequently characterized by transmission electron microscopy. Binary systems were selected based on thermodynamic models predicting stable nanograin formation with similar global compositions around 20–30 at.%. All NMMs maintained nanocrystalline grain sizes after evolution into an equiaxed structure, where the systems with highly mobile incoherent interfaces or higher energy interfaces showed a more significant increase in grain size. Furthermore, varying segregation behaviors were observed, including grain boundary (GB) segregation, precipitation, and intermetallic formation depending on the material system selected. The pathway to tailored microstructures was found to be governed by key mechanisms and factors as determined by a film’s initial characteristics, including global and local composition, interface energy, layer structure, and material selection. This work presents a global evaluation of NMM systems and demonstrates their utility as foundation materials to promote tailored nanomaterials.

Appleget, Chelsea D.↗
DOE OSTI2023

FWP FEAA149: “Next Generation Environmental Barrier Coatings”

Environmental barrier coatings (EBCs) are required coatings for utilization of SiC/SiC ceramic matrix composite (CMC) components in gas turbines, where the EBC represents the life-limiting factor for such components. EBC/CMC systems have shown success in aero-engine applications with increased turbine inlet temperatures and improved efficiencies, which are achieved through higher temperature stability, lower density, and decreased reliance on cooling air compared to traditional superalloys. While industrial gas turbines (IGTs) do not currently utilize SiC/SiC CMCs, the current shift towards low-carbon or carbon-free fuel sources for power generation could result in a need for EBC/CMC components with higher temperature capabilities. In this work, three tasks were outlined to improve understanding of EBC lifetimes to encourage use in IGTs with carbon-free fuel such as hydrogen: 1. Define the bond coating oxidation kinetics and EBC failure criteria, 2. Measure thermal expansion coefficients of each layered material, and 3. Perform advanced characterization and modeling to assess EBC lifetimes. Cyclic steam oxidation tests were conducted on various EBC/Si/SiC chemistries and EBC/SiC architectures to define substrate oxidation kinetics and EBC failure modes. An open-source code was developed to quantify the undulating thermally grown oxide thickness with thousands of measurements from specimen cross-section images. Bond coating oxidation kinetics were determined and used to develop a kinetic and thermodynamic model for predicting EBC lifetimes. High-temperature Raman spectroscopy was utilized for determining the SiO 2 thermally grown oxide phase transformation as the life-limiting feature for EBCs. Model efforts supported the claim that the SiO 2 phase transformation causes elevated stress during thermal cycling with associated cracking that decreases the adhesion strength of the EBC, eventually resulting in coating spallation. The finite element model subroutine will be made publicly available upon internal review. Further development of an EBC lifetime model for IGTs involves definition of a critical SiO 2 thickness for EBC spallation and must also consider both environmental (gas velocity, pressure, etc.) and specimen (EBC dopants, layer architectures, etc.) effects into predicted bond coating oxidation kinetics for long-lifetime components.

36 MATERIALS SCIENCE↗
DOE OSTI2023

Oracle-Preserving Latent Flows

A fundamental task in data science is the discovery, description, and identification of any symmetries present in the data. We developed a deep learning methodology for the simultaneous discovery of multiple non-trivial continuous symmetries across an entire labeled dataset. The symmetry transformations and the corresponding generators are modeled with fully connected neural networks trained with a specially constructed loss function, ensuring the desired symmetry properties. The two new elements in this work are the use of a reduced-dimensionality latent space and the generalization to invariant transformations with respect to high-dimensional oracles. The method is demonstrated with several examples on the MNIST digit dataset, where the oracle is provided by the 10-dimensional vector of logits of a trained classifier. We find classes of symmetries that transform each image from the dataset into new synthetic images while conserving the values of the logits. We illustrate these transformations as lines of equal probability (“flows”) in the reduced latent space. These results show that symmetries in the data can be successfully searched for and identified as interpretable non-trivial transformations in the equivalent latent space.

97 MATHEMATICS AND COMPUTING↗