SEARCH · Engineering Papers

Results for “Transformer model”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

SourceResearch connectionFrom yearThrough year

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 163 records · Page 9

All sources NASA NTRS DOE OSTI

DOE OSTI2025

On the Effectiveness of Neural Operators at Zero-Shot Weather Downscaling [SWR-25-20]

Code repository for the experiments performed in the paper: On the Effectiveness of Neural Operators at Zero-Shot Weather Downscaling (https://doi.org/10.1017/eds.2025.11) Overall, our work investigates the zero-shot downscaling potential of neural operators. To summarize, our contributions are: 1. We provide a comparative analysis based on two challenging weather downscaling problems, between various neural operator and non-neural-operator methods with large upsampling factors (e.g., 8x and 15x) and fine grid resolutions (e.g., 2 km × 2 km wind speed). 2. We examine whether neural operator layers provide unique advantages when testing downscaling models on upsampling factors higher than those seen during training, i.e., zero-shot downscaling. Our results instead show the surprising success of an approach that combines a powerful transformer-based model with a parameter-free interpolation step at zero-shot weather downscaling. 3. We find that this Swin-Transformer-based approach mostly outperforms all neural operator models in terms of average error metrics, whereas an enhanced super-resolution generative adversarial network (ESRGAN)-based approach is better than most models in capturing the physics of the system, and suggests their use in future work as strong baselines. However, these approaches still do not capture variations at smaller spatial scales well, including the physical characteristics of turbulence in the HR data. This suggests a potential for improvement in transformer or GAN-based methods and neural-operator-based methods for zero-shot weather downscaling.

Sinha, Saumya [National Renewable Energy Laborator↗

DOE OSTI2025

AI‐Driven Defect Engineering for Advanced Thermoelectric Materials

Thermoelectric materials offer a promising pathway to directly convert waste heat to electricity. However, achieving high performance remains challenging due to intrinsic trade-offs between electrical conductivity, the Seebeck coefficient, and thermal conductivity, which are further complicated by the presence of defects. This review explores how artificial intelligence (AI) and machine learning (ML) are transforming thermoelectric materials design. Advanced ML approaches including deep neural networks, graph-based models, and transformer architectures, integrated with high-throughput simulations and growing databases, effectively capture structure-property relationships in a complex multiscale defect space and overcome the “curse of dimensionality”. This review discusses AI-enhanced defect engineering strategies such as composition optimization, entropy and dislocation engineering, and grain boundary design, along with emerging inverse design techniques for generating materials with targeted properties. Finally, it outlines future opportunities in novel physics mechanisms and sustainability, highlighting the critical role of AI in accelerating the discovery of thermoelectric materials.

36 MATERIALS SCIENCE↗

DOE OSTI2024

Leveraging BERT and Network-Based Attention Analysis for Identifying Treatment Milestones in EHRs

This study introduces a sophisticated data-driven framework for analyzing Electronic Health Records (EHRs) using transformer-based models to identify and disentangle overlapping treatment contexts. The framework leverages a preprocessing pipeline that transforms structured procedural codes into semantically enriched descriptive text, enabling the use of attention mechanisms to cluster medical events into treatment milestones—cohesive and distinct components of care processes. The methodology is rigorously validated using synthetic datasets derived from the MIMIC-III database, designed to simulate the heterogeneity and overlapping procedural contexts characteristic of real-world EHR scenarios. Quantitative evaluation highlights the framework’s robustness in disentangling concurrent care pathways, with attention metrics and unsupervised clustering approaches demonstrating the ability to preserve intra-context relationships while distinguishing inter-context dependencies. By addressing challenges inherent in data heterogeneity, this approach provides a foundation for uncovering complex treatment patterns, advancing clinical decision-making, and optimizing resource allocation in diverse healthcare environments.

Kim, Minsu [ORNL] (ORCID:0000000224185535)↗

DOE OSTI2020

TRANSENSOR: Transformer Real-time Assessment INtelligent System with Embedded Network of Sensors and Optical Readout. Final Report

Utilities across the world are wrestling with evolving market dynamics, population growth and climate change. Distributed Energy Resources (DERs) such as solar photovoltaics, distributed generators and energy storage systems are becoming important parts of the U.S. energy mix. These factors are driving an increasing need for low-cost grid asset monitoring. Aside from being costly, traditional utility monitoring systems are not sufficiently robust and do not provide real-time visibility into the condition of grid assets such as transformers. Lack of accurate real-time measurements on performance has resulted in the use of lagging indicators, such as oil sample analysis. To address this need, an innovative embedded optical sensing technology, Transformer Real-time Assessment Intelligent System (TRANSENSOR) was developed, validated and demonstrated in this project. To date, TRANSENSOR has focused on transformers but is extendable to other grid assets. In addition to the technology being embedded into new transformers during manufacturing, a retrofit configuration that can be installed on existing transformers in the field was also developed. It is anticipated that the ability to retrofit existing transformers will help accelerate adoption of the underlying TRANSENSOR technology. Phase 1 of the project focused on laboratory development and qualification of the technology. Following iterations and exploration of relevant optical sensing modalities and multiplexed configurations of interest for the transformer environment, an effective candidate configuration was agreed upon, down-selected and custom-designed for embedding into General Electric (GE) network transformers. Two new GE 500 kilovolt ampere (kVA), 27 kilovolt (kV) distribution network transformers were built with embedded fiber-optic (FO) sensors and successfully qualified per industry standards at GE’s Shreveport, Louisiana facility during Phase 1 of the project. Following the successful completion of Phase 1, the team proceeded to Phase 2, which focused on a field demonstration of the technology. Over Phase 2, the first new GE transformer built with embedded fiber-optic sensors was installed in an above-ground cage and connected to the grid at Con Edison’s Astoria facility. A second transformer equipped with TRANSENSOR was installed in an underground vault. Additionally, an older (1982 year model) GE transformer in an above-ground cage was retrofitted with fiber- optic sensors and reconnected to the grid at ConEd’s facility. Analysis of data acquired from the sensors showed interesting correlations with transformer loading. Additionally, key events such as transformer low-voltage network connection, primary-side energizing, and a pressure loss event from an oil sampling were detected by the TRANSENSOR system. Online data processing/feature extraction algorithms for the second transformer in the underground vault detected key features and event alerts that were transmitted through a wireless 4G connection. The remote deployment concept showed promising results with data collection running for a total of 8 months across the three (3) GE transformers instrumented for the Phase 2 field demonstration. This sets the stage well for further development and commercialization.

24 POWER TRANSMISSION AND DISTRIBUTION↗

DOE OSTI2021

Transformer eXplainability and eXploration

The Transformer eXplainability and eXploration library is intended to aid in the explorability and explainability of transformer classification networks, or transformer language models with sequence classification heads. The basic function of this library is to take a trained transformer and test/train dataset and produce an ipywidget dashboard which can be displayed in a jupyter notebook or in jupyter lab.

Martindale, Nathan [Oak Ridge National Lab. (ORNL)↗

DOE OSTI2023

Anomaly Detection and Identification Using a Leave-One-Variable-Out Method

At nuclear power plants (NPPs), anomaly detection and identification (i.e., determining the causes of anomalies) are important tasks for ensuring the safe and efficient operation of NPPs. These tasks are currently labor-intensive and costly, and are made more difficult by the size and complexity of NPP systems. An alternative approach to conducting these tasks is to automate them, such as via the reconstruction-based contribution method, which is a well-researched unsupervised machine learning method that uses a data-driven model of anomaly-free behavior to detect events and then identify each variable’s contributions to those events. The present effort developed a novel contribution approach that utilized a leave-one-variable-out (LOVO) model, with which each variable is predicted using all the other variables. The novelty lay in transforming this model into a reconstruction model and modifying the identification algorithm to work with the new reconstruction model. To evaluate this method in a controlled environment, a synthetic dataset based on spring-mass-damper (SMD) systems (commonly found in mechanical engineering references) was used, with known anomalies introduced into the system. The proposed method successfully detected the anomalies and afforded insights into their causes, thus enabling the appropriate identifications to be made.

46 INSTRUMENTATION RELATED TO NUCLEAR SCIENCE AND ↗

DOE OSTI2020

A phase-field model for non-isothermal phase transformation and plasticity in polycrystalline yttria-stabilized tetragonal zirconia

Here we propose an elastoplastic phase-field (PF) model to investigate the mechanics of tetragonal-to-monoclinic phase transformation (TMPT) and elastoplastic deformation of polycrystalline yttria-stabilized tetragonal zirconia (YSTZ). A Landau polynomial with non-vanishing chemical energy at the equilibrium temperature is introduced to account for the actual formation energies of the phases. The effects of different grain orientations, latent heat, and temperature on TMPT and deformation mechanisms are considered. The suppressive transformation effects of the grain boundaries (GBs) is modeled using an inhomogeneous kinetic coefficient in the bulk and GBs. The simulation results for single crystals demonstrate the capability of the model to reproduce the orientation-dependent compressive deformation of YSTZ similar to atomistic simulations and micropillar experiments. The single crystal with [100] crystallographic orientation along the loading direction (SC[100]) displays both TMPT and plasticity, SC[101] experiences only phase transformation, while SC[001] undergoes only plastic yielding. The TMPT induced by compressive loading exhibits shape memory effect (SME) below the equilibrium transformation temperature and pseudoelasticity above it, while the critical transformation stress increases with increasing loading temperature. The irrecoverable plastic strain is found to trap a part of the monoclinic phase, which prevents a complete reverse transformation. The polycrystalline cases also display SME and PE at low and high temperatures, respectively. Due to the orientation differences between grains and the stress concentrations at geometric nonlinearities, plastic deformation occurs in polycrystalline YSTZ for an applied load less than the yield stress. The results suggest a possible limitation of plasticity and an improvement of the shape recovery of YSTZ if one can control the orientation of the grains and/or increase the density of stacking faults at the GBs during material processing.

36 MATERIALS SCIENCE↗

DOE OSTI2021

Texture Memory in Hexagonal Metals and Its Mechanism

Texture memory is a phenomenon in which retention of initial textures occurs after a complete cycle of forward and backward transformations, and it occurs in various phase-transforming materials including cubic and hexagonal metals such as steels and Ti and Zr alloys. Texture memory is known to be caused by the phenomena called variant selection, in which some of the allowed child orientations in an orientation relationship between the parent and child phases are preferentially selected. Without such variant selection, the phase transformations would randomize preferred orientations. In this article, the methods of prediction of texture memory and mechanisms of variant selections in hexagonal metals are explored. The prediction method using harmonic expansion of orientation distribution functions with the variant selection in which the Burgers orientation relationship, {110}β//{0001} α-hex <1$\bar{1}$1>β//$\langle2\bar{11}0\rangle$ α-hex , is held with two or more adjacent parent grains at the same time, called “double Burgers orientation relation (DBOR)”, is introduced. This method is shown to be a powerful tool by which to analyze texture memory and ultimately provide predictive capabilities for texture changes during phase transformations. Variation in nucleation and growth rates on special boundaries and an extensive growth of selected variants are also described. Analysis of textures of commercially pure Ti observed in situ by pulsed neutron diffraction reveals that the texture memory in CP-Ti is indeed quite well predicted by consideration of the mechanism of DBOR. The analysis also suggests that the nucleation and growth rates on the special boundary of 90° rotation about $\langle2\bar{11}0\rangle$ α-hex should be about three times larger than those of the other special boundaries, and the selected variants should grow extensively into not only one parent grain but also other grains in α-hex(hexagonal)→β(bcc) transformation. The model calculations of texture development during two consecutive cycles of α-hex→β→α-hex transformation in CP-Ti and Zr are also shown.

36 MATERIALS SCIENCE↗

DOE OSTI2021

Convex Optimization of Integrated Power-Gas Energy Flow Model With Applications to Probabilistic Energy Flow

Energy flow calculation is a fundamental problem of the integrated power and gas system (IPGS) operation and planning. However, the nonlinear gas flow model introduces major challenges to the energy flow calculation. In this paper, we propose a tractably convex optimization model to solve the energy flow problem in IPGSs. It is demonstrated that the proposed optimization model has the same optimal solution as the original nonlinear steady energy flow model. Also, piecewise linearization is adopted to tightly linearize the nonlinear objective function of the model, which transforms the formulated convex optimization into a linear program one. Thus, the computation complexity of the proposed energy flow model is significantly reduced as compared with the existing methods. In addition, the proposed model can be extended to probabilistic energy flow estimation. Extensive case studies are conducted to validate the effectiveness of the proposed energy flow model using three IPGSs.

42 ENGINEERING↗

DOE OSTI2025

RINO: Renormalization Group Invariance with No Labels

A common challenge with supervised machine learning (ML) in high energy physics (HEP) is the reliance on simulations for labeled data, which can often mismodel the underlying collision or detector response. To help mitigate this problem of domain shift, we propose RINO (Renormalization Group Invariance with No Labels), a self-supervised learning approach that can instead pretrain models directly on collision data, learning embeddings invariant to renormalization group flow scales. In this work, we pretrain a transformer-based model on jets originating from quantum chromodynamic (QCD) interactions from the JetClass dataset, emulating real QCD-dominated experimental data, and then finetune on the JetNet dataset -- emulating simulations -- for the task of identifying jets originating from top quark decays. RINO demonstrates improved generalization from the JetNet training data to JetClass data compared to supervised training on JetNet from scratch, demonstrating the potential for RINO pretraining on real collision data followed by fine-tuning on small, high-quality MC datasets, to improve the robustness of ML models in HEP.

Hao, Zichun [Caltech] (ORCID:0000000256244907)↗

DOE OSTI2022

An Accuracy-Maximization Approach for Claims Classifiers in Document Content Analytics for Cybersecurity

This paper presents our research approach and findings towards maximizing the accuracy of our classifier of feature claims for cybersecurity literature analytics, and introduces the resulting model ClaimsBERT. Its architecture, after extensive evaluations of different approaches, introduces a feature map concatenated with a Bidirectional Encoder Representation from Transformers (BERT) model. We discuss deployment of this new concept and the research insights that resulted in the selection of Convolution Neural Networks for its feature mapping aspects. We also present our results showing ClaimsBERT to outperform all other evaluated approaches. This new claims classifier represents an essential processing stage within our vetting framework aiming to improve the cybersecurity of industrial control systems (ICS). Furthermore, in order to maximize the accuracy of our new ClaimsBERT classifier, we propose an approach for optimal architecture selection and determination of optimized hyperparameters, in particular the best learning rate, number of convolutions, filter sizes, activation function, the number of dense layers, as well as the number of neurons and the drop-out rate for each layer. Fine-tuning these hyperparameters within our model led to an increase in classification accuracy from 76% obtained with BertForSequenceClassification’s original model to a 97% accuracy obtained with ClaimsBERT.

Ameri, Kimia (ORCID:0000000328791871)↗

DOE OSTI2025

Domain-specific text embedding model for accelerator physics

Accelerator physics presents unique challenges for natural language processing (NLP) due to its specialized terminology and complex concepts. A key component in overcoming these challenges is the development of robust text embedding models that transform textual data into dense vector representations, facilitating efficient information retrieval and semantic understanding. In this work, we introduce AccPhysBERT, a sentence embedding model fine-tuned specifically for accelerator physics. Our model demonstrates superior performance across a range of downstream NLP tasks, surpassing existing models in capturing the domain-specific nuances of the field. We further showcase its practical applications, including semantic paper-reviewer matching and integration into retrieval-augmented generation systems, highlighting its potential to enhance information retrieval and knowledge discovery in accelerator physics. Published by the American Physical Society 2025

Hellert, Thorsten (ORCID:0000000227970926)↗

DOE OSTI2021

A constitutive model for glass–ceramic materials

Glass–ceramics have received recent attention for use in glass–ceramic to metal hermetic seals. Due to their heterogeneous microstructure, these materials exhibit a number of advantageous responses over conventional glass based seals. Key amongst them is the possibility of a controllable thermal strain response and apparent coefficient of thermal expansion which may be used to minimize thermally induced residual stresses for aforementioned seals. Furthermore, these behaviors result from an inorganic glass matrix and variety of crystalline ceramic phases including silica polymorph(s) that may undergo reversible solid-to-solid transformations with associated inelastic strain. Correspondingly, these materials exhibit complex thermomechanical responses associated with multiple inelastic mechanisms (viscoelasticity and phase transformation). While modeling these behaviors is essential for developing and analyzing the corresponding applications, no such model exists. Therefore, in this work a three-dimensional continuum constitutive model for glass–ceramic materials combining these various inelastic mechanisms is developed via an internal state variable approach. A corresponding fully implicit three dimensional numerical formulation is also proposed and implemented. The model is used to simulate existing experiments and validate the proposed formalism. As an example, the simple seal problem of a glass–ceramic seal inside a concentric metal shell is explored. Finally, the impact of the cooling rates, viscoelastic shift factors, and inelastic strain on final residual stress state are all investigated and the differing contributions highlighted.

36 MATERIALS SCIENCE↗

DOE OSTI2022

Plant litter traits control microbial decomposition and drive soil carbon stabilization

Efforts to manage soils for carbon (C) sequestration remain limited by our understanding of how differences in plant traits and microbial traits mechanistically drive soil organic C (SOC) storage. Addressing this uncertainty is particularly critical in bioenergy agriculture, due to its potential to enhance soil C and provide a C neutral fuel. As such, we examined differences between two contrasting feedstocks, Zea mays (corn) and Miscanthus x giganteus (miscanthus), in the ability of their litter to form new chemically resistant particulate SOC vs. physically protected mineral associated SOC and used this data to improve the parameterization of a microbial SOC model. We tested a hypothesized conceptual model whereby easy to decompose corn litters drive greater microbial carbon use efficiency (CUE) and the formation of more mineral associated SOC over particulate SOC than more complex miscanthus litters. To do this, we performed a soil microcosm experiment where we added 13 C enriched aboveground and belowground litters to soils and traced the fate of the 13 C into microbial respiration and SOC pools. We found that corn litters promoted higher microbial CUE (0.37) than miscanthus litters (0.24). In turn, corn litter formed approximately 50% more mineral associated SOC than miscanthus litters. Similarly, structurally complex root litters promoted a lower CUE and formed less mineral associated SOC than leaf and shoot litters for both crops. When we used our data to parameterize the SOC model, we found that modelling microbial trait differences uniquely allowed the model to capture the fate of litter C in SOC. Collectively, we found a robust link between litter quality, microbial efficiency, and the formation of SOC. This link bridges the empirical uncertainty in how different crops can form new soil C and provides an empirical basis for modelling SOC transformations.

59 BASIC BIOLOGICAL SCIENCES↗

DOE OSTI2025

MLSPICE: Machine Learning based SPICE Modeling Platform for Power Magnetics

Electrical power converters are critical to a wide range of applications ranging from renewable integration to transportation electrification, and can be a key factor determining the size, weight, and efficiency of energy conversion systems. Magnetic components are typically the largest and least efficient components in power electronics. While there have been major strides in the modeling and analysis of power semiconductor devices and circuit simulations, the necessary advances in the design of power magnetics have lagged. In this project, we have transformed the modeling and design of power magnetics with machine learning enabled methods and catalyze simultaneous disruptive improvements for ML-based power electronics design tools. A fully automated open-source machine learning based magnetics modeling platform – the MagNet project - with innovations in full stack have been developed to greatly accelerate the design process and provide new insights to magnetic material and geometry design. The ARPA-E funded MagNet platform contains three major building blocks: 1) a ML-Integrated Data Acquisition System (MIDAS): a highly automated data acquisition testbed which is capable of measuring a large number of magnetic cores with a wide range of electrical circuit excitations; 2) a ML-integrated Core Loss Model (MICLM): a machine-learning trained modeling method for modeling the core loss and saturation effects of magnetic materials for arbitrary excitation waveforms; 3) ML-guided Magnetics SPICE Simulation Tool (PMSPICE): a fully integrated CAD tool which can simulate the magnetics in SPICE. It can help the designers to quickly model the linear and non-linear characteristics of magnetic components and evaluate their behavior in SPICE simulations. The developed MagNet system has fully demonstrated the proposed performance target and has been open sourced to the entire power electronics community to advance the modeling and design of power magnetics from many different angles.

36 MATERIALS SCIENCE↗

DOE OSTI2023

TwoFold: Highly accurate structure and affinity prediction for protein-ligand complexes from sequences

We describe our development of ab initio protein-ligand binding pose prediction models based on transformers and binding affinity prediction models based on the neural tangent kernel (NTK). Folding both protein and ligand, the TwoFold models achieve efficient and quality predictions matching state-of-the-art implementations while additionally reconstructing protein structures. In conclusion, solving NTK models points to a new use case for highly optimized linear solver benchmarking codes on HPC.

60 APPLIED LIFE SCIENCES↗

DOE OSTI2022

Quantifying Impacts of Renewable Electricity Deployment on Air Quality and Human Health in Southeast Asia Based on Aims III Scenarios

Exposure to outdoor air pollution is the largest environmental risk factor for death and disease worldwide, associated with millions of cases of excess deaths (mortality) each year. Although there are many pollutants in the air that affect our health, the most important class of pollutants is fine particulate matter, PM 2.5 , which are airborne particles of diameter ≤2.5 micrometers (µm). These particles are small enough to deposit deep in the respiratory system where they can then enter the bloodstream, traveling and causing damage to other bodily systems. Exposure to outdoor (ambient) PM 2.5 has been found to be the most important environmental risk factor for mortality in Southeast Asia, associated with 130,000 - 320,000 excess deaths in Association of Southeast Asian Nations (ASEAN) member countries in 2019. Southeast Asia, especially its mega-cities, but also other areas, has some of the worst air quality in the world. Almost all human activity emits air pollutants. Fine particulate matter is both directly emitted and formed in the atmosphere through chemical reactions, the latter of which requires modeling to predict. Power generation is one of the major sources of air pollutants that lead to elevated concentrations of fine particulate matter, including in Southeast Asia. Fossil fuel combustion for power generation, especially coal but also diesel, is the main source of air pollutant emissions from power generation. While natural gas burns cleaner than coal or diesel, in the quantities combusted for power generation in Southeast Asia, it is also a significant emitter. This study augments the ASEAN Interconnection Masterplan Study III (AIMS III) by quantifying changes to air quality and human health that result from its renewable integration and transmission interconnection scenarios. Performing this analysis requires translation of the changes in projected generation from different power sector fuel sources in the AIMS III scenarios to changes in air pollutant emissions, developing what's known as an emissions inventory for each scenario and year evaluated. We then use for the first time a new global, reduced-complexity air quality model to transform the changes in emissions to changes in air pollutant concentration of the deadliest air pollutant for human health, fine particulate matter (or PM 2.5 ). The air quality model, Global InMAP, then utilizes the location of human population in ASEAN countries to calculate exposure to PM 2.5 concentration changes and translates that to estimates of excess mortality attributable to the AIMS III scenarios.

29 ENERGY PLANNING, POLICY, AND ECONOMY↗

DOE OSTI2023

A Level Set Model for Structured Mesh Representation of Fibrous Preforms used in Composites Manufacturing

Here, a model for implicit representation of fibrous and woven preforms used for composites manufacturing is described. The method is based on a level set function defined on a structured mesh to implicitly capture the complex fiber and weave geometries. Since most software packages used to model woven preforms are based on unstructured mesh, a numerical model for transforming a discretized surface to a structured mesh representation and vice-a-versa is presented. Specific emphasis is on scalable computation of a level set function for large and complex 3D woven structures from a triangulated surface of the smallest unit (yarn). The model is applied to capture a periodic 3D layered 5-harness satin (5HS) weave geometry on a structured mesh. Geometric transformations needed to efficiently compute the level set function for the 3D structure on a high resolution mesh are described. Effects of mesh resolution on the geometric features such as number of resolved fibers, effective fiber size, and the surface area to volume ratio of the iso-surface are analyzed. Using the distance property of the level set function, an approach to evaluate the structure function resulting from transient evolving topology representing densification due to matrix phase depositions is also presented. The approach captures transition of the geometric characteristic from fiber governed at early stages of densification to yarn governed at the terminal stage.

36 MATERIALS SCIENCE↗