Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “Transformer model”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 865 records · Page 48
All sourcesNASA NTRSDOE OSTI
DOE OSTI2025

Knowledge Oriented Graph Unified Transformer (KOGUT) v0.1

KOGUT — Knowledge Oriented Graph Unified Transformer KOGUT implements the Relational Graph Transformer (RelGT) architecture for knowledge graph link prediction in biological domains, with a primary focus on microbial growth media prediction. While the original RelGT (arXiv:2505.10960) targets relational tables, time series, and multi-table databases, KOGUT adapts this architecture for heterogeneous biological knowledge graphs, providing first-in-class AI predictive models for microbial cultivation. Key Adaptations Beyond Original RelGT: - Knowledge Graph Focus: Applied to biological KGs with semantic node types (taxa, chemicals, media, phenotypes, environments) versus generic relational database tables, trained on the KG-Microbe knowledge graph (1.3M entities, 2.9M edges, 24 relation types). - Multimodal Node Encoding: Integrates node labels, categories, descriptions, and synonyms from KG metadata through learned embedding layers—adapting relational column features to graph node attributes with textual semantics. - Extended K-Hop Subgraph Strategy: Optimized neighborhood sampling (3-hop default, configurable up to 200 nodes) tuned for sparse biological networks, building on the original local-global attention framework with biological relation preservation. - Biolink Predicate Preservation: Type-specific transformations for 24 biological edge semantics (occurs_in, consumes, produces, has_phenotype, subclass_of) beyond standard relational foreign keys, enabling multi-relation link prediction. - Inductive Learning Support: Enables zero-shot predictions for novel taxa through feature-based embeddings (temperature, oxygen requirements, gram stain, cell shape), extending the original transductive relational benchmark scope to uncultured microorganisms. CheapSOTA Performance Optimizations (This Distribution): - VQ-EMA Centroid Attention: Vector quantization with exponential moving average for improved global context modeling (+5-10% MRR improvement). - HDF5 Precomputed Data Loading: One-time preprocessing of k-hop subgraphs to eliminate redundant graph traversals (2-5× training speedup). - Distributed Data Parallel Training: Multi-GPU support for scaling to larger knowledge graphs (tested on 4× NVIDIA A100 GPUs at NERSC Perlmutter). - Mixed Precision Training: Automatic mixed precision (AMP) for memory efficiency and faster training. Advantages Over Standard Knowledge Graph Embedding Models: Combines RelGT's proven multi-element tokenization (features, type, hop, structure) with graph-native biological representations, enabling interpretable link prediction across heterogeneous entities that standard embedding models (TransE, RotatE, ComplEx) and table-based transformers cannot directly model. Achieves near-perfect performance on microbial growth media prediction (MRR: 0.9966, Precision@1: 0.9932, Hit@10: 1.0000) while maintaining explainability through attention-based reasoning over biological pathways. Training Data: - KG-Microbe merged knowledge graph: 1,379,337 nodes, 2,960,472 edges - 24 biological relation types including taxonomic hierarchies, metabolic interactions, phenotype associations, and environmental relationships - Primary prediction task: Growth media suitability for microbial taxa (biolink:occurs_in, 50K edges) - Multi-relation capability: Predicts links for any of the 24 relation types, including chemical consumption/production, phenotype associations, and taxonomic classification Citation: Original RelGT Architecture: Dwivedi et al., "Relational Graph Transformer", arXiv:2505.10960, 2025 KOGUT Implementation: Knowledge Oriented Graph Unified Transformer for Microbial Growth Media Prediction Developed at Lawrence Berkeley National Laboratory (LBNL) Trained on NERSC Perlmutter supercomputer

Joachimiak, Marcin [Lawrence Berkeley National Lab↗
DOE OSTI2024

In Situ Insights into Cathode Calcination for Predictive Synthesis: Kinetic Crystallization of LiNiO 2 from Hydroxides

Abstract Calcination is a solid‐state synthesis process widely deployed in battery cathode manufacturing. However, its inherent complexity associated with elusive intermediates hinders the predictive synthesis of high‐performance cathode materials. Here, correlative in situ X‐ray absorption/scattering spectroscopy is used to investigate the calcination of nickel‐based cathodes, focusing specifically on the archetypal LiNiO 2 from Ni(OH) 2 . Combining in situ observation with data‐driven analysis reveals concurrent lithiation and dehydration of Ni(OH) 2 and consequently, the low‐temperature crystallization of layered LiNiO 2 alongside lithiated rocksalts. Following early nucleation, LiNiO 2 undergoes sluggish crystallization and structural ordering while depleting rocksalts; ultimately, it turns into a structurally‐ordered layered phase upon full lithiation but remains small in size. Subsequent high‐temperature sintering induces rapid crystal growth, accompanied by undesired delithiation and structural degradation. These observations are further corroborated by mesoscale modeling, emphasizing that, even though calcination is thermally driven and favors transformation towards thermodynamically equilibrium phases, the actual phase propagation and crystallization can be kinetically tuned via lithiation, providing freedom for structural and morphological control during cathode calcination.

36 MATERIALS SCIENCE↗
DOE OSTI2021

A Generalized Copula-Polynomial Chaos Expansion for Probabilistic Power Flow Considering Nonlinear Correlations of PV Injections

This paper develops a generalized Copula-polynomial chaos expansion (PCE) framework for power system probabilistic power flow that can handle both linear and nonlinear correlations of uncertain power injections, such as wind and PVs. A data-driven Copula statistical model is used to capture the correlations of uncertain power injections. This allows us to resort to the Rosenblatt transformation to transform correlated variables into independent ones while preserving the dependence structure. This paves the way of leveraging the PCE for surrogate modeling and uncertainty quantification of power flow results, i.e., achieving the probabilistic distributions of power flows. Simulations carried out on the IEEE 57-bus system show that the proposed framework can get much more accurate results than other alternatives with different linear and nonlinear power injection correlations.

copula↗
DOE OSTI2025

A systematic review of machine learning in groundwater monitoring

With increasing concerns about water scarcity, groundwater has become crucial since this resource provides most of the freshwater needs. However, various human and natural activities often contaminate the groundwater, making it unsuitable for use. Over the years, scientists and engineers have used many methods to predict and track groundwater contamination as part of environmental monitoring. Consequently, there is an urgent need for improved methods, particularly in the face of increasing contamination. Machine learning has sometimes been used to monitor groundwater, air quality, and climate. Traditional methods must be improved due to the complexity and large amount of environmental data. This includes using hybrid models that combine traditional and new techniques. Despite the use of machine learning in many scientific areas, there is a lack of comprehensive reviews focusing on its use in environmental monitoring, especially groundwater monitoring. We aim to fill this gap by exploring machine-learning applications in groundwater monitoring. We discuss relevant methods, their limitations, and future potential. We summarize research on automating data processing and model training using groundwater sensor data. Our research underscores the transformative potential of machine learning to revolutionize long-term groundwater monitoring and contamination detection, providing valuable insights for future research and practical applications.

AI/ML↗
DOE OSTI2022

Investigation of crossed-twin structure formation in magnesium and magnesium alloys

In this work, the effect of alloying addition on the propensity for twin-twin interactions to transform into crossed-twin structures in magnesium alloys is investigated. A full-field elasto-viscoplastic fast Fourier transform (EVP-FFT) framework combined with a discrete twin model and dislocation density-based hardening law for slip strengths is used to calculate the micromechanical fields in the crystalline matrix around the interacting twins. AZ31 and MgLi alloys are selected along with pure Mg to study the influence of plastic anisotropy in connection with alloying elements. These alloys were selected since their plastic anisotropy measure, which is defined as the ratio between the critical resolved shear stress for pyramidal $\langle$c +a$\rangle$ and basal $\langle$a$\rangle$ slip modes, spanned a wide range. To quantify the role of twin thicknesses, we probe a range of impinging twin thicknesses while fixing the recipient twin thickness. The analysis reveals that: (i) the local driving stress for crossed twin structure formation generated from the interaction of the two twins is lower in a low plastically anisotropic material, like a MgLi alloy, than a high plastically anisotropic material like pure Mg and (ii) the critical impinging twin thickness needed to form the crossed twin structure in pure Mg, AZ31 and MgLi alloys is ~0.5, ~0.75 and ~1.5 times the recipient twin thickness. We propose a relationship between the tendency for crossed twin structure formation and the experimentally observed higher ductility in MgLi alloys compared to pure Mg. Finally, one key implication of the findings is that crossed- twin structure formation can be hindered and the ductility of magnesium alloy thereby improved by properly choosing alloying elements that lower the slip strength for pyramidal $\langle$c + a$\rangle$ slip.

36 MATERIALS SCIENCE↗
DOE OSTI2022

Data augmentation for disruption prediction via robust surrogate models

The goal of this work is to generate large statistically representative data sets to train machine learning models for disruption prediction provided by data from few existing discharges. Such a comprehensive training database is important to achieve satisfying and reliable prediction results in artificial neural network classifiers. Here, we aim for a robust augmentation of the training database for multivariate time series data using Student t process regression. We apply Student t process regression in a state space formulation via Bayesian filtering to tackle challenges imposed by outliers and noise in the training data set and to reduce the computational complexity. Thus, the method can also be used if the time resolution is high. We use an uncorrelated model for each dimension and impose correlations afterwards via colouring transformations. We demonstrate the efficacy of our approach on plasma diagnostics data of three different disruption classes from the DIII-D tokamak. To evaluate if the distribution of the generated data is similar to the training data, we additionally perform statistical analyses using methods from time series analysis, descriptive statistics and classic machine learning clustering algorithms.

70 PLASMA PHYSICS AND FUSION TECHNOLOGY↗
DOE OSTI2024

Ultrafast Bragg coherent diffraction imaging of epitaxial thin films using deep complex-valued neural networks

Abstract Domain wall structures form spontaneously due to epitaxial misfit during thin film growth. Imaging the dynamics of domains and domain walls at ultrafast timescales can provide fundamental clues to features that impact electrical transport in electronic devices. Recently, deep learning based methods showed promising phase retrieval (PR) performance, allowing intensity-only measurements to be transformed into snapshot real space images. While the Fourier imaging model involves complex-valued quantities, most existing deep learning based methods solve the PR problem with real-valued based models, where the connection between amplitude and phase is ignored. To this end, we involve complex numbers operation in the neural network to preserve the amplitude and phase connection. Therefore, we employ the complex-valued neural network for solving the PR problem and evaluate it on Bragg coherent diffraction data streams collected from an epitaxial La 2-x Sr x CuO 4 (LSCO) thin film using an X-ray Free Electron Laser (XFEL). Our proposed complex-valued neural network based approach outperforms the traditional real-valued neural network methods in both supervised and unsupervised learning manner. Phase domains are also observed from the LSCO thin film at an ultrafast timescale using the complex-valued neural network.

77 NANOSCIENCE AND NANOTECHNOLOGY↗
DOE OSTI2021

The diversity and evolution of microbial dissimilatory phosphite oxidation

Significance Geochemical models of the phosphorus (P) cycle uniquely ignore microbial redox transformations. Yet phosphite is a reduced P source that has been detected in several environments at concentrations that suggest a contemporary P redox cycle. Microbial dissimilatory phosphite oxidation (DPO) converts soluble phosphite into phosphate, and a false notion of rarity has limited our understanding of its diversity and environmental distribution. Here we demonstrate that DPO is an ancient energy metabolism hosted by taxonomically diverse, autotrophic bacteria that exist globally throughout anoxic environments. DPO microorganisms are therefore likely to have provided bioavailable phosphate and fixed carbon to anoxic ecosystems throughout Earth’s history and continue to do so in contemporary environments.

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗
DOE OSTI2020

Data-driven Detection and Early Prediction of Thermoacoustic Instability in a Multi-nozzle Combustor

Thermoacoustic instability (TAI) is a critical issue in modern lean-burn gas-turbine combustors, which is induced by a strong coupling between the resonant combustor acoustics and fluctuations in the heat release rate. This instability may lead to high-amplitude pressure waves that generate undesirable noise levels as well as fatigue stresses in mechanical structures of the combustor. The intense pressure fluctuations due to TAI may also cause large flow perturbations and possibly flow reversal that may lead to flame oscillations, flame liftoff, and even flame blow-out. Hence, there is a strong need for exercising control actions in a timely fashion to mitigate the TAI phenomena. Anomaly detection is an essential prerequisite to the design of a good controller and such a detector must be able to reliably predict a forthcoming TAI. To detect and predict the onset of a TAI from an ensemble of pressure time series, this paper investigates three data-driven methods: Fast Fourier transform (FFT), symbolic time series analysis (STSA), and hidden Markov modeling (HMM). The main focus of the paper is to make a comparative evaluation of these three anomaly detection methods for classification of the current regime of operation into stable and unstable categories as well as for real-time identification of precursors to impending instabilities with short-length time series of measured variables (e.g., pressure oscillations). Finally, the results, generated on experimental data from a multi-nozzle combustor apparatus, have been compared to evaluate the performance of FFT, STSA, and HMM methods for TAI analysis.

42 ENGINEERING↗
DOE OSTI2026

Latent Twins

Over the past decade, scientific machine learning has transformed the development of mathematical and computational frameworks for analyzing, modeling, and predicting complex systems. From inverse problems to numerical partial differential equations (PDEs), dynamical systems, and model reduction, these advances have pushed the boundaries of what can be simulated. Yet they have often progressed in parallel, with representation learning and algorithmic solution methods evolving largely as separate pipelines. With Latent Twins, we propose a unifying mathematical framework that creates a hidden surrogate in latent space for the underlying equations. Whereas digital twins mirror physical systems in the digital world, Latent Twins mirror mathematical systems in a learned latent space governed by operators. Through this lens, classical modeling, inversion, model reduction, and operator approximation all emerge as special cases of a single principle. We establish the fundamental approximation properties of Latent Twins for both ordinary differential equations (ODEs) and PDEs and demonstrate the framework across three representative settings: (i) canonical ODEs, capturing diverse dynamical regimes; (ii) a PDE benchmark using the shallow-water equations, contrasting Latent Twin simulations with deep operator network and forecasts with a four-dimensional variational method baseline; and (iii) a challenging real-data geopotential reanalysis dataset, reconstructing and forecasting from sparse, noisy observations. Latent Twins provide a compact, interpretable surrogate for solution operators that evaluate across arbitrary time gaps in a single-shot, while remaining compatible with scientific pipelines such as assimilation, control, and uncertainty quantification. Looking forward, this framework offers scalable, theory-grounded surrogates that bridge data-driven representation learning and classical scientific modeling across disciplines.

Latent Twins↗
DOE OSTI2024

Inverse design for waveguide dispersion with a differentiable mode solver

Inverse design of optical components based on adjoint sensitivity analysis has the potential to address the most challenging photonic engineering problems. However, existing inverse design tools based on finite-difference-time-domain (FDTD) models are poorly suited for optimizing waveguide modes for adiabatic transformation or perturbative coupling, which lies at the heart of many important photonic devices. Among these, dispersion engineering of optical waveguides is especially challenging in ultrafast and nonlinear optical applications involving broad optical bandwidths and frequency-dependent anisotropic dielectric material response. In this work, we develop gradient back-propagation through a general-purpose electromagnetic eigenmode solver and use it to demonstrate waveguide dispersion optimization for second harmonic generation with maximized phase-matching bandwidth. This optimization of three design parameters converges in eight steps, reducing the computational cost of optimization by ∼100x compared to exhaustive search and identifying new designs for broadband optical frequency doubling of laser sources in the 1.3–1.4 µm wavelength range. Furthermore, we demonstrate that the computational cost of gradient back-propagation is independent of the number of parameters, as required for optimization of complex geometries. This technique enables practical inverse design for a broad range of previously intractable photonic devices.

Gray, Dodd (ORCID:000000030469599X)↗
DOE OSTI2026

Paradigm for universal quantum information processing with integrated acousto-optic frequency beamsplitters

Frequency-bin encoding offers tremendous potential in quantum photonic information processing, in which a single waveguide can support hundreds of lightpaths in a naturally phase-stable fashion. This stability, however, comes at a cost: arbitrary unitary operations can be realized by cascaded electro-optic phase modulators and pulse shapers, but require nontrivial numerical optimization for design and have thus far been limited to discrete tabletop components. In this article, we propose, formalize, and computationally evaluate a new paradigm for universal frequency-bin quantum information processing using acousto-optic scattering processes between distinct transverse modes. We show that controllable phase matching in intermodal processes enables 2 × 2 frequency beamsplitters and transverse-mode-dependent phase shifters, which together comprise cascadable FRequency-transverse-mODe Operations (FRODOs) that can synthesize any unitary via analytical decomposition procedures. Modeling the performance of both random gates and discrete Fourier transforms, we demonstrate the feasibility of high-fidelity quantum operations with existing integrated photonics technology, highlighting prospects of parallelizable operations achieving 100% bandwidth utilization. Our approach is realizable with CMOS technology, opening the door to scalable on-chip quantum information processing in the frequency domain.

Lukens, Joseph M. [Purdue Univ., West Lafayette, I↗
DOE OSTI2020

Generation of waves in magnetized plasma [PowerPoint]

The presentation goes through assumptions made for the study, sections on plasma dispersion, whistler modes for RBR, excitation of plane waves with antenna, dipole vs loop antenna, verification of results, excitation of whistler waves with electron beams, assumptions for the model, distribution of current in space in general case, Fourier transform of current distribution, resonant excitation of waves, and ending with a discussion of generic debunching.

70 PLASMA PHYSICS AND FUSION TECHNOLOGY↗
DOE OSTI2021

Early detection and uncertainty quantification of rapid sea-level rise from Antarctica

Focal Areas: (i) classification and/or anomaly detection methods applied towards identifying critical climate system thresholds; improved emulator design for (ii) reducing computational costs relative to full-physics models and (iii) improved uncertainty quantification workflows. Rationale: The evolution of the Antarctic ice sheet in response to climate change remains the single largest uncertainty in projecting future sea-level rise (SLR), with risk-averse projections for 2100 spanning between zero and a half meter. During the past decade, DOE has made substantial investments in new ice sheet and Earth system models needed to improve both understanding and predictive capability in this area. Yet major challenges remain, including better understanding when and under what circumstances significant increases in SLR from Antarctica may be initiated and better quantifying uncertainties in model-based SLR projections. Here, we discuss the potential for transformative advances in these areas through the application of machine learning and artificial intelligence (ML and AI, respectively).

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2021

Embedding Sensors in 3D Printed Metal Structures

The Transformational Challenge Reactor (TCR) program is leveraging recent advances in modeling and simulation, materials, and additive manufacturing (AM) technologies to design a modern nuclear reactor. Some of the main TCR technologies include in situ monitoring and the integration of sensors during the manufacturing of quality-significant nuclear reactor components. This report describes the general procedure and process optimization for embedding sensors within generic stainless steel 316 (SS316) components using laser powder bed fusion (LPBF). A more detailed, quality-significant test plan and supporting procedures are available upon request (ORNL/TM-2021/2127). LPBF involves the use of a scanning laser to selectively melt regions of a powder bed, additively building a part layer by layer. This report describes the LPBF processing technique and discusses the effects of LPBF processing parameters on the success of the sensor embedding process. Experiments used machined cavities in the form of channels in an SS316 base for the sensors to lay in while material is additively built over the top, thereby embedding them in an SS316 matrix. A preliminary investigation involved using empty SS316 sheaths as surrogates to explore the effects of various LPBF processing parameters and the dimensional requirements of the machined channels. Microstructural investigations showed that a smaller channel width/depth combination closer to the sensor’s diameter was best for the embedding process. After the desired parameters were selected, Type-K thermocouples were embedded and evaluated post-embedding using nondestructive thermal testing, as well as destructive sectioning and microscopy. Post-build characterization showed that the thermocouples were well-bonded to the SS316 matrix and were fully functional after embedding. During thermal testing to temperatures up to 500 °C, the embedded thermocouples read consistently with one another and deviated only slightly from the readings of a nonembedded thermocouple located within the furnace. This slight discrepancy was most likely due to differences in the thermal time constants for a nonembedded thermocouple vs. a thermocouple embedded in a solid SS316 block. The results presented in this report will serve as the foundation for future work that will focus on embedding sensors in relevant TCR reactor components and eventually testing those components under neutron irradiation.

36 MATERIALS SCIENCE↗
DOE OSTI2026

Colorado Residential Resiliency and Managed Charging

Validate and model impacts of electric vehicle (EV) home charging on service transformers; analyze diverse grid locations and configurations to determine 'risk factors'; finalize/deliver new design tools and typical EV load curves; improve/issue new construction standards for transformers, secondaries, and services. To ensure residential charging equity, smart charge management strategies will be studied and analyzed with the goal of creating affordable demand charges for customers.

33 ADVANCED PROPULSION SYSTEMS↗
DOE OSTI2024

Time Series Foundation Models and Deep Learning Architectures for Earthquake Temporal and Spatial Nowcasting

Advancing the capabilities of earthquake nowcasting, the real-time forecasting of seismic activities, remains crucial for reducing casualties. This multifaceted challenge has recently gained attention within the deep learning domain, facilitated by the availability of extensive earthquake datasets. Despite significant advancements, the existing literature on earthquake nowcasting lacks comprehensive evaluations of pre-trained foundation models and modern deep learning architectures; each focuses on a different aspect of data, such as spatial relationships, temporal patterns, and multi-scale dependencies. This paper addresses the mentioned gap by analyzing different architectures and introducing two innovative approaches called Multi Foundation Quake and GNNCoder. We formulate earthquake nowcasting as a time series forecasting problem for the next 14 days within 0.1-degree spatial bins in Southern California. Earthquake time series are generated using the logarithm energy released by quakes, spanning 1986 to 2024. Our comprehensive evaluations demonstrate that our introduced models outperform other custom architectures by effectively capturing temporal-spatial relationships inherent in seismic data. The performance of existing foundation models varies significantly based on the pre-training datasets, emphasizing the need for careful dataset selection. However, we introduce a novel method, Multi Foundation Quake, that achieves the best overall performance by combining a bespoke pattern with Foundation model results handled as auxiliary streams.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2022

Data augmentation for disruption prediction via robust surrogate models

The goal of this work is to generate large statistically representative datasets to train machine learning models for disruption prediction provided by data from few existing discharges. Such a comprehensive training database is important to achieve satisfying and reliable prediction results in artificial neural network classifiers. Here, we aim for a robust augmentation of the training database for multivariate time series data using Student-t process regression. We apply Student-t process regression in a state space formulation via Bayesian filtering to tackle challenges imposed by outliers and noise in the training data set and to reduce the computational complexity. Thus, the method can also be used if the time resolution is high. We use an uncorrelated model for each dimension and impose correlations afterwards via coloring transformations. We demonstrate the efficacy of our approach on plasma diagnostics data of three different disruption classes from the DIII-D tokamak. To evaluate if the distribution of the generated data is similar to the training data, we additionally perform statistical analyses using methods from time series analysis, descriptive statistics, and classic machine learning clustering algorithms.

97 MATHEMATICS AND COMPUTING↗