SEARCH · Engineering Papers

Results for “Transformer model”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

SourceResearch connectionFrom yearThrough year

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 217 records · Page 12

All sources NASA NTRS DOE OSTI

NASA NTRS2020

Assimilation of SMAP and ASCAT Soil Moisture Retrievals into the JULES Land Surface Model Using the Local Ensemble Transform Kalman Filter

A land data assimilation system is developed to merge satellite soil moisture retrievals into the Joint U.K. Land Environment Simulator (JULES) land surface model (LSM) using the Local Ensemble Transform Kalman Filter (LETKF). The system assimilates microwave soil moisture retrievals from the Soil Moisture Active Passive (SMAP) radiometer and the Advanced Scatterometer (ASCAT) after bias correction based on cumulative distribution function fitting. The soil moisture assimilation estimates are evaluated with ground-based soil moisture measurements over the continental U.S. for five consecutive warm seasons (May–September of 2015–2019). The result shows that both SMAP and ASCAT retrievals improve the accuracy of soil moisture estimates. Especially, the SMAP single-sensor assimilation experiment shows the best performance with the increase of temporal anomaly correlation by ΔR ~ 0.05 for surface soil moisture and ΔR ~ 0.03 for root-zone soil moisture compared with the LSM simulation without satellite data assimilation. SMAP assimilation is more skillful than ASCAT assimilation primarily because of the greater skill of the assimilated SMAP retrievals compared to the ASCAT retrievals. The skill improvement also depends significantly on the region; the higher skill improvement in the western U.S. compared to the eastern U.S. is explained by the Kalman gain in the two experiments. Additionally, the regional skill differences in the single-sensor assimilation experiments are attributed to the number of assimilated observations. Finally, the soil moisture assimilation estimates provide more realistic land surface information than model-only simulations for the 2015 and the 2016 western U.S. droughts, suggesting the advantage of using satellite soil moisture retrievals in the current drought monitoring system.

SMAP↗

DOE OSTI2022

Effect of crystallite geometries on electrochemical performance of porous intercalation electrodes by multiscale operando investigation

Lithium-ion batteries are yet to realize their full promise because of challenges in the design and construction of electrode architectures that allow for their entire interior volumes to be reversibly accessible for ion storage. Electrodes constructed from the same material and with the same specifications, which differ only in terms of dimensions and geometries of the constituent particles, can show surprising differences in polarization, stress accumulation and capacity fade. Here, using operando synchrotron X-ray diffraction and energy dispersive X-ray diffraction (EDXRD), we probe the mechanistic origins of the remarkable particle geometry-dependent modification of lithiation-induced phase transformations in V2O5 as a model phase-transforming cathode. A pronounced modulation of phase coexistence regimes is observed as a function of particle geometry. Specifically, a metastable phase is stabilized for nanometre-sized spherical V2O5 particles, to circumvent the formation of large misfit strains. Spatially resolved EDXRD measurements demonstrate that particle geometries strongly modify the tortuosity of the porous cathode architecture. Greater ion-transport limitations in electrode architectures comprising micrometre-sized platelets result in considerable lithiation heterogeneities across the thickness of the electrode. These insights establish particle geometry-dependent modification of metastable phase regimes and electrode tortuosity as key design principles for realizing the promise of intercalation cathodes. Designing electrode architectures for Li-ion batteries that can be reversibly accessible for ion storage can be challenging. Using operando techniques the mechanistic origin of lithiation-induced phase transformations in a V2O5 model cathode is now clarified.

Luo, Yuting↗

NASA NTRS1975

On Similarity Transformation and Geodetic Network Distortions Based on Doppler Satellite Observations

Models used in geodesy to transform two sets of coordinates are studied and distortions in geodetic networks are investigated. Commonly used transformation models are first reviewed and most of them are interpreted. Differences between various models are discussed. Pitfalls in partial solutions are then considered. It is shown that only as many chords and/or directional elements can be used in the computation as are needed to completely determine the size or shape of the polyhedron implied in the set of Cartesian coordinates. Each additional element causes the normal matrix to be singular provided that all correlations between the chords are used. A number of tables and maps indicating distortions in the NAD 27, Precise Traverse M-R '72, AUS, and SAD 69 geodetic datums are also included. The residuals of the coordinates are scanned for systematic patterns after transforming each geodetic system to the NWL9D Doppler system. Also, an attempt is made to show scale distortions in the NAD 27.

Leick, Alfred↗

NASA NTRS1979

Synthesis of a control model for a liquid nitrogen cooled, closed circuit, cryogenic nitrogen wind tunnel and its validation

The details of the efforts to synthesize a control-compatible multivariable model of a liquid nitrogen cooled, gaseous nitrogen operated, closed circuit, cryogenic pressure tunnel are presented. The synthesized model was transformed into a real-time cryogenic tunnel simulator, and this model is validated by comparing the model responses to the actual tunnel responses of the 0.3 m transonic cryogenic tunnel, using the quasi-steady-state and the transient responses of the model and the tunnel. The global nature of the simple, explicit, lumped multivariable model of a closed circuit cryogenic tunnel is demonstrated.

Balakrishna, S.↗

DOE OSTI2023

Development of a molecularly informed biogeochemical framework for reactive transport modeling of subsurface carbon inventories, transformations and fluxes (Final Report)

The overall objective of the project was to combine new molecular-level characterization strategies with soil carbon flux measurements to develop and evaluate model representations of subsurface carbon cycling. We expanded on existing studies in the East River watershed, Colorado, in collaboration with the Berkeley Lab Watershed Function Scientific Focus Area (SFA), SLAC Groundwater Quality SFA, and Rocky Mountain Biological Laboratory (RMBL) to develop an elevation and vegetation gradient that is now the subject of long-term monitoring by the USGS. To achieve the overall objective, we combined field studies of soil respiration with laboratory analyses, ranging from spectroscopy to incubation studies. This combination of techniques enabled us to develop a new understanding of the drivers of high-elevation soil respiration. In the process, we developed three new modeling approaches that improve our ability to conceptualize and ultimately to represent soil respiration in numerical models. The first approach is a plot-scale transient inverse model that can be used to determine in situ CO 2 production rates from measured concentration profiles and surface fluxes. Application of this method revealed the importance of plant phenology and deep CO 2 production in moderating CO 2 fluxes to the atmosphere. The second modeling approach is a molecular-scale tool that enables spectroscopic and elemental data for carbon speciation to be transformed into functional group abundances, or the ‘SOC-fga model’. This method uniquely enables carbon speciation to be tracked within a reactive transport framework to partition carbon among different pathways and storage zones within the soil. The third modeling tool builds strongly on the previous approaches and captures the microbial processes driving heterotrophic respiration. This ‘dormancy model’ allows the native soil microbial population to respond transiently to the presence or absence of water in order to catalyze carbon respiration. This approach was also compared to the simpler and more widely used first-order model using two experimental datasets with different temporal and spatial resolutions. Our results illustrate that the simpler first order model provides a robust and efficient representation of deep (>1 meter) soil respiration, but that shallow soils, where most respiration occurs, require explicit representation of moisture-dependent activation and dormancy rates. By assessing soil organic carbon turnover at multiple scales, we see a complex array of controls emerge. Atte scale of a hillslope, spatial heterogeneity in soil respiration rates dominates and is uncorrelated with instantaneous soil moisture and plant community. At the profile scale, the balance between plant inputs and water availability is the dominant control. At the microbial to molecular scale, physiological processes associated with carbon use and carbon speciation are important controls. These scale-dependent controls emphasize the need for new modeling approaches that examine their interactions and hierarchies.

54 ENVIRONMENTAL SCIENCES↗

DOE OSTI2025

Smart culture medium optimization for recombinant protein production: Experimental, modeling, and AI/ML-driven strategies

Recombinant protein production (RPP) is central to biotechnology, where recombinant proteins are used as either end products or catalysts in the synthesis of chemicals, fuels, and materials. Among the major cost drivers, culture medium plays a pivotal role in determining protein yield and quality. This review presents a comprehensive perspective on the critical stages of “smart” culture medium optimization: planning, screening, modeling, optimization, and validation. In the planning stage, we examine the nutritional and energetic roles of medium components, including carbon, nitrogen, amino acids, salts, and trace metals, and their impacts on culture parameters such as pH, oxidative state, and osmolality. We highlight the variability in trace metal content due to water sources, culture vessels, and raw materials, which can substantially influence RPP. The screening stage covers Design of Experiments (DoE) approaches, assessing their theoretical basis, implementation, and limitations. For modeling, we describe methods that integrate experimental data to develop predictive models for smart medium formulation. Model-based optimization strategies can then be employed to select optimal media compositions for a given application. The validation stage aims to evaluate model predictions and provide feedback for model training and refinement. Finally, we survey mechanistic and artificial intelligence/machine learning (AI/ML)-driven models as integrated, transformational tools for predictive modeling of bioprocess conditions, nutrient availability, cellular metabolism, and protein quality, with the goal of optimizing culture media to enhance protein yields while reducing costs and environmental impact. We conclude by addressing the challenges of translating laboratory-scale medium optimization to industrial-scale settings and exploring future AI/ML-driven approaches that may overcome current bottlenecks and accelerate medium design for RPP. Overall, this review provides a unified framework for advancing smart medium design in RPP.

Artificial Intelligence/Machine Learning (AI/ML)↗

DOE OSTI2024

Improved multifidelity Monte Carlo estimators based on normalizing flows and dimensionality reduction techniques

Here, we study the problem of multifidelity uncertainty propagation for computationally expensive models. In particular, we consider the general setting where the high-fidelity and low-fidelity models have a dissimilar parameterization both in terms of number of random inputs and their probability distributions, which can be either known in closed form or provided through samples. We derive novel multifidelity Monte Carlo estimators which rely on a shared subspace between the high-fidelity and low-fidelity models where the parameters follow the same probability distribution, i.e., a standard Gaussian. We build the shared space employing normalizing flows to map different probability distributions into a common one, together with linear and nonlinear dimensionality reduction techniques, active subspaces and autoencoders, respectively, which capture the subspaces where the models vary the most. We then compose the existing low-fidelity model with these transformations and construct modified models with an increased correlation with the high-fidelity model, which therefore yield multifidelity estimators with reduced variance. A series of numerical experiments illustrate the properties and advantages of our approaches.

97 MATHEMATICS AND COMPUTING↗

DOE OSTI2024

EV Forecasting-Based Model Predictive Control for Distribution System Congestion Mitigation

The uncoordinated charging of electric vehicles (EVs) in time and space brings congestion issues to the distribution network. This paper proposes an EV charging demand forecasting-based model predictive control (MPC) method for distribution system congestion management. To effectively forecast the time-series EV station charging demand, a hybrid forecasting model that integrates the long short-term memory network (LSTM) and Transformer is proposed. The Transformer-LSTM model is trained using a one-year real historical charging dataset of EV stations to forecast future charging demand in 15-minute intervals. This informs the MPC for distribution network congestion management and minimization of PV curtailment. Numerical results carried out on the modified IEEE 123-bus distribution system demonstrate that the proposed method can effectively resolve line congestion issues through EV smart charging and PV curtailment while outperforming other benchmarks.

ADVANCED PROPULSION SYSTEMS,SOLAR ENERGY↗

DOE OSTI2025

MechBERT: Language Models for Extracting Chemical and Property Relationships about Mechanical Stress and Strain

Language models are transforming materials-aware naturallanguage processing by enabling the extraction of dynamic, context-rich information from unstructured text, thus, moving beyond the limitations of traditional information-extraction methods. Moreover, small language models are on the rise because some of them can perform better than large language models (LLMs) when given domain-specific questionanswer tasks, especially about an application area that relies on a highly specialized vernacular, such as materials science. We therefore present a new class of MechBERT language models for understanding mechanical stress and strain in materials. These employ Bidirectional Encoder Representations for transformer (BERT) architectures. We showcase four MechBERT models, all of which were pretrained on a corpus of documents that are textually rich in chemicals and their stress–strain properties and were fine-tuned on question-answering tasks. We evaluated the level of performance of our models on domain-specific as well as general English-language question-answer tasks and also explored the influence of the size and type of BERT architectures on model performance. We find that our MechBERT models outperform BERT-based models of the same size and maintain relevancy better than much larger BERT-based models when tasked with domain-specific question-answering tasks within the stress–strain engineering sector. These small language models also enable much faster processing and require a much smaller fraction of data to pretrain them, affording them greater operational efficiency and energy sustainability than LLMs.

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗

DOE OSTI2024

An evaluation of GPT models for phenotype concept recognition

Clinical deep phenotyping and phenotype annotation play a critical role in both the diagnosis of patients with rare disorders as well as in building computationally-tractable knowledge in the rare disorders field. These processes rely on using ontology concepts, often from the Human Phenotype Ontology, in conjunction with a phenotype concept recognition task (supported usually by machine learning methods) to curate patient profiles or existing scientific literature. With the significant shift in the use of large language models (LLMs) for most NLP tasks, we examine the performance of the latest Generative Pre-trained Transformer (GPT) models underpinning ChatGPT as a foundation for the tasks of clinical phenotyping and phenotype annotation. The experimental setup of the study included seven prompts of various levels of specificity, two GPT models (gpt-3.5-turbo and gpt-4.0) and two established gold standard corpora for phenotype recognition, one consisting of publication abstracts and the other clinical observations. The best run, using in-context learning, achieved 0.58 document-level F1 score on publication abstracts and 0.75 document-level F1 score on clinical observations, as well as a mention-level F1 score of 0.7, which surpasses the current best in class tool. Without in-context learning, however, performance is significantly below the existing approaches. Our experiments show that gpt-4.0 surpasses the state of the art performance if the task is constrained to a subset of the target ontology where there is prior knowledge of the terms that are expected to be matched. While the results are promising, the non-deterministic nature of the outcomes, the high cost and the lack of concordance between different runs using the same prompt and input make the use of these LLMs challenging for this particular task.

59 BASIC BIOLOGICAL SCIENCES↗

DOE OSTI2025

Why Is Attention Sparse In Particle Transformer?

Transformer-based models have achieved state-of-the-art performance in jet tagging at the CERN Large Hadron Collider (LHC), with the Particle Transformer (ParT) representing a leading example of such models. A striking feature of ParT is its sparse, nearly binary, attention structure, raising questions about the origin of this behavior and whether it encodes physically meaningful correlations. In this work, we investigate the source of ParT's sparse attention by comparing models trained on multiple benchmark datasets and examine the relative contributions of the attention term and the physics-inspired interaction matrix before softmax. We find that binary sparsity arises primarily from the attention mechanism itself, with the interaction matrix playing a secondary role. Moreove, we show that ParT is able to identify key jet substructure elements, such as leptons in semileptonic top decays, even without explicit particle identification inputs. These results provide new insight into the interpretability of transformer-based jet taggers and clarify the conditions under which sparse attention patterns emerge in ParT.

Legge, Timothy [UC, San Diego]↗

DOE OSTI2025

On the Effectiveness of Neural Operators at Zero-Shot Weather Downscaling [SWR-25-20]

Code repository for the experiments performed in the paper: On the Effectiveness of Neural Operators at Zero-Shot Weather Downscaling (https://doi.org/10.1017/eds.2025.11) Overall, our work investigates the zero-shot downscaling potential of neural operators. To summarize, our contributions are: 1. We provide a comparative analysis based on two challenging weather downscaling problems, between various neural operator and non-neural-operator methods with large upsampling factors (e.g., 8x and 15x) and fine grid resolutions (e.g., 2 km × 2 km wind speed). 2. We examine whether neural operator layers provide unique advantages when testing downscaling models on upsampling factors higher than those seen during training, i.e., zero-shot downscaling. Our results instead show the surprising success of an approach that combines a powerful transformer-based model with a parameter-free interpolation step at zero-shot weather downscaling. 3. We find that this Swin-Transformer-based approach mostly outperforms all neural operator models in terms of average error metrics, whereas an enhanced super-resolution generative adversarial network (ESRGAN)-based approach is better than most models in capturing the physics of the system, and suggests their use in future work as strong baselines. However, these approaches still do not capture variations at smaller spatial scales well, including the physical characteristics of turbulence in the HR data. This suggests a potential for improvement in transformer or GAN-based methods and neural-operator-based methods for zero-shot weather downscaling.

Sinha, Saumya [National Renewable Energy Laborator↗

NASA NTRSDate not supplied

What Went Wrong: A Survey of Wildfire UAS Mishaps through Named Entity Recognition

Increasingly, unmanned aircraft systems (UAS) are being applied to wildfire incidents for tasks such as mapping, aerial ignition, and delivery. As a result, aviation incident reporting systems for wildfires are beginning to accumulate data related to UAS mishaps in wildfire response. In this research, we apply state-of-the-art natural language processing (NLP) techniques to develop a custom Named Entity Recognition (NER) model which extracts entities relevant to safety analysts. The custom NER model is built by fine-tuning an existing Bidirectional Encoder Representations from Transformers (BERT) model, resulting in a generalizable NER model that can extract engineering relevant entities including failure modes, causes, effects, control processes, and recommendations from failure-relevant text. This model performs passably, with a weighted average f1 score of 0.33 across entity types, indicating more labeled training data is needed. Extracted entities are used to form a Failure Modes and Effects Analysis (FMEA)-style survey of wildfire UAS mishaps reported using the SAFECOM system. Similar mishaps are manually clustered and reported as single rows within an FMEA. Foreach cluster, we compute frequency, severity, and overall riskin accordance with FAA standards. This methodology can beapplied as part of a broader safety management system totrack trends in mishaps (e.g., likelihood, severity) and discoverknowledge (e.g., causes, effects) that can be utilized to improvesafety outcomes and system performance.

Machine Learning↗

NASA NTRSDate not supplied

Towards an Aviation Large Language Model by Fine-tuning and Evaluating Transformers

In the aviation domain, there are many applications for machine learning and artificial intelligence tools that utilize natural language. For example, there is a desire to know the commonalities in written safety reports such as voluntary post incidents reports or create more accurate transcripts of air traffic management conversations. Another use-case is the possibility of extracting airspace procedures and constraints currently written in documents such as Letters of Agreement (LOA) which is used as the evaluation case in this paper. These applications can benefit from the use of state-of-the-art Natural Language Processing (NLP) techniques when adapted to the language/phraseology specific to the aviation domain. This paper evaluates the viability of transferring pre-trained large language models to the aviation domain by adapting transformer based models using aviation datasets. This paper utilized two datasets to adapt a ‘Robustly Optimized Bidirectional Encoder Representations from Transformers Approach’ (RoBERTa) model and two down-stream classification tasks to assess its performance. These datasets are all built upon Letters of Agreement which are Federal Aviation Administration (FAA) documents that formalize airspace operations across the national airspace system. The first two datasets are used for the adaptation of RoBERTa to the aviation domain and were of different sizes to assess the number of documents needed to adapt to the aviation domain. They contain many examples of ‘aviation English’ using domain specific terminology and phrasing which serves as a representative basis to perform the unsupervised adaptation. The second dataset is a separate set of LOA documents with two sets of classification labels to be used for evaluation; one at the document level and one at the line level. These down-stream evaluations allowed the measurement of improvement by adapting RoBERTa. The accuracy increased by 4-6% on both tasks and the F1 score on the class of interest increased by 4-8% from the adaptation.

Air Traffic Management↗

NASA NTRSDate not supplied

Towards an Aviation Large Language Model by Fine-tuning and Evaluating Transformers

Air Traffic Management↗

DOE OSTI2025

AI‐Driven Defect Engineering for Advanced Thermoelectric Materials

Thermoelectric materials offer a promising pathway to directly convert waste heat to electricity. However, achieving high performance remains challenging due to intrinsic trade-offs between electrical conductivity, the Seebeck coefficient, and thermal conductivity, which are further complicated by the presence of defects. This review explores how artificial intelligence (AI) and machine learning (ML) are transforming thermoelectric materials design. Advanced ML approaches including deep neural networks, graph-based models, and transformer architectures, integrated with high-throughput simulations and growing databases, effectively capture structure-property relationships in a complex multiscale defect space and overcome the “curse of dimensionality”. This review discusses AI-enhanced defect engineering strategies such as composition optimization, entropy and dislocation engineering, and grain boundary design, along with emerging inverse design techniques for generating materials with targeted properties. Finally, it outlines future opportunities in novel physics mechanisms and sustainability, highlighting the critical role of AI in accelerating the discovery of thermoelectric materials.

36 MATERIALS SCIENCE↗

DOE OSTI2024

Leveraging BERT and Network-Based Attention Analysis for Identifying Treatment Milestones in EHRs

This study introduces a sophisticated data-driven framework for analyzing Electronic Health Records (EHRs) using transformer-based models to identify and disentangle overlapping treatment contexts. The framework leverages a preprocessing pipeline that transforms structured procedural codes into semantically enriched descriptive text, enabling the use of attention mechanisms to cluster medical events into treatment milestones—cohesive and distinct components of care processes. The methodology is rigorously validated using synthetic datasets derived from the MIMIC-III database, designed to simulate the heterogeneity and overlapping procedural contexts characteristic of real-world EHR scenarios. Quantitative evaluation highlights the framework’s robustness in disentangling concurrent care pathways, with attention metrics and unsupervised clustering approaches demonstrating the ability to preserve intra-context relationships while distinguishing inter-context dependencies. By addressing challenges inherent in data heterogeneity, this approach provides a foundation for uncovering complex treatment patterns, advancing clinical decision-making, and optimizing resource allocation in diverse healthcare environments.

Kim, Minsu [ORNL] (ORCID:0000000224185535)↗

DOE OSTI2020

TRANSENSOR: Transformer Real-time Assessment INtelligent System with Embedded Network of Sensors and Optical Readout. Final Report

Utilities across the world are wrestling with evolving market dynamics, population growth and climate change. Distributed Energy Resources (DERs) such as solar photovoltaics, distributed generators and energy storage systems are becoming important parts of the U.S. energy mix. These factors are driving an increasing need for low-cost grid asset monitoring. Aside from being costly, traditional utility monitoring systems are not sufficiently robust and do not provide real-time visibility into the condition of grid assets such as transformers. Lack of accurate real-time measurements on performance has resulted in the use of lagging indicators, such as oil sample analysis. To address this need, an innovative embedded optical sensing technology, Transformer Real-time Assessment Intelligent System (TRANSENSOR) was developed, validated and demonstrated in this project. To date, TRANSENSOR has focused on transformers but is extendable to other grid assets. In addition to the technology being embedded into new transformers during manufacturing, a retrofit configuration that can be installed on existing transformers in the field was also developed. It is anticipated that the ability to retrofit existing transformers will help accelerate adoption of the underlying TRANSENSOR technology. Phase 1 of the project focused on laboratory development and qualification of the technology. Following iterations and exploration of relevant optical sensing modalities and multiplexed configurations of interest for the transformer environment, an effective candidate configuration was agreed upon, down-selected and custom-designed for embedding into General Electric (GE) network transformers. Two new GE 500 kilovolt ampere (kVA), 27 kilovolt (kV) distribution network transformers were built with embedded fiber-optic (FO) sensors and successfully qualified per industry standards at GE’s Shreveport, Louisiana facility during Phase 1 of the project. Following the successful completion of Phase 1, the team proceeded to Phase 2, which focused on a field demonstration of the technology. Over Phase 2, the first new GE transformer built with embedded fiber-optic sensors was installed in an above-ground cage and connected to the grid at Con Edison’s Astoria facility. A second transformer equipped with TRANSENSOR was installed in an underground vault. Additionally, an older (1982 year model) GE transformer in an above-ground cage was retrofitted with fiber- optic sensors and reconnected to the grid at ConEd’s facility. Analysis of data acquired from the sensors showed interesting correlations with transformer loading. Additionally, key events such as transformer low-voltage network connection, primary-side energizing, and a pressure loss event from an oil sampling were detected by the TRANSENSOR system. Online data processing/feature extraction algorithms for the second transformer in the underground vault detected key features and event alerts that were transmitted through a wireless 4G connection. The remote deployment concept showed promising results with data collection running for a total of 8 months across the three (3) GE transformers instrumented for the Phase 2 field demonstration. This sets the stage well for further development and commercialization.

24 POWER TRANSMISSION AND DISTRIBUTION↗