Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “Transformer model”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 145 records · Page 8
All sourcesNASA NTRSDOE OSTI
DOE OSTI2025

Comprehensive Database of Environmental Mitigations Extracted from FERC-Licensed Hydropower Projects Using Artificial Intelligence Techniques, 1998-2023

This dataset provides a comprehensive inventory of environmental mitigation measures required by Federal Energy Regulatory Commission (FERC) licensed hydropower facilities from 461 licenses that were issued from 1998 to 2023. These licenses constitute 446 of the 1015 FERC projects that were active at the end of 2023. 17,612 mentions of environmental mitigations were identified and categorized in 128 unique categories. Mitigations were identified using a Natural Language Processing (NLP) approach, specifically with a Bidirectional Encoder Representations from Transformer (BERT) model. Model-derived results were then reviewed and updated by a subject matter expert as needed. This dataset introduces important enhancements to previous efforts to inventory environmental mitigations, such as including associated license text for each mitigation, tracking the number of instances a mitigation was identified within a license, and providing improved location information. These enhancements significantly expand the dataset's utility, offering greater analytical capabilities and ensuring reproducibility. The dataset is downloadable as a zip file containing the metadata and dataset files.

Ruggles, Thomas [Oak Ridge National Laboratory (OR↗
DOE OSTI2021

Transforming a Simple Structure Model to Represent a Complex Dynamic System with Unknown Boundary Restraints

Imposing a boundary condition on a structure can significantly alter its dynamic properties. However, sometimes the specifics of the new boundary conditions are not known. When the effects of a boundary condition are uncertain or there is not enough information, engineers need to excite the complex structure to obtain these modified properties. In order to experimentally obtain the new properties, engineers need multiple experiments and many outputs for interpolation in order to sufficiently represent the entire structure. In this study, the researchers attached a stinger to a cantilever beam, acting as a new transverse restraint of unknown properties. This paper presents a conversion expression that predicts the dynamic behavior of any point in the system with the new boundary condition. This expression relies only on one impact hammer experiment with one output and the model of the stinger-free cantilever beam, referred to as the simple structure. Researchers estimated the Transfer Function (TRF) of the beam and compared it with an experimentally measured TRF to validate the method. The mean absolute error of the estimated TRF compared to the experimental TRF is 1.99 dB. This demonstrates the use of the proposed method for estimating unmeasured TRFs in a system with an uncertain boundary condition using a single input, single output (SISO) test and a model of the simple structure.

42 ENGINEERING↗
DOE OSTI2020

Broadband Characterization and Circuit Model Development of Transmission-Scale Transformers

This report describes broadband measurements of transmission-scale transformers typical in the electric power grid. This work was performed as part of the EMP Resilient Grid LDRD project at Sandia National Laboratories to generate circuit models that can be used for high-altitude electromagnetic pulse (HEMP) coupling simulations and response predictions. The objective of the work was to obtain characterization data of substation yard equipment across a frequency range relevant to HEMP. Vector network analyzer measurements up to 100 MHz were performed on two power transformers at ABB-Hitachi and a single ITEC potential transformer. Custom cable breakouts were designed to interface with the transformer terminals and provide ground connections to the chassis at the base of the transformer bushings. The three-phase terminals of the power transformers were measured as a common mode impedance using a parallel resistive splitter, and the single-phase terminals of the potential transformer were measured directly. A vector fitting algorithm was used to empirically fit circuit models to the resulting two-port networks and input impedances of the measured objects. Simplified circuit representations of the input impedances were also generated to assess the degree of precision needed for high-altitude electromagnetic pulse response predictions, which were performed in Sandia's XYCE circuit simulator platform. HEMP coupling simulations using the transformer models showed significant reduction in the voltage peak and broadening in the pulse width seen at the power transformer compared to the traveling wave voltage. This indicated the importance of the load condition when defining the coupled insult in an electric power substation. Simplified circuit models showed a similar voltage at the transformer with a smoothed waveform. The presence of potential transformers in the simulation did not significantly change the simulated voltage at the power transformer. Single-port input impedance models were also developed to define load conditions when transfer characteristics were not necessary.

24 POWER TRANSMISSION AND DISTRIBUTION↗
DOE OSTI2025

On the effectiveness of neural operators at zero-shot weather downscaling

Machine-learning (ML) methods have shown great potential for weather downscaling. These data-driven approaches provide a more efficient alternative for producing high-resolution weather datasets and forecasts compared to physics-based numerical simulations. Neural operators, which learn solution operators for a family of partial differential equations, have shown great success in scientific ML applications involving physics-driven datasets. Neural operators are grid-resolution-invariant and are often evaluated on higher grid resolutions than they are trained on, i.e., zero-shot super-resolution. Given their promising zero-shot super-resolution performance on dynamical systems emulation, we present a critical investigation of their zero-shot weather downscaling capabilities, which is when models are tasked with producing high-resolution outputs using higher upsampling factors than are seen during training. To this end, we create two realistic downscaling experiments with challenging upsampling factors (e.g., 8x and 15x) across data from different simulations: the European Centre for Medium-Range Weather Forecasts Reanalysis version 5 (ERA5) and the Wind Integration National Dataset Toolkit. While neural operator-based downscaling models perform better than interpolation and a simple convolutional baseline, we show the surprising performance of an approach that combines a powerful transformer-based model with parameter-free interpolation at zero-shot weather downscaling. We find that this Swin-Transformer-based approach mostly outperforms models with neural operator layers in terms of average error metrics, whereas an Enhanced Super-Resolution Generative Adversarial Network-based approach is better than most models in terms of capturing the physics of the ground truth data. We suggest their use in future work as strong baselines.

17 WIND ENERGY↗
DOE OSTI2020

Addressing Human and Organizational Factors in Nuclear Industry Modernization: An Operationally Focused Approach to Process and Methodology

Utility owners and operators of commercial nuclear power plants in the United States (U.S.) are and will be modernizing their nuclear power plants by performing a digital transformation involving design of an integrated set of systems that together enable a technology centric operating plant. The Plant Modernization Pathway of the U.S. Department of Energy Light Water Reactor Sustainability Program has a strategic action plan that lays the groundwork for a digital transformation of the nuclear industry. The model for this transformation is an advanced concept of operations, with an end point vision, “To achieve the maximum aggregate benefit enabled by this digital transformation.” To achieve this, the digital infrastructure for a nuclear plant must be designed as an integrated set of systems that together enable a technology centric operating model. The digital transformation process obviously needs to involve technology considerations and systems engineering, but it also needs to include human and organizational expertise. Thus, human and organizational factors, including sociotechnical systems methods and techniques (e.g., Cognitive Systems Engineering, Systems Theoretic Accident Modeling and Processes, human systems integration, and Macroergonomics) need to be considered for digital transformation projects in order to effectively integrate human and organizational expertise efforts into the new work system that results from nuclear power plant digital modernization. That is, the work system is the basic unit of sociotechnical systems analysis and contains three components: personnel, technical, and organization and management. These components should be jointly optimized with respect to the interdependence of systems performance criteria of effectiveness, efficiency and safety. Joint optimization can be achieved through the application of three human and organization functions: knowledge representation, knowledge elicitation, and cross-functional integration. This report provides a strategic framework for effective integration of human and organizational expertise within nuclear power plant digital modernization efforts.

22 GENERAL STUDIES OF NUCLEAR REACTORS↗
DOE OSTI2026

PyJMAK: An Open-Source Python Toolkit for Modeling Solid-State Metallurgical Phase Transformations

Accurate prediction of metallurgical phase transformations is an essential basis for autonomous optimization and rapid part qualification. Several methods can be used to estimate the evolution of phase fractions such as JMAK kinetics-based models, phase-field models, thermodynamic models, and data-driven machine learning models. Thermodynamic and phase-field-based methodologies solve multiphysics equations requiring numerous calibration parameters and significant computational resources. As a result, the computation domain is limited to a point or on order of micron-meters. The data-driven models rely on large datasets from experiments and simulations. While the JMAK model only provides information about phase fraction evolution, it can predict this evolution in near real-time using thermal history and thermodynamic data without restriction on the domain. JMAK models have been popularly used by researchers to model phase transformations occuring during additive manufacturing or over arbitrary temperature profiles. Commercial proprietary software such as Abaqus and Ansys or closed-source in-house implementations offer the ability to model JMAK based kinetics to predict phase transformation. However, these software packages are not open-source or freely available for use and development in conjunction with manufacturing machines, sensors, and machine learning algorithms. In addition, the use of the model is restricted by a license token. In contrast, given temperature profiles at multiple points in the domain, this Python-based PyJMAK model can compute phase evolution in parallel due to its stand-alone modular, voxel-based structure, and it can be executed on high-performance computing resources without any license restrictions.

Prabhune, Bhagya [Oak Ridge National Laboratory (O↗
DOE OSTI2022

Effect of crystallite geometries on electrochemical performance of porous intercalation electrodes by multiscale operando investigation

Lithium-ion batteries are yet to realize their full promise because of challenges in the design and construction of electrode architectures that allow for their entire interior volumes to be reversibly accessible for ion storage. Electrodes constructed from the same material and with the same specifications, which differ only in terms of dimensions and geometries of the constituent particles, can show surprising differences in polarization, stress accumulation and capacity fade. Here, using operando synchrotron X-ray diffraction and energy dispersive X-ray diffraction (EDXRD), we probe the mechanistic origins of the remarkable particle geometry-dependent modification of lithiation-induced phase transformations in V2O5 as a model phase-transforming cathode. A pronounced modulation of phase coexistence regimes is observed as a function of particle geometry. Specifically, a metastable phase is stabilized for nanometre-sized spherical V2O5 particles, to circumvent the formation of large misfit strains. Spatially resolved EDXRD measurements demonstrate that particle geometries strongly modify the tortuosity of the porous cathode architecture. Greater ion-transport limitations in electrode architectures comprising micrometre-sized platelets result in considerable lithiation heterogeneities across the thickness of the electrode. These insights establish particle geometry-dependent modification of metastable phase regimes and electrode tortuosity as key design principles for realizing the promise of intercalation cathodes. Designing electrode architectures for Li-ion batteries that can be reversibly accessible for ion storage can be challenging. Using operando techniques the mechanistic origin of lithiation-induced phase transformations in a V2O5 model cathode is now clarified.

Luo, Yuting↗
DOE OSTI2023

Development of a molecularly informed biogeochemical framework for reactive transport modeling of subsurface carbon inventories, transformations and fluxes (Final Report)

The overall objective of the project was to combine new molecular-level characterization strategies with soil carbon flux measurements to develop and evaluate model representations of subsurface carbon cycling. We expanded on existing studies in the East River watershed, Colorado, in collaboration with the Berkeley Lab Watershed Function Scientific Focus Area (SFA), SLAC Groundwater Quality SFA, and Rocky Mountain Biological Laboratory (RMBL) to develop an elevation and vegetation gradient that is now the subject of long-term monitoring by the USGS. To achieve the overall objective, we combined field studies of soil respiration with laboratory analyses, ranging from spectroscopy to incubation studies. This combination of techniques enabled us to develop a new understanding of the drivers of high-elevation soil respiration. In the process, we developed three new modeling approaches that improve our ability to conceptualize and ultimately to represent soil respiration in numerical models. The first approach is a plot-scale transient inverse model that can be used to determine in situ CO 2 production rates from measured concentration profiles and surface fluxes. Application of this method revealed the importance of plant phenology and deep CO 2 production in moderating CO 2 fluxes to the atmosphere. The second modeling approach is a molecular-scale tool that enables spectroscopic and elemental data for carbon speciation to be transformed into functional group abundances, or the ‘SOC-fga model’. This method uniquely enables carbon speciation to be tracked within a reactive transport framework to partition carbon among different pathways and storage zones within the soil. The third modeling tool builds strongly on the previous approaches and captures the microbial processes driving heterotrophic respiration. This ‘dormancy model’ allows the native soil microbial population to respond transiently to the presence or absence of water in order to catalyze carbon respiration. This approach was also compared to the simpler and more widely used first-order model using two experimental datasets with different temporal and spatial resolutions. Our results illustrate that the simpler first order model provides a robust and efficient representation of deep (>1 meter) soil respiration, but that shallow soils, where most respiration occurs, require explicit representation of moisture-dependent activation and dormancy rates. By assessing soil organic carbon turnover at multiple scales, we see a complex array of controls emerge. Atte scale of a hillslope, spatial heterogeneity in soil respiration rates dominates and is uncorrelated with instantaneous soil moisture and plant community. At the profile scale, the balance between plant inputs and water availability is the dominant control. At the microbial to molecular scale, physiological processes associated with carbon use and carbon speciation are important controls. These scale-dependent controls emphasize the need for new modeling approaches that examine their interactions and hierarchies.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2025

Smart culture medium optimization for recombinant protein production: Experimental, modeling, and AI/ML-driven strategies

Recombinant protein production (RPP) is central to biotechnology, where recombinant proteins are used as either end products or catalysts in the synthesis of chemicals, fuels, and materials. Among the major cost drivers, culture medium plays a pivotal role in determining protein yield and quality. This review presents a comprehensive perspective on the critical stages of “smart” culture medium optimization: planning, screening, modeling, optimization, and validation. In the planning stage, we examine the nutritional and energetic roles of medium components, including carbon, nitrogen, amino acids, salts, and trace metals, and their impacts on culture parameters such as pH, oxidative state, and osmolality. We highlight the variability in trace metal content due to water sources, culture vessels, and raw materials, which can substantially influence RPP. The screening stage covers Design of Experiments (DoE) approaches, assessing their theoretical basis, implementation, and limitations. For modeling, we describe methods that integrate experimental data to develop predictive models for smart medium formulation. Model-based optimization strategies can then be employed to select optimal media compositions for a given application. The validation stage aims to evaluate model predictions and provide feedback for model training and refinement. Finally, we survey mechanistic and artificial intelligence/machine learning (AI/ML)-driven models as integrated, transformational tools for predictive modeling of bioprocess conditions, nutrient availability, cellular metabolism, and protein quality, with the goal of optimizing culture media to enhance protein yields while reducing costs and environmental impact. We conclude by addressing the challenges of translating laboratory-scale medium optimization to industrial-scale settings and exploring future AI/ML-driven approaches that may overcome current bottlenecks and accelerate medium design for RPP. Overall, this review provides a unified framework for advancing smart medium design in RPP.

Artificial Intelligence/Machine Learning (AI/ML)↗
DOE OSTI2024

Improved multifidelity Monte Carlo estimators based on normalizing flows and dimensionality reduction techniques

Here, we study the problem of multifidelity uncertainty propagation for computationally expensive models. In particular, we consider the general setting where the high-fidelity and low-fidelity models have a dissimilar parameterization both in terms of number of random inputs and their probability distributions, which can be either known in closed form or provided through samples. We derive novel multifidelity Monte Carlo estimators which rely on a shared subspace between the high-fidelity and low-fidelity models where the parameters follow the same probability distribution, i.e., a standard Gaussian. We build the shared space employing normalizing flows to map different probability distributions into a common one, together with linear and nonlinear dimensionality reduction techniques, active subspaces and autoencoders, respectively, which capture the subspaces where the models vary the most. We then compose the existing low-fidelity model with these transformations and construct modified models with an increased correlation with the high-fidelity model, which therefore yield multifidelity estimators with reduced variance. A series of numerical experiments illustrate the properties and advantages of our approaches.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2024

EV Forecasting-Based Model Predictive Control for Distribution System Congestion Mitigation

The uncoordinated charging of electric vehicles (EVs) in time and space brings congestion issues to the distribution network. This paper proposes an EV charging demand forecasting-based model predictive control (MPC) method for distribution system congestion management. To effectively forecast the time-series EV station charging demand, a hybrid forecasting model that integrates the long short-term memory network (LSTM) and Transformer is proposed. The Transformer-LSTM model is trained using a one-year real historical charging dataset of EV stations to forecast future charging demand in 15-minute intervals. This informs the MPC for distribution network congestion management and minimization of PV curtailment. Numerical results carried out on the modified IEEE 123-bus distribution system demonstrate that the proposed method can effectively resolve line congestion issues through EV smart charging and PV curtailment while outperforming other benchmarks.

ADVANCED PROPULSION SYSTEMS,SOLAR ENERGY↗
DOE OSTI2025

MechBERT: Language Models for Extracting Chemical and Property Relationships about Mechanical Stress and Strain

Language models are transforming materials-aware naturallanguage processing by enabling the extraction of dynamic, context-rich information from unstructured text, thus, moving beyond the limitations of traditional information-extraction methods. Moreover, small language models are on the rise because some of them can perform better than large language models (LLMs) when given domain-specific questionanswer tasks, especially about an application area that relies on a highly specialized vernacular, such as materials science. We therefore present a new class of MechBERT language models for understanding mechanical stress and strain in materials. These employ Bidirectional Encoder Representations for transformer (BERT) architectures. We showcase four MechBERT models, all of which were pretrained on a corpus of documents that are textually rich in chemicals and their stress–strain properties and were fine-tuned on question-answering tasks. We evaluated the level of performance of our models on domain-specific as well as general English-language question-answer tasks and also explored the influence of the size and type of BERT architectures on model performance. We find that our MechBERT models outperform BERT-based models of the same size and maintain relevancy better than much larger BERT-based models when tasked with domain-specific question-answering tasks within the stress–strain engineering sector. These small language models also enable much faster processing and require a much smaller fraction of data to pretrain them, affording them greater operational efficiency and energy sustainability than LLMs.

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗
DOE OSTI2024

An evaluation of GPT models for phenotype concept recognition

Clinical deep phenotyping and phenotype annotation play a critical role in both the diagnosis of patients with rare disorders as well as in building computationally-tractable knowledge in the rare disorders field. These processes rely on using ontology concepts, often from the Human Phenotype Ontology, in conjunction with a phenotype concept recognition task (supported usually by machine learning methods) to curate patient profiles or existing scientific literature. With the significant shift in the use of large language models (LLMs) for most NLP tasks, we examine the performance of the latest Generative Pre-trained Transformer (GPT) models underpinning ChatGPT as a foundation for the tasks of clinical phenotyping and phenotype annotation. The experimental setup of the study included seven prompts of various levels of specificity, two GPT models (gpt-3.5-turbo and gpt-4.0) and two established gold standard corpora for phenotype recognition, one consisting of publication abstracts and the other clinical observations. The best run, using in-context learning, achieved 0.58 document-level F1 score on publication abstracts and 0.75 document-level F1 score on clinical observations, as well as a mention-level F1 score of 0.7, which surpasses the current best in class tool. Without in-context learning, however, performance is significantly below the existing approaches. Our experiments show that gpt-4.0 surpasses the state of the art performance if the task is constrained to a subset of the target ontology where there is prior knowledge of the terms that are expected to be matched. While the results are promising, the non-deterministic nature of the outcomes, the high cost and the lack of concordance between different runs using the same prompt and input make the use of these LLMs challenging for this particular task.

59 BASIC BIOLOGICAL SCIENCES↗
DOE OSTI2025

Why Is Attention Sparse In Particle Transformer?

Transformer-based models have achieved state-of-the-art performance in jet tagging at the CERN Large Hadron Collider (LHC), with the Particle Transformer (ParT) representing a leading example of such models. A striking feature of ParT is its sparse, nearly binary, attention structure, raising questions about the origin of this behavior and whether it encodes physically meaningful correlations. In this work, we investigate the source of ParT's sparse attention by comparing models trained on multiple benchmark datasets and examine the relative contributions of the attention term and the physics-inspired interaction matrix before softmax. We find that binary sparsity arises primarily from the attention mechanism itself, with the interaction matrix playing a secondary role. Moreove, we show that ParT is able to identify key jet substructure elements, such as leptons in semileptonic top decays, even without explicit particle identification inputs. These results provide new insight into the interpretability of transformer-based jet taggers and clarify the conditions under which sparse attention patterns emerge in ParT.

Legge, Timothy [UC, San Diego]↗