Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “Transformer model”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 451 records · Page 25
All sourcesNASA NTRSDOE OSTI
DOE OSTI2025

Denoising Seismograms in the Time Domain Using a Deep Learning Model

Deep learning has emerged as a transformative tool for enhancing the extraction of reliable information from seismograms, addressing the increasing demand for precise and efficient seismic data analysis. We introduce an innovative encoder–decoder deep learning model, named WaveDenoiser, designed for noise reduction in the time domain, thereby eliminating the need for spectrogram computations that have been used for existing deep learning tools and significantly improving processing speed. Utilizing the benchmark dataset that is Stanford Earthquake Dataset, we developed three models of varying sizes: base, medium, and large. Notably, the large (referred to as WaveDenoiser) model demonstrated superior performance, achieving a median signal‐to‐noise ratio improvement of 8.8 dB on in‐distribution unseen data (in the same geographic region) and 7.7 dB on out‐distribution unseen data (in a new geographic region), outpacing both the base and medium models. Further evaluation of the WaveDenoiser model revealed a reduction in median arrival‐time errors by 0.02 s for P waves and 0.01 s for S waves when processing waveforms prior to phase picking using PhaseNet on in‐distribution unseen data. When tested on out‐distribution unseen data, the model also effectively reduced the P‐wave median arrival‐time error by 0.02 and 0.01 s in median arrival‐time error for S waves. Importantly, the application of WaveDenoiser resulted in a significant reduction of phase picking outliers by 1.1% to 3.6% for both P and S waves. In addition, we achieved over five times acceleration in processing speed compared with the seisBench implementation of DeepDenoiser. Our findings underscore the potential of WaveDenoiser as a powerful tool for improving seismic data analysis and processing efficiency.

P-waves↗
DOE OSTI2021

HIRM v1.0: a hybrid impulse response model for climate modeling and uncertainty analyses

Simple climate models (SCMs) are frequently used in research and decision-making communities because of their flexibility, tractability, and low computational cost. SCMs can be idealized, flexibly representing major climate dynamics as impulse response functions, or process-based, using explicit equations to model possibly nonlinear climate and Earth system dynamics. Each of these approaches has strengths and limitations. Here we present and test a hybrid impulse response modeling framework (HIRM) that combines the strengths of process-based SCMs in an idealized impulse response model, with HIRM's input derived from the output of a process-based model. This structure enables the model to capture some of the major nonlinear dynamics that occur in complex climate models as greenhouse gas emissions transform to atmospheric concentration to radiative forcing to climate change. As a test, the HIRM framework was configured to emulate the total temperature of the simple climate model Hector 2.0 under the four Representative Concentration Pathways and the temperature response of an abrupt 4 times CO 2 concentration step. HIRM was able to reproduce near-term and long-term Hector global temperature with a high degree of fidelity. Additionally, we conducted two case studies to demonstrate potential applications for this hybrid model: examining the effect of aerosol forcing uncertainty on global temperature and incorporating more process-based representations of black carbon into a SCM. The open-source HIRM framework has a range of applications including complex climate model emulation, uncertainty analyses of radiative forcing, attribution studies, and climate model development.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2022

Uncertainty Quantification in High-Low Dynamic System Coupling using RAVEN and TRANSFORM

This work demonstrates new functionality and applications stemming from the development of high-fidelity to low-fidelity (high-low) coupling for system simulations and to further explore the capabilities of the Risk Analysis Virtual Environment (RAVEN) in the performance of uncertainty quantification in this kind of high-low coupled system models. The work builds from previous work on high-low coupling that utilized COBRA-TF (CTF), the high-fidelity subchannel analysis code, with a low fidelity model built in ORNL’s TRANSFORM, the system analysis code, utilizing the Functional Mock-Up Interface (FMI). Steady-state and transient analysis examples using the high/low coupled models generated from CTF and TRANSFORM/FMI are investigated. The workflows for both steady-state and transient coupled simulations are described. A steady-state parameter sweep and uncertainty analysis of the primary flow rates and reactor power are demonstrated. Likewise, a transient pump trip and power ramp sensitivity studies are also demonstrated. This work elucidates some of the potential benefits and future needs of using RAVEN for high/low system coupling analysis of energy systems. It also shows some of the difficulties that can be encountered in coupling system simulations.

Williams, Wesley↗
DOE OSTI2023

End-to-end AI framework for interpretable prediction of molecular and crystal properties

We introduce an end-to-end computational framework that allows for hyperparameter optimization using the DeepHyper library, accelerated model training, and interpretable AI inference. The framework is based on state-of-the-art AI models including CGCNN, PhysNet, SchNet, MPNN, MPNN-transformer, and TorchMD-NET. We employ these AI models along with the benchmark QM9, hMOF, and MD17 datasets to showcase how the models can predict user-specified material properties within modern computing environments. We demonstrate transferable applications in the modeling of small molecules, inorganic crystals and nanoporous metal organic frameworks with a unified, standalone framework. We have deployed and tested this framework in the ThetaGPU supercomputer at the Argonne Leadership Computing Facility, and in the Delta supercomputer at the National Center for Supercomputing Applications to provide researchers with modern tools to conduct accelerated AI-driven discovery in leadership-class computing environments. We release these digital assets as open source scientific software in GitLab, and ready-to-use Jupyter notebooks in Google Colab.

36 MATERIALS SCIENCE↗
DOE OSTI2024

Leveraging generative AI for urban digital twins: a scoping review on the autonomous generation of urban data, scenarios, designs, and 3D city models for smart city advancement

The digital transformation of modern cities by integrating advanced information, communication, and computing technologies has marked the epoch of data-driven smart city applications for efficient and sustainable urban management. Despite their effectiveness, these applications often rely on massive amounts of high-dimensional and multi-domain data for monitoring and characterizing different urban sub-systems, presenting challenges in application areas that are limited by data quality and availability, as well as costly efforts for generating urban scenarios and design alternatives. As an emerging research area in deep learning, Generative Artificial Intelligence (GenAI) models have demonstrated their unique values in content generation. This paper aims to explore the innovative integration of GenAI techniques and urban digital twins to address challenges in the planning and management of built environments with focuses on various urban sub-systems, such as transportation, energy, water, and building and infrastructure. The survey starts with the introduction of cutting-edge generative AI models, such as the Generative Adversarial Networks (GAN), Variational Autoencoders (VAEs), Generative Pre-trained Transformer (GPT), followed by a scoping review of the existing urban science applications that leverage the intelligent and autonomous capability of these techniques to facilitate the research, operations, and management of critical urban subsystems, as well as the holistic planning and design of the built environment. Based on the review, we discuss potential opportunities and technical strategies that integrate GenAI models into the next-generation urban digital twins for more intelligent, scalable, and automated smart city development and management.

3D city modeling↗
DOE OSTI2024

Learning stochastic dynamics and predicting emergent behavior using transformers

We show that a neural network originally designed for language processing can learn the dynamical rules of a stochastic system by observation of a single dynamical trajectory of the system, and can accurately predict its emergent behavior under conditions not observed during training. We consider a lattice model of active matter undergoing continuous-time Monte Carlo dynamics, simulated at a density at which its steady state comprises small, dispersed clusters. We train a neural network called a transformer on a single trajectory of the model. The transformer, which we show has the capacity to represent dynamical rules that are numerous and nonlocal, learns that the dynamics of this model consists of a small number of processes. Forward-propagated trajectories of the trained transformer, at densities not encountered during training, exhibit motility-induced phase separation and so predict the existence of a nonequilibrium phase transition. Transformers have the flexibility to learn dynamical rules from observation without explicit enumeration of rates or coarse-graining of configuration space, and so the procedure used here can be applied to a wide range of physical systems, including those with large and complex dynamical generators.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2022

Sampling lattices in semi-grand canonical ensemble with autoregressive machine learning

Calculating thermodynamic potentials and observables efficiently and accurately is key for the application of statistical mechanics simulations to materials science. However, naive Monte Carlo approaches, on which such calculations are often dependent, struggle to scale to complex materials in many state-of-the-art disciplines such as the design of high entropy alloys or multi-component catalysts. To address this issue, we adapt sampling tools built upon machine learning-based generative modeling to the materials space by transforming them into the semi-grand canonical ensemble. Furthermore, we show that the resulting models are transferable across wide ranges of thermodynamic conditions and can be implemented with any internal energy model U, allowing integration into many existing materials workflows. We demonstrate the applicability of this approach to the simulation of benchmark systems (AgPd, CuAu) that exhibit diverse thermodynamic behavior in their phase diagrams. Finally, we discuss remaining challenges in model development and promising research directions for future improvements.

36 MATERIALS SCIENCE↗
DOE OSTI2025

A generalizable machine learning-assisted fast Fourier transform algorithm to simulate the large strain phenomena in polycrystalline materials

Machine learning methods have shown initial promise in constitutive modeling for single crystals or homogenized polycrystals, delivering notable computational efficiency. However, existing machine learning-based constitutive models often lack generalizability, limiting their application across diverse boundary value problems. This study introduces a thermodynamics-informed artificial neural network model to accelerate rate-tangent crystal plasticity fast Fourier transform simulations for cross-scale deformation behaviors of polycrystals under complex loading. Our model integrates microstructural variability and local interactions effectively. To address local effects in each grain, we employ K-means clustering to group Gauss points within the microstructure into clusters assumed to be in similar mechanical states. This approach, based on self-clustering analysis, extends model scope from macroscopic stress response to the granular level, capturing mechanical responses and orientation evolution across grains. This reduces the number of nonlinear problems to solve, with cluster responses propagated throughout each group. The thermodynamics-based artificial neural network-extracted features are further processed using local material state clusters to account for history-dependent deformation and evolving microstructures. Additionally, representative volume element simulations with rate-tangent crystal plasticity fast Fourier transform provide reliable datasets for model training. The proposed model demonstrates high efficiency, accuracy, self-consistency, and enhanced generalizability in predicting strain–stress responses and orientation evolution at both individual grain and aggregate scales under complex loading conditions, such as biaxial tension and arbitrary loading scenarios.

36 MATERIALS SCIENCE↗
DOE OSTI2024

Advancing Building Energy Modeling with Large Language Models: Exploration and Case Studies

The rapid progression in artificial intelligence has facilitated the emergence of large language models like ChatGPT, offering potential applications extending into specialized engineering modeling, especially physics-based building energy modeling. This paper investigates the innovative integration of large language models with building energy modeling software, focusing specifically on the fusion of ChatGPT with EnergyPlus. A literature review is first conducted to reveal a growing trend of incorporating large language models in engineering modeling, albeit limited research on their application in building energy modeling. We underscore the potential of large language models in addressing building energy modeling challenges and outline potential applications including simulation input generation, simulation output analysis and visualization, conducting error analysis, co-simulation, simulation knowledge extraction and training, and simulation optimization. Three case studies reveal the transformative potential of large language models in automating and optimizing building energy modeling tasks, underscoring the pivotal role of artificial intelligence in advancing sustainable building practices and energy efficiency. The case studies demonstrate that selecting the right large language model techniques is essential to enhance performance and reduce engineering efforts. The findings advocate a multidisciplinary approach in future artificial intelligence research, with implications extending beyond building energy modeling to other specialized engineering modeling.

building energy modeling↗
DOE OSTI2025

Point cloud-based diffusion models for the Electron-Ion Collider

At high-energy collider experiments, generative models can be used for a wide range of tasks, including fast detector simulations, unfolding, searches of physics beyond the Standard Model, and inference tasks. In particular, it has been demonstrated that score-based diffusion models can generate high-fidelity and accurate samples of jets or collider events. This work expands on previous generative models in three distinct ways. First, our model is trained to generate entire collider events, including all particle species with complete kinematic information. We quantify how well the model learns event-wide constraints such as the conservation of momentum and discrete quantum numbers. We focus on the events at the future Electron-Ion Collider, but we expect that our results can be extended to proton-proton and heavy-ion collisions. Second, previous generative models often relied on image-based techniques. The sparsity of the data can negatively affect the fidelity and sampling time of the model. We address these issues using point clouds and a novel architecture combining edge creation with transformer modules called Point Edge Transformers. Third, we adapt the foundation model OmniLearn, to generate full collider events. This approach may indicate a transition toward adapting and fine-tuning foundation models for downstream tasks instead of training new models from scratch.

Araz, Jack Y. [Stony Brook Univ., NY (United State↗
DOE OSTI2024

Envisioning U.S. Climate Predictions and Projections to Meet New Challenges

In the face of a changing climate, the understanding, predictions, and projections of natural and human systems are increasingly crucial to prepare and cope with extremes and cascading hazards, determine unexpected feedbacks and potential tipping points, inform long-term adaptation strategies, and guide mitigation approaches. Increasingly complex socio-economic systems require enhanced predictive information to support advanced practices. Such new predictive challenges drive the need to fully capitalize on ambitious scientific and technological opportunities. These include the unrealized potential for very high-resolution modeling of global-to-local Earth system processes across timescales, reduction of model biases, enhanced integration of human systems and the Earth Systems, better quantification of predictability and uncertainties; expedited science-to-service pathways, and co-production of actionable information with stakeholders. Enabling technological opportunities include exascale computing, advanced data storage, novel observations and powerful data analytics, including artificial intelligence and machine learning. Looking to generate community discussions on how to accelerate progress on U.S. climate predictions and projections, representatives of Federally-funded U.S. modeling groups outline here perspectives on a six-pillar national approach grounded in climate science that builds on the strengths of the U.S. modeling community and agency goals. This calls for an unprecedented level of coordination to capitalize on transformative opportunities, augmenting and complementing current modeling center capabilities and plans to support agency missions. Tangible outcomes include projections with horizontal spatial resolutions finer than 10 km, representing extremes and associated risks in greater detail, reduced model errors, better predictability estimates, and more customized projections to support next generation climate services.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2022

Fabrication of Arrays of Topological Solitons in Patterned Chiral Liquid Crystals for Real-Time Observation of Morphogenesis

Topological solitons have knotted continuous field configurations embedded in a uniform background, and occur in cosmology, biology, and electromagnetism. However, real-time observation of their morphogenesis and dynamics is still challenging because their size and timescale are enormously large or tiny. Liquid crystal (LC) structures are promising candidates for a model-system to study the morphogenesis of topological solitons, enabling direct visualization due to the proper size and timescale. We report a new way is found to rationalize the real-time observation of the generation and transformation of topological solitons using cholesteric LCs confined in patterned substrates. The experimental demonstration shows the topologically protected structures arise via the transformation of topological defects. Numerical modeling based on minimization of free energy closely reconstructs the experimental findings. The fundamental insights obtained from the direct observations pose new theoretical challenges in understanding the morphogenesis of different types of topological solitons within a broad range of scales.

36 MATERIALS SCIENCE↗
DOE OSTI2024

Equity and modeling in sustainability science: Examples and opportunities throughout the process

Equity is core to sustainability, but current interventions to enhance sustainability often fall short in adequately addressing this linkage. Models are important tools for informing action, and their development and use present opportunities to center equity in process and outcomes. This Perspective highlights progress in integrating equity into systems modeling in sustainability science, as well as key challenges, tensions, and future directions. We present a conceptual framework for equity in systems modeling, focused on its distributional, procedural, and recognitional dimensions. We discuss examples of how modelers engage with these different dimensions throughout the modeling process and from across a range of modeling approaches and topics, including water resources, energy systems, air quality, and conservation. Synthesizing across these examples, we identify significant advances in enhancing procedural and recognitional equity by reframing models as tools to explore pluralism in worldviews and knowledge systems; enabling models to better represent distributional inequity through new computational techniques and data sources; investigating the dynamics that can drive inequities by linking different modeling approaches; and developing more nuanced metrics for assessing equity outcomes. We also identify important future directions, such as an increased focus on using models to identify pathways to transform underlying conditions that lead to inequities and move toward desired futures. By looking at examples across the diverse fields within sustainability science, we argue that there are valuable opportunities for mutual learning on how to use models more effectively as tools to support sustainable and equitable futures.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2025

Bridging the Gap Between Astronomical Datasets: From Proof-of-Concept to AI Model Deployment with Domain Adaptation

Artificial Intelligence is transforming astrophysics, from studying stars and galaxies to analyzing cosmic large-scale structures. However, a critical challenge arises when AI models trained on simulations or past observational data are applied to new observation— leading to domain shifts, reduced robustness, and increased uncertainty of model predictions. This talk will explore these issues, highlighting examples such as galaxy morphology classification and cosmological parameter inference, where AI struggles to adapt across different datasets. We will discuss domain adaptation as a strategy to improve model generalization and mitigate biases—essential for making AI-driven discoveries reliable. Notably, these challenges extend beyond astrophysics, affecting AI applications across physics and other scientific domains. Addressing them is essential for maximizing AI’s impact in advancing scientific research.

Ćiprijanović, Aleksandra [Fermilab]↗
DOE OSTI2025

Validating Greater Sage-Grouse Individual-based Model (IBM) Tool (Final Report)

The project focused on validating the previously developed Greater Sage-Grouse Individual-based Model (GrSG IBM; LaGory et al. 2012, 2021). The objective was to transform this predictive, spatially and temporally explicit model into a portable resource to assist siting/resource managers in proactively assessing the cumulative impacts of wind energy development on the greater sage-grouse. Utilizing a bottom-up, individual-based approach, the GrSG IBM accounts for landscape context and species behavior, aiming to reduce uncertainty in estimating development impacts and support ecologically mindful land-based wind energy development. The validation effort covered approximately 6,540 km 2 near the Seven Mile Hill Wind Project in Wyoming. The GrSG IBM tool, built on the NetLogo platform (Tisue and Wilensky 2004), was executed over a 50-year period, with the analysis focusing on years following a 10-year initialization phase. Key results demonstrated the tool’s biological soundness across five key biological metrics: non-chick age class distribution (older than 10 weeks), adult sex ratio, life expectancy, population size, and overall population growth. For instance, the tool estimated that 58.6% of the non-chick population was reproductively immature, while the reference ranges from 51.4% to 57.8% (Patterson 1952, Rogers 1964). Experts confirmed the tool’s estimate was within a reasonable range for the species. The tool estimated average life expectancy of 1.43 years, while the reference ranges from 0.9 years to 1.1 years (Ammann 1957, Hamerstrom 1949). Experts also supported the model’s life-expectancy estimate as ecologically sound for the species in the study area. In terms of population change, the model estimated an annual shift between a 0.6% decline and a 1.0% increase over 50 years. While the reference suggests 2.9% annual decline in range-wide populations (Cortes et al. 2023), that includes many at-risk populations in South Dakota and Washington, for example. Our study area—in the northeastern part of Carbon County and western-edge of Albany County, Wyoming—is one of the remaining greater sage-grouse habitats supporting some of the most stable populations. Experts confirmed that the range of the annual population change spanning from a 0.6% decline to a 1.0% increase estimated by the tool was reasonable for our study area for this reason and confirmed that aligned with population estimates from existing studies on the greater sage-grouse and wind energy development in the study area (LeBeau et al. 2017a, Smith et al. 2024). Furthermore, the project showed that temporally explicit biological metrics generated by the GrSG IBM tool can complement the USGS’ Prioritizing Restoration of Sagebrush Ecosystems Tool (PReSET; Duchardt et al. 2021) by incorporating habitat restoration strategies into seasonal habitat suitability models to visualize population responses over time.

17 WIND ENERGY↗
DOE OSTI2021

Defect-Driven Oxide Transformations and the Electrochemical Interphase

The redox reaction pathway is crucial to the sustainable production of the fuels and chemicals required for a carbon-neutral society. Our society is becoming increasingly dependent on devices using batteries and electrolyzers, all of which rely on a series of redox reactions. The overall properties of oxide materials make them very well suited for such electrochemical and catalytic applications due to their associated cationic redox properties and the static site-adsorbate interactions. As these technologies have matured, it has become apparent that defect-driven redox reactions, defect-coupled diffusion, and structural transformations that are both time- and rate-dependent are also critical materials processes. This change in focus, considering not only redox properties but also more complex, dynamic behaviors, represents a new research frontier in the molecular sciences as they are strongly linked to device operation and degradation and lie at the heart of various phenomena that take place at electrochemical interfaces. Fundamental studies of the structural, electronic, and chemical transformation mechanisms are key to the advancement of materials and technological innovations that could be implemented in various electrochemical systems. In this Account, we focus on recent studies and advances in characterizing and understanding the dynamic redox evolution and structural transformations that take place in model perovskites and layered oxides under reactive conditions and correlate them with degradation mechanisms and operations in electrolyzers and batteries. We show that the dynamic evolution of oxygen vacancies and cationic migration in the surface or bulk occurs at the solid-liquid interface, using a combination of different synchrotron-based Xray spectroscopies and scattering probes. Detailed redox-structure-reactivity correlation studies show how defects and diffusion processes can be tailored to drive various physical and chemical transformations in electrolyzers and batteries. We also highlight a strong correlation between oxygen redox reactivity and structural reorganization in both model thin films and particles, helping to bridge the gap between fundamental studies of the reaction mechanism and device applications. Finally, on the basis of these findings, we discuss strategies to probe and tune the redox reactivity and structural stability of the redox-active oxide interphase toward devising efficient pathways for energy and chemical harvesting.

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗
DOE OSTI2022

Symmetry of Cosmological Observables, a Mirror World Dark Sector, and the Hubble Constant

We find that a uniform scaling of the gravitational free-fall rates and photon-electron scattering rate leaves most dimensionless cosmological observables nearly invariant. This result opens up a new approach to reconciling cosmic microwave background and large-scale structure observations with high values of the Hubble constant H0: Find a cosmological model in which the scaling transformation can be realized without violating any measurements of quantities not protected by the symmetry. A “mirror world” dark sector allows for effective scaling of the gravitational free-fall rates while respecting the measured mean photon density today. Further model building might bring consistency with the two constraints not yet satisfied: the inferred primordial abundances of deuterium and helium.

79 ASTRONOMY AND ASTROPHYSICS↗