Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “Transformer model”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 91 records · Page 5
All sourcesNASA NTRSDOE OSTI
NASA NTRS1982

RF model of the distribution system as a communication channel, phase 2. Volume 1: Summary Report

The design, implementation, and verification of a computerized model for predicting the steady-state sinusoidal response of radial (tree) configured distribution feeders was undertaken. That work demonstrated the feasibility and validity based on verification measurements made on a limited size portion of an actual live feeder. On that basis a follow-on effort concerned with (1) extending the verification based on a greater variety of situations and network size, (2) extending the model capabilities for reverse direction propagation, (3) investigating parameter sensitivities, (4) improving transformer models, and (5) investigating procedures/fixes for ameliorating propagation trouble spots was conducted. Results are summarized.

Rustay, R. C.↗
NASA NTRS1995

Quantization Distortion in Block Transform-Compressed Data

The popular JPEG image compression standard is an example of a block transform-based compression scheme; the image is systematically subdivided into block that are individually transformed, quantized, and encoded. The compression is achieved by quantizing the transformed data, reducing the data entropy and thus facilitating efficient encoding. A generic block transform model is introduced.

block transform-compressed data quantization image↗
NASA NTRS1982

RF model of the distribution system as a communication channel, phase 2. Volume 2: Task reports

Based on the established feasibility of predicting, via a model, the propagation of Power Line Frequency on radial type distribution feeders, verification studies comparing model predictions against measurements were undertaken using more complicated feeder circuits and situations. Detailed accounts of the major tasks are presented. These include: (1) verification of model; (2) extension, implementation, and verification of perturbation theory; (3) parameter sensitivity; (4) transformer modeling; and (5) compensation of power distribution systems for enhancement of power line carrier communication reliability.

Rustay, R. C.↗
DOE OSTI2020

Report on the Use and Function of the Integrated Operations Capability Analysis Platform (ICAP) and the LWRS Innovation Portal (IP)

The commercial nuclear power industry has achieved excellent safety and reliability performance but is struggling to survive in an electricity market that is increasingly dominated by subsidized renewables and cheap natural gas. These challenges have forced utilities to explore previously uncharted avenues to drastically reduce the operations and maintenance costs of their plants which are the primary drivers of the total cost to produce electricity. The petrochemical industry faced similar challenges some years ago as the costs of extraction and processing were rising while their reservoirs were being depleted along with a drop-in commodity prices that resulted in unsustainable operations. In this challenging climate, they developed a business model called Integrated Operations (IO) that sought to utilize technology to enable news ways of working through the integration of people, technology, process and governance changes. The Light Water Reactor Sustainability (LWRS) program, working with IFE have developed an operating model via transferable learnings from the North Sea O&G industry. This framework is termed “Integrated Operations for Nuclear” (ION). ION is a transformation model that integrates the benefits and features of four principal factors: People, Technology, Process and Governance. The purpose of this report is to describe how to generate an ION business process analysis and utilize this information to reduce O&M costs. In order to make this job easier, DOE LWRS has created a suite of tools that will allow a person who is involved in a transformation effort at their company to build a solid documented business case for embarking on a major transformation effort. These tools, the Integrated Operations Capability Analysis Model (ICAP) and the LWRS Innovation Portal (IP) are described herein with instructions on their use. Instructions on how the interface with the EPRI Business Case Analysis Method (BCAM) are also provided.

21 SPECIFIC NUCLEAR REACTORS AND ASSOCIATED PLANTS↗
DOE OSTI2021

Development of an Assessment Methodology That Enables the Nuclear Industry to Evaluate Adoption of Advanced Automation

Nuclear power has a crucial role in providing safe, reliable, and economical carbon-free electricity for today and the future. For continued operation, many of the existing United States nuclear power plants will begin the subsequent license renewal process for extending their operating license periods. As plants extend their expected operating lifetimes, there is a significant opportunity to modernize. These plants have a much stronger business case with these extended mission periods to modernize and significantly enhance their economic viability in current and future energy markets by implementing digital technologies that support innovation, efficiency gains, and business-model transformation. Ensuring continued safety and reliability is crucial. Transformative digital technologies—including automation—that fundamentally change the concept of operation for the nuclear power plant operating model requires a critical focus on the human and technology integration element. Further, the nuclear industry has historically been reluctant to modernize due to having a risk adverse culture and lack of clarity for a transformative new state vision (Joe & Remer, 2019; Thomas et al., 2020). Common barriers include (1) the perceived value and return on investment (ROI) of digital technology, (2) the perceived risk associated with licensing, regulatory, and cybersecurity, and (3) insufficient guidance for performing digital modifications to power generation systems. This work presents a methodology to address these barriers and support the industry in adopting advanced automation and digital technology through developing a transformative vision and implementation strategy that will address the human and technology integration element. This research leverages previous LWRS Program and industry results. It draws specifically on previous LWRS Program research in the areas of advanced alarm systems, computer-based procedures, model informed decision support, and advanced human-system interface displays (e.g., overviews and task-based). The modernization methodology can be used to guide transformative thinking when integrating a set of vendor-specific capabilities to support a new concept of operations and a utility’s end-state vision. The results of this research are organized into six major sections: - Section 1 introduces the need for supporting large-scale digital modifications that will renew the technology base for extended operating life beyond 60 years - Section 2 describes the challenges that the nuclear industry is enduring with modernizing. - Section 3 summarizes the primary standards and guidance. - Section 4 presents earlier work from the LWRS Program regarding the development of a transformative conceptual design for an advanced control room of a hybrid plants. - Section 5 presents a methodology that is designed at addressing the challenges in the industry today in achieving a transformative new state vision and concept of operations. - Conclusions and next steps of this research are provided in Section 6.

99 GENERAL AND MISCELLANEOUS↗
DOE OSTI2024

Integrated edge-to-exascale workflow for real-time steering in neutron scattering experiments

We introduce a computational framework that integrates artificial intelligence (AI), machine learning, and high-performance computing to enable real-time steering of neutron scattering experiments using an edge-to-exascale workflow. Focusing on time-of-flight neutron event data at the Spallation Neutron Source, our approach combines temporal processing of four-dimensional neutron event data with predictive modeling for multidimensional crystallography. At the core of this workflow is the Temporal Fusion Transformer model, which provides voxel-level precision in predicting 3D neutron scattering patterns. The system incorporates edge computing for rapid data preprocessing and exascale computing via the Frontier supercomputer for large-scale AI model training, enabling adaptive, data-driven decisions during experiments. This framework optimizes neutron beam time, improves experimental accuracy, and lays the foundation for automation in neutron scattering. Although real-time experiment steering is still in the proof-of-concept stage, the demonstrated potential of this system offers a substantial reduction in data processing time from hours to minutes via distributed training, and significant improvements in model accuracy, setting the stage for widespread adoption across neutron scattering facilities and more efficient exploration of complex material systems.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2024

Quadrature Based Neural Network Learning of Stochastic Hamiltonian Systems

Hamiltonian Neural Networks (HNNs) provide structure-preserving learning of Hamiltonian systems. In this paper, we extend HNNs to structure-preserving inversion of stochastic Hamiltonian systems (SHSs) from observational data. We propose the quadrature-based models according to the integral form of the SHSs’ solutions, where we denoise the loss-by-moment calculations of the solutions. The integral pattern of the models transforms the source of the essential learning error from the discrepancy between the modified Hamiltonian and the true Hamiltonian in the classical HNN models into that between the integrals and their quadrature approximations. This transforms the challenging task of deriving the relation between the modified and the true Hamiltonians from the (stochastic) Hamilton–Jacobi PDEs, into the one that only requires invoking results from the numerical quadrature theory. Meanwhile, denoising via moments calculations gives a simpler data fitting method than, e.g., via probability density fitting, which may imply better generalization ability in certain circumstances. Numerical experiments validate the proposed learning strategy on several concrete Hamiltonian systems. The experimental results show that both the learned Hamiltonian function and the predicted solution of our quadrature-based model are more accurate than that of the corrected symplectic HNN method on a harmonic oscillator, and the three-point Gaussian quadrature-based model produces higher accuracy in long-time prediction than the Kramers–Moyal method and the numerics-informed likelihood method on the stochastic Kubo oscillator as well as other two stochastic systems with non-polynomial Hamiltonian functions. Moreover, the Hamiltonian learning error εH arising from the Gaussian quadrature-based model is lower than that from Simpson’s quadrature-based model. These demonstrate the superiority of our approach in learning accuracy and long-time prediction ability compared to certain existing methods and exhibit its potential to improve learning accuracy via applying precise quadrature formulae.

Mathematics↗
DOE OSTI2025

Enhancing Unknown Waveform Detection by Learning Intra and Inter-domain Dependencies with Advanced Attention Fusion Mechanisms

Detection of unknown waveforms in mission-critical communications is a crucial area of interest for the Department of Energy (DoE). Traditional methods and recent deep learning-based approaches often assume that the training set includes all possible classes, which is impractical for detecting new waveforms. This limitation gives rise to the problem of open-set recognition (OSR), which involves correctly identifying known classes while detecting and rejecting unknown or unseen classes. To address this limitation, we propose a novel dual-domain complex-valued neural architecture that jointly processes time-domain and frequency-domain signal representations using transformer mechanisms. A transformer model is a deep learning architecture that uses self-attention mechanisms to process and learn relationships in sequential data. Our model employs a cosine similarity loss to extract domain-specific features and incorporates a transformer architecture in the latent space to weigh the importance of different features from the time and frequency domains. The transformer layer includes stacked self-attention and cross-attention modules to learn intra-domain and inter-domain dependencies, creating a more holistic signal representation. An attention-based fusion module intelligently combines the time and frequency-domain features using multi-head attention, enabling the network to learn the optimal feature for each domain in each input signal. Quantitative results demonstrate the impact of these architectural choices on overall performance, showing significant improvement after incorporating self and cross-attention modules and using complex attention fusion over simple weighted fusion. Our ongoing work will focus on addressing the limitations of threshold-based OSR methods by developing a novel generative framework that integrates a conditional diffusion probabilistic model (DPM). DPM is a generative framework that learns to synthesize complex data by reversing a gradual noising process using a neural network trained to denoise step-by-step. Our goal is to leverage the inherent strengths of DPMs for identifying unknown signals more robustly. One primary advantage of using a DPM is its ability to provide a more reliable anomaly score based on the model's reconstruction error, rather than relying solely on classifier confidence. Additionally, the iterative denoising process of DPMs makes this approach naturally resilient to low Signal-to-Noise Ratio (SNR) conditions, where traditional methods often fail. By implementing this generative framework, we aim to enhance the model's capability to accurately detect unknown waveforms and maintain performance in challenging environments.

99 - GENERAL AND MISCELLANEOUS↗
NASA NTRS2003

An Analysis of Model Scale Data Transformation to Full Scale Flight Using Chevron Nozzles

Ground-based model scale aeroacoustic data is frequently used to predict the results of flight tests while saving time and money. The value of a model scale test is therefore dependent on how well the data can be transformed to the full scale conditions. In the spring of 2000, a model scale test was conducted to prove the value of chevron nozzles as a noise reduction device for turbojet applications. The chevron nozzle reduced noise by 2 EPNdB at an engine pressure ratio of 2.3 compared to that of the standard conic nozzle. This result led to a full scale flyover test in the spring of 2001 to verify these results. The flyover test confirmed the 2 EPNdB reduction predicted by the model scale test one year earlier. However, further analysis of the data revealed that the spectra and directivity, both on an OASPL and PNL basis, do not agree in either shape or absolute level. This paper explores these differences in an effort to improve the data transformation from model scale to full scale.

Brown, Clifford↗
NASA NTRS1979

In-flight simulation with pilot-center of gravity offset and velocity mismatch

Similarity transformations which preserve modal characteristics and pilot's acceleration cues in in-flight simulation are presented. The model transformation for lateral acceleration matching is developed. A velocity-mismatch example, based on a VRA simulation of the Space Shuttle, illustrates that acceleration matching is achieved at the expense of mismatching in cues which are secondary to the simulated piloting task, while primarily cues are preserved. The approach is applicable for both implicit and explicit model-following, and it can easily be extended to the longitudinal case.

Stengel, R. F.↗
DOE OSTI2025

Deep learning model for fast, science-based forecasting of fluid migration along faults in geologic carbon storage scenarios

Effective long-term geologic storage depends on robust site selection and credible, science-based forecasting of subsurface behavior to ensure storage integrity. For this work, we develop a deep learning–based reduced-order model (ROM) to quantify potential carbon dioxide (CO₂) and brine migration through geological faults. The ROM combines a Transformer model for binary classification and a Stacked Ensemble for regression, trained on a comprehensive dataset generated from 1400 physics-based reservoir simulations. Key geologic and operational parameters—including fault geometry, reservoir structure, and injection conditions—were systematically varied to capture a wide range of fluid migration scenarios. The ROM accurately predicts the onset of migration, cumulative migration volumes of both CO₂ and brine, and associated migration rates, as compared to an independent set of validation simulations, while significantly reducing computational cost compared to traditional simulation methods. Model performance was evaluated across diverse fault configurations, revealing that shallow reservoir geometry and fault angle are among the most influential factors governing migration behavior. Sensitivity analysis using SHapley Additive exPlanations (SHAP) provided interpretability, revealing distinct patterns in how geological and operational features drive transient versus cumulative migration outcomes. The ROM’s ability to rapidly simulate fault migration scenarios enables efficient sensitivity analyses, scenario evaluations, and decision support for site selection and monitoring design. This approach enhances the safety, scalability, and long-term operational performance of geologic carbon storage (GCS) systems by providing a robust, interpretable tool for predicting subsurface fluid migration and assessing fault-related migration potential.

42 ENGINEERING↗
DOE OSTI2025

Integrating Analytical Solutions and U-Net Model for Predicting Groundwater Contaminant Plumes in Pump-and-Treat Systems

Pump-and-treat (P&T) is a common technique for groundwater remediation involving the extraction and treatment of contaminated water above ground. Optimizing the design and operation of the P&T well network is essential for maximizing the system’s effectiveness and efficiency. However, this optimization often necessitates many model evaluations, leading to computationally demanding tasks. This study introduces a novel approach that integrates analytical solutions for groundwater dynamics with the U-Net (Ronneberger et al., 2015) deep learning framework to predict groundwater contaminant plume migration under dynamic pumping conditions. By incorporating the Thiem equation (Thiem, 1906) into the input preprocessing, the U-Net model transforms sparse well data into a continuous spatial field that captures the hydraulic impacts of pumping activities. This integration enables the model to leverage both deep learning capabilities and classical physics-based groundwater theories, enhancing prediction accuracy and computational efficiency. These advancements can facilitate rapid, large-scale evaluations of P&T optimization simulations, allowing for timely and effective decision-making in well placement and system management. We demonstrate the model's robust performance across both simplified transient 2D models and a more complex 3D heterogeneous site model at the 200 West P&T facility at the Hanford Site. The U-Net-based model offers substantial computational advantages, reducing simulation times significantly compared to full physics-based models and providing a powerful tool for rapid site evaluation and P&T system optimization, such as evaluating alternative P&T well network designs. Our findings highlight the potential of advanced machine learning models to significantly enhance the efficiency and sustainability of groundwater remediation efforts, offering a novel application of U-Net architecture in environmental science.

Pump-and-treat↗
DOE OSTI2026

Scalable Federated Learning for Scientific Foundation Models on Leadership-Class Systems

Federated learning (FL) at leadership-class HPC systems remains largely unexplored, despite growing interest in deploying federated workflows on modern HPC systems. This paper provides the first system-level empirical characterization of federated fine-tuning of pretrained foundation models on an exascale supercomputer under a multi-node deployment. Using up to 96 concurrent FL clients deployed across Frontier nodes, we study the impact of client scale, model size, data heterogeneity, partial participation, and differential privacy on runtime, communication overhead, and convergence stability. Our results show that pretrained transformer models remain robust to heterogeneity, client dropout, and privacy noise, while system efficiency degrades rapidly with scale as synchronizat and orchestration dominate runtime. We further demonstrate that system-aware execution strategies, including intra-node aggregation and early aggregation, significantly reduce wall-clock time without degrading model quality. These findings establish a practical performance baseline and inform the design of communication-efficient FL systems on leadership-class HPC platforms.

Kotevska, Olivera [ORNL] (ORCID:0000000316772243)↗
DOE OSTI2025

Spatially Aware Linear Transformer (SAL-T) for Particle Jet Tagging

Transformers are very effective in capturing both global and local correlations within high-energy particle collisions, but they present deployment challenges in high-data-throughput environments, such as the CERN LHC. The quadratic complexity of transformer models demands substantial resources and increases latency during inference. In order to address these issues, we introduce the Spatially Aware Linear Transformer (SAL-T), a physics-inspired enhancement of the linformer architecture that maintains linear attention. Our method incorporates spatially aware partitioning of particles based on kinematic features, thereby computing attention between regions of physical significance. Additionally, we employ convolutional layers to capture local correlations, informed by insights from jet physics. In addition to outperforming the standard linformer in jet classification tasks, SAL-T also achieves classification results comparable to full-attention transformers, while using considerably fewer resources with lower latency during inference. Experiments on a generic point cloud classification dataset (ModelNet10) further confirm this trend. Our code is available at https://github.com/aaronw5/SAL-T4HEP.

Wang, Aaron [Illinois U., Chicago] (ORCID:00000003↗
DOE OSTI2024

ThermoPore: Predicting part porosity based on thermal images using deep learning

Part qualification is often a critical and labor-intensive process in additive manufacturing, particularly in the detection of defects such as porosity, which stands to benefit significantly from advancements in machine learning. We present a deep learning approach for quantifying and localizing ex-situ porosity within Laser Powder Bed Fusion fabricated samples utilizing in-situ thermal image monitoring data. Our goal is to build the real time porosity map of parts based on thermal images acquired during the build. The quantification task builds upon the established Convolutional Neural Network model architecture to predict pore count and the localization task leverages the spatial and temporal attention mechanisms of the novel Video Vision Transformer model to indicate areas of expected porosity. Our model for porosity quantification achieved a R 2 score of 0.57 and our model for porosity localization produced an average Intersection over Union (IoU) score of 0.32 and a maximum of 1.0. This work is setting the foundations of part porosity “Digital Twins” based on additive manufacturing monitoring data and can be applied downstream to reduce time-intensive post-inspection and testing activities during part qualification and certification. In addition, we seek to accelerate the acquisition of crucial insights normally only available through ex-situ part evaluation by means of machine learning analysis of in-situ process monitoring data.

Deep learning↗
DOE OSTI2025

Operator learning for energy-efficient building ventilation control with computational fluid dynamics simulation of a real-world classroom

Energy-efficient ventilation control plays an important role in reducing building energy consumption while ensuring occupant health and comfort. While Computational Fluid Dynamics (CFD) simulations provide detailed and physically accurate representations of indoor airflow, their high computational cost limits their use in real-time building control. In this work, we present a neural operator learning framework that combines the physical accuracy of CFD with the computational efficiency of machine learning to enable building ventilation control with the high-fidelity fluid dynamics models. Our method jointly optimizes the airflow supply rates and vent angles to reduce energy use and adhere to air quality constraints. We train an ensemble of neural operator transformer models to learn the mapping from building control actions to airflow fields using high-resolution CFD data. This learned neural operator is then embedded in an optimization-based control framework for building ventilation control. Experimental results show that our approach achieves significant energy savings compared to maximum airflow rate control, rule-based control, as well as data-driven control methods using spatially averaged CO 2 prediction and deep learning–based reduced-order models, while consistently maintaining safe indoor air quality. These results highlight the practicality and scalability of our method in maintaining energy efficiency and indoor air quality in real-world buildings.

32 ENERGY CONSERVATION, CONSUMPTION, AND UTILIZATI↗
DOE OSTI2021

Retaining Systems Engineering Model Meaning Through Transformation: Demo 2

Digital engineering strategies typically assume that digital engineering models interoperate seamlessly across the multiple different engineering modeling software applications involved, such as model- based systems engineering (MBSE), mechanical computer-aided design (MCAD), electrical computer-aided design (ECAD), and other engineering modeling applications. The presumption is that the data schema in these modeling software applications are structured in the familiar flat- tabular schema like any other software application. Engineering domain-specific applications (e.g., systems, mechanical, electrical, simulation) are typically designed to solve domain-specific problems, necessarily excluding explicit representations of non-domain information to help the engineer focus on the domain problems (system definition, design, simulation). Such exclusions become problematic in inter-domain information exchange. The obvious assumptions of one domain might not be so obvious to experts in another domain. Ambiguity in domain-specific language can erode the ability to enable different domain modeling applications to interoperate, unless the underlying language is understood and used as the basis for translation from one application to another. The engineering modeling software application industry has struggled for decades to enable these applications to interoperate. Industry standards have been developed, but they have not unified the industry. Why is this? The authors assert that the industry has relied on traditional database integration methods. The basic issue prohibiting successful application integration then is that traditional database-driven integration does not consider the distinct languages of each domain. An engineering models meaning is expressed through the underlying language of that engineering domain. In essence, traditional integration methods do not retain the semantic context (meaning) of the model. The basis of this research stems from the widely held assumption that systems engineering models are (or can be) structured according to the underlying semantic ontology of the model. This assumption can be imagined from two thoughts. 1) Digital systems engineering models are often represented using graph theory (the graph of a complex systems model can contain millions of nodes and edges). When examining the nodes one at a time and following the outbound edges of each node one by one, one can end up with rudimentary statements about the model (i.e., node A relates to node B), as in a semantic graph. 2) Likewise, from the study of natural languages, a sentence can be structured into unambiguous triples of subject-predicate-object within formal and highly expressive semantic ontologies. The rudimentary statements about a systems model discerned with graph theory closely mimic the triples used in the ontologies that try to structure natural languages. In other words, a systems models semantic graph can be (or is) structured into an ontology. Additionally, it is well established in industry that through natural language processing (NLP), which provides the means to create language structures, that computers can interpret ontological graphs. Therefore, the authors hypothesized that if the integrity of the underlying semantic structure of a systems model is retained, the contextual meaning of the model is retained. By structuring system models into the triples of the underlying ontology during the transformation from one MBSE application to another, the authors have provided a proof of the concept that the meaning of a system model can be retained during transformation. The authors assert that this is the missing ingredient in effective systems model-to-model interoperability. ACKNOWLEDGEMENTS The authors would like to thank the FY19 Model Interoperability team members who provided a solid foundation for the FY20 team to leverage: John McCloud, for the work he did to guide us toward the right use of technology that will appropriately discover and manipulate ontologies. Carlos Tafoya, for the work he did to develop an application programming interface (API)/Adapter that would export ontology-based data from GENESYS. Peter Chandler, for the work he did to architect our overall integration solution, with an eye toward the future that would influence a large-scale federated production-level systems engineering digital model ecosystem.

42 ENGINEERING↗
DOE OSTI2024

The Case for Co-Designing Model Architectures with Hardware

While GPUs are responsible for training the vast majority of state-of-the-art deep learning models, the implications of their architecture are often overlooked when designing new deep learning (DL) models. As a consequence, modifying a DL model to be more amenable to the target hardware can significantly improve the runtime performance of DL training and inference. In this paper, we provide a set of guidelines for users to maximize the runtime performance of their transformer models. These guidelines have been created by carefully considering the impact of various model hyperparameters controlling model shape on the efficiency of the underlying computation kernels executed on the GPU. We find the throughput of models with “efficient” model shapes is up to 39% higher while preserving accuracy compared to models with a similar number of parameters but with unoptimized shapes.

Yin, Junqi↗