Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “Transformer model”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 379 records · Page 21
All sourcesNASA NTRSDOE OSTI
DOE OSTI2022

Integration of Omics into a New Comprehensive Rate Law for Competitive Terminal Electron-Accepting Processes in Reactive Transport Models: Application to N, Fe, S, and Contaminant Transformations in Stream and Wetland Sediments

Surface waters represent important sources of alternate energy and drinking water in the United States, and characterizing the biogeochemical processes that affect surface water quality is relevant to the DOE-BER mission. Sediment biogeochemical processes regulate the release of carbon (C), nutrients, and contaminants to surface waters and thus influence water quality. Sediment biogeochemical processes are dynamic and affected by the deposition and remobilization of solid material and changes in environmental conditions driven by water discharge variations. Wetlands are important natural filters of surface waters which may either trap, metabolize, or mobilize nutrients and contaminants. Despite their importance, biogeochemical processes regulating nutrient and contaminant release and C transformation in stream and wetland sediments cannot be predicted accurately by current mathematical models. These reactive transport models largely rely on detectable changes in geochemical conditions to activate metabolic processes, do not accurately account for the competition between microbial processes, and poorly constrain effects of hydrological perturbations on biogeochemical processes. In this BER-SBR exploratory project, metagenomic and geochemical signatures were combined to identify microbially-mediated redox processes in anaerobic stream and wetland sediments from the Savannah River Site (SRS, ANL SFA) and East Fork Poplar Creek (EFPC, ORNL SFA) that play important roles in C, uranium (U), and mercury (Hg) transformations. In addition, sediment incubations were conducted to examine the competition between anaerobic respiration processes Finally, new rate laws were developed for reactive transport models that rely on complementary metagenomic and geochemical signatures to identify the underlying anaerobic microbial processes in stream and wetland sediments, describe the competition between the dominant metabolic processes involved in nutrient release and U and Hg mobilization, and more accurately quantify carbon transformation and the response of microbial processes to changes in redox conditions associated with hydrological forcing. These rate laws were optimized in batch reactors with SRS wetland sediments, where iron and sulfate reduction dominate. Anaerobic carbon remineralization processes followed the expected thermodynamic sequence of microbial respiration with depth in both sediments, except that geochemical signals indicated that sulfate reduction was inactive in EFPC sediments and moderate in SRS wetland sediments. Estimates indicated that microbial iron reduction contributed to at least half of the production of reduced iron in these sediments. Incubations demonstrated that nitrate reduction, denitrification, and dissimilatory nitrate reduction to ammonium were active in the natural EFPC sediment and activated upon nitrate amendment in these nitrate-rich sediments. In turn, these processes were outcompeted by the addition of either iron oxides or sulfate as alternative terminal electron acceptors. Although geochemical products of sulfate reduction were not detected in the incubations, the abundance of sulfate reduction genes increased with depth in the sediment and was equally more pronounced in treatments amended with either iron oxides or sulfate. Simultaneously, anaerobic sulfide oxidizing bacteria coupling sulfide oxidation to DNRA (and not conventional denitrification) were apparently enriched over time, regardless of the treatments. These findings indicate that sulfate reduction is important in freshwater stream sediments and probably catalyzed by a cryptic sulfur cycle involving nitrogen species, in which the sulfur products from sulfate reduction are immediately removed by side reactions and not detectable by geochemical measurements alone. Similar experiments in SRS sediments, however, demonstrated little interaction between nitrogen and sulfur cycling microorganisms. Sulfate reduction was impacted by the addition of more thermodynamically favorable electron acceptors, suggesting either that iron-reducing microorganisms outcompeted sulfate-reducing microorganisms for organic substrate, depleting the stock of electron donor available for sulfate reduction, or that the cryptic sulfur cycle was shunted by the precipitation of FeS generated as a result of the abiotic reduction of iron oxides by dissolved sulfide. A diagnostic modeling exercise was conducted to further investigate the competition between terminal electron accepting processes. As conventional kinetic models typically do not account for cryptic cycles and used inaccurate formulations to describe competition between microbial communities, new metabolic rate laws were developed that explicitly express the electron acceptor-specific enzyme of each energetically favorable metabolic process based on gene abundance detected in the incubations. The model was tested with the sediment slurry incubation data to determine whether substrate competition could explain the decrease in sulfate reduction observed in the presence of iron oxide competitor. The model was able to reproduce geochemical concentrations really well in each treatment once the model was calibrated with the unamended control, suggesting that microbial competition was indeed driven by thermodynamic considerations. Overall, carbon remineralization processes and rates will be reproduced much more realistically with the new metabolic rate laws.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2021

Enriching Load Data Using Micro-PMUs and Smart Meters

In modern distribution systems, load uncertainty can be fully captured by micro-PMUs, which can record high-resolution data; however, in practice, micro-PMUs are installed at limited locations in distribution networks due to budgetary constraints. In contrast, smart meters are widely deployed but can only measure relatively low-resolution energy consumption, which cannot sufficiently reflect the actual instantaneous load volatility within each sampling interval. In this paper, we have proposed a novel approach for enriching load data for service transformers that only have low-resolution smart meters. The key to our approach is to statistically recover the high-resolution load data, which is masked by the low-resolution data, using trained probabilistic models of service transformers that have both high- and low-resolution data sources, i.e., micro-PMUs and smart meters. The overall framework consists of two steps: first, for the transformers with micro-PMUs, a Gaussian Process is leveraged to capture the relationship between the maximum/minimum load and average load within each low-resolution sampling interval of smart meters; a Markov chain model is employed to characterize the transition probability of known high-resolution load. Next, the trained models are used as teachers for the transformers with only smart meters to decompose known low-resolution load data into targeted high-resolution load data. The enriched data can recover instantaneous load uncertainty and significantly enhance distribution system observability and situational awareness. Here, we have verified the proposed approach using real high- and low-resolution load data.

24 POWER TRANSMISSION AND DISTRIBUTION↗
DOE OSTI2026

Surrogate model evaluation and building energy benchmarking for commercial buildings

Building energy consumption benchmarking involves challenges associated with various energy patterns for different building types; heating, ventilating, and air-conditioning (HVAC) system types; and climates. Given significant variation in energy use patterns, accurate prediction of long-term energy use using surrogate models remains challenging. Multiple linear regression (MLR) is commonly used for building energy benchmarking because of its simple structure; however, it lacks accuracy compared to other black-box models. Although many studies have compared surrogate models and offer guidance on model selection based on metrics, they do not provide detailed analysis on improving the surrogate model accuracy. In this paper, we implement a surrogate model using polynomial ridge regression (i.e., MLR with interaction terms combined with ridge regularization) for small office and retail strip mall buildings across six HVAC system types and all climate zones, for electricity and natural gas in baseline and proposed scenarios. A simulation workflow is developed using OpenStudio TM /EnergyPlus TM to generate simulation data using measures over a wide range of efficiency inputs. Enhancements based on statistical insights are used for improving the model accuracy using filters, input transformations, and change points. Surrogate models achieved average coefficient of variation of the root mean squared error (CVRMSE) values of 2.17, 1.06, 2.05, and 3.26 for proposed electricity, proposed natural gas, baseline electricity, and baseline natural gas, respectively, with enhancements reducing CVRMSE by an average of 14.9% across all combinations. We provide model interpretation via Shapley additive explanations to determine which input variables most influence energy consumption and provide supportive arguments for enhancements.

32 ENERGY CONSERVATION, CONSUMPTION, AND UTILIZATI↗
DOE OSTI2025

HydraGNN v4.0

The new version of HydraGNN v4.0 provides additional core capabilities, such as: Inclusion of multi-body atomistic cluster expansion MACE, polarizable atom interaction neural network PAINN, and equivariant principal neighborhood aggregation (PNAEq) among the message passing layers supported -Inclusion of graph transformers to directly model long-range interactions between nodes that are distant in the graph topology Integration of graph transformers with message passing layers by combining the graph embedding generated by the two mechanisms, which allows for an improved expressivity of the HydraGNN architecture Improved re-implementation of multi-task learning (MTL) to allow its use for stabilized training across imbalanced, multi-source, multi-fidelity data Introduction of multi-task parallelism, a newly proposed type of model parallelism specifically for MTL architectures, which allows to dispatch different output decoding heads to different GPU devices Integration of multi-task parallelism with pre-existing distributed data parallelism to enable a 2D parallelization for distributed training Improved portability of the distributed training across Intel GPUs, which has been testes on ALCF exascale supercomputer Aurora Inclusion of 2-level fine-grained energy profilers portable across NVIDIA, AMD, and Intel GPUs to monitor the power and energy consumption associated with different functions executed by the HydraGNN code during data pre-load and training Restructuring of previous examples and inclusion of new sets of examples to illustrate the download, preprocess, and training of HydraGNN models on new large-scale open-source datasets for atomistic materials modeling (e.g., Alexandria, Transition1x, OMat24, OMol25)

Lupo Pasini, Massimiliano [Oak Ridge National Labo↗
NASA NTRS1987

Tail modeling in a stretched magnetosphere. I - Methods and transformations

A new method is developed for representing the magnetospheric field B as a distorted dipole field. Because Delta-B = 0 must be maintained, such a distortion may be viewed as a transformation of the vector potential A. The simplest form is a one-dimensional 'stretch transformation' along the x axis, concisely represented by the 'stretch function' f(x), which is also a convenient tool for representing features of the substorm cycle. One-dimensional stretch transformations are extended to spherical, cylindrical, and parabolic coordinates and then to arbitrary coordinates. It is shown that distortion transformations can be viewed as mappings of field lines from one pattern to another; the final result only requires knowledge of the field and not of the potentials. General transformations in Cartesian and arbitrary coordinates are derived, and applications to field modeling, field line motion, MHD modeling, and incompressible fluid dynamics are considered.

Stern, David P.↗
NASA NTRSDate not supplied

Lessons Learned from Medical System Foundation Development for Long-Duration Lunar Orbit and Lunar Surface Missions

The Human Research Program (HRP) Exploration Medical Capability (ExMC) Element has been tasked with the development of Medical System Foundations for Level of Care IV for both short-duration lunar orbital missions and, subsequently, long-duration lunar orbital and surface operations missions. These Medical System Foundations serve as a framework to aid in early medical system design and mission planning. The content of both Foundation models is similar, consisting of a concept of operations, functional decomposition, clinical content (medical conditions, capabilities, and resources), technical requirements (interface, non-functional, and functional), and traces between these components and to the NASA standards documents and parent-level (Program- and Vehicle habitat system-level) requirements. Additionally, the development of both Foundations employed systems engineering principles and a model-based systems engineering (MBSE) approach. Throughout the development of these Foundations, ExMC has strived to improve the efficiency and robustness of its processes and to be more responsive to change (i.e., in design reference mission parameters and assumptions) and to stakeholders’ feedback. The most significant improvements made between the short- and long-duration Foundation models during this transformation process are the following: • Replacement of the traditional document-based ConOps with a model-based ConOps according to MBSE principles, which facilitated more efficient understanding of the material and the consolidation of all relevant information into a centralized location. • Utilization of an agile approach with tasks organized into sprints. This approach enabled solicitation of more frequent usability feedback from stakeholders, incorporation of more human factors reviews into the sprints, and more efficient tasking of team members. This presentation will discuss the journey of developing both Foundation models, as well as the lessons learned and resulting improvements made between the Short- and Long-Duration models.

M Kaetzer↗
NASA NTRS1995

One-Equation Turbulence Models in Aerodynamics

The present paper shows under which assumptions one-equation models can be derived from two-equation models. Based on that transformation a new one-equation turbulence model is derived that basically behaves like a two-equation model. The new model is compared in detail against existing models

Menter, F. R.↗
DOE OSTI2026

wa-hls4ml: A Benchmark and Surrogate Models for hls4ml Resource and Latency Estimation

As machine learning (ML) is increasingly implemented in hardware to address real-time challenges in scientific applications, the development of advanced toolchains has significantly reduced the time required to iterate on various designs. These advancements have solved major obstacles, but also exposed new challenges. For example, processes that were not previously considered bottlenecks, such as hardware synthesis, are becoming limiting factors in the rapid iteration of designs. To mitigate these emerging constraints, multiple efforts have been undertaken to develop an ML-based surrogate model that estimates resource usage of ML accelerator architectures. We introduce wa-hls4ml, a benchmark for ML accelerator resource and latency estimation, and its corresponding initial dataset of over 680,000 fully connected and convolutional neural networks, all synthesized using hls4ml and targeting Xilinx FPGAs. The benchmark evaluates the performance of resource and latency predictors against several common ML model architectures, primarily originating from scientific domains, as exemplar models, and the average performance across a subset of the dataset. Additionally, we introduce GNN- and transformer-based surrogate models that predict latency and resources for ML accelerators. We present the architecture and performance of the models and find that the models generally predict latency and resources for the 75% percentile within several percent of the synthesized resources on the synthetic test dataset.

Hawks, Benjamin [Fermilab] (ORCID:0000000157000288↗
DOE OSTI2025

wa-hls4ml: A Benchmark and Surrogate Models for hls4ml Resource and Latency Estimation

As machine learning (ML) is increasingly implemented in hardware to address real-time challenges in scientific applications, the development of advanced toolchains has significantly reduced the time required to iterate on various designs. These advancements have solved major obstacles, but also exposed new challenges. For example, processes that were not previously considered bottlenecks, such as hardware synthesis, are becoming limiting factors in the rapid iteration of designs. To mitigate these emerging constraints, multipleefforts have been undertaken to develop an ML-based surrogate model that estimates resource usage of synthesized ML accelerator architectures. We introduce wa-hls4ml, a benchmark for ML accelerator resource and latency estimation, and its corresponding initial dataset of over 680 000 fully connected and convolutional neural networks, all synthesized using hls4ml and targeting Xilinx FPGAs. The benchmark evaluates the performance of resource and latency predictors against several common ML model architectures, primarily originating from scientific domains, as exemplar models, and the average performance across a subset of the dataset. Additionally, we introduce GNN- and transformer-based surrogate models that predict latency and resources for ML accelerators. We present the architecture and performance of the models and find that the models generally predict latency and resources for the 75% percentile within several percent of the synthesized resources on the synthetic test dataset.

Hawks, Benjamin G. [Fermilab]↗
DOE OSTI2024

Automated Framework for Groundwater Monitoring Using DWT with LSTM and Transformers

Environmental monitoring is critical for safeguarding public health and ecological well-being. Traditional data structuring and workflow monitoring methods consume significant time and effort, hindering timely insights and effective decision-making. Our study addresses this challenge by presenting an AI framework that automates data cleaning, structuring, and modeling processes, specifically targeting applications in groundwater monitoring. By leveraging automation for data processing and model training, our framework establishes a novel and efficient paradigm for environmental monitoring, with its potential application to the vast network of over a hundred Department of Energy Environmental Management (DoE-EM) cleanup sites across the country. It analyzes data streams from a network of groundwater Internet-of-Things (IoT) sensors deployed at the Savannah River Site (SRS) for prediction modeling. This allows human experts to focus on analysis and decision-making, ultimately leading to better environmental outcomes.The framework employs multivariate time-series forecasting methods to study and model the behavior of varying chemical analytes. The continuous learning process is enabled by utilizing deep learning techniques. It allows the framework to become more nuanced in its analysis over time, adapting to the specific characteristics of the environmental site and the evolving nature of contaminant behavior. Deep learning models known for sequence modeling, LSTM, and Transformers are employed for time series forecasting. Data processing and structuring are essential components significantly impacting the final model's performance. This hypothesis was proven by presenting a comparative analysis of model performance with processed and unprocessed data. The feature engineering approach utilized was the Discrete Wavelet Transform, which works well with time series data.

Discrete Wavelet Transform (DWT)↗
DOE OSTI2024

Distribution Transformer Demand: Understanding Demand Segmentation, Drivers, and Management Through 2050

The National Renewable Energy Laboratory (NREL) has been working closely with the U.S. Department of Energy's Office of Electricity (OE) to understand the critical drivers and potential means of managing distribution transformer demand through 2050. This effort has consulted with utility representative organizations and transformer manufacturers to understand the problem, characterized the in-service assets, and modeled future demand. Distribution transformers, or service transformers, range from 10 to 5,000 kilovolt-amperes (kVA), have a high-side voltage of less than 34.5 kilovolts, and have step-down power delivery for customer end use. This research will help the manufacturing sector understand production requirements and better inform utility strategies for managing their demand.

24 POWER TRANSMISSION AND DISTRIBUTION↗
NASA NTRS1987

The cortex transform - Rapid computation of simulated neural images

With a goal of providing means for accelerating the image processing, machine vision, and testing of human vision models, an image transform was designed, which makes it possible to map an image into a set of images that vary in resolution and orientation. Each pixel in the output may be regarded as the simulated response of a neuron in human visual cortex. The transform is amenable to a number of shortcuts that greatly reduce the amount of computation.

Watson, Andrew B.↗
NASA NTRS2024

Harnessing Collaborative Learning Automata to Guide Multi-objective Optimization based Inverse Analysis for Structural Damage Identification

Structural damage identification based on physical models is often transformed into an optimization problem that minimizes the difference between measurement information of structure being monitored and the model prediction in the parametric space. However, the objective function in this context often exhibits multimodality, involving high-dimensional variables due to the reliance on finite element models for damage identification. These features pose challenges to optimization algorithms, where entrapment in local solutions can lead to false positives and false negatives in damage identification. In this research, we propose a reinforcement learning based multi-swarm optimizer to tackle such challenges in pursuit of a small yet diverse solution set that can capture the true damage scenario as one of the solutions. The proposed method leverages the flexibility of the particle swarm optimizer and incorporates novel strategies of metaheuristics to realize targeted improvement. To enable the particle swarm to adaptively select the appropriate search strategy based on the current environment, we adopt the learning automata technique, which sidesteps the need for reward strategy selection that is usually ad hoc at each step of the search. The integration harnesses the automatic learning and self-adaptation capabilities of learning automata, enabling the particles to navigate based on environmental signals. This leads to accumulated probabilities tied to advantageous movements, fostering an adaptive exploration of particles in the search space. The proposed approach is first validated through implementing into benchmark test cases with comparisons. It is then applied to structural damage identification with piezoelectric admittance experimental signals. `The results highlight the capability of the algorithm to identify a small solution set with high accuracy to match the actual damage scenario.

Yang Zhang↗
DOE OSTI2022

A Length Adaptive Algorithm-Hardware Co-design of Transformer on FPGA Through Sparse Attention and Dynamic Pipelining

Transformers are considered one of the most important deep learning models since 2018, in part because it establishes state-of-the-art (SOTA) records and could potentially replace existing Deep Neural Networks (DNNs). Despite the remarkable triumphs, the prolonged turnaround time of Transformer models is a widely recognized roadblock. The variety of sequence lengths imposes additional computing overhead where inputs need to be zero-padded to the maximum sentence length in the batch to accommodate the parallel computing platforms. This paper targets the field-programmable gate array (FPGA) and proposes a coherent sequence length adaptive algorithm–hardware co-design for Transformer acceleration. Particularly, we develop a hardware-friendly sparse attention operator and a length-aware hardware resource scheduling algorithm. The proposed sparse attention operator brings the complexity of attention-based models down to linear complexity and alleviates the off-chip memory traffic. The proposed length-aware resource hardware scheduling algorithm dynamically allocates the hardware resources to fill up the pipeline slots and eliminates bubbles for NLP tasks. Experiments show that our design has very small accuracy loss and has 80.2 × and 2.6 × speedup compared to CPU and GPU implementation, and 4 × higher energy efficiency than state-of-the-art GPU accelerator optimized via CUBLAS GEMM.

Peng, Hongwu↗
NASA NTRS1994

On the Connection Between One-and Two-Equation Models of Turbulence

A formalism will be presented that allows the transformation of two-equation eddy viscosity turbulence models into one-equation models. The transformation is based on an assumption that is widely accepted over a large range of boundary layer flows and that has been shown to actually improve predictions when incorporated into two-equation models of turbulence. Based on that assumption, a new one-equation turbulence model will be derived. The new model will be tested in great detail against a previously introduced one-equation model and against its parent two-equation model.

Menter, F. R.↗
NASA NTRS2001

Stable Algorithm For Estimating Airdata From Flush Surface Pressure Measurements

An airdata estimation and evaluation system and method, including a stable algorithm for estimating airdata from nonintrusive surface pressure measurements. The airdata estimation and evaluation system is preferably implemented in a flush airdata sensing (FADS) system. The system and method of the present invention take a flow model equation and transform it into a triples formulation equation. The triples formulation equation eliminates the pressure related states from the flow model equation by strategically taking the differences of three surface pressures, known as triples. This triples formulation equation is then used to accurately estimate and compute vital airdata from nonintrusive surface pressure measurements.

Whitmore, Stephen, A.↗
DOE OSTI2025

Ground-state-based model reduction with unitary circuits

Here, we present a method to numerically obtain low-energy effective models based on a unitary transformation of the ground state. The algorithm finds a unitary circuit that transforms the ground state of the original model to a projected wavefunction with only the low-energy degrees of freedom. The effective model can then be derived using the unitary transformation encoded in the circuit. We test our method on the one-dimensional and two-dimensional square-lattice Hubbard model at half-filling, and obtain more accurate effective spin models than the standard perturbative approach.

Hubbard model↗