SEARCH · Engineering Papers

Results for “Transformer model”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

SourceResearch connectionFrom yearThrough year

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 397 records · Page 22

All sources NASA NTRS DOE OSTI

DOE OSTI2025

Physics-informed transformation toward improving the machine-learned NLTE models of ICF simulations

The integration of machine-learning techniques into inertial confinement fusion (ICF) simulations has emerged as a powerful approach for enhancing computational efficiency. By replacing the costly nonlocal thermodynamic equilibrium (NLTE) model with machine-learning models, significant reductions in calculation time have been achieved. However, determining how to optimize machine-learning-based NLTE models in order to match ICF simulation dynamics remains challenging, underscoring the need for physically relevant error metrics and strategies to enhance model accuracy with respect to these metrics. Thus, we propose novel physics-informed transformations designed to emphasize energy transport, use these transformations to establish new error metrics, and demonstrate that they yield smaller errors within reduced principal-component spaces compared to conventional transformations. Published by the American Physical Society 2025

70 PLASMA PHYSICS AND FUSION TECHNOLOGY↗

DOE OSTI2025

Rapid wavefield forecasting for earthquake early warning via deep sequence to sequence learning

We propose a deep learning model, WaveCastNet, to forecast high-dimensional wavefields. WaveCastNet integrates a convolutional long expressive memory architecture into a sequence-to-sequence forecasting framework, enabling it to model long-term dependencies and multiscale patterns in both space and time. By sharing weights across spatial and temporal dimensions, WaveCastNet requires significantly fewer parameters than more resource-intensive models such as transformers, resulting in faster inference times. Crucially, WaveCastNet also generalizes better than transformers to rare and critical seismic scenarios, such as high-magnitude earthquakes. Here, we show the ability of the model to predict the intensity and timing of destructive ground motions in real time, using simulated data from the San Francisco Bay Area. Furthermore, we demonstrate its zero-shot capabilities by evaluating WaveCastNet on real earthquake data. Our approach does not require estimating earthquake magnitudes and epicenters, steps that are prone to error in conventional methods, nor does it rely on empirical ground-motion models, which often fail to capture strongly heterogeneous wave propagation effects.

Geophysics↗

DOE OSTI2025

Model form and sensitivity analysis of CALPHAD-based nucleation models in b-stabilized Ti alloys

Accurate prediction of α-phase nucleation and growth in β-stabilized titanium alloys is crucial for designing heat treatments to optimize mechanical properties in additively manufactured lightweight components. Ideally, predictions of nucleation and growth would incorporate both top-down observations of past experimental heat treatments and bottom-up modeling of phase transformations; however, the appropriate method of combining these information sources is not self-evident. Combining top-down and bottom-up information requires a unified form of model that can connect between spatiotemporal scales, as well as sets of fitting parameters that can be identified by each data source. The selection of which parameters to fit to which data source can be made based on expert opinion, or by performing a sensitivity analysis. In solid-solid nucleation, direct observation of the nucleation and growth process is challenging. Most data on the heat treatment-controlled phase transformations are not in-situ. To predict the process and outcome of the nucleation, growth and coarsening of precipitates, theoretical models of the nucleation pathway are used to bridge the gap. Many sources of uncertainty affect the modeling of this nucleation process. It can be influenced by small variations in the thermomechanical processing history, chemical composition, and initial microstructure. If molecular dynamics (MD) simulations are used to determine thermodynamic quantities and inform CALPHAD modeling, additional uncertainty can be introduced and accounted for using Bayesian methods. Top-down uncertainties require additional steps to quantify. The influence of nucleation model form on the sensitivity of predictions to input parameters and physical conditions is the focus of this study. Classical nucleation theory (CNT) allows modeling to formulate the nucleation as homogeneous or, more commonly, heterogeneous. Non-classical nucleation models are also increasingly explored as a means of reconciling top-down and bottom-up data. In this study, the sensitivity of the intragranular nucleation of α in a β-annealed, slow-cooled aging (BASCA) heat treatment of β-stabilized Ti5553 alloy is explored using CNT and both heterogeneous and homogeneous assumptions. The Kampmann-Wagner Numerical model of precipitate nucleation and growth is employed. Using open-source tools (pyCalphad and thermodynamic modeling of TiMo as a surrogate system, a sensitivity analysis is performed to measure variations in key parameters, including chemical driving force, interfacial energy, and diffusivity, as they relate to predictions of precipitate number density. The inclusion of top-down and bottom-up data in selection of nucleation model form is discussed.

Rodriguez Negron, A. M.↗

NASA NTRS2013

Formal Validation of Fault Management Design Solutions

The work presented in this paper describes an approach used to develop SysML modeling patterns to express the behavior of fault protection, test the model's logic by performing fault injection simulations, and verify the fault protection system's logical design via model checking. A representative example, using a subset of the fault protection design for the Soil Moisture Active-Passive (SMAP) system, was modeled with SysML State Machines and JavaScript as Action Language. The SysML model captures interactions between relevant system components and system behavior abstractions (mode managers, error monitors, fault protection engine, and devices/switches). Development of a method to implement verifiable and lightweight executable fault protection models enables future missions to have access to larger fault test domains and verifiable design patterns. A tool-chain to transform the SysML model to jpf-Statechart compliant Java code and then verify the generated code via model checking was established. Conclusions and lessons learned from this work are also described, as well as potential avenues for further research and development.

Statechart↗

NASA NTRS1992

Modelling a single phase voltage controlled rectifier using Laplace transforms

The development of a 20 kHz, AC power system by NASA for large space projects has spurred a need to develop models for the equipment which will be used on these single phase systems. To date, models for the AC source (i.e., inverters) have been developed. It is the intent of this paper to develop a method to model the single phase voltage controlled rectifiers which will be attached to the AC power grid as an interface for connected loads. A modified version of EPRI's HARMFLO program is used as the shell for these models. The results obtained from the model developed in this paper are quite adequate for the analysis of problems such as voltage resonance. The unique technique presented in this paper uses the Laplace transforms to determine the harmonic content of the load current of the rectifier rather than a curve fitting technique. Laplace transforms yield the coefficient of the differential equations which model the line current to the rectifier directly.

Kraft, L. Alan↗

NASA NTRS1981

The design of exact nonlinear model followers

A practical approach to the design of control systems for strongly nonlinear, multivariable, time-dependent plants is described. The structure of the control system is that of an exact model follower. The model dynamics are decoupled, linear, constant, and of the order of the plant. The plant state and controls are transformed so that the plant, when viewed through these transformations, looks like the simple model. Regulation of disturbances is accomplished by means of the transformed state and controls. Conditions for transformability into linear models, the appropriate models, and the construction of the transformations are discussed. The approach is illustrated on a trajectory autopilot for a helicopter.

Meyer, G.↗

DOE OSTI2023

Mitigation of spatial nonstationarity with vision transformers

Spatial nonstationarity, the location variance of features’ statistical distributions, is ubiquitous in many natural settings. For example, in geological reservoirs rock matrix porosity varies vertically due to geomechanical compaction trends, in mineral deposits grades vary due to sedimentation and concentration processes, in hydrology rainfall varies due to the atmosphere and topography interactions, and in metallurgy crystalline structures vary due to differential cooling. Conventional geostatistical modeling workflows rely on the assumption of stationarity to be able to model spatial features for geostatistical inference. Nevertheless, this is often not a realistic assumption when dealing with nonstationary spatial data and this has motivated a variety of nonstationary spatial modeling workflows such as trend and residual decomposition, cosimulation with secondary features, and spatial segmentation and independent modeling over stationary subdomains. The advent of deep learning technologies has enabled new workflows for modeling spatial relationships. However, there is a paucity of demonstrated best practice and general guidance on mitigation of spatial nonstationarity with deep learning in the geospatial context. We demonstrate the impact of two common types of geostatistical spatial nonstationarity on deep learning model prediction performance and propose the mitigation of such impacts using self-attention (vision transformer) models. We demonstrate the utility of vision transformers for the mitigation of nonstationarity with relative errors as low as 10%, exceeding the performance of alternative deep learning methods such as convolutional neural networks. We establish best practice by demonstrating the ability of self-attention networks for modeling large-scale spatial relationships in the presence of commonly observed geospatial nonstationarity.

58 GEOSCIENCES↗

DOE OSTI2024

Short-Term Probabilistic Solar Forecasting via Reinforcement Learning over ECMWF

In this paper, we present an innovative reinforcement learning approach for short-term solar forecasting, leveraging data from the European Centre for Medium-Range Weather Forecasts (ECMWF). The methodology begins with the application of the System Advisor Model (SAM) to transform various ECMWF numerical weather prediction members into predictive photovoltaic power generation. To enhance the precision of deterministic forecasting, we introduce a dynamic model selection algorithm based on Q-learning. This algorithm dynamically identifies and utilizes the most accurate ensemble member for forecasting purposes. Furthermore, we employ a support vector regression surrogate model with a Gaussian distribution to generate probabilistic forecasts, providing a holistic view of solar energy generation uncertainty. To expedite the training process and make it more practical for real-world applications, we integrate a rolling update workflow. This innovative workflow reduces the training period from months to a mere 19 days, making our method highly efficient. Numerical results of the case study show that in comparison to benchmark models, the proposed method improves the deterministic and probabilistic solar forecasting accuracy by up to 40.84% and 48.42%, respectively.

ensemble forecasting↗

DOE OSTI2022

AbBERT: Learning Antibody Humanness via Masked Language Modeling

Understanding the degree of humanness of antibody sequences is critical to the therapeutic antibody development process to reduce the risk of failure modes like immunogenicity or poor manufacturability. We introduce AbBERT, a transformer-based language model trained on up to 20 million unpaired heavy/light chain sequences from the Observed Antibody Space database. We first validate AbBERT using a novel “multi-mask” scoring procedure to demonstrate high accuracy in predicting complementary determining regions—including the challenging hypervariable H3 region. We then demonstrate several uses of AbBERT at various points along the antibody design process. AbBERT enhances in silico antibody optimization via deep reinforcement learning by utilizing its learned embeddings as additional observations during optimization. Within a larger computational antibody design platform, AbBERT has been successfully applied as an additional design objective, where it displays strong correlations with computational tools predicting antibody structural stability. Finally, mutant antibody sequences that have been scored as unfavorable by AbBERT have shown corresponding low yields when expressed in cells. These use cases demonstrate the power of language modeling within computational antibody design.

Bioinformatics↗

NASA NTRS1980

Interactive display of molecular models using a microcomputer system

A simple, microcomputer-based, interactive graphics display system has been developed for the presentation of perspective views of wire frame molecular models. The display system is based on a TERAK 8510a graphics computer system with a display unit consisting of microprocessor, television display and keyboard subsystems. The operating system includes a screen editor, file manager, PASCAL and BASIC compilers and command options for linking and executing programs. The graphics program, written in USCD PASCAL, involves the centering of the coordinate system, the transformation of centered model coordinates into homogeneous coordinates, the construction of a viewing transformation matrix to operate on the coordinates, clipping invisible points, perspective transformation and scaling to screen coordinates; commands available include ZOOM, ROTATE, RESET, and CHANGEVIEW. Data file structure was chosen to minimize the amount of disk storage space. Despite the inherent slowness of the system, its low cost and flexibility suggests general applicability.

Egan, J. T.↗

DOE OSTI2025

Sub-microsecond Transformers for Jet Tagging on FPGAs

We present the first sub-microsecond transformer implementation on an FPGA achieving competitive performance for state-of-the-art high-energy physics benchmarks. Transformers have shown exceptional performance on multiple tasks in modern machine learning applications, including jet tagging at the CERN Large Hadron Collider (LHC). However, their computational complexity prohibits use in real-time applications, such as the hardware trigger system of the collider experiments up until now. In this work, we demonstrate the first application of transformers for jet tagging on FPGAs, achieving $\mathcal{O}(100)$ nanosecond latency with superior performance compared to alternative baseline models. We leverage high-granularity quantization and distributed arithmetic optimization to fit the entire transformer model on a single FPGA, achieving the required throughput and latency. Furthermore, we add multi-head attention and linear attention support to hls4ml, making our work accessible to the broader fast machine learning community. This work advances the next-generation trigger systems for the High Luminosity LHC, enabling the use of transformers for real-time applications in high-energy physics and beyond.

Laatu, Lauri [Imperial Coll., London]↗

NASA NTRS1990

Pyramid image codes

All vision systems, both human and machine, transform the spatial image into a coded representation. Particular codes may be optimized for efficiency or to extract useful image features. Researchers explored image codes based on primary visual cortex in man and other primates. Understanding these codes will advance the art in image coding, autonomous vision, and computational human factors. In cortex, imagery is coded by features that vary in size, orientation, and position. Researchers have devised a mathematical model of this transformation, called the Hexagonal oriented Orthogonal quadrature Pyramid (HOP). In a pyramid code, features are segregated by size into layers, with fewer features in the layers devoted to large features. Pyramid schemes provide scale invariance, and are useful for coarse-to-fine searching and for progressive transmission of images. The HOP Pyramid is novel in three respects: (1) it uses a hexagonal pixel lattice, (2) it uses oriented features, and (3) it accurately models most of the prominent aspects of primary visual cortex. The transform uses seven basic features (kernels), which may be regarded as three oriented edges, three oriented bars, and one non-oriented blob. Application of these kernels to non-overlapping seven-pixel neighborhoods yields six oriented, high-pass pyramid layers, and one low-pass (blob) layer.

Watson, Andrew B.↗

NASA NTRS1979

Advanced vehicle separation apparatus

A method of obtaining test data from two independent models or bodies in a conventional wind tunnel is described. The system makes efficient use of wind tunnel test time with computer control performing complex coordinate transformations necessary for model positioning. The apparatus is designed to be used in any of the three Unitary Wind Tunnels at NASA-Ames Research Center. Mechanical design details and a brief description of the control system for the separation apparatus are presented.

Ospring, M. J.↗

NASA NTRS1991

Methodology for object-oriented real-time systems analysis and design: Software engineering

Successful application of software engineering methodologies requires an integrated analysis and design life-cycle in which the various phases flow smoothly 'seamlessly' from analysis through design to implementation. Furthermore, different analysis methodologies often lead to different structuring of the system so that the transition from analysis to design may be awkward depending on the design methodology to be used. This is especially important when object-oriented programming is to be used for implementation when the original specification and perhaps high-level design is non-object oriented. Two approaches to real-time systems analysis which can lead to an object-oriented design are contrasted: (1) modeling the system using structured analysis with real-time extensions which emphasizes data and control flows followed by the abstraction of objects where the operations or methods of the objects correspond to processes in the data flow diagrams and then design in terms of these objects; and (2) modeling the system from the beginning as a set of naturally occurring concurrent entities (objects) each having its own time-behavior defined by a set of states and state-transition rules and seamlessly transforming the analysis models into high-level design models. A new concept of a 'real-time systems-analysis object' is introduced and becomes the basic building block of a series of seamlessly-connected models which progress from the object-oriented real-time systems analysis and design system analysis logical models through the physical architectural models and the high-level design stages. The methodology is appropriate to the overall specification including hardware and software modules. In software modules, the systems analysis objects are transformed into software objects.

Schoeffler, James D.↗

DOE OSTI2022

VWC-BERT: Scaling Vulnerability–Weakness–Exploit Mapping on Modern AI Accelerators

Defending cybersystems needs accurate mapping of software and hardware vulnerabilities to generalized descriptions of weaknesses, and weaknesses to exploits. These mappings enable cyber defenders to build plans for effective defense and assessment of potential risks to a cybersystem. With close to 170k vulnerabilities, manual mapping is not a feasible option. However, automated mapping is challenging due to limited training data, computational intractability, and limitations in computational natural language processing. Tools based on breakthroughs in Transformer-based language models have been demonstrated to classify vulnerabilities with high accuracy. We make three key contributions in this paper: (1) We present a new framework, \VWCBERT, that augments the Transformer-based hierarchical multi-class classification framework of Das et al. (\textsc{V2W-BERT}) with the ability to map weaknesses to exploits. (2) We implement \VWCBERT~ on modern AI accelerator platforms using two data parallel techniques for the pre-training phase and demonstrate nearly linear speedups across NVIDIA and Graphcore accelerator platforms. We observe nearly linear speedups for up to 16 V100 and 8 A100 GPUs, and about 3.4$\times$ speedup for A100 relative to V100 GPUs. We also observe excellent speedups on Graphcore, with $5.7\times$ speedup on 128 IPUs relative to 16 IPUs. Enabled by scaling, we also demonstrate higher accuracy using a larger language model, RoBERTa-Large. We show up to 87\% accuracy for strict and up to 98\% accuracy for relaxed classification. (3) We develop a novel parallel link manager for the link prediction phase and demonstrate up to 21$\times$ speedup with 16 V100 GPUs relative to one V100 GPU, and thus reducing the runtime from 2.5 hours to 10 minutes. We believe that generalizability and scalability of \VWCBERT~ will benefit both the theoretical development and practical deployment of novel cyberdefense solutions and vulnerability classification.

Das, Siddhartha Shankar↗

NASA NTRS1992

Model based matching using simulated annealing and a minimum representation size criterion

We define the model based matching problem in terms of the correspondence and transformation that relate the model and scene, and the search and evaluation measures needed to find the best correspondence and transformation. Simulated annealing is proposed as a method for search and optimization, and the minimum representation size criterion is used as the evaluation measure in an algorithm that finds the best correspondence. An algorithm based on simulated annealing is presented and evaluated. This algorithm is viewed as a part of an adaptive, hierarchical approach which provides robust results for a variety of model based matching problems.

Ravichandran, B.↗

DOE OSTI2024

A crystal-plasticity-informed Gaussian Process Regression model to capture anisotropy in single crystal shape memory alloys

This work presents a machine learning (ML) framework that model the anisotropic actuation responses in a shape memory alloy. A Gaussian Process Regression (GPR) based ML model is trained on a set of different crystal orientations subjected to different actuation conditions. The training employed thermo-mechanical responses from a crystal-plasticity model that captures phase-transformation, stress-induced plasticity, and transformation-induced plasticity. Further, on training the GPR-ML model at fixed stress level for different orientations, it captured the thermo-mechanical responses accounting for the anisotropy, and predicted responses for new orientations with good accuracy. The GPR-ML model is able to capture the transformation temperature variations even when trained using multiple stress levels, and the transformation strain showed significant deviations. The developed GPR-ML model gave reasonable predictions for an unexplored sample set of orientations and loading conditions.

36 MATERIALS SCIENCE↗

DOE OSTI2022

Exploring Functional Materials by Understanding Beam‐Sample Interactions

Abstract Ultra‐low‐dose electron diffraction is performed with a double metal cyanide catalyst (DMC) to understand how electron irradiation stimulates structural alterations in functional materials. The commonly fading diffraction patterns with dose accumulation depend on the irradiated area and the beam current even when below 50 femto Amperes. Heat generation is observed and modeled by statistical, inelastic scattering events to describe how phonon excitations modulate radiation hardness. Specifically, the characteristic 1/e‐decay of Bragg intensities from DMC is delayed from 6 to 30 eÅ −2 at room temperature, which is comparable to the effect of embedding radiation soft matter in ice. DMC's radiation hardness is enhanced by a latency dose that forms during a phase transformation. This unifying model predicts that a critical dose rate exists for any material that varies between 0.1 and 10 4 eÅ −2 s −1 because of a material dependent competition of heat generation and spread. It shows that Brillouin scattering causes time dependent perturbations in electron irradiated solids that trigger time‐temperature‐transformations on a time scale of nanoseconds to microseconds at room temperature, which is not included in traditional models describing the decay of Bragg intensities by radiolysis.

Kisielowski, Christian↗