Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “neural generative models”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 37 records · Page 2
All sourcesNASA NTRSDOE OSTI
DOE OSTI2024

LENS: Learning Enabled Network Synthesis

RTRC and UMD have developed novel machine learning based methods under the ARPA-E DIFFERENTIATE program for rapid acceleration of hypothesis generation in complex architecture design spaces involving both discrete choices of component inclusion and interconnection and continuous parametric decisions. The project named Learning Enabled Network Synthesis (LENS) further demonstrated the developed methods on challenging electrical power converter design problems by identifying the most suitable circuit topologies and simultaneously selecting the most appropriate components to achieve optimized design of power converter with improved performances. We demonstrated that LENS could enable exploration of very large design space of circuit topologies and components by addressing the limitations of conventional design process in non-linear, high switching speed, multi-dimensional power converter design and optimization. The key innovation developed in LENS is the seamless integration of statistical learning and logical reasoning techniques and building on the individual strengths of these techniques for rapid hypothesis discovery. The main component of LENS comprises of: 1) Graph Reasoning Engine (GRE) to enforce composition rules that rapidly reject all discrete architectures that are composed incorrectly and generates an adaptive database of feasible designs which can be used by ML modules, 2) Graph Generative Learning module which is a deep neural network based generative model for graph architectures which can enable design space exploration beyond the dataset generated by the GRE, 3) Graph Reduced Order Model (ROM) for graph domains for accelerating computation of output metrics, and 4) Active learning and Rule Discovery module for sample efficient learning and extracting logical rules from the learned ML models which will be integrated in the GRE to enhance the filtering effectiveness. LENS approach can be applied to any design domains where designs can be represented as multi-attribute graphs. The LENS team integrated the various technical innovations listed above into an optimization pipeline and exercised the optimization pipeline on the converter design problem. The LENS project demonstrated that the developed AI/ML technologies can be used to generate novel converter circuits >45x faster than experts on chosen use-cases. This can enable faster design space exploration and identification of new designs which are not considered by experts due to the increasing design space complexity. This has significant potential impact on the public and energy needs of the country. It is currently estimated that 30% of all electrical powers generated passes through power converters. The future estimate is that 80% of all power generated would be passing through converters. LENS fills a critical gap in this space since by accelerating the design process the designers would be able to generate more efficient converters which can lead to significant energy savings for the country.

42 ENGINEERING↗
DOE OSTI2022

Dynamic Parameter Estimation with Physics-based Neural Ordinary Differential Equations

Accurate estimation of dynamic parameters of gen-erators is crucial to building a reliable model for dynamical studies and reliable operation of the power system. This paper develops a physics-based neural ordinary differential equations (ODE) approach to learn the parameters of generator dynamic model using phasor measurement units (PMU) data. We design a physics-based neural network to represent the swing equations of the power system dynamics. A loss function is defined as the difference between dynamic simulation results from the physics-based neural networks and pseudo PMU measurements. The parameters of generator dynamic model are iteratively updated using the neural ODEs and the adjoint method. By exploiting the mini-batch scheme in neural ODE training, the parameter estimation performance is significantly improved. Numerical study results on a 3-machine 9-bus system show that the proposed algorithm outperforms state-of-the-art baseline method in both computation time and dynamic parameter estimation accuracy.

24 POWER TRANSMISSION AND DISTRIBUTION↗
DOE OSTI2021

Verification of Data-Driven Models of Physical Phenomena using Interpretable Approximation

Machine-learned models, specifically neural networks, are increasingly used as “closures” or “constitutive models” in engineering simulators to represent fine-scale physical phenomena that are too computationally expensive to resolve explicitly. However, these neural net models of unresolved physical phenomena tend to fail unpredictably and are therefore not used in mission-critical simulations. In this report, we describe new methods to authenticate them, i.e., to determine the (physical) information content of their training datasets, qualify the scenarios where they may be used and to verify that the neural net, as trained, adhere to physics theory. We demonstrate these methods with neural net closure of turbulent phenomena used in Reynolds Averaged Navier-Stokes equations. We show the types of turbulent physics extant in our training datasets, and, using a test flow of an impinging jet, identify the exact locations where the neural network would be extrapolating i.e., where it would be used outside the feature-space where it was trained. Using Generalized Linear Mixed Models, we also generate explanations of the neural net (à la Local Interpretable Model agnostic Explanations) at prototypes placed in the training data and compare them with approximate analytical models from turbulence theory. Finally, we verify our findings by reproducing them using two different methods.

42 ENGINEERING↗
DOE OSTI2026

Learning turbulent flows with generative models for super resolution and sparse flow reconstruction

Neural operators are promising surrogates for dynamical systems but when trained with standard L 2 losses they tend to oversmooth fine-scale turbulent structures. Here, we show that combining operator learning with generative modeling overcomes this limitation. We consider three practical turbulent-flow challenges where conventional neural operators fail: spatio-temporal super-resolution, forecasting, and sparse flow reconstruction. For Schlieren jet super-resolution, an adversarially trained neural operator (adv-NO) reduces the energy-spectrum error by 15 × while preserving sharp gradients at neural operator-like inference cost. For 3D homogeneous isotropic turbulence, adv-NO trained on only 160 timesteps from a single trajectory forecasts accurately for five eddy-turnover times and offers 114 × wall-clock speed-up at inference than the baseline diffusion-based forecasters, enabling near-real-time rollouts. For reconstructing cylinder wake flows from highly sparse Particle Tracking Velocimetry-like inputs, a conditional generative model infers full 3D velocity and pressure fields with correct phase alignment and statistics. These advances enable accurate reconstruction and forecasting at low compute cost, bringing near-real-time analysis and control within reach in experimental and computational fluid mechanics.

Fluid dynamics↗
DOE OSTI2023

A publicly available PyTorch-$\mathrm{ABAQUS}$ $\mathrm{UMAT}$ deep-learning framework for level-set plasticity

Here this paper introduces a publicly available PyTorch-ABAQUS deep-learning framework of a family of plasticity models where the yield surface is implicitly represented by a scalar-valued function. In particular, our focus is to introduce a practical framework that can be deployed for engineering analysis that employs a user-defined material subroutine (UMAT/VUMAT) for ABAQUS, which is written in FORTRAN. To accomplish this task while leveraging the back-propagation learning algorithm to speed up the neural-network training, we introduce an interface code where the weights and biases of the trained neural networks obtained via the PyTorch library can be automatically converted into a generic FORTRAN code that can be a part of the UMAT/VUMAT algorithm. To enable third-party validation, we purposely make all the data sets, source code used to train the neural-network-based constitutive models, and the trained models available in a public repository. Furthermore, the practicality of the workflow is then further tested on a dataset for anisotropic yield function to showcase the extensibility of the proposed framework. A number of representative numerical experiments are used to examine the accuracy, robustness and reproducibility of the results generated by the neural network models.

36 MATERIALS SCIENCE↗
DOE OSTI2020

Technical note: Deep learning for creating surrogate models of precipitation in Earth system models

Abstract. We investigate techniques for using deep neural networks to produce surrogatemodels for short-term climate forecasts. A convolutional neural network istrained on 97 years of monthly precipitation output from the 1pctCO2 run (theCO 2 concentration increases by 1 % per year) simulated by the second-generation Canadian Earth System Model (CanESM2). The neural network clearly outperforms a persistence forecast anddoes not show substantially degraded performance even when the forecast lengthis extended to 120 months. The model is prone to underpredicting precipitationin areas characterized by intense precipitation events. Scheduled sampling(forcing the model to gradually use its own past predictions rather than groundtruth) is essential for avoiding amplification of early forecasting errors.However, the use of scheduled sampling also necessitates preforecasting(generating forecasts prior to the first forecast date) to obtain adequateperformance for the first few prediction time steps. We document the trainingprocedures and hyperparameter optimization process for researchers who wish toextend the use of neural networks in developing surrogate models.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2022

Making Invisible Visible: Data-Driven Seismic Inversion With Spatio-Temporally Constrained Data Augmentation

Deep learning and data-driven approaches have shown great potential in scientific domains. The promise of data-driven techniques relies on the availability of a large volume of high-quality training datasets. Due to the high cost of obtaining data through expensive physical experiments, instruments, and simulations, data augmentation techniques for scientific applications have emerged as a new direction for obtaining scientific data recently. However, existing data augmentation techniques originating from computer vision yield physically unacceptable data samples that are not helpful for the domain problems that we are interested in. In this article, we develop new data augmentation techniques based on convolutional neural networks. Specifically, our generative models leverage different physics knowledge (such as governing equations, observable perception, and physics phenomena) to improve the quality of the synthetic data. To validate the effectiveness of our data augmentation techniques, we apply them to solve a subsurface seismic full-waveform inversion using simulated CO 2 leakage data. Our interest is to invert for subsurface velocity models associated with very small CO 2 leakage. We validate the performance of our methods using comprehensive numerical tests. Here via comparison and analysis, we show that data-driven seismic imaging can be significantly enhanced by using our data augmentation techniques. Particularly, the imaging quality has been improved by 15% in test scenarios of general-sized leakage and 17% in small-sized leakage when using an augmented training set obtained with our techniques.

58 GEOSCIENCES↗
DOE OSTI2020

Fifth-Generation District Heating and Cooling Substations: Demand Response with Artificial Neural Network-Based Model Predictive Control

District heating and cooling (DHC) is considered one of the most sustainable technologies to meet the heating and cooling demands of buildings in urban areas. The fifth-generation district heating and cooling (5GDHC) concept, often referred to as ambient loops, is a novel solution emerging in Europe and has become a widely discussed topic in current energy system research. 5GDHC systems operate at a temperature close to the ground and include electrically driven heat pumps and associated thermal energy storage in a building-sited energy transfer station (ETS) to satisfy user comfort. This work presents new strategies for improving the operation of these energy transfer stations by means of a model predictive control (MPC) method based on recurrent artificial neural networks. The results show that, under simple time-of-use utility rates, the advanced controller outperforms a rule-based controller for smart charging of the domestic hot water (DHW) thermal energy storage under specific boundary conditions. By exploiting the available thermal energy storage capacity, the MPC controller is capable of shifting up to 14% of the electricity consumption of the ETS from on-peak to off-peak hours. Therefore, the advanced control implemented in 5GDHC networks promotes coupling between the thermal and the electric sector, producing flexibility on the electric grid.

32 ENERGY CONSERVATION, CONSUMPTION, AND UTILIZATI↗
DOE OSTI2020

Inversion of Time-Lapse Seismic Reservoir Monitoring Data Using CycleGAN: A Deep Learning-Based Approach for Estimating Dynamic Reservoir Property Changes

Carbon capture and storage is being pursued globally as a geoengineering measure for reducing the emission of anthropogenic CO 2 the atmosphere. Comprehensive monitoring, verification, and accounting programs must be established for demonstrating the safe storage of injected CO 2 . One of the most commonly deployed monitoring techniques is time-lapse seismic reservoir monitoring (also known as 4-D seismic), which involves comparing 3-D seismic survey data taken at the same study site but over different times. Analyses of 4-D seismic data volumes can help improve the quality of storage reservoir characterization, track the movement of injected CO 2 plume, and identify potential CO 2 spillover/leakage from the storage reservoirblue. However, the derivation of high-resolution CO 2 saturation maps from 4-D seismic data is a highly nonlinear and ill-posed inverse problem, often requiring significant computational effort. In this research, we apply a physics-based deep learning method to facilitate the solution of both the forward and inverse problems in seismic inversion while honoring physical constraints. A cycle generative adversarial neural network (CycleGAN) model is trained to learn the bidirectional functional mappings between the reservoir dynamic property changes and seismic attribute changes, such that both forward and inverse solutions can be obtained efficiently from the trained model. We show that our CycleGAN-based approach not only improves the reliability of 4-D seismic inversion but also expedites the quantitative interpretation. Our deep learning-based workflow is generic and can be readily used for reservoir characterization and reservoir model updates involving the use of 4-D seismic data.

58 GEOSCIENCES↗
DOE OSTI2020

Deblending galaxies with variational autoencoders: A joint multiband, multi-instrument approach

ABSTRACT Blending of galaxies has a major contribution in the systematic error budget of weak-lensing studies, affecting photometric and shape measurements, particularly for ground-based, deep, photometric galaxy surveys, such as the Rubin Observatory Legacy Survey of Space and Time (LSST). Existing deblenders mostly rely on analytic modelling of galaxy profiles and suffer from the lack of flexible yet accurate models. We propose to use generative models based on deep neural networks, namely variational autoencoders (VAE), to learn probabilistic models directly from data. We train a VAE on images of centred, isolated galaxies, which we reuse, as a prior, in a second VAE-like neural network in charge of deblending galaxies. We train our networks on simulated images including six LSST bandpass filters and the visible and near-infrared bands of the Euclid satellite, as our method naturally generalizes to multiple bands and can incorporate data from multiple instruments. We obtain median reconstruction errors on ellipticities and r-band magnitude between ±0.01 and ±0.05, respectively, in most cases, and ellipticity multiplicative bias of 1.6 per cent for blended objects in the optimal configuration. We also study the impact of decentring and prove the method to be robust. This method only requires the approximate centre of each target galaxy, but no assumptions about the number of surrounding objects, pointing to an iterative detection/deblending procedure we leave for future work. Finally, we discuss future challenges about training on real data and obtain encouraging results when applying transfer learning.

Arcelin, Bastien↗
DOE OSTI2026

An Iterative Machine Learning Framework for Event Classification and Monte Carlo Tuning in SpinQuest

The E1039/SpinQuest experiment at Fermi National Accelerator Laboratory uses a 120~GeV proton beam from the Main Injector incident on transversely polarized proton and deuteron targets, using $NH_3$ and $ND_3$, respectively. In addition to measuring the Sivers asymmetry in Drell--Yan $pp$ and $pd$ scattering from sea quarks, SpinQuest will study transverse-spin effects, particularly the transverse single-spin asymmetry (TSSA) in $J/\psi$ production. The angular distributions from the $J/\psi$ decay could play an important role in understanding the gluon contribution to the proton spin structure. However, before extracting these angular distributions, it is necessary to isolate signal events originating from the target from events produced by other sources and from the combinatorial background. To effectively and accurately classify the target events, it is important to ensure that the simulated events are properly tuned to the experimental physics channels. We have introduced an iterative technique to match simulated and experimental events and to classify the physics channels using deep neural networks and a generative model based on normalizing flows.

Hossain, Forhad [Virginia U. (main)] (ORCID:000000↗
DOE OSTI2021

A systematic approach to generating accurate neural network potentials: the case of carbon

Abstract Availability of affordable and widely applicable interatomic potentials is the key needed to unlock the riches of modern materials modeling. Artificial neural network-based approaches for generating potentials are promising; however, neural network training requires large amounts of data, sampled adequately from an often unknown potential energy surface. Here we propose a self-consistent approach that is based on crystal structure prediction formalism and is guided by unsupervised data analysis, to construct an accurate, inexpensive, and transferable artificial neural network potential. Using this approach, we construct an interatomic potential for carbon and demonstrate its ability to reproduce first principles results on elastic and vibrational properties for diamond, graphite, and graphene, as well as energy ordering and structural properties of a wide range of crystalline and amorphous phases.

Chemistry↗
DOE OSTI2020

Deep Generative Models that Solve PDEs: Distributed Computing for Training Large Data-Free Models

Recent progress in scientific machine learning (SciML) has opened up the possibility of training novel neural network architectures that solve complex partial differential equations (PDEs). Several (nearly data free) approaches have been recently reported that successfully solve PDEs, with examples including deep feed forward networks, generative networks, and deep encoder-decoder networks. However, practical adoption of these approaches is limited by the difficulty in training these models, especially to make predictions at large output resolutions (≥1024×1024). Here we report on a software framework for data parallel distributed deep learning that resolves the twin challenges of training these large SciML models - training in reasonable time as well as distributing the storage requirements. Our framework provides several out of the box functionality including (a) loss integrity independent of number of processes, (b) synchronized batch normalization, and (c) distributed higher-order optimization methods. We show excellent scalability of this framework on both cloud as well as HPC clusters, and report on the interplay between bandwidth, network topology and bare metal vs cloud. We deploy this approach to train generative models of sizes hitherto not possible, showing that neural PDE solvers can be viably trained for practical applications. We also demonstrate that distributed higher-order optimization methods are 2-3× faster than stochastic gradient-based methods and provide minimal convergence drift with higher batch-size.

PDEs↗
DOE OSTI2023

Laser Wakefield Accelerator modelling with Variational Neural Networks

A machine learning model was created to predict the electron spectrum generated by a GeV-class laser wakefield accelerator. The model was constructed from variational convolutional neural networks, which mapped the results of secondary laser and plasma diagnostics to the generated electron spectrum. An ensemble of trained networks was used to predict the electron spectrum and to provide an estimation of the uncertainty of that prediction. It is anticipated that this approach will be useful for inferring the electron spectrum prior to undergoing any process that can alter or destroy the beam. In addition, the model provides insight into the scaling of electron beam properties due to stochastic fluctuations in the laser energy and plasma electron density.

43 PARTICLE ACCELERATORS↗
DOE OSTI2023

Diffusion Probabilistic Modeling for Video Generation

Denoising diffusion probabilistic models are a promising new class of generative models that mark a milestone in high-quality image generation. This paper showcases their ability to sequentially generate video, surpassing prior methods in perceptual and probabilistic forecasting metrics. We propose an autoregressive, end-to-end optimized video diffusion model inspired by recent advances in neural video compression. The model successively generates future frames by correcting a deterministic next-frame prediction using a stochastic residual generated by an inverse diffusion process. We compare this approach against six baselines on four datasets involving natural and simulation-based videos. We find significant improvements in terms of perceptual quality and probabilistic frame forecasting ability for all datasets.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2025

Using Machine Learning to Generate a GISS ModelE Calibrated Physics Ensemble (CPE)

A neural network (NN) surrogate of the NASA GISS ModelE atmosphere (version E3) is trained on a perturbed parameter ensemble (PPE) spanning 45 physics parameters and 36 outputs. The NN is leveraged in a Markov Chain Monte Carlo (MCMC) Bayesian parameter inference framework to generate a second posterior constrained ensemble coined a “calibrated physics ensemble,” or CPE. The CPE members are characterized by diverse parameter combinations and are, by definition, close to top-of-atmosphere radiative balance, and must broadly agree with numerous hydrologic, energy cycle and radiative forcing metrics simultaneously. Global observations of numerous cloud, environment, and radiation properties (provided by global satellite products) are crucial for CPE generation. The inference framework explicitly accounts for discrepancies (or biases) in satellite products during CPE generation. We demonstrate that product discrepancies strongly impact calibration of important model parameter settings (e.g., convective plume entrainment rates; fall speed for cloud ice). Structural improvements new to E3 are retained across CPE members (e.g., stratocumulus simulation). Notably, the framework improved the simulation of shallow cumulus and Amazon rainfall while not degrading radiation fields, an upgrade that neither default parameters nor Latin Hypercube parameter searching achieved. Analyses of the initial PPE suggested several parameters were unimportant for output variation. However, many “unimportant” parameters were needed for CPE generation, a result that brings to the forefront how parameter importance should be determined in PPEs. From the CPE, two diverse 45-dimensional parameter configurations are retained to generate radiatively-balanced, auto-tuned atmospheres that were used in two E3 submissions to CMIP6.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2023

Efficient Super-Resolution of Near-Surface Climate Modeling Using the Fourier Neural Operator

Downscaling methods are critical in efficiently generating high-resolution atmospheric data. However, state-of-the-art statistical or dynamical downscaling techniques either suffer from the high computational cost of running a physical model or require high-resolution data to develop a downscaling tool. Here, we demonstrate a recently proposed zero-shot super-resolution method, the Fourier neural operator (FNO), to efficiently perform downscaling without the need for high-resolution data. Because the FNO learns dynamics in Fourier space, FNO is a resolution-invariant emulator; it can be trained at a coarse resolution and produces emulation at any high resolution. We applied FNO to downscale a 4-km resolution Weather Research and Forecasting (WRF) Model simulation of near-surface heat-related variables over the Great Lakes region. The FNO is driven by the atmospheric forcings, and topographic features used in the WRF model at the same resolution. We incorporated a physics-constrained loss in FNO by using the Clausius-Clapeyron relation to better constrain the relations among the emulated states. Trained on merely 600 WRF snapshots at 4-km resolution, the FNO shows comparable performance with a widely used convolutional network, U-Net, achieving averaged modified Kling-Gupta Efficiency of 0.88 and 0.94 on the test dataset for temperature and pressure, respectively. We then employed the FNO to produce 1-km emulations to reproduce the fine climate features. Further, by taking the WRF simulation as ground truth, we show consistent performances at the two resolutions, suggesting the reliability of FNO in producing high-resolution dynamics. Our study demonstrates the potential of using FNO for zero-shot super-resolution in generating first-order estimation on atmospheric modeling.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2022

Deep learning to estimate permeability using geophysical data

Time-lapse electrical resistivity tomography (ERT) is a popular geophysical method to estimate three-dimensional (3D) permeability fields from electrical potential difference measurements. Traditional inversion and data assimilation methods are used to ingest this ERT data into hydrogeophysical models to estimate permeability. Due to ill-posedness and the curse of dimensionality, existing inversion strategies provide poor estimates and low resolution of the 3D permeability field. Recent advances in deep learning provide us with powerful algorithms to overcome this challenge. This paper presents a deep learning (DL) framework to estimate the 3D subsurface permeability from time-lapse ERT data. To test the feasibility of the proposed framework, we train DL-enabled inverse models on simulation data. Each measurement in both synthetic and field data is standardized by removing the mean and scaling the time-series to unit variance. This pre-processing step is necessary to bring simulation data closer to field observations. Subsurface process models based on hydrogeophysics are used to generate this synthetic data. Training performed on limited simulation data resulted in the DL model over-fitting. An advanced data augmentation based on mixup is implemented to generate additional training samples to overcome this issue. This mixup technique creates weakly labeled (low-fidelity) samples from strongly labeled (high-fidelity) data. The weakly labeled training data is then used to develop DL-enabled inverse models and reduce over-fitting. As both time-lapse ERT (1133048 features/realization) and 3D permeability (585453 features/realization) data samples are from a high-dimensional space, principal component analysis (PCA) is employed to reduce dimensionality. Encoded ERT and encoded permeability are generated using the trained PCA estimators. A deep neural network is then trained to map the encoded ERT to encoded permeability. This mixup training and unsupervised learning allowed us to build a fast and reasonably accurate DL-based inverse model under limited simulation data. Results show that proposed weak supervised learning can capture salient spatial features in the 3D permeability field. Quantitatively, the average mean squared error (in terms of the natural log) on the strongly labeled training, validation, and test datasets is less than 0.5. The R 2 -score (global metric) is greater than 0.75, and the percent error in each cell (local metric) is less than 10%. Finally, an added benefit in terms of computational cost is that the proposed DL-based inverse model is at least O(10 4 ) times faster than running a forward model once it is trained. Data generation, DL model training, and hyperparameter tuning to identify optimal neural network architectures utilized high-performance computing resources while the DL inference is performed on a standard laptop. Approximately, O(10 5 ) processor hours are used for generating data and DL tuning and training. We acknowledge that the data generation and DL model development are expensive. But once a DL model is trained, it can be re-used for inversion rapidly for the given system, with set physics and domain. Note that traditional inversion may require multiple forward model simulations (e.g., in the order of 10 to 1000), which are very expensive. This computational savings ≈ O(10 5 ) – O(10 7 )) makes the proposed DL-based inverse model attractive for subsurface imaging and real-time ERT monitoring applications due to fast and yet reasonably accurate estimations of permeability field.

58 GEOSCIENCES↗