Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “convolutional neural network model”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 343 records · Page 19
All sourcesNASA NTRSDOE OSTI
DOE OSTI2024

Accelerating defect predictions in semiconductors using graph neural networks

First-principles computations reliably predict the energetics of point defects in semiconductors but are constrained by the expense of using large supercells and advanced levels of theory. Machine learning models trained on computational data, especially ones that sufficiently encode defect coordination environments, can be used to accelerate defect predictions. Here, we develop a framework for the prediction and screening of native defects and functional impurities in a chemical space of group IV, III–V, and II–VI zinc blende semiconductors, powered by crystal Graph-based Neural Networks (GNNs) trained on high-throughput density functional theory (DFT) data. Using an innovative approach of sampling partially optimized defect configurations from DFT calculations, we generate one of the largest computational defect datasets to date, containing many types of vacancies, self-interstitials, anti-site substitutions, impurity interstitials and substitutions, as well as some defect complexes. We applied three types of established GNN techniques, namely crystal graph convolutional neural network, materials graph network, and Atomistic Line Graph Neural Network (ALIGNN), to rigorously train models for predicting defect formation energy (DFE) in multiple charge states and chemical potential conditions. We find that ALIGNN yields the best DFE predictions with root mean square errors around 0.3 eV, which represents a prediction accuracy of 98% given the range of values within the dataset, improving significantly on the state-of-the-art. We further show that GNN-based defective structure optimization can take us close to DFT-optimized geometries at a fraction of the cost of full DFT. The current models are based on the semi-local generalized gradient approximation-Perdew–Burke–Ernzerhof (PBE) functional but are highly promising because of the correlation of computed energetics and defect levels with higher levels of theory and experimental data, the accuracy and necessity of discovering novel metastable and low energy defect structures at the PBE level of theory before advanced methods could be applied, and the ability to train multi-fidelity models in the future with new data from non-local functionals. The DFT-GNN models enable prediction and screening across thousands of hypothetical defects based on both unoptimized and partially optimized defective structures, helping identify electronically active defects in technologically important semiconductors.

Rahman, Md Habibur (ORCID:000000027705984X)↗
DOE OSTI2020

Technical note: Deep learning for creating surrogate models of precipitation in Earth system models

Abstract. We investigate techniques for using deep neural networks to produce surrogatemodels for short-term climate forecasts. A convolutional neural network istrained on 97 years of monthly precipitation output from the 1pctCO2 run (theCO 2 concentration increases by 1 % per year) simulated by the second-generation Canadian Earth System Model (CanESM2). The neural network clearly outperforms a persistence forecast anddoes not show substantially degraded performance even when the forecast lengthis extended to 120 months. The model is prone to underpredicting precipitationin areas characterized by intense precipitation events. Scheduled sampling(forcing the model to gradually use its own past predictions rather than groundtruth) is essential for avoiding amplification of early forecasting errors.However, the use of scheduled sampling also necessitates preforecasting(generating forecasts prior to the first forecast date) to obtain adequateperformance for the first few prediction time steps. We document the trainingprocedures and hyperparameter optimization process for researchers who wish toextend the use of neural networks in developing surrogate models.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2021

Computationally Efficient Multiscale Neural Networks Applied to Fluid Flow in Complex 3D Porous Media

Abstract The permeability of complex porous materials is of interest to many engineering disciplines. This quantity can be obtained via direct flow simulation, which provides the most accurate results, but is very computationally expensive. In particular, the simulation convergence time scales poorly as the simulation domains become less porous or more heterogeneous. Semi-analytical models that rely on averaged structural properties (i.e., porosity and tortuosity) have been proposed, but these features only partly summarize the domain, resulting in limited applicability. On the other hand, data-driven machine learning approaches have shown great promise for building more general models by virtue of accounting for the spatial arrangement of the domains’ solid boundaries. However, prior approaches building on the convolutional neural network (ConvNet) literature concerning 2D image recognition problems do not scale well to the large 3D domains required to obtain a representative elementary volume (REV). As such, most prior work focused on homogeneous samples, where a small REV entails that the global nature of fluid flow could be mostly neglected, and accordingly, the memory bottleneck of addressing 3D domains with ConvNets was side-stepped. Therefore, important geometries such as fractures and vuggy domains could not be modeled properly. In this work, we address this limitation with a general multiscale deep learning model that is able to learn from porous media simulation data. By using a coupled set of neural networks that view the domain on different scales, we enable the evaluation of large ( $$>512^3$$ > 512 3 ) images in approximately one second on a single graphics processing unit. This model architecture opens up the possibility of modeling domain sizes that would not be feasible using traditional direct simulation tools on a desktop computer. We validate our method with a laminar fluid flow case using vuggy samples and fractures. As a result of viewing the entire domain at once, our model is able to perform accurate prediction on domains exhibiting a large degree of heterogeneity. We expect the methodology to be applicable to many other transport problems where complex geometries play a central role.

36 MATERIALS SCIENCE↗
NASA NTRS2022

Revisiting the Ground Magnetic Field Perturbations Challenge: A Machine Learning Perspective

Forecasting ground magnetic field perturbations has been a long-standing goal of the space weather community. The availability of ground magnetic field data and its potential to be used in geomagnetically induced current studies, such as risk assessment, have resulted in several forecasting efforts over the past few decades. One particular community effort was the Geospace Environment Modeling (GEM) challenge of ground magnetic field perturbations that evaluated the predictive capacity of several empirical and first principles models at both mid- and high-latitudes in order to choose an operative model. In this work, we use three different deep learning models-a feed-forward neural network, a long short-term memory recurrent network and a convolutional neural network-to forecast the horizontal component of the ground magnetic field rate of change (dB H /dt) over 6 different ground magnetometer stations and to compare as directly as possible with the original GEM challenge. We find that, in general, the models are able to perform at similar levels to those obtained in the original challenge, although the performance depends heavily on the particular storm being evaluated. We then discuss the limitations of such a comparison on the basis that the original challenge was not designed with machine learning algorithms in mind.

Victor A. Pinto↗
DOE OSTI2021

Attend and Decode: 4D fMRI Task State Decoding Using Attention Models

Source code for Brain Attend and Decode paper. Functional magnetic resonance imaging (fMRI) is a neuroimaging modality that captures the blood oxygen level in a subject's brain while the subject either rests or performs a variety of functional tasks under different conditions. Given fMRI data, the problem of inferring the task, known as task state decoding, is challenging due to the high dimensionality (hundreds of million sampling points per datum) and complex spatio-temporal blood flow patterns inherent in the data. In this work, we propose to tackle the fMRI task state decoding problem by casting it as a 4D spatiotemporal classification problem. We present a novel architecture called Brain Attend and Decode (BAnD), that uses residual convolutional neural networks for spatial feature extraction and self-attention mechanisms for temporal modeling. We achieve significant performance gain compared to previous works on a 7-task benchmark from the large-scale Human Connectome Project-Young Adult (HCP-YA) dataset. We also investigate the transferability of BAnD's extracted features on unseen HCP tasks, either by freezing the spatial feature extraction layers and retraining the temporal model, or finetuning the entire model. The pre-trained features from BAnD are useful on similar tasks while finetuning them yields competitive results on unseen tasks/conditions.

Ng, BrendaM.↗
NASA NTRSDate not supplied

Flight Trajectory Prediction Based on Hybrid-Recurrent Networks

The development of future technologies for the National Airspace System (NAS) will be reliant on a new communications infrastructure capable of managing the limited available spectrum for communications among aircraft and ground systems. Emerging approaches to autonomous allocation of aviation spectrum mostlyrely on machine learning techniques, where 4D (longitude, latitude, altitude, time) trajectory prediction is an important data input to enable real-time resource allocation. This study explores and evaluates effective data sources and deep recurrent neural network techniques when determining flight trajectories. Specifically, data are collected and evaluated in a 100-day and 14-day period. Sources of data include NASA Sherlock Data Warehouse, MIT Lincoln Labs Corridor Integrated Weather Service (CIWS), and assorted NOAA weather datasets. Deep learning models for 4D predictions all utilize a hybrid-recurrent technique. A baseline model is considered via the convolutional-LSTM design from the existing literature. The modified design considers Gated Recurrent Units (GRU), Independently Recurrent Neural Networks (IndRNN), and stand-alone self-attention layers. Results indicatethe effectiveness of LSTM and GRUcells for state-of-the-art data processing (interpolation). Additionally, GRUs may be quickly trained with limited data, allowing for exacting improvements with optimizer selection. Attention mechanisms provide notable performance improvements to convolutional layers and may extend dimensional capabilities of a learning model. Finally, NOAA measurements provide only a supplemental value, requiring support from tailored measurements for Air Traffic Management.

Nathan Schimpf↗
DOE OSTI2022

Physics-Based Method for Generating Fully Synthetic IV Curve Training Datasets for Machine Learning Classification of PV Failures

Classification machine learning models require high-quality labeled datasets for training. Among the most useful datasets for photovoltaic array fault detection and diagnosis are module or string current-voltage (IV) curves. Unfortunately, such datasets are rarely collected due to the cost of high fidelity monitoring, and the data that is available is generally not ideal, often consisting of unbalanced classes, noisy data due to environmental conditions, and few samples. In this paper, we propose an alternate approach that utilizes physics-based simulations of string-level IV curves as a fully synthetic training corpus that is independent of the test dataset. In our example, the training corpus consists of baseline (no fault), partial soiling, and cell crack system modes. The training corpus is used to train a 1D convolutional neural network (CNN) for failure classification. The approach is validated by comparing the model’s ability to classify failures detected on a real, measured IV curve testing corpus obtained from laboratory and field experiments. Results obtained using a fully synthetic training dataset achieve identical accuracy to those obtained with use of a measured training dataset. When evaluating the measured data’s test split, a 100% accuracy was found both when using simulations or measured data as the training corpus. When evaluating all of the measured data, a 96% accuracy was found when using a fully synthetic training dataset. The use of physics-based modeling results as a training corpus for failure detection and classification has many advantages for implementation as each PV system is configured differently, and it would be nearly impossible to train using labeled measured data.

Hopwood, Michael W. (ORCID:0000000161901767)↗
DOE OSTI2024

Multi deep learning-based stochastic microstructure reconstruction and high-fidelity micromechanics simulation of time-dependent ceramic matrix composite response

A multi deep learning-based framework is developed for efficient, automated microstructure reconstruction and generation of stochastic representative volume elements (SRVEs) with periodic boundary conditions (PBCs) for accurate modeling of ceramic matrix composite (CMC) response. The methodology comprises a convolutional neural network coupled with regression layers to act as a vanilla regression network for semantic segmentation of the microstructure, allowing accurate characterization of the phases and their distributions at the microscale. Scanning electron microscope and confocal microscope are used to obtain C/SiNC and SiC/SiNC CMCs micrographs for vanilla regression testing. Microstructure variability in terms of fiber volume fraction and porosity are quantified through the output regression layer, ensuring accurate representation of material variability in SRVE construction. Generative adversarial network (GAN) and its variants are designed to produce high-fidelity SRVE, spanning CMCs microstructure variability space. A circular padding algorithm is developed to generate SRVEs with PBCs during training of GANs. The accuracy of the generated SRVEs is established through micromechanics simulations, where an efficient formulation of the high-fidelity generalized methods of cells (HFGMC) approach is used to compute the effective mechanical properties. Furthermore, an iterative algorithm is implemented in the HFGMC solver to simulate time-dependent deformation of SiC/SiNC subjected to creep loading conditions.

36 MATERIALS SCIENCE↗
DOE OSTI2023

Inference of the optical depth to reionization τ from Planck CMB maps with convolutional neural networks

The optical depth to reionization, τ, is the least constrained parameter of the cosmological Λ cold dark matter (ΛCDM) model. To date, its most precise value is inferred from large-scale polarized cosmic microwave background (CMB) power spectra from the High Frequency Instrument (HFI) aboard the Planck satellite. These maps are known to contain significant contamination by residual non-Gaussian systematic effects, which are hard to model analytically. Therefore, robust constraints on τ are currently obtained through an empirical cross-spectrum likelihood built from simulations. In this paper, we present a likelihood-free inference of τ from polarized Planck HFI maps which, for the first time, is fully based on neural networks (NNs). NNs have the advantage of not requiring an analytical description of the data and can be trained on state-of-the-art simulations, combining the information from multiple channels. By using Gaussian sky simulations and Planck SRoll2 simulations, including CMB, noise, and residual instrumental systematic effects, we trained, tested, and validated NN models considering different setups. We inferred the value of τ directly from Stokes Q and U maps at ~4° pixel resolution, without computing angular power spectra. On Planck data, we obtained τ NN = 0.0579 ± 0.0082, which is compatible with current EE cross-spectrum results but with a ~30% larger uncertainty, which can be assigned to the inherent nonoptimality of our estimator and to the retraining procedure applied to avoid biases. While this paper does not improve on current cosmological constraints on τ, our analysis represents a first robust application of NN-based inference on real data, and highlights its potential as a promising tool for complementary analysis of near-future CMB experiments, also in view of the ongoing challenge to achieve the first detection of primordial gravitational waves.

79 ASTRONOMY AND ASTROPHYSICS↗
DOE OSTI2020

Machine learning enabled discovery of application dependent design principles for two-dimensional materials

Abstract The unique electronic and mechanical properties of two-dimensional (2D) materials make them promising next-generation candidates for a variety of applications. Large-scale searches for high-performing 2D materials are limited to calculating descriptors with computationally demanding first-principles density functional theory. In this work, we alleviate this issue by extending and generalizing crystal graph convolutional neural networks to systems with planar periodicity and train an ensemble of models to predict thermodynamic, mechanical and electronic properties. We carry out a screening of nearly 45,000 structures for two separate applications: mechanical strength and photovoltaics. By collecting statistics of the screened candidates, we investigate structural and compositional design principles that impact the properties of the structures surveyed. Our approach recovers some well-accepted design rules: hybrid organic-inorganic perovskites with lead and tin tend to be good candidates for solar cell applications and titanium based MXenes usually have high stiffness coefficients. Interestingly, other members of the group 4 elements also contribute to increasing the mechanical strength of MXenes. For all-inorganic perovskites, we discover some compositions that have not been deeply studied in the field of photovoltaics and thus open up paths for further investigation. We open-source the code-base to spur further development in this space.

Venturi, Victor (ORCID:0000000156727549)↗
DOE OSTI2022

Neuromorphic Accelerator for Deep Spiking Neural Networks with NVM Crossbar Arrays

In this paper, we present a scalable digital hardware accelerator based on non-volatile memory arrays capable of realizing deep convolutional spiking neural networks (SNNs). Our design studies are conducted using a compact model for spin-transfer torque random access memory (STT-RAM) devices. Large networks are realized by tiling multiple cores which communicate by transmitting spike packets via an on-chip routing network. Compared to an equivalent SRAM based core design, we show that the STT-RAM based design achieves nearly 15X higher GSOPS (Synaptic Operations per Second) per Watt per mm 2 making it a promising platform for realizing systems with significant area and power limitations.

Kulkarni, Shruti↗
DOE OSTI2020

Radar Super Resolution using a Deep Convolutional Neural Network

Super-resolution involves synthetically increasing the resolution of gridded data beyond its native resolution. Typically, this is done using interpolation schemes, which estimate sub-grid scale values from neighboring data, and perform the same operation everywhere regardless of the large-scale context, or by requiring a network of radars with overlapping fields of view. Recently, significant progress has been made in single image super resolution using convolutional neural networks. Conceptually, a neural network may be able to learn relations between large scale precipitation features and the associated sub-pixel scale variability and outperform interpolation schemes. Here, we use a deep convolutional neural network to artificially enhance the resolution of NEXRAD PPI scans. The model is trained on 6-months of reflectivity observations from the Langley Hill WA (KLGX) radar, and we find that it substantially outperforms common interpolation schemes for x4 and x8 resolution increases based on several objective error and perceptual quality metrics.

radar, machine learning, super resolution, Remote ↗
DOE OSTI2023

Di-CNN: Domain-Knowledge-Informed Convolutional Neural Network for Manufacturing Quality Prediction

In manufacturing, convolutional neural networks (CNNs) are widely used on image sensor data for data-driven process monitoring and quality prediction. However, as purely data-driven models, CNNs do not integrate physical measures or practical considerations into the model structure or training procedure. Consequently, CNNs’ prediction accuracy can be limited, and model outputs may be hard to interpret practically. This study aims to leverage manufacturing domain knowledge to improve the accuracy and interpretability of CNNs in quality prediction. A novel CNN model, named Di-CNN, was developed that learns from both design-stage information (such as working condition and operational mode) and real-time sensor data, and adaptively weighs these data sources during model training. It exploits domain knowledge to guide model training, thus improving prediction accuracy and model interpretability. A case study on resistance spot welding, a popular lightweight metal-joining process for automotive manufacturing, compared the performance of (1) a Di-CNN with adaptive weights (the proposed model), (2) a Di-CNN without adaptive weights, and (3) a conventional CNN. The quality prediction results were measured with the mean squared error (MSE) over sixfold cross-validation. Model (1) achieved a mean MSE of 6.8866 and a median MSE of 6.1916, Model (2) achieved 13.6171 and 13.1343, and Model (3) achieved 27.2935 and 25.6117, demonstrating the superior performance of the proposed model.

47 OTHER INSTRUMENTATION↗
DOE OSTI2025

Online thermal profile prediction for large format additive manufacturing: A hybrid CNN-LSTM based approach

Large format additive manufacturing (LFAM) is an advanced 3D printing technique that efficiently fabricates large-scale components through a layer-by-layer extrusion and deposition process. Accurate surface layer temperature monitoring is essential to prevent manufacturing failures and ensure final product quality. Traditional physics-based offline approaches for simulating thermal behavior are often inefficient and complex, posing challenges on real-time, in-situ monitoring. Here, to address this, we propose a data-driven hybrid CNN-LSTM model to predict sequential thermal images of arbitrary length using real-time infrared thermal imaging. In this approach, a Convolutional Neural Networks (CNN) is trained offline to capture spatial features, reduce dimensional complexity, and enhance time efficiency, while a stacked Long Short-Term Memory (LSTM) is applied online to capture temporal information for improved prediction of future thermal behavior in subsequent printing layers. Model performance is evaluated using MSE, SSIM, and PSNR metrics and is benchmarked against stacked LSTM and convolutional LSTM models, demonstrating superior accuracy and applicability. Additionally, to mitigate noise from moving extruders and gantry backgrounds in thermal images, a fine-tuned semantic segmentation model is implemented offline to extract printing geometry, enabling precise temperature tracking along the tool path for further thermal analysis. The frameworks developed in this study significantly advance temperature monitoring, thermal analysis, and in-situ manufacturing control for LFAM, bridging the gap between theoretical modeling and practical application.

Geometry extraction↗
DOE OSTI2021

A Visual Designer of Layer‐wise Relevance Propagation Models

Abstract Layer‐wise Relevance Propagation (LRP) is an emerging and widely‐used method for interpreting the prediction results of convolutional neural networks (CNN). LRP developers often select and employ different relevance backpropagation rules and parameters, to compute relevance scores on input images. However, there exists no obvious solution to define a “best” LRP model. A satisfied model is highly reliant on pertinent images and designers' goals. We develop a visual model designer, named as VisLRPDesigner, to overcome the challenges in the design and use of LRP models. Various LRP rules are unified into an integrated framework with an intuitive workflow of parameter setup. VisLRPDesigner thus allows users to interactively configure and compare LRP models. It also facilitates relevance‐based visual analysis with two important functions: relevance‐based pixel flipping and neuron ablation. Several use cases illustrate the benefits of VisLRPDesigner. The usability and limitation of the visual designer is evaluated by LRP users.

Huang, Xinyi↗
DOE OSTI2023

Exploring Randomly Wired Neural Networks for Climate Model Emulation

Exploring the climate impacts of various anthropogenic emissions scenarios is key to making informed decisions for climate change mitigation and adaptation. State-of-the-art Earth system models can provide detailed insight into these impacts but have a large associated computational cost on a per-scenario basis. This large computational burden has driven recent interest in developing cheap machine learning models for the task of climate model emulation. In this paper, we explore the efficacy of randomly wired neural networks for this task. We describe how they can be constructed and compare them with their standard feedforward counterparts using the ClimateBench dataset. Specifically, we replace the serially connected dense layers in multilayer perceptrons, convolutional neural networks, and convolutional long short-term memory networks with randomly wired dense layers and assess the impact on model performance for models with 1 million and 10 million parameters. We find that models with less-complex architectures see the greatest performance improvement with the addition of random wiring (up to 30.4% for multilayer perceptrons). Furthermore, of 24 different model architecture, parameter count, and prediction task combinations, only one had a statistically significant performance deficit in randomly wired networks relative to their standard counterparts, with 14 cases showing statistically significant improvement. We also find no significant difference in prediction speed between networks with standard feedforward dense layers and those with randomly wired layers. These findings indicate that randomly wired neural networks may be suitable direct replacements for traditional dense layers in many standard models.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2023

Dilated causal convolutional neural networks for forecasting zone airflow to estimate short-term energy consumption

Here this paper investigates the use of dilated causal convolutional neural networks for fine- grained temporal forecasting of building zone states. Specifically, we build and evaluate models using a small set of exogenous features (e.g., external temperature) to autoregressively predict zone airflow setpoints every minute for a 24-hour prediction window. We carefully explore the trade-off between generality and specificity in these models, training and evaluating them based on zone, zone type, month, season, and combinations thereof. When evaluated for a commercial office building in Eastern Washington with 16 zones served by variable air volume air handling units, we find that the highest performance comes from a zone-specific, season-agnostic approach; with it, we obtain an R 2 of 0.704 (averaged over zones) and an average normalized root mean square error (nRMSE) of 0.111. In contrast, the most general model (trained across all zones and seasons) yields an R 2 of only 0.416 and a nRMSE of 0.168, while a baseline zone-specific reduced order model obtains 0.443 R 2 and 0.159 nRMSE. We also report on factors affecting airflow forecasting performance, on the ability of models trained on a specific zone to generalize to other zones, and on the capability of those models trained on a specific month to generalize to other months.

32 ENERGY CONSERVATION, CONSUMPTION, AND UTILIZATI↗
DOE OSTI2020

Development of a Convolutional Neural Network Classifier for Data Starved Spectra - 20199

The Institute for Clean Energy Technology (ICET) at Mississippi State University is exploring the utility of machine learning in augmenting its mobile radiation surveying platforms, which are currently being developed as means to survey depleted uranium contaminated areas in support of remediation and decommissioning efforts. Mobile survey platforms provide a means to efficiently scan large areas of interest while reducing human exposure to radiation and other hazards. The survey platforms can also be used for scanning for any gamma emitting isotope in addition to depleted uranium. The spectral data that the platforms collect may be data starved with relatively low counts and poorly defined spectral features depending on the speed of the platforms and scintillation detector selection. Such data-starved spectra are difficult to use for isotope identification, requiring advanced knowledge of the possible radionuclides that could be present and environmental factors that could attenuate signals or introduce background noise. These factors in combination with the volume of survey data increases the time it takes to perform analysis of survey data when the source type is unknown. There are a number of algorithms in the field of machine learning that can be used to classify data that would be challenging and time-consuming for a human to identify. Supervised machine learning algorithms train models based on extensive amounts of human-labeled training data. Once sufficiently trained, these models can be used to quickly make high-fidelity predictions on new data. Convolutional neural networks are machine learning algorithms that excel in learning representations of 'shapes'. They do this by taking numerical input data and convolving them with spatial feature detectors referred to as filters. These filters are incrementally adjusted to reduce the prediction error on the data during the backpropagation step of training. Discussed in this paper is the development of a convolutional neural network classifier (CNNC) that can utilize spectral survey data for source discrimination and isotope identification. Bench-top laboratory experiments data using LaBr{sub 3}(Ce) scintillation detectors were used to train and evaluate the performance of the developed CNNC. The CNNC is capable of discriminating a variety of gamma emitting source types, differentiating different forms of uranium (depleted vs. natural), and estimating the amount of uranium for a known geometry. The discussed CNNC may be useful in scenarios where survey systems are deployed in situations where hazardous radioactive material maybe present, but the type is unknown. When used in remediation applications the CNNC can be used to screen-out false positives, helping reduce remediation costs. (authors)

07 ISOTOPE AND RADIATION SOURCES↗