Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “deep learning method”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 1,081 records · Page 60
All sourcesNASA NTRSDOE OSTI
DOE OSTI2021

Class imbalance in out-of-distribution datasets: Improving the robustness of the TextCNN for the classification of rare cancer types

In the last decade, the widespread adoption of electronic health record documentation has created huge opportunities for information mining. Natural language processing (NLP) techniques using machine and deep learning are becoming increasingly widespread for information extraction tasks from unstructured clinical notes. Disparities in performance when deploying machine learning models in the real world have recently received considerable attention. In the clinical NLP domain, the robustness of convolutional neural networks (CNNs) for classifying cancer pathology reports under natural distribution shifts remains understudied. In this research, we aim to quantify and improve the performance of the CNN for text classification on out-of-distribution (OOD) datasets resulting from the natural evolution of clinical text in pathology reports. We identified class imbalance due to different prevalence of cancer types as one of the sources of performance drop and analyzed the impact of previous methods for addressing class imbalance when deploying models in real-world domains. Our results show that our novel class-specialized ensemble technique outperforms other methods for the classification of rare cancer types in terms of macro F1 scores. We also found that traditional ensemble methods perform better in top classes, leading to higher micro F1 scores. Based on our findings, we formulate a series of recommendations for other ML practitioners on how to build robust models with extremely imbalanced datasets in biomedical NLP applications.

60 APPLIED LIFE SCIENCES↗
DOE OSTI2022

High-dimensional anomaly detection with radiative return in $\mathcal{e}$⁺$\mathcal{e}$⁻ collisions

Experiments at a future $\mathcal{e}$⁺$\mathcal{e}$⁻ collider will be able to search for new particles with masses below the nominal centre-of-mass energy by analyzing collisions with initial-state radiation (radiative return). We show that machine learning methods that use imperfect or missing training labels can achieve sensitivity to generic new particle production in radiative return events. In addition to presenting an application of the classification without labels (CWoLa) search method in $\mathcal{e}$⁺$\mathcal{e}$⁻ collisions, our study combines weak supervision with variable-dimensional information by deploying a deep sets neural network architecture. We have also investigated some of the experimental aspects of anomaly detection in radiative return events and discuss these in the context of future detector design.

72 PHYSICS OF ELEMENTARY PARTICLES AND FIELDS↗
DOE OSTI2021

DeepGRN: prediction of transcription factor binding site across cell-types using attention-based deep neural networks

Abstract Background Due to the complexity of the biological systems, the prediction of the potential DNA binding sites for transcription factors remains a difficult problem in computational biology. Genomic DNA sequences and experimental results from parallel sequencing provide available information about the affinity and accessibility of genome and are commonly used features in binding sites prediction. The attention mechanism in deep learning has shown its capability to learn long-range dependencies from sequential data, such as sentences and voices. Until now, no study has applied this approach in binding site inference from massively parallel sequencing data. The successful applications of attention mechanism in similar input contexts motivate us to build and test new methods that can accurately determine the binding sites of transcription factors. Results In this study, we propose a novel tool (named DeepGRN) for transcription factors binding site prediction based on the combination of two components: single attention module and pairwise attention module. The performance of our methods is evaluated on the ENCODE-DREAM in vivo Transcription Factor Binding Site Prediction Challenge datasets. The results show that DeepGRN achieves higher unified scores in 6 of 13 targets than any of the top four methods in the DREAM challenge. We also demonstrate that the attention weights learned by the model are correlated with potential informative inputs, such as DNase-Seq coverage and motifs, which provide possible explanations for the predictive improvements in DeepGRN. Conclusions DeepGRN can automatically and effectively predict transcription factor binding sites from DNA sequences and DNase-Seq coverage. Furthermore, the visualization techniques we developed for the attention modules help to interpret how critical patterns from different types of input features are recognized by our model.

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗
DOE OSTI2024

Unconventional Wells Interference: Supervised Machine Learning for Detecting Fracture Hits

The primary objective of the study was development of a machine learning (ML)-based workflow for fracture hit (“frac hit”) detection and monitoring using shale oil-field data such as drilling surveys, production history (oil and produced water), pressure, and fracking start time and duration records. The ML method takes advantage of long short-term memory (LSTM) and multilayer perceptron (MLP) neural networks to identify the frac hits due to hydraulic communication between the fracking child well(s) and the producing parent well(s) within the same pad (intra-pad interaction) and/or on different pads (inter-pad interaction). It utilizes time series of pressure and production data from within a pad and from adjacent pads. The workflow can capture time variable features of frac hits when the model architecture is deep and wide enough, with enough trainable parameters for deep learning and feature extraction, as demonstrated in this paper by using training and testing subsets of the field data from selected neighboring pads with over a couple of hundred wells. The study was focused on frac-hit interaction among paired wells and demonstrated that the ML model, once trained, can predict the frac-hit probability.

58 GEOSCIENCES↗
DOE OSTI2022

Structure-Informed Graph Learning of Networked Dependencies for Online Prediction of Power System Transient Dynamics

Online transient analysis plays an increasingly important role in dynamic power grids as the renewable generation continues growing. Traditional numerical methods for transient analysis not only are computationally intensive but also require precise contingency information as input, and therefore, are not suitable for online applications. Existing online transient assessment studies focus on the determination of post-contingency system stability or stability margin. Here, this paper develops a novel graph-learning framework, Deep-learning Neural Representation or DNR, for online prediction, of the time-series trajectories of the system states using initial system responses that can be measured by phasor measurement units (PMUs). The proposed DNR framework consists of two sequential modules: a Network Constructor that captures network dependencies among generators, and a Dynamics Predictor that predicts the system trajectories. The key to improved prediction performance is the introduction of the spatio-temporal message-passing operations into graph neural networks with structural knowledge. Its effectiveness and scalability are validated through comparative studies, demonstrating the prediction performance under different contingency scenarios for systems of different sizes. This framework provides a solution to online predicting post-fault system dynamics based on real-time PMU measurements. Additionally, it can also be applied to facilitate the offline transient simulation without simulating the entire trajectories.

24 POWER TRANSMISSION AND DISTRIBUTION↗
DOE OSTI2022

To image, or not to image: class-specific diffractive cameras with all-optical erasure of undesired objects

Abstract Privacy protection is a growing concern in the digital era, with machine vision techniques widely used throughout public and private settings. Existing methods address this growing problem by, e.g., encrypting camera images or obscuring/blurring the imaged information through digital algorithms. Here, we demonstrate a camera design that performs class-specific imaging of target objects with instantaneous all-optical erasure of other classes of objects. This diffractive camera consists of transmissive surfaces structured using deep learning to perform selective imaging of target classes of objects positioned at its input field-of-view. After their fabrication, the thin diffractive layers collectively perform optical mode filtering to accurately form images of the objects that belong to a target data class or group of classes, while instantaneously erasing objects of the other data classes at the output field-of-view. Using the same framework, we also demonstrate the design of class-specific permutation and class-specific linear transformation cameras, where the objects of a target data class are pixel-wise permuted or linearly transformed following an arbitrarily selected transformation matrix for all-optical class-specific encryption, while the other classes of objects are irreversibly erased from the output image. The success of class-specific diffractive cameras was experimentally demonstrated using terahertz (THz) waves and 3D-printed diffractive layers that selectively imaged only one class of the MNIST handwritten digit dataset, all-optically erasing the other handwritten digits. This diffractive camera design can be scaled to different parts of the electromagnetic spectrum, including, e.g., the visible and infrared wavelengths, to provide transformative opportunities for privacy-preserving digital cameras and task-specific data-efficient imaging.

47 OTHER INSTRUMENTATION↗
DOE OSTI2022

A Decision-Making Machine Learning Approach in Hermite Spectral Approximations of Partial Differential Equations

The accuracy and effectiveness of Hermite spectral methods for the numerical discretization of partial differential equations on unbounded domains are strongly affected by the amplitude of the Gaussian weight function employed to describe the approximation space. This is particularly true if the problem is under-resolved, i.e., there are no enough degrees of freedom. The issue becomes even more crucial when the equation under study is time-dependent, forcing in this way the choice of Hermite functions where the corresponding weight depends on time. In order to adapt dynamically the approximation space, it is here proposed an automatic decision-making process that relies on machine learning techniques, such as deep neural networks and support vector machines. The algorithm is numerically tested with success on a simple 1D problem, but the main goal is its exportability in the context of more serious applications. Here we also show at the end an application in the framework of plasma physics.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2022

Rethinking the ill-posedness of the spectral function reconstruction — Why is it fundamentally hard and how Artificial Neural Networks can help

Reconstructing hadron spectral functions through Euclidean correlation functions are of the important missions in lattice QCD calculations. However, in a Källen–Lehmann (KL) spectral representation, the reconstruction is observed to be ill-posed in practice. It is usually ascribed to the fewer observation points compared to the number of points in the spectral function. In this paper, by solving the eigenvalue problem of continuous KL convolution, we show analytically that the ill-posedness of the inversion is fundamental and it exists even for continuous correlation functions. We discussed how to introduce regulators to alleviate the predicament, in which include the Artificial Neural Networks (ANNs) representations recently proposed by the Authors in another study. The uniqueness of solutions using ANNs representations is manifested analytically and validated numerically. Reconstructed spectral functions using different regularization schemes are also demonstrated, together with their eigen-mode decomposition. We observe that components with large eigenvalues can be reliably reconstructed by all methods, whereas those with low eigenvalues need to be constrained by regulators.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2022

Synthesis of CdZnTeSe single crystals for room temperature radiation detector fabrication: mitigation of hole trapping effects using a convolutional neural network

In this article, we report the growth of Cd 0.9 Zn 0.1 Te 0.97 Se 0.03 (CZTS) wide bandgap semiconductor single crystals for room temperature gamma-ray detection using a modified vertical Bridgman method. Charge transport properties measured in the radiation detectors, fabricated from the grown CZTS crystals, indicated signs of hole trapping. Hole traps inhibit high-resolution radiation detection especially for energetic gamma rays. Machine learning (ML) applications are gaining tremendous mpetus in improving device and sensor performance by compensating for limi tations arising from such intrinsic material properties. In this article, we describe a deep convolutional neural network (CNN) that has demonstrated remarkable efficiency in identifying the energy of a gamma photon detected by a CZTS detector. The CNN has been trained using simulated data that resemble output pulses from actual CZTS detectors when exposed to 662-keV gamma photons. The device properties required for the simulation have been derived from radiation detection measurements on a real Cd 0.9 Zn 0.1 Te 0.97 Se 0.03 detector fabricated in our laboratory. The CNN has been trained with detector pulses arising through photoelectric (PE) and Compton scattering (CS) separately. The percentage error in predicting the detected energies, within an extremely small duration of 0.28 ms, was found to be lower than 0.1% for gamma energies above 50 keV and for training datasets con taining PE and CS events separately. The CNN was also validated for a mixed PE and CS dataset to obtain a prediction error of 1%. Additionally, the effect of detector resolution on the efficiency of the CNN was also explored.

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗
DOE OSTI2023

Parameters, Properties, and Process: Conditional Neural Generation of Realistic SEM Imagery Toward ML-Assisted Advanced Manufacturing

Abstract The research and development cycle of advanced manufacturing processes traditionally requires a large investment of time and resources. Experiments can be expensive and are hence conducted on relatively small scales. This poses problems for typically data-hungry machine learning tools which could otherwise expedite the development cycle. We build upon prior work by applying conditional generative adversarial networks (GANs) to scanning electron microscope (SEM) imagery from an emerging advanced manufacturing process, shear-assisted processing and extrusion (ShAPE). We generate realistic images conditioned on temper and either experimental parameters or material properties. In doing so, we are able to integrate machine learning into the development cycle, by allowing a user to immediately visualize the microstructure that would arise from particular process parameters or properties. This work forms a technical backbone for a fundamentally new approach for understanding manufacturing processes in the absence of first-principle models. By characterizing microstructure from a topological perspective, we are able to evaluate our models’ ability to capture the breadth and diversity of experimental scanning electron microscope (SEM) samples. Our method is successful in capturing the visual and general microstructural features arising from the considered process, with analysis highlighting directions to further improve the topological realism of our synthetic imagery.

36 MATERIALS SCIENCE↗
DOE OSTI2020

Improving the accessibility and transferability of machine learning algorithms for identification of animals in camera trap images: MLWIC2

Motion-activated wildlife cameras (or “camera traps”) are frequently used to remotely and noninvasively observe animals. The vast number of images collected from camera trap projects has prompted some biologists to employ machine learning algorithms to automatically recognize species in these images, or at least filter-out images that do not contain animals. These approaches are often limited by model transferability, as a model trained to recognize species from one location might not work as well for the same species in different locations. Furthermore, these methods often require advanced computational skills, making them inaccessible to many biologists. We used 3 million camera trap images from 18 studies in 10 states across the United States of America to train two deep neural networks, one that recognizes 58 species, the “species model,” and one that determines if an image is empty or if it contains an animal, the “empty-animal model.” Our species model and empty-animal model had accuracies of 96.8% and 97.3%, respectively. Furthermore, the models performed well on some out-of-sample datasets, as the species model had 91% accuracy on species from Canada (accuracy range 36%–91% across all out-of-sample datasets) and the empty-animal model achieved an accuracy of 91%–94% on out-of-sample datasets from different continents. Our software addresses some of the limitations of using machine learning to classify images from camera traps. By including many species from several locations, our species model is potentially applicable to many camera trap studies in North America. We also found that our empty-animal model can facilitate removal of images without animals globally. We provide the trained models in an R package (MLWIC2: Machine Learning for Wildlife Image Classification in R), which contains Shiny Applications that allow scientists with minimal programming experience to use trained models and train new models in six neural network architectures with varying depths.

59 BASIC BIOLOGICAL SCIENCES↗
DOE OSTI2022

B-DeepONet: An enhanced Bayesian DeepONet for solving noisy parametric PDEs using accelerated replica exchange SGLD

Here, the Deep Operator Network (DeepONet) is a neural network architecture used to approximate operators, including the solution operator of parametric PDEs. DeepONets have shown remarkable approximation ability. However, the performance of DeepONets deteriorates when the training data is polluted with noise, a scenario that occurs in practice. To handle noisy data, we propose a Bayesian DeepONet based on replica exchange Langevin diffusion (reLD). Replica exchange uses two particles. The first particle trains a DeepONet to exploit the loss landscape and make predictions. The other particle trains a different DeepONet to explore the loss landscape and escape local minima via swapping. Compared to DeepONets trained with state-of-the-art gradient-based algorithms (e.g., Adam), the proposed Bayesian DeepONet greatly improves the training convergence for noisy scenarios and accurately estimates the uncertainty. To further reduce the high computational cost of the reLD training of DeepONets, we propose (1) an accelerated training framework that exploits the DeepONet's architecture to reduce its computational cost up to 25% without compromising performance and (2) a transfer learning strategy that accelerates training DeepONets for PDEs with different parameter values. Finally, we illustrate the effectiveness of the proposed Bayesian DeepONet using four parametric PDE problems.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2026

Visualizing Millisecond Atomic Dynamics of Nanocrystals in Liquid

Atomic structures of nanomaterials are inherently dynamic and continuously reshaped through interactions with chemical species and external stimuli. Such dynamics are further amplified as the size and dimensionality of nanomaterials decrease. Despite advances in analytical methods, it remains challenging to capture the structural dynamics of nanomaterials in reactive environments with both atomic spatial resolution and commensurate temporal resolution. Here, in this study, we directly visualize atomic-scale dynamics of gold (Au) nanocrystals in reactive liquid environments with millisecond-speed liquid-cell electron microscopy (EM) and deep-learning denoising. We uncover reversible fluctuations in the local crystallinity of Au nanocrystals dependent on the surrounding chemical environment. These transient fluctuations, driven by interactions at nanocrystal–liquid interfaces, critically influence the dissolution kinetics and grain boundary relaxation. By overcoming the spatiotemporal limitations in conventional liquid-cell EM, our findings provide insights into how transient nanoscale structures dictate the stability and reactivity of nanomaterials.

Kang, Sungsu [University of Chicago, IL (United St↗
DOE OSTI2025

Near-zero photon bioimaging by fusing deep learning and ultralow-light microscopy

Enhancing the reliability and reproducibility of optical microscopy by reducing specimen irradiance continues to be an important biotechnology target. As irradiance levels are reduced, however, the particle nature of light is heightened, giving rise to Poisson noise, or photon sparsity that restricts only a few (0.5%) image pixels to comprise a photon. Photon sparsity can be addressed by collecting approximately 200 photons per pixel; this, however, requires long acquisitions and, as such, suboptimal imaging rates. Here, we introduce near-zero photon bioimaging, a method that operates at kHz rates and 10,000-fold lower irradiance than standard microscopy. To achieve this level of performance, we uniquely combined a judiciously designed epifluorescence microscope enabling ultralow background levels and AI that learns to reconstruct biological images from as low as 0.01 photons per pixel. We demonstrate that near-zero photon bioimaging captures the structure of multicellular and subcellular features with high fidelity, including features represented by nearly zero photons. Beyond optical microscopy, the near-zero photon bioimaging paradigm can be applied in remote sensing, covert applications, and biomedical imaging that utilize damaging or quantum light.

AI↗
DOE OSTI2023

Group-equivariant autoencoder for identifying spontaneously broken symmetries

We introduce the group-equivariant autoencoder (GE autoencoder), a deep neural network (DNN) method that locates phase boundaries by determining which symmetries of the Hamiltonian have spontaneously broken at each temperature. We use group theory to deduce which symmetries of the system remain intact in all phases, and then use this information to constrain the parameters of the GE autoencoder such that the encoder learns an order parameter invariant to these “never-broken” symmetries. This procedure produces a dramatic reduction in the number of free parameters such that the GE-autoencoder size is independent of the system size. We include symmetry regularization terms in the loss function of the GE autoencoder so that the learned order parameter is also equivariant to the remaining symmetries of the system. By examining the group representation by which the learned order parameter transforms, we are then able to extract information about the associated spontaneous symmetry breaking. We test the GE autoencoder on the 2D classical ferromagnetic and antiferromagnetic Ising models, finding that the GE autoencoder (1) accurately determines which symmetries have spontaneously broken at each temperature; (2) estimates the critical temperature in the thermodynamic limit with greater accuracy, robustness, and time efficiency than a symmetry-agnostic baseline autoencoder; and (3) detects the presence of an external symmetry-breaking magnetic field with greater sensitivity than the baseline method. Lastly, we describe various key implementation details, including a quadratic-programming-based method for extracting the critical temperature estimate from trained autoencoders and calculations of the DNN initialization and learning rate settings required for fair model comparisons.

42 ENGINEERING↗
DOE OSTI2023

Embedding hard physical constraints in neural network coarse-graining of three-dimensional turbulence

In recent years, deep learning approaches have shown much promise in modeling complex systems in the physical sciences. A major challenge in deep learning of partial differential equations is enforcing physical constraints and boundary conditions. In this work, we propose a general framework to directly embed the notion of an incompressible fluid into convolutional neural networks, and apply this to coarse-graining of turbulent flow. These physics-embedded neural networks leverage interpretable strategies from numerical methods and computational fluid dynamics to enforce physical laws and boundary conditions by taking advantage the mathematical properties of the underlying equations. Here, we demonstrate results on three-dimensional fully developed turbulence, showing that this technique drastically improves local conservation of mass, without sacrificing performance according to several other metrics characterizing the fluid flow.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2025

Selection Algorithm Improvement for MicroBooNE

Data selection is an extremely important part of data analysis for any experiment. Finding a physics result is often the result of sifting through a massive amount of data, keeping data that we believe to be signal and throwing out data we do not. This process is called data selection. Creating a selection algorithm is an intensive process that must balance keeping enough data to have statistics and maximizing the signal purity of that data. In this study, we used three different reconstruction tools, Pandora, WireCell, and LANTERN, for the MicroBooNE experiment in conjunction to improve the selection algorithm for analysis. For the case of this study, we look into the charged current N proton 0 pions (CCNp0$\pi$) interaction channel. This is the dominant channel for the Short Baseline Neutrino (SBN) program and is expected to be a large contributor to the Deep Underground Neutrino Experiment (DUNE). We first investigated each of the three tools to find out more about their strengths and weaknesses as reconstructions. We then put together a direct comparison of the three methods to find which method or combination of methods would return the best result for us. While the study is ongoing, we have learned a lot about data selection for the experiment and the differences between the reconstruction tools.

Dillon, Brayden [Michigan State U.]↗
DOE OSTI2021

On the Stochastic Stability of Deep Markov Models

Deep Markov models (DMM) are generative models which are scalable and expressive generalization of Markov models for representation, learning, and inference problems. DMMs using deep neural networks to parametrize the transition of Markov probability distributions have recently been shown to provide more expressiveness in modeling sequential data and dynamical system responses. However, the fundamental stochastic stability guarantees of such models have not been thoroughly investigated. In this paper, we present a rigorous analytical method to prove the necessary and sufficient conditions of DMM's stochastic stability. This task is achieved by spectral analysis of the efficiently computed Jacobians of probabilistic maps modeled by deep neural networks. We make theoretical connections between the eigenvalues of neural network's weights and the different activation function types used on the stability and overall dynamic behavior of DMMs with Gaussian distributions. We empirically substantiate our theoretical results on stochastic stability and eigenvalue spectra via several numerical experiments. Formal stability guarantees of DMMs can substantially improve their robustness and trustworthiness, necessary for reliable use in safety-critical real-world applications.

Drgona, Jan↗