Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “Deep Neural Network”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 1,351 records · Page 75
All sourcesNASA NTRSDOE OSTI
DOE OSTI2024

A post-merger enhancement only in star-forming Type 2 Seyfert galaxies: the deep learning view

ABSTRACT Supermassive black holes require a reservoir of cold gas at the centre of their host galaxy in order to accrete and shine as active galactic nuclei (AGN). Major mergers have the ability to drive gas rapidly inwards, but observations trying to link mergers with AGN have found mixed results due to the difficulty of consistently identifying galaxy mergers in surveys. This study applies deep learning to this problem, using convolutional neural networks trained to identify simulated post-merger galaxies from survey-realistic imaging. This provides a fast and repeatable alternative to human visual inspection. Using this tool, we examine a sample of ∼8500 Seyfert 2 galaxies ($L[\mathrm{O\, {\small III}}] \sim 10^{38.5 - 42}$ erg s−1) at z < 0.3 in the Sloan Digital Sky Survey and find a merger fraction of $2.19_{-0.17}^{+0.21}$ per cent compared with inactive control galaxies, in which we find a merger fraction of $2.96_{-0.20}^{+0.26}$ per cent, indicating an overall lack of mergers among AGN hosts compared with controls. However, matching the controls to the AGN hosts in stellar mass and star formation rate reveals that AGN hosts in the star-forming blue cloud exhibit a ∼2 × merger enhancement over controls, while those in the quiescent red sequence have significantly lower relative merger fractions, leading to the observed overall deficit due to the differing M*–SFR distributions. We conclude that while mergers are not the dominant trigger of all low-luminosity, obscured AGN activity in the nearby Universe, they are more important to AGN fuelling in galaxies with higher cold gas mass fractions as traced through star formation.

Avirett-Mackenzie, M. S.↗
DOE OSTI2021

Extracting the Galactic Center excess’ source-count distribution with neural nets

The two leading hypotheses for the Galactic Center excess (GCE) in the Fermi data are an unresolved population of faint millisecond pulsars (MSPs) and dark-matter (DM) annihilation. The dichotomy between these explanations is typically reflected by modeling them as two separate emission components. However, point sources (PSs) such as MSPs become statistically degenerate with smooth Poisson emission in the ultrafaint limit (formally where each source is expected to contribute much less than one photon on average), leading to an ambiguity that can render questions such as whether the emission is PS-like or Poissonian in nature ill defined. We present a conceptually new approach that describes the PS and Poisson emission in a unified manner and only afterwards derives constraints on the Poissonian component from the so obtained results. For the implementation of this approach, we leverage deep learning techniques, centered around a neural network-based method for histogram regression that expresses uncertainties in terms of quantiles. We demonstrate that our method is robust against a number of systematics that have plagued previous approaches, in particular DM/PS misattribution. In the Fermi data, we find a faint GCE described by a median source-count distribution (SCD) peaked at a flux of ~ 4 x 10 -11 counts cm -2 s -1 (corresponding to ~ 3-4 expected counts per PS), which would require N ~ $\mathscr{O}$(10 4 ) sources to explain the entire excess (median value N = 29,300 across the sky). Although faint, this SCD allows us to derive the constraint η ρ ≤ 66% for the Poissonian fraction of the GCE flux η ρ at 95% confidence, suggesting that a substantial amount of the GCE flux is due to PSs.

71 CLASSICAL AND QUANTUM MECHANICS, GENERAL PHYSIC↗
DOE OSTI2024

NuclPred v1

This tool takes a genome assembly as input and predicts per-site nucleosome occupancy as output. Trained on physical maps of nucleosome binding preferences across the fungal kingdom, NuclPred can be applied broadly across fungi (and other eukaryotes). This breadth, combined with its accuracy, means it could have both basic and applied biological implications, for example in understanding eukaryotic gene regulation and genetic engineering. Almost universally across eukaryotes, nucleosomes - each wrapping ~150 base pairs of DNA - serve to package DNA inside the nucleus, with major consequences on DNA access, gene activity and DNA integration. NuclPred was generated using a supervised deep learning approach combining convolutional and recurrent neural networks to take DNA features (nucleotides, GC content and structural information) as input, then use that information to predict the physical attractiveness DNA sequences might have for forming nucleosomes. With this information at hand, researchers can design more efficient CRISPR constructs, explore the interplay between DNA signatures and other regulators impact nucleosome locations, predict expression patterns, etc. This tool will be published as part of a manuscript currently under revision at iScience (draft attached).

Mondo, Stephen↗
DOE OSTI2025

GenomeFace v1.0

GenomeFace is meta-genome binning software. Metagenomic binning, the process of grouping DNA sequences into taxonomic units, is critical for understanding the functions, interactions, and evolutionary dynamics of microbial communities. We propose a deep learning approach to binning using two neural networks, one based on composition and another on environmental abundance, dynamically weighting the contribution of each based on characteristics of the input data. Trained on over 43,000 prokaryotic genomes, our network for composition-based binning is inspired by metric learning techniques used for facial recognition. Using a task-specific, multi-GPU accelerated algorithm to cluster the embeddings produced by our network, our binner leverages marker genes observed to be universally present in nearly all taxa to grade and select optimal clusters of sequences from a hierarchy of candidates. We evaluate our approach on four simulated datasets with known ground truth. Our linear time integration of marker genes recovers more near complete genomes than state of the art but computationally infeasible solutions using them, while being over an order of magnitude faster. Finally, we demonstrate the scalability and acuity of our approach by testing it on three of the largest metagenome assemblies ever performed. Compared to other binners, we produced 47%-183% more near complete genomes. From these datasets, we find over the genomes of over 3000 new candidate species which have never been previously cataloged, representing a potential 4% expansion of the known bacterial tree of life.

Lettich, Richard [Lawrence Berkeley National Labor↗
DOE OSTI2024

UNNT: A novel Utility for comparing Neural Net and Tree-based models

The use of deep learning (DL) is steadily gaining traction in scientific challenges such as cancer research. Advances in enhanced data generation, machine learning algorithms, and compute infrastructure have led to an acceleration in the use of deep learning in various domains of cancer research such as drug response problems. In our study, we explored tree-based models to improve the accuracy of a single drug response model and demonstrate that tree-based models such as XGBoost (eXtreme Gradient Boosting) have advantages over deep learning models, such as a convolutional neural network (CNN), for single drug response problems. However, comparing models is not a trivial task. To make training and comparing CNNs and XGBoost more accessible to users, we developed an open-source library called UNNT (A novel Utility for comparing Neural Net and Tree-based models). The case studies, in this manuscript, focus on cancer drug response datasets however the application can be used on datasets from other domains, such as chemistry.

59 BASIC BIOLOGICAL SCIENCES↗
DOE OSTI2020

Vision-Guided Automation and Assistance

Robotic automation and assistance is an excellent way to increase efficiency and safety when working in a potentially hazardous workspace. However, many traditional automation approaches rely on the precise placement of items to be manipulated. The scope of such automated tasks is also limited to the manipulation of one type of object since there is no way to distinguish between different objects. The addition of a vision pipeline, where robot vision is used to localize and recognize objects, can allow the robot to perform more complex tasks in a variable workspace. In my vision pipeline, a depth camera is used to generate a 3D point cloud of the workspace. Objects to be manipulated are isolated in the point cloud by filtering out shared surfaces and using Euclidean cluster extraction to group clusters of points. The locations of the objects are then estimated from the point cloud of the isolated objects. The localization of objects with robot vision allows for variable workspaces, reducing the overhead associated with requiring all objects to be precisely placed in the workspace. For object recognition, deep learning is used to train a neural network that classifies objects from the raw point clouds. More complex tasks that require multiple objects to be manipulated differently can be achieved through the vision pipeline. Through its ability to visualize the workspace, robot vision greatly improves efficiency and safety of automated and assistive tasks.

42 ENGINEERING↗
DOE OSTI2022

Graph Neural Networks for Charged Particle Tracking on FPGAs

The determination of charged particle trajectories in collisions at the CERN Large Hadron Collider (LHC) is an important but challenging problem, especially in the high interaction density conditions expected during the future high-luminosity phase of the LHC (HL-LHC). Graph neural networks (GNNs) are a type of geometric deep learning algorithm that has successfully been applied to this task by embedding tracker data as a graph—nodes represent hits, while edges represent possible track segments—and classifying the edges as true or fake track segments. However, their study in hardware- or software-based trigger applications has been limited due to their large computational cost. In this paper, we introduce an automated translation workflow, integrated into a broader tool called hls4ml , for converting GNNs into firmware for field-programmable gate arrays (FPGAs). We use this translation tool to implement GNNs for charged particle tracking, trained using the TrackML challenge dataset, on FPGAs with designs targeting different graph sizes, task complexites, and latency/throughput requirements. This work could enable the inclusion of charged particle tracking GNNs at the trigger level for HL-LHC experiments.

Elabd, Abdelrahman↗
DOE OSTI2026

Machine Learning-Enabled Wearable Piezoelectric Acoustic Sensor for Real-Time Breast Abnormality Detection

In contemporary society, breast health has become a significant public health concern, particularly among women. According to statistics from the World Health Organization, both the incidence and mortality rates of breast tumors have steadily increased in recent years. Therefore, effective early-stage screening and postoperative monitoring are essential for maintaining breast health. However, conventional clinical diagnostic modalities are typically bulky, operationally complex, and unsuitable for continuous real-time monitoring, which limits their use in portable and everyday health management applications. To address these limitations, this study proposes a machine learning-integrated wearable piezoelectric sensing platform as an auxiliary tool for breast health assessment. The device consists of a PDMS matching layer embedded with flexible silver nanowires, a P(VDF-TrFE) piezoelectric layer, and a multi-channel low-noise signal acquisition circuit. It is capable of acquiring acoustic echo signals from tissue-mimicking environments and automatically evaluating signal validity using a convolutional neural network (CNN). By integrating piezoelectric sensing with deep learning-based signal analysis, the proposed system achieves a signal-to-noise ratio exceeding 70 dB and a real-time classification accuracy above 96% under controlled conditions. These results demonstrate that the platform provides a compact, portable, and intelligent approach for wearable sensing of mechanical heterogeneity and highlight its potential for future development in continuous biomedical monitoring technologies.

He, Shuaitong↗
DOE OSTI2022

Photometric redshift estimation with convolutional neural networks and galaxy images: Case study of resolving biases in data-driven methods

Deep-learning models have been increasingly exploited in astrophysical studies, but these data-driven algorithms are prone to producing biased outputs that are detrimental for subsequent analyses. In this work, we investigate two main forms of biases: class-dependent residuals, and mode collapse. We do this in a case study, in which we estimate photometric redshift as a classification problem using convolutional neural networks (CNNs) trained with galaxy images and associated spectroscopic redshifts. We focus on point estimates and propose a set of consecutive steps for resolving the two biases based on CNN models, involving representation learning with multichannel outputs, balancing the training data, and leveraging soft labels. The residuals can be viewed as a function of spectroscopic redshift or photometric redshift, and the biases with respect to these two definitions are incompatible and should be treated individually. We suggest that a prerequisite for resolving biases in photometric space is resolving biases in spectroscopic space. Experiments show that our methods can better control biases than benchmark methods, and they are robust in various implementing and training conditions with high-quality data. Our methods hold promises for future cosmological surveys that require a good constraint of biases, and they may be applied to regression problems and other studies that make use of data-driven models. Nonetheless, the bias-variance tradeoff and the requirement of sufficient statistics suggest that we need better methods and optimized data usage strategies.

79 ASTRONOMY AND ASTROPHYSICS↗
DOE OSTI2023

Ultra-low latency recurrent neural network inference on FPGAs for physics applications with hls4ml

Abstract Recurrent neural networks have been shown to be effective architectures for many tasks in high energy physics, and thus have been widely adopted. Their use in low-latency environments has, however, been limited as a result of the difficulties of implementing recurrent architectures on field-programmable gate arrays (FPGAs). In this paper we present an implementation of two types of recurrent neural network layers—long short-term memory and gated recurrent unit—within the hls4ml framework. We demonstrate that our implementation is capable of producing effective designs for both small and large models, and can be customized to meet specific design requirements for inference latencies and FPGA resources. We show the performance and synthesized designs for multiple neural networks, many of which are trained specifically for jet identification tasks at the CERN Large Hadron Collider.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2022

BUTTER-E - Energy Consumption Data for the BUTTER Empirical Deep Learning Dataset

The BUTTER-E - Energy Consumption Data for the BUTTER Empirical Deep Learning Dataset adds node-level energy consumption data from watt-meters to the primary sweep of the BUTTER - Empirical Deep Learning Dataset. This dataset contains energy consumption and performance data from 63,527 individual experimental runs spanning 30,582 distinct configurations: 13 datasets, 20 sizes (number of trainable parameters), 8 network "shapes", and 14 depths on both CPU and GPU hardware collected using node-level watt-meters. This dataset reveals the complex relationship between dataset size, network structure, and energy use, and highlights the impact of cache effects. BUTTER-E is intended to be joined with the BUTTER dataset (see "BUTTER - Empirical Deep Learning Dataset on OEDI" resource below) which characterizes the performance of 483k distinct fully connected neural networks but does not include energy measurements.

Array↗
DOE OSTI2024

Physics-Guided, Physics-Informed, and Physics-Encoded Neural Networks and Operators in Scientific Computing: Fluid and Solid Mechanics

Abstract Advancements in computing power have recently made it possible to utilize machine learning and deep learning to push scientific computing forward in a range of disciplines, such as fluid mechanics, solid mechanics, materials science, etc. The incorporation of neural networks is particularly crucial in this hybridization process. Due to their intrinsic architecture, conventional neural networks cannot be successfully trained and scoped when data are sparse, which is the case in many scientific and engineering domains. Nonetheless, neural networks provide a solid foundation to respect physics-driven or knowledge-based constraints during training. Generally speaking, there are three distinct neural network frameworks to enforce the underlying physics: (i) physics-guided neural networks (PgNNs), (ii) physics-informed neural networks (PiNNs), and (iii) physics-encoded neural networks (PeNNs). These methods provide distinct advantages for accelerating the numerical modeling of complex multiscale multiphysics phenomena. In addition, the recent developments in neural operators (NOs) add another dimension to these new simulation paradigms, especially when the real-time prediction of complex multiphysics systems is required. All these models also come with their own unique drawbacks and limitations that call for further fundamental research. This study aims to present a review of the four neural network frameworks (i.e., PgNNs, PiNNs, PeNNs, and NOs) used in scientific computing research. The state-of-the-art architectures and their applications are reviewed, limitations are discussed, and future research opportunities are presented in terms of improving algorithms, considering causalities, expanding applications, and coupling scientific and deep learning solvers.

Computer Science↗
DOE OSTI2022

Learning high-dimensional parametric maps via reduced basis adaptive residual networks

We propose a scalable framework for the learning of high-dimensional parametric maps via adaptively constructed residual network (ResNet) maps between reduced bases of the inputs and outputs. When just few training data are available, it is beneficial to have a compact parametrization in order to ameliorate the ill-posedness of the neural network training problem. By linearly restricting high-dimensional maps to informed reduced bases of the inputs, one can compress high-dimensional maps in a constructive way that can be used to detect appropriate basis ranks, equipped with rigorous error estimates. A scalable neural network learning framework is thus to learn the nonlinear compressed reduced basis mapping. Unlike the reduced basis construction, however, neural network constructions are not guaranteed to reduce errors by adding representation power, making it difficult to achieve good practical performance. Inspired by recent approximation theory that connects ResNets to sequential minimizing flows, we present an adaptive ResNet construction algorithm. This algorithm allows for depth-wise enrichment of the neural network approximation, in a manner that can achieve good practical performance by first training a shallow network and then adapting. We prove universal approximation of the associated neural network class for $L^2_v$ functions on compact sets. Our overall framework allows for constructive means to detect appropriate breadth and depth, and related compact parametrizations of neural networks, significantly reducing the need for architectural hyperparameter tuning. Numerical experiments for parametric PDE problems and a 3D CFD wing design optimization parametric map demonstrate that the proposed methodology can achieve remarkably high accuracy for limited training data, and outperformed other neural network strategies we compared against.

42 ENGINEERING↗
DOE OSTI2021

DeepMerge – II. Building robust deep learning algorithms for merging galaxy identification across domains

In astronomy, neural networks are often trained on simulation data with the prospect of being used on telescope observations. Unfortunately, training a model on simulation data and then applying it to instrument data leads to a substantial and potentially even detrimental decrease in model accuracy on the new target dataset. Simulated and instrument data represent different data domains, and for an algorithm to work in both, domain-invariant learning is necessary. Here we employ domain adaptation techniques— Maximum Mean Discrepancy (MMD) as an additional transfer loss and Domain Adversarial Neural Networks (DANNs)— and demonstrate their viability to extract domain-invariant features within the astronomical context of classifying merging and non-merging galaxies. Additionally, we explore the use of Fisher loss and entropy minimization to enforce better in-domain class discriminability. We show that the addition of each domain adaptation technique improves the performance of a classifier when compared to conventional deep learning algorithms. We demonstrate this on two examples: between two Illustris-1 simulated datasets of distant merging galaxies, and between Illustris-1 simulated data of nearby merging galaxies and observed data from the Sloan Digital Sky Survey. The use of domain adaptation techniques in our experiments leads to an increase of target domain classification accuracy of up to ~20%. With further development, these techniques will allow astronomers to successfully implement neural network models trained on simulation data to efficiently detect and study astrophysical objects in current and future large-scale astronomical surveys.

galaxies: interactions↗
NASA NTRSDate not supplied

Towards Physics Guided Optical Flow for Tracking Atmospheric Motion

Observations of atmospheric 3D winds are critical for improving short-range and long-range forecasting. Such advancement in forecasting directly applies to research in a number of areas including convective processes, wildfire plumes and tornado prediction. Atmospheric Motion Vectors (AMVs) provide a passively sensed approach to quantifying horizontal motion and cloud heights, which are typically sourced from geostationary sensors due to the availability of high frequency observations. Recent work has shown that estimating AMVs by tracking individual pixels with dense optical flow is a promising new direction. In this work, we compare state-of-the-art convolutional neural networks for optical flow in a physics-guided deep learning framework for predicting AMVs. The approach is semi-supervised and uses physically informed wind vectors from high-resolution numerical simulations for supervised learning followed by fine-tuning though warping and reconstruction of full-disk geostationary images (GOES-16). In the vertical direction, we use labels from the CALIPSO low-earth orbit satellite to predict cloud height from 16-band geostationary images with a neural network. We present results for both tasks on held-out time periods and secondary datasets.

Geostationary↗
DOE OSTI2020

Vegetation Detection Using Deep Learning and Conventional Methods

Land cover classification with the focus on chlorophyll-rich vegetation detection plays an important role in urban growth monitoring and planning, autonomous navigation, drone mapping, biodiversity conservation, etc. Conventional approaches usually apply the normalized difference vegetation index (NDVI) for vegetation detection. In this paper, we investigate the performance of deep learning and conventional methods for vegetation detection. Two deep learning methods, DeepLabV3+ and our customized convolutional neural network (CNN) were evaluated with respect to their detection performance when training and testing datasets originated from different geographical sites with different image resolutions. A novel object-based vegetation detection approach, which utilizes NDVI, computer vision, and machine learning (ML) techniques, is also proposed. The vegetation detection methods were applied to high-resolution airborne color images which consist of RGB and near-infrared (NIR) bands. RGB color images alone were also used with the two deep learning methods to examine their detection performances without the NIR band. The detection performances of the deep learning methods with respect to the object-based detection approach are discussed and sample images from the datasets are used for demonstrations.

58 GEOSCIENCES↗
DOE OSTI2024

Neural Networks to Find the Optimal Forcing for Offsetting the Anthropogenic Climate Change Effects

Abstract Of great relevance to climate engineering is the systematic relationship between the radiative forcing to the climate system and the response of the system, a relationship often represented by the linear response function (LRF) of the system. However, estimating the LRF often becomes an ill-posed inverse problem due to high-dimensionality and nonunique relationships between the forcing and response. Recent advances in machine learning make it possible to address the ill-posed inverse problem through regularization and sparse system fitting. Here, we develop a convolutional neural network (CNN) for regularized inversion. The CNN is trained using the surface temperature responses from a set of Green’s function perturbation experiments as imagery input data together with data sample densification. The resulting CNN model can infer the forcing pattern responsible for the temperature response from out-of-sample forcing scenarios. This promising proof of concept suggests a possible strategy for estimating the optimal forcing to negate certain undesirable effects of climate change. The limited success of this effort underscores the challenges of solving an inverse problem for a climate system with inherent nonlinearity. Significance Statement Predicting the climate response for a given climate forcing is a direct problem, while inferring the forcing for a given desired climate response is often an inverse, ill-posed, problem, posing a new challenge to the climate community. This study makes the first attempt to infer the radiative forcing for a given target pattern of global surface temperature response using a deep learning approach. The resulting deeply trained convolutional neural network inversion model shows promise in capturing the forcing pattern corresponding to a given surface temperature response, with a significant implication on the design of an optimal solar radiation management strategy for curbing global warming. This study also highlights the technical challenges that future research should prioritize in seeking feasible solutions to the inverse climate problem.

Ren, Huiying↗
DOE OSTI2024

Two-Step Hyperparameter Optimization Method: Accelerating Hyperparameter Search by Using a Fraction of a Training Dataset

Abstract Hyperparameter optimization (HPO) is an important step in machine learning (ML) model development, but common practices are archaic—primarily relying on manual or grid searches. This is partly because adopting advanced HPO algorithms introduces added complexity to the workflow, leading to longer computation times. This poses a notable challenge to ML applications, as suboptimal hyperparameter selections curtail the potential of ML model performance, ultimately obstructing the full exploitation of ML techniques. In this article, we present a two-step HPO method as a strategic solution to curbing computational demands and wait times, gleaned from practical experiences in applied ML parameterization work. The initial phase involves a preliminary evaluation of hyperparameters on a small subset of the training dataset, followed by a reevaluation of the top-performing candidate models postretraining with the entire training dataset. This two-step HPO method is universally applicable across HPO search algorithms, and we argue it has attractive efficiency gains. As a case study, we present our recent application of the two-step HPO method to the development of neural network emulators for aerosol activation. Although our primary use case is a data-rich limit with many millions of samples, we also find that using up to 0.0025% of the data—a few thousand samples—in the initial step is sufficient to find optimal hyperparameter configurations from much more extensive sampling, achieving up to 135× speedup. The benefits of this method materialize through an assessment of hyperparameters and model performance, revealing the minimal model complexity required to achieve the best performance. The assortment of top-performing models harvested from the HPO process allows us to choose a high-performing model with a low inference cost for efficient use in global climate models (GCMs).

97 MATHEMATICS AND COMPUTING↗