Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “Neural Networks (Computer)”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 955 records · Page 53

Pulling out all the tops with computer vision and deep learning

We apply computer vision with deep learning — in the form of a convolutional neural network (CNN) — to build a highly effective boosted top tagger. Previous work (the “DeepTop” tagger of Kasieczka et al) has shown that a CNN-based top tagger can achieve comparable performance to state-of-the-art conventional top taggers based on high-level inputs. Here, we introduce a number of improvements to the DeepTop tagger, including architecture, training, image preprocessing, sample size and color pixels. Our final CNN top tagger outperforms BDTs based on high-level inputs by a factor of ~ 2–3 or more in background rejection, over a wide range of tagging efficiencies and fiducial jet selections. As reference points, we achieve a QCD background rejection factor of 500 (60) at 50%top tagging efficiency for fully-merged (non-merged) top jets with pT in the 800–900 GeV (350–450 GeV) range. Our CNN can also be straightforwardly extended to the classification of other types of jets, and the lessons learned here may be useful to others designing their own deep NNs for LHC applications.

97 MATHEMATICS AND COMPUTING↗

Short-Term Forecasting of Thermostatic and Residential Loads Using Long Short-Term Memory Recurrent Neural Networks

Internet of Things (IoT) devices in smart grids enable intelligent energy management for grid managers and personalized energy services for consumers. Investigating a smart grid with IoT devices requires a simulation framework with IoT devices modeling. However, there lack comprehensive study on the modeling of IoT devices in smart grids. This paper investigates the IoT device modeling of a thermostatic load and implements the recurrent neural networks model for short-term load forecasting in this IoT-based thermostatic load. The recurrent neural network structure is leveraged to build a load forecasting model on temporal correlation. The temporal recurrent neural network layers including long short-term memory cells are employed to learn the data from both the simulation platform and New South Wales residential datasets. The simulation results are provided for demonstration.

electric load forecasting↗

Physics-Informed Neural Networks for PDE-Constrained Optimization and Control

The goal of optimal control is to determine a sequence of inputs for maximizing or minimizing a given performance criterion subject to the dynamics and constraints of the system under observation. This work introduces Control Physics-Informed Neural Networks (PINNs), which simultaneously learn both the system states and the optimal control signal in a single-stage framework that leverages the system’s underlying physical laws. While prior approaches often follow a two-stage process-modeling, the system first and then devising its control—the presented novel framework embeds the necessary optimality conditions directly into the network architecture and loss function. We demonstrate the effectiveness of the novel methodology by solving various open-loop optimal control problems governed by analytical, one-dimensional, and two-dimensional partial differential equations (PDEs).

97 MATHEMATICS AND COMPUTING↗

Modeling Cell Migration with Convolutional Neural Network and Deep Reinforcement Learning

Cell migration modeling is a longstanding biological challenge, which is regulated by a highly complex set of regulatory mechanisms at multiple scales in a developmental system. This study presents a generic framework for regulatory mechanisms discovery during cell migration. This framework uses convolutional neural networks and reinforcement learning to better study navigation rules and mechanisms during cell migration. This framework adopts a flexible model-free approach that directly takes raw images as the sensory input. It can better handle simulation scenarios that involve cell division during embryogenesis. Computational experiments also prove that this model achieves better performance than a previous model with a fully connected neural network.

Wang, Zi↗

Extending Power of Nature from Binary Problems to Real-Valued Graph Learning in Real World

Nature performs complex computations constantly at clearly lower cost and higher performance than digital computers. It is crucial to understand how to harness the unique computational power of nature in Machine Learning (ML). In the past decade, besides the development of Neural Networks (NNs), the community has also relentlessly explored nature-powered ML paradigms. Although most of them are still predominantly theoretical, a new practical paradigm enabled by the recent advent of CMOS-compatible room-temperature nature-based computers has emerged. By harnessing the nature's power of entropy increase, this paradigm can solve binary learning problems delivering immense speedup and energy savings compared with NNs, while maintaining comparable accuracy. Regrettably, its values to the real world are highly constrained by its binary nature. A clear pathway to its extension to real-valued problems remains elusive. This paper aims to unleash this pathway by proposing a novel end-to-end Nature-Powered Graph Learning (NP-GL) framework. Specifically, through a three-dimensional co-design, NP-GL can leverage the nature's power of entropy increase to efficiently solve real-valued graph learning problems. Experimental results across 4 real-world applications with 6 datasets demonstrate that NP-GL delivers, on average, 6970X speedup and 10^5x energy consumption reduction with comparable or even higher accuracy than Graph Neural Networks (GNNs).

artificial intelligence↗

Complex multicomponent spectrum analysis with Deep Neural Network

Here, in this paper, we present the use of deep neural networks to estimate physical parameters from complex optical emission spectra of the D β /H β transition. Specifically, we focus on estimating the radio frequency electric field vector of the lower hybrid wave and isotope ratio within the scrape-off-layer plasma of the WEST tokamak. Fitting the spectral data using a traditional non-linear least squares analysis requires many free parameters and is computationally expensive, rendering the data unusable for real-time control. By implementing relatively small neural networks, the physical parameters can be directly extracted from the spectral data with reasonable accuracy in a few milliseconds. The deep neural network prediction can serve as input for a reduced model using least-squares fitting or for real-time control. We show that deep neural networks can be an effective tool for analyzing complex multicomponent spectra, providing a speedup of more than 10 5 times compared to least residual analysis, with an accuracy of 0.5% for the isotope ratio, and 0.09 kV/cm and 0.38 kV/cm for the RF radial and poloidal electric field respectively.

47 OTHER INSTRUMENTATION↗

Computational Approaches for Clean Energy Materials

Currently, 80% of the global final energy consumption occurs in form of fuels and only 20% as electricity. On the other hand, renewable energy additions come almost exclusively in the form of electricity (dominantly photovoltaics and wind). Thus, a successful energy transition will require enormous growth in renewables, sufficient to convert excess electricity into fuels, as well as the development of non-electricity based solar fuel technologies. As much as photovoltaic capacities have grown over the past 20 years, it is far from clear that current technologies and materials are up to the task to grow from here by yet another factor 100 until 2050. Therefore, sustained research efforts on emerging inorganic semiconductors for solar electricity and fuels are essential for facing the double challenge of climate change and energy security. Computational materials science can make important contributions, guiding and supporting research activities through both materials search and discovery and through detailed studies that help to develop a mechanistic understanding of materials performance and bottlenecks. This presentation will highlight three recent computational projects with relevance for photovoltaics and solar fuels (1) Defect graph neural networks (dGNN) for materials discovery in solar thermochemical hydrogen (STCH) [1]. The dGNN approach facilitates broad and fast materials screening for defect properties. (2) Modeling highly off-stoichiometric systems by evaluating the free energy of defect interaction [2]. This approach allows quantitative prediction of H2 production in complex STCH oxides. (3) First-principles atomic structure prediction for interfaces [3]. This work showed how an atomically thin CdCl2 interlayer phase enables in principle ideal electron transport across the incommensurate SnO2/CdTe interface. [1] M.D. Witman, A. Goyal, T. Ogitsu, A.H. McDaniel, S. Lany, Nat. Comput. Sci. (2023). https://doi.org/10.1038/s43588-023-00495-2. [2] A. Goyal, M.D. Sanders, R.P. O'Hayre, S. Lany, PRX Energy 3, 013008 (2024). https://doi.org/10.1103/PRXEnergy.3.013008. [3] A. Sharan, M. Nardone, D. Krasikov, N. Singh, S. Lany, Appl. Phys. Rev. 9, 041411 (2022). https://doi.org/10.1063/5.0104008.

density functional theory↗

Scalable balanced training of conditional generative adversarial neural networks on image data

Here, we propose a distributed approach to train deep convolutional generative adversarial neural network (DC-CGANs) models. Our method reduces the imbalance between generator and discriminator by partitioning the training data according to data labels, and enhances scalability by performing a parallel training where multiple generators are concurrently trained, each one of them focusing on a single data label. Performance is assessed in terms of inception score, Fréchet inception distance, and image quality on MNIST, CIFAR10, CIFAR100, and ImageNet1k datasets, showing a significant improvement in comparison to state-of-the-art techniques to training DC-CGANs. Weak scaling is attained on all the four datasets using up to 1000 processes and 2000 NVIDIA V100 GPUs on the OLCF supercomputer Summit.

97 MATHEMATICS AND COMPUTING↗

A domain wall-magnetic tunnel junction artificial synapse with notched geometry for accurate and efficient training of deep neural networks

Inspired by the parallelism and efficiency of the brain, several candidates for artificial synapse devices have been developed for neuromorphic computing, yet a nonlinear and asymmetric synaptic response curve precludes their use for backpropagation, the foundation of modern supervised learning. Spintronic devices—which benefit from high endurance, low power consumption, low latency, and CMOS compatibility—are a promising technology for memory, and domain-wall magnetic tunnel junction (DW-MTJ) devices have been shown to implement synaptic functions such as long-term potentiation and spike-timing dependent plasticity. In this work, we propose a notched DW-MTJ synapse as a candidate for supervised learning. Using micromagnetic simulations at room temperature, we show that notched synapses ensure the non-volatility of the synaptic weight and allow for highly linear, symmetric, and reproducible weight updates using either spin transfer torque (STT) or spin–orbit torque (SOT) mechanisms of DW propagation. We use lookup tables constructed from micromagnetics simulations to model the training of neural networks built with DW-MTJ synapses on both the MNIST and Fashion-MNIST image classification tasks. Accounting for thermal noise and realistic process variations, the DW-MTJ devices achieve classification accuracy close to ideal floating-point updates using both STT and SOT devices at room temperature and at 400 K. Our work establishes the basis for a magnetic artificial synapse that can eventually lead to hardware neural networks with fully spintronic matrix operations implementing machine learning.

42 ENGINEERING↗

DLSIA: Deep Learning for Scientific Image Analysis

DLSIA (Deep Learning for Scientific Image Analysis) is a Python-based machine learning library that empowers scientists and researchers across diverse scientific domains with a range of customizable convolutional neural network (CNN) architectures for a wide variety of tasks in image analysis to be used in downstream data processing. DLSIA features easy-to-use architectures, such as autoencoders, tunable U-Nets and parameter-lean mixed-scale dense networks (MSDNets). Additionally, this article introduces sparse mixed-scale networks (SMSNets), generated using random graphs, sparse connections and dilated convolutions connecting different length scales. For verification, several DLSIA-instantiated networks and training scripts are employed in multiple applications, including inpainting for X-ray scattering data using U-Nets and MSDNets, segmenting 3D fibers in X-ray tomographic reconstructions of concrete using an ensemble of SMSNets, and leveraging autoencoder latent spaces for data compression and clustering. As experimental data continue to grow in scale and complexity, DLSIA provides accessible CNN construction and abstracts CNN complexities, allowing scientists to tailor their machine learning approaches, accelerate discoveries, foster interdisciplinary collaboration and advance research in scientific image analysis.

97 MATHEMATICS AND COMPUTING↗

FALCON: Framework for Anomaly Detection in Industrial Control Systems

Industrial Control Systems (ICS) are used to control physical processes in critical infrastructure. These systems are used in a wide variety of operations such as water treatment, power generation and distribution, and manufacturing. While the safety and security of these systems are of serious concern, recent reports have shown an increase in targeted attacks aimed at manipulating physical processes to cause catastrophic consequences. This trend emphasizes the need for algorithms and tools that provide resilient and smart attack detection mechanisms to protect ICS. In this paper, we propose an anomaly detection framework for ICS based on a deep neural network. The proposed methodology uses dilated convolution and long short-term memory (LSTM) layers to learn temporal as well as long term dependencies within sensor and actuator data in an ICS. The sensor/actuator data are passed through a unique feature engineering pipeline where wavelet transformation is applied to the sensor signals to extract features that are fed into the model. Additionally, this paper explores four variations of supervised deep learning models, as well as an unsupervised support vector machine (SVM) model for this problem. The proposed framework is validated on Secure Water Treatment testbed results. This framework detects more attacks in a shorter period of time than previously published methods.

97 - MATHEMATICS AND COMPUTING↗

Deep Learning-Assisted Real-Time Forward Modeling of Electromagnetic Logging in Complex Formations

Higher dimensional (i.e., 2-D and 3-D) modeling is indispensable to correctly evaluate the responses of electromagnetic logging tools in complex formation environments. However, limited by the high computational cost of rigorous modeling such as finite difference method and the finite element method, the real-time applications in the well logging industry primarily rely on the 1-D forward solver, which would result in erroneous formation evaluation for complex scenarios. As a result, aiming at realizing fast modeling for electromagnetic logging tools in complex formations, this paper proposes a general framework assisted by deep neural networks (DNNs). The framework consists of three modules: earth model classification, parameter extraction, and surrogate construction. Separate DNNs are trained and tested for different modules. The accuracy and efficiency of the DNN assisted fast modeling are validated by several experiments. Here, this study finds that the fast modeling assisted by DNNs is able to calculate the tool responses and reconstruct the subsurface formations in real-time.

97 MATHEMATICS AND COMPUTING↗

TNet: A Model-Constrained Tikhonov Network Approach for Inverse Problems

Deep learning (DL), in particular deep neural networks, by default is purely data-driven and in general does not require physics. This is the strength of DL but also one of its key limitations when applied to science and engineering problems in which underlying physical properties—such as stability, conservation, and positivity—and accuracy are required. DL methods in their original forms are often not capable of respecting the underlying mathematical models or achieving desired accuracy even in big-data regimes. On the other hand, many data-driven science and engineering problems, such as inverse problems, typically have limited experimental or observational data, and DL would overfit the data in this case. Leveraging information encoded in the underlying mathematical models, we argue, not only compensates for missing information in low data regimes but also provides opportunities to equip DL methods with the underlying physics, hence promoting better generalization. This paper develops a model-constrained DL approach and its variant TNet—a Tikhonov neural network—which are capable of learning not only information hidden in the training data but also in the underlying mathematical models to solve inverse problems governed by partial differential equations in low data regimes. We provide the constructions and some theoretical results for the proposed approaches for both linear and nonlinear inverse problems. Since TNet is designed to learn inverse solutions with Tikhonov regularization, it is interpretable: in fact it recovers Tikhonov solutions for linear cases while potentially approximating Tikhonov solutions for nonlinear inverse problems. We also prove that data randomization can enhance not only the smoothness of the networks but also their generalizations. Comprehensive numerical results confirm the theoretical findings and show that with even as little as 1 training data sample for one-dimensional (1D) deconvolution, 5 for an inverse 2D heat conductivity problem, 100 for inverse initial conditions for a time-dependent 2D Burgers’s equation, and 50 for inverse initial conditions for 2D Navier–Stokes equations, TNet solutions can be as accurate as Tikhonov solutions while being several orders of magnitude faster. Furthermore, this is possible owing to the model-constrained term, replications, and randomization.

97 MATHEMATICS AND COMPUTING↗

Differentiable Neural Architecture, Mixed Precision and Accelerator Co-Search

Quantization, effective Neural Network architecture, and efficient accelerator hardware are three important design paradigms to maximize accuracy and efficiency. Mixed Precision Quantization is a process of assigning different precision to different Neural Network layers for optimized inference. Neural Architecture Search (NAS) is a process of automatically designing the neural network for a task and can also be extended to search for the precision of each weight and activation matrix. In this paper, we develop the following three methods: (i) Fast Differentiable Hardware-aware Mixed Precision Quantization Search method to find optimal precision, (ii) Joint Differentiable hardware-aware Architecture and Mixed Precision Quantization Co-search, (iii) Joint Accelerator, Architecture, and Precision triple co-search to find best possibilities in all the three worlds. We demonstrate the effectiveness of our proposed methods targeting Bitfusion accelerator by searching mixed precision models on MobilenetV2. We achieve better accuracy-latency trade-off models than the manually designed and previously proposed search methods.

97 MATHEMATICS AND COMPUTING↗

Knowledge-guided learning with curated prior genetic biomarkers for robust model interpretation

Abstract Motivation Knowledge-guided learning offers effective and robust model training strategies in data-scarce settings by incorporating established domain knowledge, thereby enhancing generalization, robustness, and interpretability. By contrast, conventional deep learning approaches rely purely on data-driven learning, which can limit robust model interpretability, particularly in high-dimensional settings with limited size samples. In computational biology, knowledge-guided learning has primarily leveraged network- and structural-based knowledge, leading to biologically interpretable representations and enhanced predictive performance compared to conventional approaches. However, curated biomarkers, one of the most accessible forms of biological knowledge, remain largely unexplored within knowledge-guided paradigms. Results In this study, we propose a model-agnostic training paradigm, Biomarker-driven Explainable Prior-guided Learning (BioExPL), that can be applied to any neural networks that incorporates curated prior knowledge. BioExPL enforces neural networks to reflect curated biomarker priors in their latent representations through a novel knowledge-alignment loss. BioExPL consistently demonstrated significantly improved predictive performance and enhanced model interpretability with minimized computational overhead in simulation studies and intensive experiments on multiple cancer datasets. BioExPL not only integrates prior curated knowledge into the model but also accurately identifies unknown associated signals additionally. BioExPL is model-agnostic and domain-independent, enabling its integration into diverse neural network architectures. Availability and implementation The open-source is publicly available at: https://github.com/datax-lab/BioExPL.

Baek, Beomsu [Department of Computer Science, Univ↗

The expressivity of classical and quantum neural networks on entanglement entropy

Abstract Analytically continuing the von Neumann entropy from Rényi entropies is a challenging task in quantum field theory. While then-th Rényi entropy can be computed using the replica method in the path integral representation of quantum field theory, the analytic continuation can only be achieved for some simple systems on a case-by-case basis. In this work, we propose a general framework to tackle this problem using classical and quantum neural networks with supervised learning. We begin by studying several examples with known von Neumann entropy, where the input data is generated by representing$${\text {Tr}}\rho _A^n$$ Tr ρ A n with a generating function. We adopt KerasTuner to determine the optimal network architecture and hyperparameters with limited data. In addition, we frame a similar problem in terms of quantum machine learning models, where the expressivity of the quantum models for the entanglement entropy as a partial Fourier series is established. Our proposed methods can accurately predict the von Neumann and Rényi entropies numerically, highlighting the potential of deep learning techniques for solving problems in quantum information theory.

Physics↗

Integrated data-driven and experimental approaches to accelerate lead optimization targeting SARS-CoV- 2 main protease

Identification of potential therapeutic candidates can be expedited by integrating computational modeling with domain aware machine learning (ML) approaches followed by experimental validation. Generative deep learning models have been recently developed that can generate thousands of new candidates, but their physiochemical properties are typically not optimized. Using our deep learning models and a scaffold as a starting point, we generated tens of thousands of compounds for SARS-CoV-2 M pro that preserve the core scaffold. Here we utilized and implemented several computational tools such as structural alert and toxicity analysis, high throughput virtual screening, ML-based 3D quantitative structure–activity relationships, multi-parameter optimization, and graph neural networks on libraries of generated candidates to predict biological activity and binding affinity a priori. From these collective computational results, eight promising candidates were identified and tested experimentally using Native Mass Spectrometry (MS) and FRET-based functional assays. Two compounds, with quinazoline-2-thiol and acetylpiperidine core moiety showed IC 50 values in the low micromolar range: 2.95±0.0017 µM and 3.41±0.0015 µM, respectively. The molecular dynamics simulations further highlight that binding of these compounds results in allosteric modulations in the chain B and the interface domains of the M pro . The key fragments from these top hits can be used as input for closed loop lead optimization in the integrated pipeline.

60 APPLIED LIFE SCIENCES↗