Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “Learning”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 739 records · Page 41

ShapeShop: Towards Understanding Deep Learning Representations via Interactive Experimentation

Deep learning is the driving force behind many recent technologies; however, deep neural networks are often viewed as “black-boxes” due to their internal complexity that is hard to understand. Little research focuses on helping people explore and understand the relationship between a user’s data and the learned representations in deep learning models. We present our ongoing work, ShapeShop, an interactive system for visualizing and understanding what semantics a neural network model has learned. Built using standard web technologies, ShapeShop allows users to experiment with and compare deep learning models to help explore the robustness of image classifiers.

Hohman, Frederick M.↗

Learning from Dynamic User Interaction Graphs to Forecast Diverse Social Behavior

Most of the existing graph analytics for understanding social be- havior focuses on learning from static rather than dynamic graphs using hand-crafted network features or recently emerged graph embeddings learned independently from a downstream predictive task, and solving predictive (e.g., link prediction) rather than fore- casting tasks directly. To address these limitations, we propose (1) a novel task – forecasting user interactions over dynamic social graphs, and (2) a novel deep learning, multi-task, node-aware at- tention model that focuses on forecasting social interactions, going beyond recently emerged approaches for learning dynamic graph embeddings. Our model relies on graph convolutions and recurrent layers to forecast future social behavior and interaction patterns in dynamic social graphs. We evaluate our model on the ability to forecast the number of retweets and mentions of a specific news source on Twitter (focusing on deceptive and credible news sources) with R2 of 0.79 for retweets and 0.81 for mentions. An additional evaluation includes model forecasts of user-repository interactions on GitHub and comments to a specific video on YouTube with a mean absolute error close to 2% and R2 exceeding 0.69. Our results demonstrate that learning from connectivity information over time in combination with node embeddings yields better forecasting results than when we incorporate the state-of-the-art graph em- beddings e.g., Node2Vec and DeepWalk into our model. Finally, we perform in-depth analyses to examine factors that influence model performance across tasks and different graph types e.g., the influence of training and forecasting windows as well as graph topological properties.

Shrestha, Prasha↗

Quantum Transfer Learning to Boost Dementia Detection

Dementia is a devastating condition with profound implications for individuals, families, and healthcare systems. Early and accurate detection of dementia is critical for timely intervention and improved patient outcomes. While classical machine learning and deep learning approaches have been explored extensively for dementia prediction, these solutions often struggle with high-dimensional biomedical data and large-scale datasets, quickly reaching computational and performance limitations. To address this challenge, quantum machine learning (QML) has emerged as a promising paradigm, offering faster training and advanced pattern recognition capabilities. This work aims to demonstrate the potential of quantum transfer learning (QTL) to enhance the performance of a weak classical deep learning model applied to a binary classification task for dementia detection. Besides, we show the effect of noise on the QTL-based approach, investigating the reliability and robustness of this method. Using the OASIS 2 dataset, we show how quantum techniques can transform a suboptimal classical model into a more effective solution for biomedical image classification, highlighting their potential impact on advancing healthcare technology.

Bhowmik, Sounak [University of Tennessee, Knoxvill↗

Prediction of Tropical Cyclone Genesis from Mesoscale Convective Systems using Machine Learning

Tropical cyclone (TC) genesis is a problem of great significance in climate and weather research. Although various environmental conditions necessary for TC genesis have been recognized for a long time, prediction of TC genesis remains a challenge due to complex and stochastic processes involved during TC genesis. Different from traditional statistical and dynamical modeling of TC genesis, in this research, a machine learning framework is developed to determine whether a mesoscale convective system (MCS) would evolve into a tropical cyclone. The machine learning models 1) are built upon a number of essential environmental predictors associated with MCS/TC, 2) predict whether MCSs can become TCs at different lead times, 3) provide information about the relative importance of each predictor, which can be conducive to discovering new aspects of TC genesis. The results indicate that the machine learning classifier, AdaBoost, is able to achieve a 97.2% F1-score accuracy in predicting TC genesis over the entire tropics at a 6-hour lead time using a comprehensive set of environmental predictors. A robust performance can still be attained when the lead time is extended to 12 hours, 24 hours, and 48 hours, and when this machine learning classifier is separately applied to the North Atlantic Ocean and the Western North Pacific Ocean. In contrast, the conventional approach based on genesis potential index can only have no more than 80% F1-score accuracy. Moreover, the machine learning classifier suggests that the low-level vorticity and genesis potential index are the most important predictors to TC genesis, which is consistent with previous discoveries.

54 ENVIRONMENTAL SCIENCES↗

Network Scale Travel Time Prediction using Deep Learning

In recent years, deep learning models have been receiving increased attention within the artificial intelligence (AI) community because of their high prediction accuracy. In this paper, two deep learning models, long short-term memory (LSTM) and convolutional neural network (CNN), are proposed to predict travel time in a road network. One major advantage of using deep learning for travel time prediction is that it can make accurate predictions for all the segments in the transportation network with a single model structure, instead of building customized models for each segment separately. The proposed models were evaluated on a transportation network in the City of Saint Louis, Missouri. The prediction results show that deep learning can provide accurate prediction for both congested and uncongested traffic conditions, and can successfully capture the traffic dynamics of unexpected incidents or special events. The study findings show that deep learning offers a promising approach to real-time prediction of travel times on a network scale.

33 ADVANCED PROPULSION SYSTEMS↗

Application of an automated machine learning-genetic algorithm (AutoML-GA) coupled with computational fluid dynamics simulations for rapid engine design optimization

In recent years, the use of machine learning-based surrogate models for computational fluid dynamics (CFD) simulations has emerged as a promising technique for reducing the computational cost associated with engine design optimization. However, such methods still suffer from drawbacks. One main disadvantage is that the default machine learning (ML) hyperparameters are often severely suboptimal for a given problem. This has often been addressed by manually trying out different hyperparameter settings, but this solution is ineffective in case of a high-dimensional hyperparameter space. Besides this problem, the amount of data needed for training is also not known a priori. In response to these issues that need to be addressed, the present work describes and validates an automated active learning approach, AutoML-GA, for surrogate-based optimization of internal combustion engines. In this approach, a Bayesian optimization technique is used to find the best machine learning hyperparameters based on an initial dataset obtained from a small number of CFD simulations. Subsequently, a genetic algorithm is employed to locate the design optimum on the ML surrogate surface. In the vicinity of the design optimum, the solution is refined by repeatedly running CFD simulations at the projected optima and adding the newly obtained data to the training dataset. It is demonstrated that AutoML-GA leads to a better optimum with a lower number of CFD simulations, compared to the use of default hyperparameters. The proposed framework offers the advantage of being a more hands-off approach that can be readily utilized by researchers and engineers in industry who do not have extensive machine learning expertise.

Owoyele, Opeoluwa↗

Sparse coding of pathology slides compared to transfer learning with deep neural networks

Background Histopathology images of tumor biopsies present unique challenges for applying machine learning to the diagnosis and treatment of cancer. The pathology slides are high resolution, often exceeding 1GB, have non-uniform dimensions, and often contain multiple tissue slices of varying sizes surrounded by large empty regions. The locations of abnormal or cancerous cells, which may constitute a small portion of any given tissue sample, are not annotated. Cancer image datasets are also extremely imbalanced, with most slides being associated with relatively common cancers. Since deep representations trained on natural photographs are unlikely to be optimal for classifying pathology slide images, which have different spectral ranges and spatial structure, we here describe an approach for learning features and inferring representations of cancer pathology slides based on sparse coding. Results We show that conventional transfer learning using a state-of-the-art deep learning architecture pre-trained on ImageNet (RESNET) and fine tuned for a binary tumor/no-tumor classification task achieved between 85% and 86% accuracy. However, when all layers up to the last convolutional layer in RESNET are replaced with a single feature map inferred via a sparse coding using a dictionary optimized for sparse reconstruction of unlabeled pathology slides, classification performance improves to over 93%, corresponding to a 54% error reduction. Conclusions We conclude that a feature dictionary optimized for biomedical imagery may in general support better classification performance than does conventional transfer learning using a dictionary pre-trained on natural images.

59 BASIC BIOLOGICAL SCIENCES↗

Time-lapse seismic data inversion for estimating reservoir parameters using deep learning

Geologic carbon sequestration involves the injection of captured carbon dioxide ([Formula: see text]) into subsurface formations for long-term storage. The movement and fate of the injected [Formula: see text] plume is of great concern to regulators because monitoring helps to identify potential leakage zones and determines the possibility of safe long-term storage. To address this concern, we design a deep-learning framework for [Formula: see text] saturation monitoring to determine the geologic controls on the storage of the injected [Formula: see text]. We use different combinations of porosities and permeabilities for a given reservoir to generate saturation and velocity models. We train the deep-learning model with a few time-lapse seismic images and their corresponding changes in saturation values for a particular [Formula: see text] injection site. The deep-learning model learns the mapping from the change in the time-lapse seismic response to the change in [Formula: see text] saturation during the training phase. We then apply the trained model to data sets comprising different time-lapse seismic image slices (corresponding to different time instances) generated using different porosity and permeability distributions that are not part of the training to estimate the [Formula: see text] saturation values along with the plume extent. Our algorithm provides a deep-learning assisted framework for the direct estimation of [Formula: see text] saturation values and plume migration in heterogeneous formations using the time-lapse seismic data. Our method improves the efficiency of time-lapse inversion by streamlining the large number of intermediate steps in the conventional time-lapse inversion workflow. This method also helps to incorporate the geologic uncertainty for a given reservoir by accounting for the statistical distribution of porosity and permeability during the training phase. Tests on different examples verify the effectiveness of our approach.

Geochemistry & Geophysics↗

Learning genetic perturbation effects with variational causal inference

Advances in sequencing technologies have enhanced the understanding of gene regulation in cells. In particular, Perturb-seq has enabled high-resolution profiling of the transcriptomic response to genetic perturbations at the single-cell level. This understanding has implications in functional genomics and potentially for identifying therapeutic targets. Various computational models have been developed to predict perturbational effects. While deep learning models excel at interpolating observed perturbational data, they tend to overfit in the lack of enough data and may not generalize well to unseen perturbations. In contrast, mechanistic models, such as linear causal models based on gene regulatory networks, hold greater potential for extrapolation, as they encapsulate regulatory information that can predict responses to unseen perturbations. However, their application has been limited to small studies due to overly simplistic assumptions, making them less effective in handling noisy, large-scale single-cell data. We propose a hybrid approach that combines a mechanistic causal model with variational deep learning, termed Single Cell Causal Variational Autoencoder (SCCVAE). The mechanistic model employs a learned regulatory network to represent perturbational changes as shift interventions that propagate through the learned network. SCCVAE integrates this mechanistic causal model into a variational autoencoder, generating rich, comprehensive transcriptomic responses. Our results indicate that SCCVAE exhibits superior performance over current state-of-the-art baselines for extrapolating to predict unseen perturbational responses. Additionally, for the observed perturbations, the latent space learned by SCCVAE allows for the identification of functional perturbation modules and simulation of single-gene knockdown experiments of varying penetrance, presenting a robust tool for interpreting and interpolating perturbational responses at the single-cell level.

59 BASIC BIOLOGICAL SCIENCES↗

Transfer learning for metamaterial design and simulation

Abstract We demonstrate transfer learning as a tool to improve the efficacy of training deep learning models based on residual neural networks (ResNets). Specifically, we examine its use for study of multi-scale electrically large metasurface arrays under open boundary conditions in electromagnetic metamaterials. Our aim is to assess the efficiency of transfer learning across a range of problem domains that vary in their resemblance to the original base problem for which the ResNet model was initially trained. We use a quasi-analytical discrete dipole approximation (DDA) method to simulate electrically large metasurface arrays to obtain ground truth data for training and testing of our deep neural network. Our approach can save significant time for examining novel metasurface designs by harnessing the power of transfer learning, as it effectively mitigates the pervasive data bottleneck issue commonly encountered in deep learning. We demonstrate that for the best case when the transfer task is sufficiently similar to the target task, a new task can be effectively trained using only a few data points yet still achieve a test mean absolute relative error of 3 % with a pre-trained neural network, realizing data reduction by a factor of 1000.

Peng, Rixi↗

A review on the application of machine learning for combustion in power generation applications

Abstract Although the world is shifting toward using more renewable energy resources, combustion systems will still play an important role in the immediate future of global energy. To follow a sustainable path to the future and reduce global warming impacts, it is important to improve the efficiency and performance of combustion processes and minimize their emissions. Machine learning techniques are a cost-effective solution for improving the sustainability of combustion systems through modeling, prediction, forecasting, optimization, fault detection, and control of processes. The objective of this study is to provide a review and discussion regarding the current state of research on the applications of machine learning techniques in different combustion processes related to power generation. Depending on the type of combustion process, the applications of machine learning techniques are categorized into three main groups: (1) coal and natural gas power plants, (2) biomass combustion, and (3) carbon capture systems. This study discusses the potential benefits and challenges of machine learning in the combustion area and provides some research directions for future studies. Overall, the conducted review demonstrates that machine learning techniques can play a substantial role to shift combustion systems towards lower emission processes with improved operational flexibility and reduced operating cost.

Engineering↗

Non-Constant Learning Rates in Retrospective Experience Curve Analyses and their Correlation to Deployment Programs

A key challenge for policy-makers and technology market forecasters is to estimate future technology costs and in particular the rate of cost reduction versus production volume. A related, critical question is what role should state and federal governments have in advancing energy efficient and renewable energy technologies? This work provides retrospective experience curves and learning rates for several energy-related technologies, each of which have a known history of federal and state deployment programs. We derive learning rates for eight technologies including energy efficient lighting technologies, stationary fuel cell systems, and residential solar photovoltaics, and provide an overview and timeline of historical deployment programs such as state and federal standards and state and national incentive programs for each technology. Piecewise linear regimes are observed in a range of technology experience curves, and public investments or deployment programs are found to be strongly correlated to an increase in learning rate across multiple technologies. A downward bend in the experience curve is found in 5 out of the 8 energy-related technologies presented here (electronic ballasts, magnetic ballasts, compact fluorescent lighting, general service fluorescent lighting, and the installed cost of solar PV). In each of the five downward-bending experience curves, we believe that an increase in the learning rate can be linked to deployment programs to some degree. This work sheds light on the endogenous versus exogenous contributions to technological innovation and highlights the impact of exogenous government sponsored deployment programs. This work can inform future policy investment direction and can shed light on market transformation and technology learning behavior.

29 ENERGY PLANNING, POLICY, AND ECONOMY↗

Brentwood Lessons Learned Project Report

The purpose of this report is to document lessons learned in the installation of the hydrogen fueling station at the National Park Service Brentwood site in Washington, D.C., to help further the deployment of hydrogen infrastructure required to support hydrogen and other fuel cell technologies. Hydrogen fueling is the most difficult infrastructure component to build and permit. Hydrogen fueling can include augmenting hydrogen fueling capability to existing conventional fuel fueling stations as well as building brand new hydrogen fueling stations. This report was produced as part of the Brentwood Lessons Learned project. The project consisted of transplanting an existing modular hydrogen fueling station from Connecticut to the National Park Service Brentwood site. This relocation required design and construction at the Brentwood site to accommodate the existing station design as well as installation and validation of the updated station. One of the most important lessons learned was that simply moving an existing modular station to an operating site was not necessarily straight-forward - performing the relocation required significant effort and cost. The station has to function at the selected operating site and this functionality requires a power supply, building supports connecting to an existing alarm system, electrical grounding and lighting, providing nitrogen for purging, and providing deionized water if an electrolyzer is part of the station package. Most importantly, the station has to fit into the existing site both spatially and operationally and not disrupt existing operations at the site. All of this coordination and integration requires logistical planning and project management. The idea that a hydrogen fueling station can be simply dropped onto a site and made immediately operational is generally not realistic. Other important lessons learned include that delineating the boundaries of the multiple jurisdictions that have authority over a project for all parties involved in the project are key to an efficient approval process; and site investigation is necessary when integrating a new station design onto an existing site, particularly an older existing site that may have limited documentation on the site history and operations. The lessons learned for permitting and subcontracting construction work can be applied to other similar sites and to commercial sites.

30 DIRECT ENERGY CONVERSION↗

Quantifying Uncertainty to Improve Decision Making in Machine Learning

Data-driven modeling, including machine learning methods, continue to play an increasing role in society. Data-driven methods impact decision making for applications ranging from everyday determinations about which news people see and control of self-driving cars to high-consequence national security situations related to cyber security and analysis of nuclear weapons reliability. Although modern machine learning methods have made great strides in model induction and show excellent performance in a broad variety of complex domains, uncertainty remains an inherent aspect of any data-driven model. In this report, we provide an update to the preliminary results on uncertainty quantification for machine learning presented in SAND2017-6776. Specifically, we improve upon the general problem definition and expand upon the experiments conducted for the earlier re- port. Most importantly, we summarize key lessons learned about how and when uncertainty quantification can inform decision making and provide valuable insights into the quality of learned models and potential improvements to them.

97 MATHEMATICS AND COMPUTING↗

Understanding the Machine Learning Needs of ECP Applications

In order to support the codesign needs of ECP applications in current and future hardware in the area of machine learning, the ExaLearn team at Sandia studied the different machine learning use cases in three different ECP applications. This report is a summary of the needs of the three applications. The Sandia ExaLearn team will develop a proxy application representative of ECP application needs, specifically the ExaSky and EXAALT ECP projects. The proxy application will allow us to demonstrate performance portable kernels within machine learning codes. Furthermore, current training scalability of machine learning networks in these applications is negatively affected by large batch sizes. Training throughput of the network will increase as batch size increases, but network accuracy and generalization worsens. The proxy application will contain hybrid model- and data-parallelism to improve training efficiency while maintaining network accuracy. The proxy application will also target optimizing 3D convolutional layers, specific to scientific machine learning, which have not been as thoroughly explored by industry.

97 MATHEMATICS AND COMPUTING↗

Joint Analysis of Program Data Representations using Machine Learning for Improved Software Assurance and Development Capabilities

We explore the use of multiple deep learning models for detecting flaws in software programs. Current, standard approaches for flaw detection rely on a single representation of a software program (e.g., source code or a program binary). We illustrate that, by using techniques from multimodal deep learning, we can simultaneously leverage multiple representations of software programs to improve flaw detection over single representation analyses. Specifically, we adapt three deep learning models from the multimodal learning literature for use in flaw detection and demonstrate how these models outperform traditional deep learning models. We present results on detecting software flaws using the Juliet Test Suite and Linux Kernel.

97 MATHEMATICS AND COMPUTING↗

Machine Learning for Turbulence Modeling

This work was conducted as part of a Harry S. Truman Fellowship Laboratory Directed Research and Development project. The goal was to use machine learning methods to provide uncertainty quantification and model improvements for Reynolds Averaged Navier Stokes (RANS) turbulence models. For applications of interest in energy, safety, and security, it is critical to be able to model turbulence accurately. Current RANS models are unreliable for many flows of engineering relevance. Machine learning provides an avenue for developing improved models based on the data generated by high fidelity simulations. In this project, machine learning methods were used to predict when current RANS models would fail. They were also used to develop improved RANS closure models. A key aim was developing a tight feedback loop between scientific domain knowledge and data driven methods. To this end, a methodology for incorporating known invariance constraints into the machine learning models was proposed and evaluated. This work demonstrated that incorporating known constraints into the data driven models provided improved performance and reduced computational cost. This research represents one of the first applications of deep learning to turbulence modeling.

42 ENGINEERING↗

Hypothesis testing via AI: Generating physically interpretable models of scientific data with machine learning (Full Technical Report)

Deep learning has demonstrated an exceptional ability to solve complex tasks (an engineering success); however, it has done so at the expense of the ability to generate new knowledge (a scientific failure). We propose an alternative framework—entitled Deep Symbolic Regression (DSR)—in which artificial neural networks (NNs) rapidly generate hypotheses about physical relationships among inputs. This framework bypasses the need to interpret an NN altogether, while still leveraging the representational power of deep learning. The resulting models are tractable mathematical expressions, which are inherently and readily human interpretable and can provide insights into underlying physical phenomena. Further, we fold this methodology into the scientific process by allowing the scientist to directly integrate a priori knowledge and beliefs to accelerate learning. We demonstrate this methodology on symbolic regression—the problem of rediscovering underlying expressions describing a dataset—and achieve state-of-the-art performance across a wide variety of symbolic regression problems. Further, we generalize our DSR framework to apply to the more general class of symbolic optimization problems, in which one seeks to optimize a sequence of symbols or “tokens” under a black-box reward function. Examples of other symbolic optimization problems include neural architecture search and computational antibody design. Our generalized tool, Deep Symbolic Optimization (DSO), has been demonstrated on the task of learning symbolic control policies for reinforcement learning environments, and has been adopted as an enabling capability for computational antibody design.

97 MATHEMATICS AND COMPUTING↗