SEARCH · Engineering Papers
Results for “Transfer Learning Model”
Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.
Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.
Generalization of ML-Based Surrogate Models to Dynamic Injection Schedules using Transfer Learning
For presentation at the FECM/ NETL Carbon Management Project Review Meeting, Pittsburgh, PA, August 28-September 1, 2023.
Active Learning for Rapid Targeted Synthesis of Compositionally Complex Alloys
The next generation of advanced materials is tending toward increasingly complex compositions. Synthesizing precise composition is time-consuming and becomes exponentially demanding with increasing compositional complexity. An experienced human operator does significantly better than a novice but still struggles to consistently achieve precision when synthesis parameters are coupled. The time to optimize synthesis becomes a barrier to exploring scientifically and technologically exciting compositionally complex materials. This investigation demonstrates an active learning (AL) approach for optimizing physical vapor deposition synthesis of thin-film alloys with up to five principal elements. We compared AL-based on Gaussian process (GP) and random forest (RF) models. The best performing models were able to discover synthesis parameters for a target quinary alloy in 14 iterations. We also demonstrate the capability of these models to be used in transfer learning tasks. RF and GP models trained on lower dimensional systems (i.e., ternary, quarternary) show an immediate improvement in prediction accuracy compared to models trained only on quinary samples. Furthermore, samples that only share a few elements in common with the target composition can be used for model pre-training. We believe that such AL approaches can be widely adapted to significantly accelerate the exploration of compositionally complex materials.
Bridging the gap between experiments and simulations using machine learning
The physics of inertial confinement fusion is rich and complex. Simulation codes that are used to design experiments are computationally expensive and lack the predictive capability required for extensive parameter exploration in search of a high-performing design for laser direct drive. In this work we use deep learning to build a fast emulator of experiments. To facilitate the development of the deep-learning model, an autoencoder is used to reduce the dimensionality of the input space. Two deep learning models are developed. One model is trained on a vast array of simulation data and is subsequently calibrated to expensive and limited experimental data using a technique known as “transfer learning.” The other model is trained on a statistical model and is subsequently calibrated using experimental data. A comparative study of the two predictive models is carried out. The models potentially reproduce key experimental observables with high accuracy and unprecedented inference times relative to those achieved with simulation codes. These models facilitate rapid exploration of a high dimensional input parameter space.
Investigating explainable transfer learning for battery lifetime prediction under state transitions
Battery lifetime prediction at early cycles is crucial for researchers and manufacturers to examine product quality and promote technology development. Machine learning has been widely utilized to construct data-driven solutions for high-accuracy predictions. However, the internal mechanisms of batteries are sensitive to many factors, such as charging/discharging protocols, manufacturing/storage conditions, and usage patterns. These factors will induce state transitions, thereby decreasing the prediction accuracy of data-driven approaches. Transfer learning is a promising technique that overcomes this difficulty and achieves accurate predictions by jointly utilizing information from various sources. Hence, we develop two transfer learning methods, Bayesian Model Fusion and Weighted Orthogonal Matching Pursuit, to strategically combine prior knowledge with limited information from the target dataset to achieve superior prediction performance. From our results, our transfer learning methods reduce root-mean-squared error by 41% through adapting to the target domain. Furthermore, the transfer learning strategies identify the variations of impactful features across different sets of batteries and therefore disentangle the battery degradation mechanisms and the root cause of state transitions from the perspective of data mining. These findings suggest that the transfer learning strategies proposed in our work are capable of acquiring knowledge across multiple data sources for solving specialized issues.
Transfer learning driven design optimization for inertial confinement fusion
Transfer learning is a promising approach to create predictive models that incorporate simulation and experimental data into a common framework. In this technique, a neural network is first trained on a large database of simulations and then partially retrained on sparse sets of experimental data to adjust predictions to be more consistent with reality. Previously, this technique has been used to create predictive models of Omega [Humbird et al., IEEE Trans. Plasma Sci. 48, 61–70 (2019)] and NIF [Humbird et al., Phys. Plasmas 28, 042709 (2021); Kustowski et al., Mach. Learn. 3, 015035 (2022)] inertial confinement fusion (ICF) experiments that are more accurate than simulations alone. Here, in this work, we conduct a transfer learning driven hypothetical ICF campaign in which the goal is to maximize experimental neutron yield via Bayesian optimization. The transfer learning model achieves yields within 5% of the maximum achievable yield in a modest-sized design space in fewer than 20 experiments. Furthermore, we demonstrate that this method is more efficient at optimizing designs than traditional model calibration techniques commonly employed in ICF design. Such an approach to ICF design could enable robust optimization of experimental performance under uncertainty.
Iterative ML and Experiments for Emerging VOCs
SAND2026-17074O Iterative ML and Experiments for Emerging VOCs is a tool that analyzes and predicts the behaviors of SARS-CoV-2 variants. It processes experimental data on ACE2 (the receptor for the SARS-CoV-2 virus that allows it to infect the cell) and antibody binding using machine learning models, including neural networks, to forecast ACE2 interactions and variant expression. The tool employs transfer learning and global epistasis modeling, integrating public datasets with proprietary data to enhance prediction accuracy. Additionally, it fits concentration-response curves to determine dissociation constants and generates visualizations to support research findings, thereby aiding in the identification of new antibodies for emerging variants of concern. Sandia National Laboratories is a multimission laboratory managed and operated by National Technology & Engineering Solutions of Sandia, LLC, a wholly owned subsidiary of Honeywell International Inc., for the U.S. Department of Energy’s National Nuclear Security Administration under contract DE-NA0003525.
Quasi-Classical Trajectory Calculation of Rate Constants Using an Ab Initio Trained Machine Learning Model (aML-MD) with Multifidelity Data
Machine learning (ML) provides a great opportunity for the construction of models with improved accuracy in classical molecular dynamics (MD). However, the accuracy of a ML trained model is limited by the quality and quantity of the training data. Generating large sets of accurate ab initio training data can require significant computational resources. Furthermore, inconsistent or incompatible data with different accuracies obtained using different methods may lead to biased or unreliable ML models that do not accurately represent the underlying physics. Recently, transfer learning showed its potential for avoiding these problems as well as for improving the accuracy, efficiency, and generalization of ML models using multifidelity data. In this work, ab initio trained ML-based MD (aML-MD) models are developed through transfer learning using DFT and multireference data from multiple sources with varying accuracy within the Deep Potential MD framework. Further, the accuracy of the force field is demonstrated by calculating rate constants for the H + HO 2 → H 2 + 3 O 2 reaction using quasi-classical trajectories. We show that the aML-MD model with transfer learning can accurately predict the rate constants while reducing the computational cost by more than five times compared to the use of more expensive quantum chemistry training data sets. Hence, the aML-MD model with transfer learning shows great potential in using multifidelity data to reduce the computational cost involved in generating the training set for these potentials.
Combining multitask and transfer learning with deep Gaussian processes for autotuning-based performance engineering
We combine deep Gaussian processes (DGPs) with multitask and transfer learning for the performance modeling and optimization of HPC applications. Deep Gaussian processes merge the uncertainty quantification advantage of Gaussian processes (GPs) with the predictive power of deep learning. Multitask and transfer learning allow for improved learning efficiency when several similar tasks are to be learned simultaneously and when previous learned models are sought to help in the learning of new tasks, respectively. A comparison with state-of-the-art autotuners shows the advantage of our approach on two application problems. In this article, we combine DGPs with multitask and transfer learning to allow for both an improved tuning of an application parameters on problems of interest but also the prediction of parameters on any potential problem the application might encounter.
Combining protein sequences and structures with transformers and equivariant graph neural networks to predict protein function
Abstract Motivation Millions of protein sequences have been generated by numerous genome and transcriptome sequencing projects. However, experimentally determining the function of the proteins is still a time consuming, low-throughput, and expensive process, leading to a large protein sequence-function gap. Therefore, it is important to develop computational methods to accurately predict protein function to fill the gap. Even though many methods have been developed to use protein sequences as input to predict function, much fewer methods leverage protein structures in protein function prediction because there was lack of accurate protein structures for most proteins until recently. Results We developed TransFun—a method using a transformer-based protein language model and 3D-equivariant graph neural networks to distill information from both protein sequences and structures to predict protein function. It extracts feature embeddings from protein sequences using a pre-trained protein language model (ESM) via transfer learning and combines them with 3D structures of proteins predicted by AlphaFold2 through equivariant graph neural networks. Benchmarked on the CAFA3 test dataset and a new test dataset, TransFun outperforms several state-of-the-art methods, indicating that the language model and 3D-equivariant graph neural networks are effective methods to leverage protein sequences and structures to improve protein function prediction. Combining TransFun predictions and sequence similarity-based predictions can further increase prediction accuracy. Availability and implementation The source code of TransFun is available at https://github.com/jianlin-cheng/TransFun.
Optimization of Thermal Conductance at Interfaces Using Machine Learning Algorithms
We report optimization of thermal transport across the interface of two different materials is critical to micro-/nanoscale electronic, photonic, and phononic devices. Although several examples of compositional intermixing at the interfaces having a positive effect on interfacial thermal conductance (ITC) have been reported, an optimum arrangement has not yet been determined because of the large number of potential atomic configurations and the significant computational cost of evaluation. On the other hand, computation-driven materials design efforts are rising in popularity and importance. Yet, the scalability and transferability of machine learning models remain as challenges in creating a complete pipeline for the simulation and analysis of large molecular systems. In this work we present a scalable Bayesian optimization framework, which leverages dynamic spawning of jobs through the Message Passing Interface (MPI) to run multiple parallel molecular dynamics simulations within a parent MPI job to optimize heat transfer at the silicon and aluminum (Si/Al) interface. We found a maximum of 50% increase in the ITC when introducing a two-layer intermixed region that consists of a higher percentage of Si. Because of the random nature of the intermixing, the magnitude of increase in the ITC varies. We observed that both homogeneity/heterogeneity of the intermixing and the intrinsic stochastic nature of molecular dynamics simulations account for the variance in ITC.
Domain Shift Analysis in Chest Radiographs Classification in a Veterans Healthcare Administration Population
This study aims to assess the impact of domain shift on chest X-ray classification accuracy and to analyze the influence of ground truth label quality and demographic factors such as age group, sex, and study year. We used a DenseNet121 model pre-trained MIMIC-CXR dataset for deep learning-based multi-label classification using ground truth labels from radiology reports extracted using the CheXpert and CheXbert Labeler. We compared the performance of the 14 chest X-ray labels on the MIMIC-CXR and Veterans Healthcare Administration chest X-ray dataset (VA-CXR). The validation of ground truth and the assessment of multi-label classification performance across various NLP extraction tools revealed that the VA-CXR dataset exhibited lower disagreement rates than the MIMIC-CXR datasets. Additionally, there were notable differences in AUC scores between models utilizing CheXpert and CheXbert. When evaluating multi-label classification performance across different datasets, minimal domain shift was observed in the unseen VA dataset, except for the label “Enlarged Cardiomediastinum.” The subgroup with the most significant variations in multi-label classification performance was study year. These findings underscore the importance of considering domain shift in chest X-ray classification tasks, paying particular attention to the temporality of the exam. Our study reveals the significant impact of domain shift and demographic factors on chest X-ray classification, emphasizing the need for improved transfer learning and robust model development. Addressing these challenges is crucial for advancing medical imaging research and improving patient care.
Expanding the Domain of Applicability of Machine Learning Models with Limited Data for Drug Property Prediction
Accurate machine learning models for predicting small molecule interactions with biological targets are essential for therapeutic discovery, biothreat response, and computational drug design, but their performance is often limited for understudied targets with sparse experimental data. To address this challenge, we developed and evaluated methods to improve molecular property prediction under low-data conditions, using the NimA-related kinase (NEK) family as a proof-of-concept. This work focused on two complementary goals within the ATOM Modeling PipeLine (AMPL) and the Generative Molecular Design (GMD) loop: expanding model applicability through transfer learning, representation learning, feature scaling, sampling strategies, and active-learning-inspired compound selection; and enabling efficient virtual screening to prioritize compounds that balance predicted activity, design objectives, and synthetic accessibility.
Array-Based Machine Learning for Functional Group Detection in Electron Ionization Mass Spectrometry
Mass spectrometry is a ubiquitous technique capable of complex chemical analysis. The fragmentation patterns that appear in mass spectrometry are an excellent target for artificial intelligence methods to automate and expedite the analysis of data to identify targets such as functional groups. To develop this approach, we trained models on electron ionization (a reproducible hard fragmentation technique) mass spectra so that not only the final model accuracies but also the reasoning behind model assignments could be evaluated. The convolutional neural network (CNN) models were trained on 2D images of the spectra using transfer learning of Inception V3, and the logistic regression models were trained using array-based data and Scikit Learn implementation in Python. Our training dataset consisted of 21,166 mass spectra from the United States’ National Institute of Standards and Technology (NIST) Webbook. The data was used to train models to identify functional groups, both specific (e.g., amines, esters) and generalized classifications (aromatics, oxygen-containing functional groups, and nitrogen-containing functional groups). We found that the highest final accuracies on identifying new data were observed using logistic regression rather than transfer learning on CNN models. It was also determined that the mass range most beneficial for functional group analysis is 0–100 m/z. We also found success in correctly identifying functional groups of example molecules selected from both the NIST database and experimental data. Beyond functional group analysis, we also have developed a methodology to identify impactful fragments for the accurate detection of the models’ targets. The results demonstrate a potential pathway for analyzing and screening substantial amounts of mass spectral data.
Applying deep learning methods to develop new models of molecular charge transfer, nonadiabatic dynamics, and nonlinear spectroscopy in the condensed phase
Photon- and field-induced charge transfer has central importance in the generation and storage of electricity, the novel properties of materials, photo-induced catalysis, and electro-optic activity (e.g., photovoltaic cells, fuel cells, and organic chromophores for use in optical fibers and light-emission diodes). These non-equilibrium electronic and chemical transformations are probed by ultrafast, nonlinear spectroscopies. Accurate simulations play a crucial role in our ability to understand, optimize, and control these transformations. This project applies modern deep learning and machine learning (ML) methods to dramatically improve models of electronic dynamics, electronic-nuclear dynamics, and spectroscopic measurements for improved simulations of chemistry in complex environments, far from equilibrium phenomena, and processes in extreme environments, such as materials exposed to strong or resonant fields. This project develops accurate neural net models that go beyond predictive capability to also provide new insight into the fundamental physics underlying electron and nuclear dynamics. To achieve its objectives, this project explores and develops customized versions of high-capacity deep learning algorithms/models. These techniques are developed with an emphasis on fundamental chemical insight, not just predictive accuracy, to assist the development of the next generation of quantum simulation methods.
Toward a Generalizable Prediction Model of Molten Salt Mixture Density with Chemistry‐Informed Transfer Learning
Optimally designing applications of molten salts requires knowledge of their thermophysical properties over a wide range of temperatures and compositions. There exist significant gaps in existing databases and this data can be challenging to experimentally measure due to high temperatures, salt corrosivity, and salt hygroscopicity. Existing databases have been used to create Redlich–Kister (RK) models for mixture density showing improved accuracy with respect to ideal mixing assumptions, but these models require subcomponent data measurements for each new system, therefore lacking generality. In order to address generalizability and data sparsity, a transfer learning procedure is proposed to train deep neural networks (DNNs) using a combination of semi‐empirical relationships (RK), data from the thermophysical arm of the molten salt thermal properties database and universal ab initio properties of component mixtures taken from the joint automated repository for various integrated simulations (JARVIS) classical force‐field inspired descriptors database to predict density in molten salts. Herein, it is shown that DNNs predict molten salt density with an r 2 over 0.99 and a mean absolute percentage error under 1%, outperforming alternative methods.
Post Irradiation Examination Dislocation Defect Detection Software
This software provides dislocation-type defect identification and segmentation using a standard open source computer vision model, YOLOv8, that leverages transfer learning to create a highly effective dislocation defect quantification tool while using only a minimal number of expert annotated micrographs for training. This model demonstrates the ability to segment both dislocation lines and loops concurrently in micrographs with high pixel noise levels and on multiple alloys. It includes multiple layers of frozen layers used for transfer learning from multidisciplinary data and is extensible to alloys that are not included in the training dataset.
Hierarchical transfer learning: an agile and equitable strategy for machine-learning interatomic models
Machine-learned interatomic models are growing in popularity due to their ability to afford near quantum-accurate predictions for complex phenomena with orders-of-magnitude greater computational efficiency. However, these models struggle when applied to systems of many element types due to the approximately exponential increase in number of parameters that must be determined. To mitigate this challenge, we present a new hierarchical transfer learning approach that allows the fitting problem to be decomposed into smaller independent and reusable parameter blocks that enable development of explicitly chemically extensible ML-IAM. Application of this strategy is demonstrated for C and N mixtures under conditions ranging from nominally ambient to ~10,000 K and 200 GPa for compositions from 0 to 100% N. Ultimately, this strategy makes model generation for chemically complex systems more tractable and efficient, facilitates comprehensive model validation, and makes ML-IAM development for problems of this nature more accessible to users with limited access to extreme computing infrastructure.