Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “trainability”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 55 records · Page 3
All sourcesNASA NTRSDOE OSTI
DOE OSTI2023

Theory of overparametrization in quantum neural networks

The prospect of achieving quantum advantage with quantum neural networks (QNNs) is exciting. Understanding how QNN properties (for example, the number of parameters $M$) affect the loss landscape is crucial to designing scalable QNN architectures. Here we rigorously analyze the overparametrization phenomenon in QNNs, defining overparametrization as the regime where the QNN has more than a critical number of parameters $M_c$ allowing it to explore all relevant directions in state space. In this study, our main results show that the dimension of the Lie algebra obtained from the generators of the QNN is an upper bound for $M_c$, and for the maximal rank that the quantum Fisher information and Hessian matrices can reach. Underparametrized QNNs have spurious local minima in the loss landscape that start disappearing when $M$ ≥ $M_c$. Thus, the overparametrization onset corresponds to a computational phase transition where the QNN trainability is greatly improved. We then connect the notion of overparametrization to the QNN capacity, so that when a QNN is overparametrized, its capacity achieves its maximum possible value.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2022

Quantum machine learning for chemistry and physics

Machine learning (ML) has emerged as a formidable force for identifying hidden but pertinent patterns within a given data set with the objective of subsequent generation of automated predictive behavior. In recent years, it is safe to conclude that ML and its close cousin, deep learning (DL), have ushered in unprecedented developments in all areas of physical sciences, especially chemistry. Not only classical variants of ML, even those trainable on near-term quantum hardwares have been developed with promising outcomes. Such algorithms have revolutionized materials design and performance of photovoltaics, electronic structure calculations of ground and excited states of correlated matter, computation of force-fields and potential energy surfaces informing chemical reaction dynamics, reactivity inspired rational strategies of drug designing and even classification of phases of matter with accurate identification of emergent criticality. In this review we shall explicate a subset of such topics and delineate the contributions made by both classical and quantum computing enhanced machine learning algorithms over the past few years. We shall not only present a brief overview of the well-known techniques but also highlight their learning strategies using statistical physical insight. The objective of the review is not only to foster exposition of the aforesaid techniques but also to empower and promote cross-pollination among future research in all areas of chemistry which can benefit from ML and in turn can potentially accelerate the growth of such algorithms.

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗
DOE OSTI2024

Quantum subspace expansion in the presence of hardware noise

Finding ground state energies on current quantum processing units (QPUs) using algorithms such as the variational quantum eigensolver (VQE) continues to pose challenges. Hardware noise severely affects both the expressivity and trainability of parameterized quantum circuits, limiting them to shallow depths in practice. Here, we demonstrate that both issues can be addressed by synergistically integrating VQE with a quantum subspace expansion, allowing for an optimal balance between quantum and classical computing capabilities and costs. We perform a systematic benchmark analysis of the iterative quantum-assisted eigensolver in the presence of hardware noise. We determine ground state energies of 1D and 2D mixed-field Ising spin models on noisy simulators and the IBM QPUs ibmq_quito (5 qubits) and ibmq_guadalupe (16 qubits). To maximize accuracy, we propose a suitable criterion to select the subspace basis vectors according to the trace of the noisy overlap matrix. Finally, we show how to systematically approach the exact solution by performing controlled quantum error mitigation based on probabilistic error reduction on the noisy backend fake_guadalupe.

71 CLASSICAL AND QUANTUM MECHANICS, GENERAL PHYSIC↗
DOE OSTI2026

Pre-training Vision Models for the Classification of Alerts from Wide-field Time-domain Surveys

Modern wide-field time-domain surveys facilitate the study of transient, variable and moving phenomena by conducting image differencing and relaying alerts to their communities. Machine learning tools have been used on data from these surveys and their precursors for more than a decade, and convolutional neural networks (CNNs), which make predictions directly from input images, saw particularly broad adoption through the 2010s. Since then, continually rapid advances in computer vision have transformed the standard practices around using such models. It is now commonplace to use standardized architectures pre-trained on large corpora of everyday images (e.g., ImageNet). In contrast, time-domain astronomy studies still typically design custom CNN architectures and train them from scratch. Here, we explore the effects of adopting various pre-training regimens and standardized model architectures on the performance of alert classification. We find that the resulting models match or outperform a custom, specialized CNN like what is typically used for filtering alerts. Moreover, our results show that pre-training on galaxy images from Galaxy Zoo tends to yield better performance than pre-training on ImageNet or training from scratch. We observe that the design of standardized architectures are much better optimized than the custom CNN baseline, requiring significantly less time and memory for inference despite having more trainable parameters. On the eve of the Legacy Survey of Space and Time and other image-differencing surveys, these findings advocate for a paradigm shift in the creation of vision models for alerts, demonstrating that greater performance and efficiency, in time and in data, can be achieved by adopting the latest practices from the computer vision field.

79 ASTRONOMY AND ASTROPHYSICS↗
DOE OSTI2021

Large gradients via correlation in random parameterized quantum circuits

Scaling of variational quantum algorithms to large problem sizes requires efficient optimization of random parameterized quantum circuits. For such circuits with uncorrelated parameters, the presence of exponentially vanishing gradients in cost function landscapes is an obstacle to optimization by gradient descent methods. In this work, we prove that reducing the dimensionality of the parameter space by utilizing circuit modules containing spatially or temporally correlated gate layers can allow one to circumvent the vanishing gradient phenomenon. Here, examples are drawn from random separable circuits and asymptotically optimal variational versions of Grover's algorithm based on the quantum alternating operator ansatz. In the latter scenario, our bounds on cost function variation imply a transition between vanishing gradients and efficient trainability as the number of layers is increased toward $\mathcal{O}\left({2}^{n/2}\right)$, the optimal oracle complexity of quantum unstructured search.

71 CLASSICAL AND QUANTUM MECHANICS, GENERAL PHYSIC↗
DOE OSTI2022

Photometric redshifts from SDSS images with an interpretable deep capsule network

ABSTRACT Studies of cosmology, galaxy evolution, and astronomical transients with current and next-generation wide-field imaging surveys like the Rubin Observatory Legacy Survey of Space and Time are all critically dependent on estimates of photometric redshifts. Capsule networks are a new type of neural network architecture that is better suited for identifying morphological features of the input images than traditional convolutional neural networks. We use a deep capsule network trained on ugriz images, spectroscopic redshifts, and Galaxy Zoo spiral/elliptical classifications of ∼400 000 Sloan Digital Sky Survey galaxies to do photometric redshift estimation. We achieve a photometric redshift prediction accuracy and a fraction of catastrophic outliers that are comparable to or better than current methods for SDSS main galaxy sample-like data sets (r ≤ 17.8 and zspec ≤ 0.4) while requiring less data and fewer trainable parameters. Furthermore, the decision-making of our capsule network is much more easily interpretable as capsules act as a low-dimensional encoding of the image. When the capsules are projected on a two-dimensional manifold, they form a single redshift sequence with the fraction of spirals in a region exhibiting a gradient roughly perpendicular to the redshift sequence. We perturb encodings of real galaxy images in this low-dimensional space to create synthetic galaxy images that demonstrate the image properties (e.g. size, orientation, and surface brightness) encoded by each dimension. We also measure correlations between galaxy properties (e.g. magnitudes, colours, and stellar mass) and each capsule dimension. We publicly release our code, estimated redshifts, and additional catalogues at https://biprateep.github.io/encapZulate-1.

79 ASTRONOMY AND ASTROPHYSICS↗
DOE OSTI2026

Space-Charge-Limited van der Waals Spin Transistor

Integrating semiconducting and magnetic materials could combine transistorlike operation with nonvolatility and enable architectures such as logic in memory. Here, in this work, we employ correlated electrical transport and scanning nitrogen-vacancy center magnetic imaging to elucidate a spin transistor concept that amalgamates vertical and lateral hopping transport inside a 2D antiferromagnetic semiconductor, mechanistically distinct from vertical tunneling devices. Our device, based on a monolayer-bilayer junction in CrSBr, displays giant, gate-tunable magnetoresistance driven by the dual action of electrostatic doping on space-charge-limited lateral conduction and interlayer exchange coupling. Moreover, we visualize a field-trainable, layer-sharing effect that selects between coherent or domain-wall reversal at the spin-flip transition, enabling multilevel, memristive conductance states. Our layer-dependent space charge mechanism for convergent electrical and magnetic control opens opportunities to address limitations in contemporary computing.

2-dimensional systems↗
DOE OSTI2023

Proximity-induced equilibrium supercurrent and perfect superconducting diode effect due to band asymmetry

Herein we theoretically investigate the consequences of proximity-induced conventional superconductivity in metals that break time-reversal and inversion symmetries through their energy dispersion. We discover behaviors impossible in an isolated superconductor such as an equilibrium supercurrent that apparently violates a no-go theorem and, at suitable topological defects, nonconservation of electric charge reminiscent of the chiral anomaly. The equilibrium supercurrent is expected to be trainable by a helical electromagnetic field in the normal state. Remarkably, if the band asymmetry exceeds the critical current of the parent superconductor in appropriate units, we predict a perfect superconducting diode effect with diode coefficient unity. We propose toroidal metals such as UNi 4 ⁢B and metals with directional scalar spin chiral order as potential platforms.

75 CONDENSED MATTER PHYSICS, SUPERCONDUCTIVITY AND↗
DOE OSTI2021

Universal Compiling and (No-)Free-Lunch Theorems for Continuous-Variable Quantum Learning

Quantum compiling, where a parameterized quantum circuit is trained to learn a target unitary, is an important primitive for quantum computing that can be used as a subroutine to obtain optimal circuits or as a tomographic tool to study the dynamics of an experimental system. While much attention has been paid to quantum compiling on discrete-variable hardware, less has been paid to compiling in the continuous-variable paradigm. Here we motivate several, closely related, short-depth continuous-variable algorithms for quantum compilation. We analyze the trainability of our proposed cost functions and numerically demonstrate our algorithms by learning arbitrary Gaussian operations and Kerr nonlinearities. We further make connections between this framework and quantum learning theory in the continuous-variable setting by deriving no-free-lunch theorems. These generalization bounds demonstrate a linear resource reduction for learning Gaussian unitaries using entangled coherent-Fock states and an exponential resource reduction for learning arbitrary unitaries using two-mode-squeezed states.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2022

Group-Invariant Quantum Machine Learning

Quantum machine learning (QML) models are aimed at learning from data encoded in quantum states. Recently, it has been shown that models with little to no inductive biases (i.e., with no assumptions about the problem embedded in the model) are likely to have trainability and generalization issues, especially for large problem sizes. As such, it is fundamental to develop schemes that encode as much information as available about the problem at hand. In this work we present a simple, yet powerful, framework where the underlying invariances in the data are used to build QML models that, by construction, respect those symmetries. These so-called group-invariant models produce outputs that remain invariant under the action of any element of the symmetry group $\mathfrak{G}$ associated with the dataset. We present theoretical results underpinning the design of $\mathfrak{G}$-invariant models, and exemplify their application through several paradigmatic QML classification tasks, including cases when $\mathfrak{G}$ is a continuous Lie group and also when it is a discrete symmetry group. Notably, our framework allows us to recover, in an elegant way, several well-known algorithms for the literature, as well as to discover new ones. Taken together, we expect that our results will help pave the way towards a more geometric and group-theoretic approach to QML model design.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2024

Theory for Equivariant Quantum Neural Networks

Quantum neural network architectures that have little to no inductive biases are known to face trainability and generalization issues. Inspired by a similar problem, recent breakthroughs in machine learning address this challenge by creating models encoding the symmetries of the learning task. This is materialized through the usage of equivariant neural networks the action of which commutes with that of the symmetry. In this work, we import these ideas to the quantum realm by presenting a comprehensive theoretical framework to design equivariant quantum neural networks (EQNNs) for essentially any relevant symmetry group. We develop multiple methods to construct equivariant layers for EQNNs and analyze their advantages and drawbacks. Our methods can find unitary or general equivariant quantum channels efficiently even when the symmetry group is exponentially large or continuous. As a special implementation, we show how standard quantum convolutional neural networks (QCNNs) can be generalized to group-equivariant QCNNs where both the convolution and pooling layers are equivariant to the symmetry group. We then numerically demonstrate the effectiveness of a S U ( 2 ) -equivariant QCNN over symmetry-agnostic QCNN on a classification task of phases of matter in the bond-alternating Heisenberg model. Our framework can be readily applied to virtually all areas of quantum machine learning. Lastly, we discuss about how symmetry-informed models such as EQNNs provide hopes to alleviate central challenges such as barren plateaus, poor local minima, and sample complexity. Published by the American Physical Society 2024

71 CLASSICAL AND QUANTUM MECHANICS, GENERAL PHYSIC↗
DOE OSTI2021

Physics-Informed Evolutionary Strategy Based Control for Mitigating Delayed Voltage Recovery

Here, in this work we propose a novel data-driven, real-time power system voltage stability control method based on the physics-informed guided meta evolutionary strategy (ES). The main objective is to quickly provide an adaptive control strategy to secure system voltage stability. The problem is challenging due to the high-dimensional feature of the power system model and the fast-changing and uncertain nature of power system operation scenarios. To this end, a model-free and derivative-free guided ES method is applied. The method is further combined with a meta-learning strategy to make the learnt control policy automatically adapted to unseen operation conditions and fault scenarios, which is highly desired for real-time emergency control. Last but not least, physical knowledge is embedded in the above method through a trainable action mask technique to rule out unnecessary load shedding actions for better learning and control performance. Case studies on the IEEE 300-bus system and comparisons with other state-of-the-art benchmark methods verify the superiority of the proposed physics-informed guided meta ES method in realizing fast and adaptive power system voltage stability control.

42 ENGINEERING↗
DOE OSTI2025

A Two-Stage Quantum Reinforcement Learning Method for Multi-Objective Transmission Switching

Multi-objective transmission switching (MO-TS) problems involve the strategic reconfiguration of network topology to simultaneously optimize multiple objectives. As the system scale increases, finding feasible solutions becomes increasingly challenging due to the problem's nonlinearity and high computational complexity. To address these challenges, this paper proposes a two-stage quantum reinforcement learning method that leverages potential quantum advantages for MO-TS. In the first stage, candidate switching lines are identified using a graph-theoretical approach to reduce the problem's dimensionality. The second stage introduces a quantum-classical reinforcement learning framework, where a learnable measurement-based CNN-ResVQC architecture is developed to effectively reduce the input dimension for quantum processing, mitigate vanishing gradients, and enhance trainability while improving the quantum circuit's flexibility in modeling complex decision policies for MO-TS. Numerical studies on IEEE 14-bus, 57-bus, and 118-bus systems demonstrate that the proposed algorithm achieves superior training stability and faster convergence with approximately 1% of the network parameters required by classical algorithms, highlighting its effectiveness, efficiency, and scalability. Furthermore, the practicality is validated through its stable convergence under three common quantum noise channels.

99 GENERAL AND MISCELLANEOUS↗
DOE OSTI2022

Analyses of internal structures and defects in materials using physics-informed neural networks

Characterizing internal structures and defects in materials is a challenging task, often requiring solutions to inverse problems with unknown topology, geometry, material properties, and nonlinear deformation. Here, we present a general framework based on physics-informed neural networks for identifying unknown geometric and material parameters. By using a mesh-free method, we parameterize the geometry of the material using a differentiable and trainable method that can identify multiple structural features. We validate this approach for materials with internal voids/inclusions using constitutive models that encompass the spectrum of linear elasticity, hyperelasticity, and plasticity. We predict the size, shape, and location of the internal void/inclusion as well as the elastic modulus of the inclusion. Our general framework can be applied to other inverse problems in different applications that involve unknown material properties and highly deformable geometries, targeting material characterization, quality assurance, and structural design.

36 MATERIALS SCIENCE↗
DOE OSTI2021

MLPF: efficient machine-learned particle-flow reconstruction using graph neural networks

In general-purpose particle detectors, the particle-flow algorithm may be used to reconstruct a comprehensive particle-level view of the event by combining information from the calorimeters and the trackers, significantly improving the detector resolution for jets and the missing transverse momentum. In view of the planned high-luminosity upgrade of the CERN Large Hadron Collider (LHC), it is necessary to revisit existing reconstruction algorithms and ensure that both the physics and computational performance are sufficient in an environment with many simultaneous proton–proton interactions (pileup). Machine learning may offer a prospect for computationally efficient event reconstruction that is well-suited to heterogeneous computing platforms, while significantly improving the reconstruction quality over rule-based algorithms for granular detectors. We introduce MLPF, a novel, end-to-end trainable, machine-learned particle-flow algorithm based on parallelizable, computationally efficient, and scalable graph neural network optimized using a multi-task objective on simulated events. We report the physics and computational performance of the MLPF algorithm on a Monte Carlo dataset of top quark–antiquark pairs produced in proton–proton collisions in conditions similar to those expected for the high-luminosity LHC. The MLPF algorithm improves the physics response with respect to a rule-based benchmark algorithm and demonstrates computationally scalable particle-flow reconstruction in a high-pileup environment.

71 CLASSICAL AND QUANTUM MECHANICS, GENERAL PHYSIC↗
DOE OSTI2020

Toward Large-Scale Image Segmentation on Summit

Semantic segmentation of images is an important computer vision task that emerges in a variety of application domains such as medical imaging, robotic vision and autonomous vehicles to name a few. While these domain-specific image analysis tasks involve relatively small image sizes (~ 10 2 × 10 2 ), there are many applications that need to train machine learning models on image data with extents that are orders of magnitude larger (~10 4 × 10 4 ). Training deep neural network (DNN) models on large extent images is extremely memory-intensive and often exceeds the memory limitations of a single graphical processing unit, a hardware accelerator of choice for computer vision workloads. Here, an efficient, sample parallel approach to train U-Net models on large extent image data sets is presented. Its advantages and limitations are analyzed and near-linear strong-scaling speedup demonstrated on 256 nodes (1536 GPUs) of the Summit supercomputer. Using a single node of the Summit supercomputer, an early evaluation of a recently released model parallel framework called GPipe is demonstrated to deliver ~ 2X speedup in executing a U-Net model with an order of magnitude larger number of trainable parameters than reported before. Performance bottlenecks for pipelined training of U-Net models are identified and mitigation strategies to improve the speedups are discussed. Together, these results open up the possibility of combining both approaches into a unified scalable pipelined and data parallel algorithm to efficiently train U-Net models with very large receptive fields on data sets of ultra-large extent images.

Seal, Sudip↗
DOE OSTI2025

ATTNChecker: Highly-Optimized Fault Tolerant Attention for Large Language Model Training

Large Language Models (LLMs) have demonstrated remarkable performance in various natural language processing tasks. However, the training of these models is computationally intensive and susceptible to faults, particularly in the attention mechanism, which is a critical component of transformer-based LLMs. In this paper, we investigate the impact of faults on LLM training, focusing on INF, NaN, and near-INF values in the computation results with systematic fault injection experiments. We observe the propagation patterns of these errors, which can trigger non-trainable states in the model and disrupt training, forcing the procedure to load from checkpoints. To mitigate the impact of these faults, we propose ATTNChecker, the first Algorithm-Based Fault Tolerance (ABFT) technique tailored for the attention mechanism in LLMs. ATTNChecker is designed based on fault propagation patterns of LLM and incorporates performance optimization to adapt to both system reliability and model vulnerability while providing lightweight protection for fast LLM training. Evaluations on four LLMs show that ATTNChecker on average incurs on average 7% overhead on training while detecting and correcting all extreme errors. Compared with the state-of-the-art checkpoint/restore approach, ATTNChecker reduces recovery overhead by up to 49×.

Liang, Yuhang [University of Alabama - Birmingham]↗
DOE OSTI2025

CAFE AU LAIT: Compute-Aware Federated Augmented Low-Rank AI Training

Federated finetuning is crucial for unlocking the knowledge embedded in pretrained Large Language Models (LLMs) when data are geographically distributed across clients. Unlike finetuning with data from a single institution, federated finetuning allows collaboration across multiple institutions, enabling the utilization of diverse and decentralized datasets while preserving data privacy. Given the high computing costs of LLM training and the emphasis on energy efficiency in Federated Learning (FL), Low-Rank Adaptation (LoRA) has emerged as a widely adopted algorithm due to its significantly reduced number of trainable parameters. However, this assumes that all data silos have the necessary computing resources to compute local updates of LLMs. Nevertheless, in practice, the computing resources across clients are highly heterogeneous: while some may have access to hundreds of GPUs, others might have limited or no GPU access. Recently, federated finetuning using synthetic data has been proposed, allowing clients to participate in a collaborative training run without training LLMs locally. However, our experimental results reveal a performance gap between models trained using synthetic data and those trained using local updates. Motivated by the observed heterogeneity in computing resources and the performance gap, we propose a novel two-stage algorithm that leverages the storage and computing capabilities of a strong server. In the first stage, under the coordination of the strong server, clients with limited computing resources collaborate to generate synthetic data, which is transferred to and stored on the strong server. In the second stage, the strong server uses this synthetic data on behalf of the resource-constrained clients to perform federated LoRA finetuning alongside clients with sufficient computing resources. This approach ensures that all clients can participate in the finetuning process. Experimental results demonstrate that incorporating local updates from even a small fraction of clients improves performance compared to using synthetic data for all clients. Furthermore, we incorporate the Gaussian mechanism in both stages to guarantee client-level differential privacy.

Wang, Jiayi [ORNL]↗