SEARCH · Engineering Papers
Results for “pruning”
Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.
Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.
“Understanding Robustness Lottery”: A Geometric Visual Comparative Analysis of Neural Network Pruning Approaches
Deep learning approaches have provided state-of-the-art performance in many applications by relying on large and overparameterized neural networks. However, such networks are very brittle and are difficult to deploy on resource-limited platforms. Model pruning, i.e., reducing the size of the network, is a widely adopted strategy that can lead to a more robust and compact model. Many heuristics exist for model pruning, but our understanding of the pruning process remains limited due to the black-box nature of a neural network model. Empirical studies show that some heuristics improve performance whereas others can make models more brittle. Here, this work aims to shed light on how different pruning methods alter the network’s internal feature representation and the corresponding impact on model performance. To facilitate a comprehensive comparison and characterization of the high-dimensional model feature space, we introduce a visual geometric analysis of feature representations. We evaluated a set of critical geometric concepts decomposed from the commonly adopted classification loss and used them to design a visualization system to compare and highlight the impact of pruning on model performance and feature representation. The proposed tool provides an environment for an in-depth comparison of pruning methods and a comprehensive understanding of how the model responds to common data corruption. By leveraging the proposed visualization, machine learning researchers can reveal the similarities between pruning methods and redundancy in robustness evaluation benchmarks, obtain geometric insights about the differences between pruned models that achieve superior robustness performance, and identify samples that are robust or fragile to model pruning and common data corruption.
Ps and Qs: Quantization-Aware Pruning for Efficient Low Latency Neural Network Inference
Efficient machine learning implementations optimized for inference in hardware have wide-ranging benefits, depending on the application, from lower inference latency to higher data throughput and reduced energy consumption. Two popular techniques for reducing computation in neural networks are pruning, removing insignificant synapses, and quantization, reducing the precision of the calculations. In this work, we explore the interplay between pruning and quantization during the training of neural networks for ultra low latency applications targeting high energy physics use cases. Techniques developed for this study have potential applications across many other domains. We study various configurations of pruning during quantization-aware training, which we term quantization-aware pruning, and the effect of techniques like regularization, batch normalization, and different pruning schemes on performance, computational complexity, and information content metrics. We find that quantization-aware pruning yields more computationally efficient models than either pruning or quantization alone for our task. Further, quantization-aware pruning typically performs similar to or better in terms of computational efficiency compared to other neural architecture search techniques like Bayesian optimization. Surprisingly, while networks with different training configurations can have similar performance for the benchmark application, the information content in the network can vary significantly, affecting its generalizability.
network-pruner (Neural network pruning analysis) [SWR-25-113]
This repository implements an iterative magnitude pruning algorithm for pruning neural networks in PyTorch. The pruning method involves gradually removing less significant weights from the model to achieve a specified sparsity, followed by fine-tuning the pruned model to recover performance.
Attack-Resilient Weighted $\ell_{1}$ Observer with Prior Pruning
Security related questions for Cyber Physical Systems (CPS) have attracted much research attention in searching for novel methods for attack-resilient control and/or estimation. Specifically, false data injection attacks (FDIAs) have been shown to be capable of bypassing bad data detection (BDD), while arbitrarily compromising the integrity of state estimators and robust controller even with very sparse measurements corruption. Moreover, based on the inherent sparsity of pragmatic attack signals, ℓ1 -minimization scheme has been used extensively to improve the design of attack-resilient estimators. For this, the theoretical maximum for the percentage of compromised nodes that can be accommodated has been shown to be 50%. In order to guarantee correct state recoveries for larger percentage of attacked nodes, researchers have begun to incorporate prior information into the underlying resilient observer design framework. For the most pragmatic cases, this prior information is often obtained through some data-driven machine learning process. Existing results have shown strong positive correlation between the tolerated attack percentages and the precision of the prior information. In this paper, we present a pruning method to improve the precision of the prior information, given corresponding stochastic uncertainty characteristics of the underlying machine learning model. Then a weighted ℓ1 -minimization is proposed based on the pruned prior. The theoretical and simulation results show that the pruning method significantly improves the observer performance for much larger attack percentages, even when moderately accurate machine learning model used.
SymbolNet: neural symbolic regression with adaptive dynamic pruning for compression
Abstract Compact symbolic expressions have been shown to be more efficient than neural network (NN) models in terms of resource consumption and inference speed when implemented on custom hardware such as field-programmable gate arrays (FPGAs), while maintaining comparable accuracy (Tsoi et al 2024 EPJ Web Conf. 295 09036). These capabilities are highly valuable in environments with stringent computational resource constraints, such as high-energy physics experiments at the CERN Large Hadron Collider. However, finding compact expressions for high-dimensional datasets remains challenging due to the inherent limitations of genetic programming (GP), the search algorithm of most symbolic regression (SR) methods. Contrary to GP, the NN approach to SR offers scalability to high-dimensional inputs and leverages gradient methods for faster equation searching. Common ways of constraining expression complexity often involve multistage pruning with fine-tuning, which can result in significant performance loss. In this work, we propose S y m b o l N e t , a NN approach to SR specifically designed as a model compression technique, aimed at enabling low-latency inference for high-dimensional inputs on custom hardware such as FPGAs. This framework allows dynamic pruning of model weights, input features, and mathematical operators in a single training process, where both training loss and expression complexity are optimized simultaneously. We introduce a sparsity regularization term for each pruning type, which can adaptively adjust its strength, leading to convergence at a target sparsity ratio. Unlike most existing SR methods that struggle with datasets containing more than O ( 10 ) inputs, we demonstrate the effectiveness of our model on the LHC jet tagging task (16 inputs), MNIST (784 inputs), and SVHN (3072 inputs).
Accelerating Transformer-based Deep Learning Models on FPGAs using Column Balanced Block Pruning
Although Transformer-based language representations achieve state-of-the-art accuracy on various natural language processing (NLP) tasks, the large model size has been challenging the resource constrained computing platforms. Weight pruning, as a popular and effective technique in reducing the number of weight parameters and accelerating the Transformer, has been investigated on GPUs. However, the Transformer acceleration using weight pruning on field-programmable gate array (FPGAs) remains unexplored. This paper investigates the column balanced block-wise pruning on Transformer and designs an FPGA acceleration engine to customize the balanced blockwise matrix multiplication. We implement the Transformer model with proper hardware scheduling, and the experiments show that the Transformer inference on FPGA achieves 10.35 ms latency with the batch size of 32, which is 10.96 × speed up comparing to CPU platform and 2.08 × speed up comparing to GPU platform.
Resilient Observer Design for Cyber-Physical Systems with Data-Driven Measurement Pruning
Resilient observer design for Cyber-Physical Systems (CPS) in the presence of adversarial false data injection attacks (FDIA) is an active area of research. The existing state-of-the-art algorithms tend to break down as more and more knowledge of the system is built into the attack model; also as the percentage of attacked nodes increases. From the view of optimization theory, the problem is often cast as a classical error correction problem for which a theoretical limit of has been established as the maximum percentage attacked nodes for which state recovery is guaranteed. Beyond this limit, the performance of -minimization based schemes, for instance, deteriorates rapidly. Similar performance degradation occurs for other types of resilient observers beyond certain percentages of attacked nodes. In order to increase the corresponding percentage of attacked nodes for which state recoveries can be guaranteed, researchers have begun to incorporate prior information into the underlying resilient observer design framework. For the most pragmatic cases, this prior information is often obtained through a data-driven machine learning process. Existing results have shown a strong positive correlation between the maximum attacked percentages that can be tolerated and the accuracy of the data-driven model. Motivated by these results, this chapter examines the case for pruning algorithms designed to improve the Positive Prediction Value (PPV) of the resulting prior information, given stochastic uncertainty characteristics of the underlying machine learning model. Theoretical quantification of the achievable improvement is given. Simulation results show that the pruning algorithm significantly increases the maximum correctable percentage of attacked nodes, even for machine learning model whose prediction power is comparable to the random flip of a coin.
Fast Iterative Multi-site Hosting Capacity Analysis for Distribution Systems With Search Space Pruning
Interconnection studies for distributed energy resources (DERs) is a time-intensive process, primarily due to the necessity of solving large number of power flow scenarios. Hosting capacity analysis (HCA) is a time-consuming aspect of interconnection studies that is divided into single-site HCA (SHCA) and multi-site HCA (MHCA). From a computational and understandable standpoint, the industry seeks iteration-based solutions for SHCA, although it doesn't maximize the total DER hosting capacity (DERHC) of the grid, as MHCA does. While non-iterative solutions are available for MHCA, they involve a trade-off between the modeling accuracy of the distribution system, solution quality, and ease of understanding. In this work, we present a fast iterative solution for MHCA, reducing computational complexity by eliminating the need to solve power flows for a large amount of search space, thus making iterative solutions feasible. This iterative approach guarantees both a global optimal solution with sufficient time and a fast, close-to-optimal solution through efficient search space pruning. It also easily integrates with existing utility HCA tools. The results are demonstrated on select locations in the IEEE-123 bus system for community-scale interconnection studies. We highlight the benefits of skipping the need to solve millions of power flows, all while maximizing the grid's total DERHC.
Enhancement of Distribution System State Estimation Using Pruned Physics-Aware Neural Networks
Realizing complete observability in the three-phase distribution system remains a challenge that hinders the implementation of classic state estimation algorithms. In this paper, a new method, called the pruned physics-aware neural network (P2N2), is developed to improve the voltage estimation accuracy in the distribution system. The method relies on the physical grid topology, which is used to design the connections between different hidden layers of a neural network model. To verify the proposed method, a numerical simulation based on one- year smart meter data of load consumptions for three-phase power flow is developed to generate the measurement and voltage state data. The IEEE 123-node system is selected as the test network to benchmark the proposed algorithm against the classic weighted least squares (WLS). Numerical results show that P2N2 outperforms WLS in terms of data redundancy and estimation accuracy.
prune_quant_vit
This tool provides codes to prune and quantize vision transformers (ViTs). The tool will allow developers and researchers to speed up inference of ViTs and deploy them on CPUs.
Enhancement of Distribution System State Estimation Using Pruned Physics-Aware Neural Networks: Preprint
Realizing complete observability in the three-phase distribution system remains a challenge that hinders the implementation of classical state estimation algorithms. In this paper, a new method so-called pruned physics-aware neural network (P2N2) is developed to improve the voltage estimation accuracy in the distribution system. The method relies on the physical grid topology, which is used to design the connections between different hidden layers of a neural network model. To verify the proposed method, a numerical simulation based on one-year smart meter data of load consumptions for threephase power flow is developed to generate the measurement and voltage state data. The IEEE 123 node system is selected as the test network to benchmark the proposed algorithm against the classical weighted least squares (WLS). Numerical results show that P2N2 outperforms WLS, in terms of data redundancy and estimation accuracy.
Fine-Pruning: A biologically inspired algorithm for personalization of machine learning models
Not Available
Locally purified maximally mixed states at scale: Entanglement pruning and symmetries
Locally Purified Density Operators (LPDOs) are state-of-the-art tensor network ansatze candidates that efficiently represent mixed quantum states at scale. However, given their non-uniqueness, their representational complexity is generally sub-optimal in practical computations. Here, in this work we perform a comprehensive numerical and analytical analysis and resolve this issue in the experimentally relevant limit where noise depolarizes the density operator into a maximally mixed state. To resolve the sub-optimality issue, we analyze two numerical tools, one analytic method, and detail the relations between them. The numerical tools used are fidelity-preserving truncations and isometric gauge transformations leveraging Riemannian optimizations over entropic objective functions. In addition, by invoking the injectivity and symmetry constraints of the maximally mixed LPDO, we also present analytical closed-form expressions for the disentangler and discuss their relation to numerical optimizers. Further, away from the maximally mixed state, our simulations highlight how the truncation threshold smoothly interpolate, as a function of depolarization, between established matrix product results and our new results. Our work shows how, by minimizing the resources required to represent key states of practical interest in experiment, the efficiency of tensor network algorithms can be substantially increased. This paves the path for uncovering tensor network’s fundamental scalability limits and latent potential in representing the wide locus of mixed quantum states that are accessible on near-term quantum devices.
Adaptive pruning-based optimization of parameterized quantum circuits
Abstract Variational hybrid quantum–classical algorithms are powerful tools to maximize the use of noisy intermediate-scale quantum devices. While past studies have developed powerful and expressive ansatze, their near-term applications have been limited by the difficulty of optimizing in the vast parameter space. In this work, we propose a heuristic optimization strategy for such ansatze used in variational quantum algorithms, which we call ‘parameter-efficient circuit training (PECT)’. Instead of optimizing all of the ansatz parameters at once, PECT launches a sequence of variational algorithms, in which each iteration of the algorithm activates and optimizes a subset of the total parameter set. To update the parameter subset between iterations, we adapt the Dynamic Sparse Reparameterization scheme which was originally proposed for training deep convolutional neural networks. We demonstrate PECT for the Variational Quantum Eigensolver, in which we benchmark unitary coupled-cluster ansatze including UCCSD and k -UpCCGSD, as well as the Low-Depth Circuit Ansatz (LDCA), to estimate ground state energies of molecular systems. We additionally use a layerwise variant of PECT to optimize a hardware-efficient circuit for the Sycamore processor to estimate the ground state energy densities of the one-dimensional Fermi-Hubbard model. From our numerical data, we find that PECT can enable optimizations of certain ansatze that were previously difficult to converge and more generally can improve the performance of variational algorithms by reducing the optimization runtime and/or the depth of circuits that encode the solution candidate(s).
The Generalization-Stability Tradeoff In Neural Network Pruning
Explore the source record for details and available documents.
Calibrate and Prune: Improving Reliability of Lottery Tickets Through Prediction Calibration
Explore the source record for details and available documents.
PQV-Mobile: A Combined Pruning and Quantization Toolkit to Optimize Vision Transformers for Mobile Applications
Explore the source record for details and available documents.