Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “hyperparameter”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 19 records
All sourcesNASA NTRSDOE OSTI
DOE OSTI2024

Two-Step Hyperparameter Optimization Method: Accelerating Hyperparameter Search by Using a Fraction of a Training Dataset

Abstract Hyperparameter optimization (HPO) is an important step in machine learning (ML) model development, but common practices are archaic—primarily relying on manual or grid searches. This is partly because adopting advanced HPO algorithms introduces added complexity to the workflow, leading to longer computation times. This poses a notable challenge to ML applications, as suboptimal hyperparameter selections curtail the potential of ML model performance, ultimately obstructing the full exploitation of ML techniques. In this article, we present a two-step HPO method as a strategic solution to curbing computational demands and wait times, gleaned from practical experiences in applied ML parameterization work. The initial phase involves a preliminary evaluation of hyperparameters on a small subset of the training dataset, followed by a reevaluation of the top-performing candidate models postretraining with the entire training dataset. This two-step HPO method is universally applicable across HPO search algorithms, and we argue it has attractive efficiency gains. As a case study, we present our recent application of the two-step HPO method to the development of neural network emulators for aerosol activation. Although our primary use case is a data-rich limit with many millions of samples, we also find that using up to 0.0025% of the data—a few thousand samples—in the initial step is sufficient to find optimal hyperparameter configurations from much more extensive sampling, achieving up to 135× speedup. The benefits of this method materialize through an assessment of hyperparameters and model performance, revealing the minimal model complexity required to achieve the best performance. The assortment of top-performing models harvested from the HPO process allows us to choose a high-performing model with a low inference cost for efficient use in global climate models (GCMs).

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2020

Hyperparameter Optimization in Binary Communication Networks for Neuromorphic Deployment

Training neural networks for neuromorphic deployment is non-trivial. There have been a variety of approaches proposed to adapt back-propagation or back-propagation-like algorithms appropriate for training. Considering that these networks often have very different performance characteristics than traditional neural networks, it is often unclear how to set either the network topology or the hyperparameters to achieve optimal performance. In this work, we introduce a Bayesian approach for optimizing the hyperparameters of an algorithm for training binary communication networks that can be deployed to neuromorphic hardware. We show that by optimizing the hyperparameters on this algorithm for each dataset, we can achieve improvements in accuracy over the previous state-of-the-art for this algorithm on each dataset (by up to 15 percent). This jump in performance continues to emphasize the potential when converting traditional neural networks to binary communication applicable to neuromorphic hardware.

Parsa, Maryam↗
DOE OSTI2023

Protecting and Defending against Autonomous Control Systems and Digital Twin Cyber Attacks: Response Strategy for Hyperparameter attacks of Digital Twin Machine Learning Models in Nuclear Power Plants (Final)

Navigating through the complex tapestry of technological advancements, "Response Strategy for Hyperparameter attacks of Digital Twin Machine Learning Model in Nuclear Power Plants" stands at the intersection of cybersecurity and nuclear power plant operations, embarking on a journey through the intricacies of securing digital twins against malicious cyber activities. As nuclear power plants progressively integrate digital twin technology and machine learning models to optimize operations and ensure system reliability, they inadvertently expose themselves to a new spectrum of vulnerabilities, notably in the realm of hyperparameter attacks. Hyperparameters, integral in machine learning model tuning and optimal performance of digital twins, have emerged as a target for adversaries aiming to destabilize the predictive capabilities and therefore, the operational accuracy of these digital entities within critical infrastructures like nuclear plants. This paper, therefore, meticulously threads the needle through the development of a robust response strategy, poised to shield these digital reflections against calculated hyperparameter manipulations, ensuring that the digital twin can effectively and securely function as a reliable proxy for its physical counterpart. The ensuing sections delve into the orchestrated maelstrom of multi-rate time-changing intelligent coordinated hyperparameter attacks and the implementation of event-triggered predictive control, laying down a structured, predictive, and responsive framework that safeguards the nexus where the digital and physical realms of nuclear power plants coalesce. The operational integrity of digital twins in nuclear power plants depends critically on the security of machine learning hyperparameters. This study makes two different contributions. First, a decision-based idea known as a multi-rate time changing intelligent coordinated hyperparameter attack is put forth. In this attack, many hyperparameters are repeatedly changed using both random and intelligent optimal techniques by the attacker. These assaults introduce varied rates at different attack steps, compromise various amounts of hyperparameters, and improve stealth and flexibility. Second, a technique is developed for event triggered predictive control to rapidly respond to potential hyperparameter attacks. This control integrates a sliding window framework, retaining a history of previous data points and employing linear regression to predict the next data point from the current dataset. The control gain K is determined using the Lyapunov-Krasovskii method, and subsequently, an action is developed. Finally, the outcome of the simulation demonstrates the viability of the proposed method for defending nuclear power plant digital twins from hyperparameter attacks.

22 GENERAL STUDIES OF NUCLEAR REACTORS↗
DOE OSTI2026

The Sensitivity of Variational Bayesian Neural Network Performance to Hyperparameters

In scientific applications, predictive modeling is often of limited use without accurate uncertainty quantification (UQ) to indicate when a model may be extrapolating or when more data needs to be collected. Bayesian Neural Networks (BNNs) produce predictive uncertainty by propagating uncertainty in neural network (NN) weights and offer the promise of obtaining not only an accurate predictive model but also accurate UQ. However, in practice, obtaining accurate UQ with BNNs is difficult due in part to the approximations used for model training (such as those made in variational inference) and in part to the need to choose a suitable set of hyperparameters; these hyperparameters outnumber those needed for traditional NNs and often have opaque effects on the results. We aim to shed light on the effects of hyperparameter choices for variational BNNs by performing a global sensitivity analysis of variational BNN performance under varying hyperparameter settings. Our results indicate that many of the hyperparameters interact with each other to affect both predictive accuracy and UQ. For improved usage of variational BNNs in real-world applications, we suggest that thorough hyperparameter tuning, including tuning of prior hyperparameters and loss function parameters, is essential for accurate UQ in variational BNNs.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2024

Streamlining Ocean Dynamics Modeling with Fourier Neural Operators: A Multiobjective Hyperparameter and Architecture Optimization Approach

Training an effective deep learning model to learn ocean processes involves careful choices of various hyperparameters. We leverage DeepHyper’s advanced search algorithms for multiobjective optimization, streamlining the development of neural networks tailored for ocean modeling. The focus is on optimizing Fourier neural operators (FNOs), a data-driven model capable of simulating complex ocean behaviors. Selecting the correct model and tuning the hyperparameters are challenging tasks, requiring much effort to ensure model accuracy. DeepHyper allows efficient exploration of hyperparameters associated with data preprocessing, FNO architecture-related hyperparameters, and various model training strategies. We aim to obtain an optimal set of hyperparameters leading to the most performant model. Moreover, on top of the commonly used mean squared error for model training, we propose adopting the negative anomaly correlation coefficient as the additional loss term to improve model performance and investigate the potential trade-off between the two terms. The numerical experiments show that the optimal set of hyperparameters enhanced model performance in single timestepping forecasting and greatly exceeded the baseline configuration in the autoregressive rollout for long-horizon forecasting up to 30 days. Utilizing DeepHyper, we demonstrate an approach to enhance the use of FNO in ocean dynamics forecasting, offering a scalable solution with improved precision.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2021

Multi-Objective Hyperparameter Optimization for Spiking Neural Network Neuroevolution

Neuroevolution has had significant success over recent years, but there has been relatively little work applying neuroevolution approaches to spiking neural networks (SNNs). SNNs are a type of neural network that includes temporal processing component, are not easily trained using other methods, and can be deployed into energy-efficient neuromorphic hardware. In this work, we investigate two evolutionary approaches for training SNNs. We explore the impact of the hyperparameters of the evolutionary approaches, including tournament size, population size, and representation type, on the performance of the algorithms. We present a multi-objective Bayesian-based hyperparameter optimization approach to tune the hyperparameters to produce the most accurate and smallest SNNs. We show that the hyperparameters can significantly affect the performance of these algorithms. We also perform sensitivity analysis and demonstrate that every hyperparameter value has the potential to perform well, assuming other hyperparameter values are set correctly.

Parsa, Maryam↗
NASA NTRSDate not supplied

A Convex Optimization Approach to Improving Suboptimal Hyperparameters of Sliced Normal Distributions

Sliced Normal (SN) distributions are a generalization of Gaussian distributions where the quadratic argument of the exponential is replaced with a sum of squares polynomial. SNs may be used to represent the distribution of a diverse set of random variables including multi-modal, non-symmetric, and skewed distributions. Unfortunately, the likelihood function of a SN includes a normalization constant and the inclusion of this normalization constant makes the likelihood a non-convex function of the hyperparameters which define the SN. In previous work, suboptimal fitting of the hyperparameters was performed by transforming the given data into a higher dimensional monomial basis and selecting the optimal hyperparameters of a Gaussian fit in this space. However, this approach did not account for the effect of lifting on the normalization constant. Indeed, it was observed that as the number of monomials is increased the likelihood of the Sliced Normal can decrease. In this paper, we increase the likelihood of Sliced Normals found using the previous method by developing a convex formulation which scales the covariance matrix of the Gaussian fit such that the likelihood of the Sliced Normal is maximized. The result is significant improvements of the log likelihood of fitted SN distributions, including a significant increase, especially for problems with 500+ monomials.

Convex optimization approach to improving suboptim↗
DOE OSTI2021

Adaptive hyperparameter updating for training restricted Boltzmann machines on quantum annealers

Restricted Boltzmann Machines (RBMs) have been proposed for developing neural networks for a variety of unsupervised machine learning applications such as image recognition, drug discovery, and materials design. The Boltzmann probability distribution is used as a model to identify network parameters by optimizing the likelihood of predicting an output given hidden states trained on available data. Training such networks often requires sampling over a large probability space that must be approximated during gradient based optimization. Quantum annealing has been proposed as a means to search this space more efficiently which has been experimentally investigated on D-Wave hardware. D-Wave implementation requires selection of an effective inverse temperature or hyperparameter (β) within the Boltzmann distribution which can strongly influence optimization. Here, we show how this parameter can be estimated as a hyperparameter applied to D-Wave hardware during neural network training by maximizing the likelihood or minimizing the Shannon entropy. We find both methods improve training RBMs based upon D-Wave hardware experimental validation on an image recognition problem. Neural network image reconstruction errors are evaluated using Bayesian uncertainty analysis which illustrate more than an order magnitude lower image reconstruction error using the maximum likelihood over manually optimizing the hyperparameter. The maximum likelihood method is also shown to out-perform minimizing the Shannon entropy for image reconstruction.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2021

HYPPO: A Surrogate-Based Multi-Level Parallelism Tool for Hyperparameter Optimization

We present a new software, HYPPO, that enables the automatic tuning of hyperparameters of various deep learning (DL) models. Unlike other hyperparameter optimization (HPO) methods, HYPPO uses adaptive surrogate models and directly accounts for uncertainty in model predictions to find accurate and reliable models that make robust predictions. Using asynchronous nested parallelism, we are able to significantly alleviate the computational burden of training complex architectures and quantifying the uncertainty. HYPPO is implemented in Python and can be used with both TensorFlow and PyTorch libraries. We demonstrate various software features on time-series prediction and image classification problems as well as a scientific application in computed tomography image reconstruction. Finally, we show that (1) we can reduce by an order of magnitude the number of evaluations necessary to find the most optimal region in the hyperparameter space and (2) we can reduce by two orders of magnitude the throughput for such HPO process to complete.

adaptation models↗
DOE OSTI2022

HPS-RL: Hyperparameter tuning for deep RL applications (HPS-RL) v1

Genetic Algorithms meets Deep RL for Hyperparameters Hyperparameter optimization and architecture search can easily become cumbersome and finding the right hyperparameters can seriously impact the robustness of the deep RL application being developed. We use genetic algorithms to evolve optimum deep RL architectures in a scalable manner. HPS-RL is designed to work with multiple gym enviornments, allow users to test their own optimization functions and tune multi-objective parameters in multiple deep RL algorithms. HPS-RL uses multi-threading and is being extended with mpipy for distributed processing on HPC. https://arxiv.org/abs/2201.11182

Kiran, Mariam↗
DOE OSTI2024

The Value of Hyperparameter Optimization in Phase-Picking Neural Networks

The effectiveness of neural networks for picking seismic phase arrival times has been demonstrated through several case studies, and seismic monitoring programs are starting to adopt the technology into their workflows. However, published models were designed and trained using rather arbitrary choices of hyperparameters, limiting their performance. In this study, we use phase picks from both routine and template-matching analyses from multiple regions (Ridgecrest, California; Kilauea, Hawaii; Yellowstone, Wyoming–Montana–Idaho) to test a hyperparameter optimization scheme for phase-picking neural networks and to evaluate their performance. We show that a published model, namely PhaseNet (Zhu and Beroza, 2019), can be simplified and improved with reasonable effort and there are preferred choices of hyperparameters that increase the performance. We also show that models optimized based on the arrival times reported in routine event catalogs consistently perform well when picking arrival times of smaller events, which is crucial for many tasks from microseismicity to explosion monitoring.

58 GEOSCIENCES↗
DOE OSTI2025

DeepHyper: A Python Package for Massively Parallel Hyperparameter Optimization in Machine Learning

Machine learning models are increasingly applied across scientific disciplines, yet their effectiveness often hinges on heuristic decisions—such as data transformations, training strategies, and model architectures—that are not learned by the models themselves. Automating the selection of these heuristics and analyzing their sensitivity is crucial for building robust and efficient learning workflows. DeepHyper addresses this challenge by democratizing hyperparameter optimization, providing accessible tools to streamline and enhance machine learning workflows from a laptop to the largest supercomputer in the world. Building on top of hyperparameter optimization, it unlocks new capabilities around ensembles of models for improved accuracy and uncertainty quantification. All of these organized around efficient parallel computing.

ensemble↗
DOE OSTI2023

Multiobjective Hyperparameter Optimization for Deep Learning Interatomic Potential Training Using NSGA-II

Deep neural network (DNN) potentials are an emerging tool for simulation of dynamical atomistic systems, with the promise of quantum mechanical accuracy at speedups of 10000$\times$. As with other DNN methods, hyperparameters used during training can make a substantial difference in model accuracy, and optimal settings vary with dataset. To enable rapid tuning of hyperparameters for DNN potential training, we developed a scalable multiobjective optimization evolutionary algorithm for supercomputers and tested it on the Summit system at the Oak Ridge Leadership Computing Facility (OLCF). The multiobjective approach is required due to the coupling of two learned values defining the potential: the energy and force. Using a large-scale implementation of the NSGA-II algorithm adapted for training DNN potentials, we discovered several optimal multiobjective combinations, including best choices of activation functions, learning rate scaling scheme, and pairing of the two radial cutoffs used in the three dimensional descriptor function.

Coletti, Mark↗
DOE OSTI2020

Genetic Algorithm for Hyperparameter Optimization in Gaussian Process Modeling

A genetic algorithm is developed and applied to optimize hyperparameters of convolutional recursively determined dual neural network-Gaussian process (NNGP) kernels. As a specific application of the combined GPNN-GA algorithm, it is applied to image classification in publicly available data of Hyper Suprime-Cam Subaru Strategic Program. Matthews correlation coefficient is calculated based on results of binary star-galaxy classification and used as a fitting function of the GA module of the algorithm. The simulation results confirm significant improvement of the classification accuracy with optimized hyperparameters.

79 ASTRONOMY AND ASTROPHYSICS↗
DOE OSTI2021

Hyperparameter Setting for a Marked Multidimensional Hawkes Process with Dissimilar Decays

We provide further details for using Lim, et al.'s marked multidimensional Hawkes processes with dissimilar decays. We first describe what makes these different from other Hawkes processes, then describe each model hyperparameter and how to initialize it informed by the input data and any prior biases. We derived tighter bounds than Lim, et al. for faster convergence of rejection sampling. The resulting hyperparameters and bounds have been helpful against both synthetic and real-world datasets.

73 NUCLEAR PHYSICS AND RADIATION PHYSICS↗
DOE OSTI2020

Bi-cross validation of spectral clustering hyperparameters

One challenge impeding the analysis of terabyte scale X-ray scattering data from the Linac Coherent Light Source (LCLS) is determining the number of clusters required for the execution of traditional clustering algorithms. Here, we demonstrate that the previous work using bi-cross validation to determine the number of singular vectors directly maps to the spectral clustering problem of estimating both the number of clusters and hyperparameter values. Here by applying this method to LCLS X-ray scattering data enables the identification of dropped shots without manually setting boundaries on detector fluence and provides a path toward identifying rare and anomalous events.

71 CLASSICAL AND QUANTUM MECHANICS, GENERAL PHYSIC↗
DOE OSTI2021

Accurate and Accelerated Neuromorphic Network Design Leveraging A Bayesian Hyperparameter Pareto Optimization Approach

Neuromorphic systems allow for extremely efficient hardware implementations for neural networks (NNs). In recent years, several algorithms have been presented to train spiking NNs (SNNs) for neuromorphic hardware. However, SNNs often provide lower accuracy than their artificial NNs (ANNs) counterparts or require computationally expensive and slow training/inference methods. To close this gap, designers typically rely on reconfiguring SNNs through adjustments in the neuron/synapse model or training algorithm itself. Nevertheless, these steps incur significant design time, while still lacking the desired improvement in terms of training/inference times (latency). Designing SNNs that can mimic the accuracy of ANNs with reasonable training times is an exigent challenge in neuromorphic computing. In this work, we present an alternative approach that looks at such designs as an optimization problem rather than algorithm or architecture redesign. We develop a versatile multiobjective hyperparameter optimization (HPO) for automatically tuning HPs of two state-of-the-art SNN training algorithms, SLAYER and HYBRID. We emphasize that, to the best of our knowledge, this is the first work trying to improve SNNs’ computational efficiency, accuracy, and training time using an efficient HPO. We demonstrate significant performance improvements for SNNs on several datasets without the need to redesign or invent new training algorithms/architectures. Our approach results in more accurate networks with lower latency and, in turn, higher energy efficiency than previous implementations. In particular, we demonstrate improvement in accuracy and more than 5× reduction in the training/inference time for the SLAYER algorithm on the DVS Gesture dataset. In the case of HYBRID, we demonstrate 30% reduction in timesteps while surpassing the accuracy of the state-of-the-art networks on CIFAR10. Further, our analysis suggests that even a seemingly minor change in HPs could change the accuracy by 5 - 6×.

Parsa, Maryam↗
DOE OSTI2024

Hyperparameter Studies for Vision Transformers Trained on High-Fidelity Simulations

This library is a collection of python modules that define, train, and analyze vision-transformer (ViT) machine learning models. The code implements, with mild modifications, ViT models that have been made publicly available through publication and GitHub code. The training data for these models is hydrodynamic simulation output in the form of numpy arrays. This library contains code to train these ViT models on the hydrodynamic simulation output with a variety of hyperparameters, and to compare the results of such models. Furthermore, the library contains definitions of simple convolutional neural network (CNN) machine learning architectures which can be trained on the same hydrodynamic simulation output. These are included as a reference point to compare the ViT models to. Additionally, the library includes trained ViT and CNN models and example input data for demonstration purposes. The code is based on the PyTorch python library.

Callis, Skylar↗