Engineering PapersSearch

SEARCH · Engineering Papers

Results for “hyperparameter optimization”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 19 records

Automated Data Accountability for Missions in Mars Rover Data

As the Mars Curiosity Rover transmits data to the JPL Ground Data System (GDS), it frequently observes data loss and corruption, requiring re-transmits from the rover and Ground Data System Analysts (GDSA) to monitor the downlink process. As new missions are launched, the GDSA team redistributes analysts to these new missions, causing shortages in previous missions. The GDSA team can significantly benefit from the automation and optimization of the downlink process of telemetry data. In fact, there is a need for a better understanding of why the data is corrupted, so that the GDSA team can best determine the root cause of the issues in the GDS. This paper presents machine learning and deep learning based approaches to automate and optimize the detection of data loss. We first created a pipeline to automatically accumulate data from the telemetry databases (MAROS, Telemetry Data Storage, and GDS Elastic Search Database) in the downlink process. With our newly created datasets, we perform feature selection to supplement the GDSA understanding of the downlink process and provide supplemental analysis on the importance of different features. We implement various machine learning and deep learning based models, including support vector machines, ensemble methods, and deep neural networks and evaluate their accuracies in identifying whether a downlink process is complete or incomplete. We utilize fast hyperparameter optimization methods that allow our models to quickly be re-trained, allowing them to quickly be tuned and optimized on daily incoming data in real time. This hyperparameter optimization also allows our methods to be quickly integrated into other JPL missions. Our results show that our best-performing machine learning and deep learning based models outperform the existing GDSA detection software by 6 accuracy points and can aid analysts by providing insights into the data accountability problem. Since these various machine learning and deep learning approaches vary significantly in interpretability, we provide a discussion on the tradeoffs between their performance and trustworthiness in helping detect issues in data transmission.

Divsalar, Dariush

Reinforcement Learning Applied to Cognitive Space Communications

The future of space exploration depends on robust, reliable communication systems. As the number of such communication systems increase, automation is fast becoming a requirement to achieve this goal. A reinforcement learning solution can be employed as a possible automation method for such systems. The goal of this study is to build a reinforcement learning algorithm which optimizes data throughput of a single actor. A training environment was created to simulate a link within the NASA Space Communication and Navigation (SCaN) infrastructure, using state of the art simulation tools developed by the SCaN Center for Engineering, Networks, Integration, and Communications (SCENIC) laboratory at NASA Glenn Research Center to obtain the closest possible representation of the real operating environment. Reinforcement learning was then used to train an agent inside this environment to maximize data throughput. The simulation environment contained a single actor in low earth orbit capable of communicating with twenty-five ground stations that compose the Near-Earth Network (NEN). Initial experiments showed promising training results, so additional complexity was added by augmenting simulation data with link fading profiles obtained from real communication events with the International Space Station. A grid search was performed to find the optimal hyperparameters and model architecture for the agent. Using the results of the grid search, an agent was trained on the augmented training data. Testing shows that the agent performs well inside the training environment and can be used as a foundation for future studies with added complexity and eventually tested in the real space environment.

Schubert, Carson D.

An Ensemble Neural Network Model for Predicting Rare-Earth Oxide and Silicate Heat Capacities at High Temperature

In this work, a neural network model was developed to predict the constant pressure heat capacity for materials in the rare-earth oxide—silica material space. Several model architectures were trained and tested on heat capacity data generated from first-principles density functional theory calculations. Hyperparameter optimization was performed, and the optimal model was selected for heat capacity predictions. The optimal model architecture was found to have a root-mean-squared error of 5.12 ± 3.37 J/mol-K. The optimal model architecture was then used in a bagging ensemble model trained using the leave-one-group-out method to provide error estimates for model predictions. The out-of-bag score for the ensemble model was 0.997. The predicted heat capacities agree well with the DFT and experimental results and were computed orders of magnitude faster than DFT simulations. Machine learning shows the potential to provide a suitable surrogate model for thermochemical property predictions for candidate environmental barrier coating materials but refining of input material features and model architectures could further improve accuracy for these models.

environmental barrier coatings

Lunar Browser Utilization of Machine Learning for Trajectory Solution Production

This paper describes the application of machine learning tools to produce Earth-Moon spacecraft trajectories with applications to NASA’s Commercial Lunar Payload Services (CLPS) and Artemis Human Landing System (HLS) programs. Existing trajectory solutions are used to train and test machine learning models to predict essential details of a trajectory sequence from Earth-launch to Low-Lunar Orbit, populating a database of solutions with future launch dates. The machine learning model will implement hyperparameter optimization for further re-training to improve model performance. Accurate predictive models decrease the time required to produce solutions and are readily implemented in the Lunar Browser tool.

Trajectory Design

A Convex Optimization Approach to Improving Suboptimal Hyperparameters of Sliced Normal Distributions

Sliced Normal (SN) distributions are a generalization of Gaussian distributions where the quadratic argument of the exponential is replaced with a sum of squares polynomial. SNs may be used to represent the distribution of a diverse set of random variables including multi-modal, non-symmetric, and skewed distributions. Unfortunately, the likelihood function of a SN includes a normalization constant and the inclusion of this normalization constant makes the likelihood a non-convex function of the hyperparameters which define the SN. In previous work, suboptimal fitting of the hyperparameters was performed by transforming the given data into a higher dimensional monomial basis and selecting the optimal hyperparameters of a Gaussian fit in this space. However, this approach did not account for the effect of lifting on the normalization constant. Indeed, it was observed that as the number of monomials is increased the likelihood of the Sliced Normal can decrease. In this paper, we increase the likelihood of Sliced Normals found using the previous method by developing a convex formulation which scales the covariance matrix of the Gaussian fit such that the likelihood of the Sliced Normal is maximized. The result is significant improvements of the log likelihood of fitted SN distributions, including a significant increase, especially for problems with 500+ monomials.

Convex optimization approach to improving suboptim

Automated Pneumothorax Diagnosis using Deep Neural Networks

Thoracic ultrasound can provide information leading to rapid diagnosis of pneumothorax with improved accuracy over the standard physical examination and with higher sensitivity than anteroposterior chest radiography. However, the clinical We have Furthermore, remote environments, such as the battlefield or deep-space exploration, may lack expertise for diagnosing developed an automated image interpretation pipeline for the analysis of thoracic ultrasound data and the classification of pneumothorax events to provide decision support in such situations. Our pipeline consists of image preprocessing, data augmentation, and deep learning architectures for medical diagnosis. In this work, we demonstrate that robust, accurate interpretation of chest images and video can be achieved using deep neural networks. A number of novel image processing techniques were employed to achieve this result. Affine transformations were applied for data augmentation. Hyperparameters were optimized for learning rate, dropout regularization, batch size, and epoch iteration by a sequential model-based Bayesian approach. In addition, we utilized pretrained architecturesinterpretation of a patient medical image is highly operator dependent. certain pathologies., applying transfer learning and fine-tuning techniques to fully connected layers. Our pipeline yielded binary classification validation accuracies of 98.3% for M-mode images and 99.8% with B-mode video frames.

US Army collaboration

Exploring Applications of Machine Learning for Wildfire Monitoring and Detection using Unmanned Aerial Vehicles

Wildfires are increasing in frequency and severity around the world, including the United States. The losses caused by wildfires could be mitigated if high-risk areas, hotspots, and flare-ups could be monitored continuously, such as through the use of Unmanned Aerial Vehicles (UAVs). This paper documents exploratory efforts using machine learning to determine efficient flight paths for UAVs and to detect wildfires using image classification. On path planning, three machine learning techniques—Genetic Algorithm, Simulated Annealing, and Dynamic Programming—were explored. Genetic Algorithm was found to be an effective approach for path planning for wildfire monitoring and surveillance by UAVs. For a scenario of 25 locations in a circular arrangement, the algorithm was able to return the optimal path. The accuracy and execution time was found to be sensitive to the algorithm hyperparameters selected, which was especially evident in scenarios with hundreds or thousands of locations. Simulated Annealing was also found to be an effective approach for UAV path planning, with a major benefit of avoiding getting trapped in local minima and being straightforward to implement. Like Genetic Algorithm, the performance of Simulated Annealing was also found to be sensitive to the algorithm hyperparameters selected. By comparison, Dynamic Programming guarantees optimality for any number of locations, but it was found to be less practical in terms of execution time for scenarios with more than about a couple dozen locations. On wildfire detection, image classification using deep learning with a convolutional neural network was explored. Transfer learning was found to be a useful technique to efficiently train deep learning models. Also, it was determined that GPU processing can increase training speed by an order of magnitude, which enables significantly faster development. For a validation test set of 500 images, there were only two false negatives and zero false positives. These results demonstrate that detecting wildfires in static cameras using machine learning is feasible and establish a baseline for using images captured by UAVs in flight for wildfire detection.

Wildfire management

Towards an Aviation Large Language Model by Fine-tuning and Evaluating Transformers

In the aviation domain, there are many applications for machine learning and artificial intelligence tools that utilize natural language. For example, there is a desire to know the commonalities in written safety reports such as voluntary post incidents reports or aerial wildfire operations reports to better understand the risks present. Another use-case is the possibility of extracting airspace procedures and constraints currently written in documents such as Letters of Agreement. These applications can benefit from the use of state-of-the-art natural language processing techniques when adapted to the language/phraseology specific to the aviation domain. This paper evaluates the viability of adaptation of NLP tools to the aviation domain by fine-tuning transformer based models using aviation data sets. In 2018, a novel language model based on neural units (also called transformers) was created and became known as “Bidirectional Encoder Representations from Transformers” or BERT. This architecture combined with large amounts of English training data and innovative semi-supervised training tasks set the standard for what would later emerge as Large Language Models. The performance of these models was further improved by hyperparameter tuning and refinement of the semi-supervised training task and resulted in “Robustly Optimized BERT Pre-training Approach through hyperparameter tuning” or RoBERTa models. These pre-trained Large Language Models proved to be useful for a wide variety of natural language processing tasks such as text classification and question answering through a process called fine-tuning. The transformer architecture with pre-trained weights served as the basis with the last few layers replaced with layers fine-tuned to perform a new task e.g., a layer that provides a label for the entire input text. This process of fine-tuning can also be used to adapt the models to new domains; e.g., BioBERT started with the pre-trained BERT model and was completed by additional fine-tuning and training on biomedical documents. Transformer-based architectures can also be used to create rich representations of text called embeddings which can serve as the input to other machine learning models. This allows simpler algorithms such as logistic regression to use context-rich representations of the text while still remaining quick to train and evaluate. In the world of aviation, there is a growing demand for natural language processing and understanding but the domain presents unique challenges. Due to the technical content (and specialized language) of most aviation documents, fine-tuning pre-trained Large Language Models to specific tasks has not met the benchmark on natural language processing tasks set by simpler models trained from scratch on the data. To address this deficiency, this paper evaluates the improvements from fine-tuning a Large Language Model on a large set of aviation documents using the original semi-supervised training tasks before performing specific natural language tasks. In fine-tuning, a domain-specific dataset is used on the original training task but with the pre-trained Large Language Model instead of starting from a random initialization. This approach allows the model to be adapted to the specific domain language without discarding the information gained from training on general English data. This paper utilized two major dataset types to train and assess the RoBERTa fine-tuning performance. The first are 7,057 Letters of Agreement which are Federal Aviation Administration (FAA) documents that formalize airspace operations across the national airspace system. They contain many examples of ‘aviation English’ using domain specific terminology and phrasing which serves as a representative basis to perform the semi-supervised fine-tuning. The second type is the 494 document classification labels to be used for evaluation. This down-stream evaluation aims to show the performance of the fine-tuned model, better understand how much data is needed for an effective fine-tuning, and how fine-tuning can be adapted for different applications in-the domain. After semi-supervised training, evaluation begins by encoding the documents for classification using the fine-tuned RoBERTa model. Then a logistic regression classifier is trained to label the document type and compared against our ground truth labels. This currently leads to a 82.8% accuracy on 10-fold cross validation showing improvement over baseline RoBERTa which achieved 81.0%. We plan to measure the improvements on additional tasks and it is expected that these improvements will lead to more robust models that can tackle the natural language processing challenges present in aviation datasets.

ATM

Predicting Unreinforced Fabric Mechanical Behavior with Recurrent Neural Networks

Unreinforced woven fabrics are widely employed in various high-performance applications, including parachute deployment systems, airbags, and ballistic armor. The analysis of such materials is inherently complex due to the multiscale structure of these materials, and the dependence of macroscale behavior on changes that occur at lower scales. Previously, NASA’s Multiscale Analysis Tool (NASMAT) showed its capability in predicting unreinforced fabric behavior at the macroscale by capturing finite rotations that occur at the mesoscale. Though effective, the tool can face high computational cost for large, complex problems, motivating the need for the development of a surrogate model that can capture the same behavior. A recurrent neural network (RNN) was developed and trained on virtual NASMAT data to mimic the physics-based solutions while improving the computational runtime. The architecture of the RNN to best simulate the fabric behavior was carefully crafted based on heuristic knowledge of predicting physics-based temporal data, manual hyperparameter case studies, and Hyperband optimization.. The resultant model was able to predict a variety of stress-strain curves for fabrics with different mesoscale geometries, and was further validated by comparing to experimental data for the K706 style Kevlar plain-weave fabric, demonstrating the ability of the model to effectively capture the geometric changes in the fabric without explicitly calculating them, as is done in NASMAT. Furthermore, the tool showed its ability to improve on the runtime by a factor of 10 for fabric solutions compared to the multiscale tool, which would further enable the simulation of complex loading scenarios on unreinforced fabrics.

Fabric

A Generalized Approach to Aircraft Trajectory Prediction via Supervised Deep Learning

As research advances diverse forms and missions of aircraft, the National Airspace System (NAS) will become increasingly crowded, limiting current communications resources to accommodate aviation operations. Ongoing research proposes a paradigm of airspace communications, such that resources are autonomously and dynamically allocated via intelligent agents; this allocation requires accurate representations of the NAS, including the predicted positions of aircraft. State-of-the-art research emphasizes the importance of a hybrid-recurrent framework for trajectory prediction and compares the impact of commonly considered weather data on prediction accuracy. However, current research has been limited in its scope of efforts, frequently utilizing a unique flight route, architecture, set of weather data, and date range. This article considers the challenges of generalizing hybrid-recurrent predictive models for flight trajectories. Results illustrate an increase in error variance when identical models are trained over a generalized set of flights; this may be mitigated with careful tuning of hyperparameters, both in the network structure and optimization algorithms. Even so, an irreducible vertical error was identified, resulting from the complex takeoff and landing procedures which can not be correlated to functions of weather or additional assumptions of aircraft behavior. Finally, the use of a test route indicates that generalized models still do not possess sufficient knowledge for general aircraft predictions, with mean error increases ranging from 70-500%. These results illustrate the need for continued efforts on improving model versatility, as well as potential limitations for spectrum allocation near airports and other centers.

Nathan Schimpf

Tuning a variational autoencoder for data accountability problem in the Mars Science Laboratory ground data system

The Mars Curiosity rover is frequently sending back engineering and science data that goes through a pipeline of systems before reaching its final destination at the mission operations center making it prone to volume loss and data corruption. A ground data system analysis (GDSA) team is charged with the monitoring of this flow of information and the detection of anomalies in that data in order to request a re-transmission when necessary. This work presents ∆-MADS, a derivative-free optimization method applied for tuning the architecture and hyperparameters of a variational autoencoder trained to detect the data with missing patches in order to assist the GDSA team in their mission.

Lakhmiri, Dounia

Use of Design of Experiments and Rule-Based Inference in Determining Neural Network Architectures for Loss of Control Detection

In this work, we describe methods for selecting the neural network architectures and input spaces to implement belief state inference on generic commercial transport aircraft. First, we highlight a case study on the planning, execution, and analysis of a set of experiments to determine the configurations of a conditional variational autoencoder (CVAE). We present a structured method that can be used in a number of aerospace applications, to optimize the structure and training parameters of the CVAE for belief state inference, using Design of Experiments (DOE) statistical methodologies. The motivation for this specific DOE was to identify the appropriate hyperparameters for measuring the CVAE reconstruction probability and latent space, such that the measurements can be used to infer qualitative state changes for the aircraft. We demonstrate that this process yields information about a trained neural network’s utility for this specific application, along with a quantifiable range of certainty. We execute 84 experiments using loss-of-control flight maneuver data from a NASA T-2 aircraft, demonstrating that this empirical process allows us to construct cheap and simple models with specific attributes amenable to belief state inference in aerospace applications. While theoretically, we could create a single CVAE with an input space the size of all measurable flight variables and environmental dynamics, it becomes intractable to use such a neural network in an in-situ intelligent multi-agent system. Using the recommendations from our case study, we introduce a technical approach for feasibly describing the belief space by (1) identifying significant statistical relationships among flight variables using rule induction, (2) using a set of rules that cover all features to define the input space of multiple CVAEs, and (3) forming a belief space based on the joint probability density of their collective latent spaces. This results in a series of relatively small matrix multiplications that can be performed in real time, as opposed to large matrix computations in a single CVAE. We demonstrate the application of this approach on the T-2 flight loss-of control experiments, using the architecture and hyperparameter recommendations from the case study. We compare the utilities of an individual CVAE trained on all flight variables and multiple CVAEs defined on subsets of flight variables for detecting qualitative changes in flight. We demonstrate that the use of multiple CVAEs with smaller input spaces permits the CVAE to capture more granular relationships in the latent space, permitting better state space characterization and loss-of-control detection.

Design of experiments

Use of Design of Experiments in Determining Neural Network Architectures for Loss of Control Detection

We describe empirical methods for selecting a neural network architecture to implement belief state inference on generic commercial transport aircraft. We highlight a case study on the planning, execution, and analysis of a set of experiments to determine the configurations of a conditional variational autoencoder (CVAE). Our main contribution is the application of a structured method that can be used for machine learning in many aerospace applications. This method optimizes the structure and training parameters of a neural network for belief state inference, using Design of Experiments (DOE) statistical methodologies. The motivation for this specific DOE analysis was to identify the appropriate hyperparameters for measuring the CVAE reconstruction probability and latent space, such that the measurements can be used to infer qualitative state changes for the aircraft. We demonstrate that this process yields information about a trained neural network’s utility for this specific application, along with a quantifiable range of certainty. We execute 84 experiments using loss-of-control flight maneuver data from the NASA T-2 aircraft, demonstrating that this empirical process allows us to construct cheap and simple models with specific attributes amenable to belief state inference in aerospace applications.

Loss of Control

Use of Design of Experiments in Determining Neural Network Architectures for Loss of Control Detection

We describe empirical methods for selecting a neural network architecture to implement belief state inference on generic commercial transport aircraft. We highlight a case study on the planning, execution, and analysis of a set of experiments to determine the configurations of a conditional variational autoencoder (CVAE). Our main contribution is the application of a structured method that can be used for machine learning in many aerospace applications. This method optimizes the structure and training parameters of a neural network for belief state inference, using Design of Experiments (DOE) statistical methodologies. The motivation for this specific DOE analysis was to identify the appropriate hyperparameters for measuring the CVAE reconstruction probability and latent space, such that the measurements can be used to infer qualitative state changes for the aircraft. We demonstrate that this process yields information about a trained neural network’s utility for this specific application, along with a quantifiable range of certainty. We execute 84 experiments using loss-of-control flight maneuver data from the NASA T 2 aircraft, demonstrating that this empirical process allows us to construct cheap and simple models with specific attributes amenable to belief state inference in aerospace applications.

Loss of Control

Use of Design of Experiments in Determining Neural Network Architectures for Loss of Control Detection

Abstract—We describe empirical methods for selecting a neural network architecture to implement belief state inference on generic commercial transport aircraft. We highlight a case study on the planning, execution, and analysis of a set of experiments to determine the configurations of a conditional variational autoencoder (CVAE). Our main contribution is the application of a structured method that can be used for machine learning in many aerospace applications. This method optimizes the structure and training parameters of a neural network for belief state inference, using Design of Experiments (DOE) statistical methodologies. The motivation for this specific DOE analysis was to identify the appropriate hyperparameters for measuring the CVAE reconstruction probability and latent space, such that the measurements can be used to infer qualitative state changes for the aircraft. We demonstrate that this process yields information about a trained neural network’s utility for this specific application, along with a quantifiable range of certainty. We execute 84 experiments using loss-of-control flight maneuver data from the NASA T-2 aircraft, demonstrating that this empirical process allows us to construct cheap and simple models with specific attributes amenable to belief state inference in aerospace applications.

neural networks

Monte Carlo Tree Search Methods for the Earth-Observing Satellite Scheduling Problem

This work explores on-board planning for the single spacecraft, multiple ground station Earth-observing satellite scheduling problem through artificial neural network function approximation of state–action value estimates generated by Monte Carlo tree search (MCTS). An extensive hyperparameter search is conducted for MCTS on the basis of performance, safety, and downlink opportunity utilization to determine the best hyperparameter combination for data generation. A hyperparameter search is also conducted on neural network architectures. The learned behavior of each network is explored, and each network architecture’s robustness to orbits and epochs outside of the training distributions is investigated. Furthermore, each algorithm is compared with a genetic algorithm, which serves to provide a baseline for optimality. MCTS is shown to compute near-optimal solutions in comparison to the genetic algorithm. The state–action value networks are shown to match or exceed the performance of MCTS in six orders of magnitude less execution time, showing promise for execution on board spacecraft.

Adam P. Herrmann

Machine Learning based Aircraft Performance Model Estimation for Trajectory Prediction

The accurate prediction of aircraft trajectory by ground-based decision support tools is a critical component of air traffic management in the US National Airspace System (NAS). Accurate predictions of where the aircraft will be in the future or when they will arrive at specific locations (e.g., fixes) is a key enabler for sequencing and efficient arrival management of flights. Traditional physics based aircraft trajectory prediction relies on a simplified point-mass total energy model whose parameters are referred to as Aircraft Performance Model (APM) parameters. Even though the performance coefficients and weight of an aircraft are a vital part of the aircraft performance model’s predictions and accuracy, these coefficients are proprietary in nature and therefore, unavailable to decision-support tools. Current approaches freeze some coefficients to default base of aircraft data (BADA) values and optimize others. However, the APM parameters are highly coupled by the flight dynamics and prioritizing one parameter over others leads to bias and skewed predictions. To alleviate this problem, we provide a combined optimization framework to predict all the critical (thrust, drag and weight) APM parameters. This paper is focused on training Machine Learning (ML) models that map historical flights to optimized APM parameters that provide the best fit (in terms of prediction error). Our dataset obtained from NASA’s Sherlock data warehouse is comprised of thousands of historical flights and includes weather and track data collected from 2019. Using different subsets of relevant features (e.g., aircraft type), we trained several ML models to estimate the aircraft’s take off weight, drag polar coefficients (both parasitic and lift induced), and thrust settings (multiplier applied to the maximum engine thrust). The chosen flights are from three of the most common aircraft types (B738, B737, and A320) arriving at four airports (LAX, DEN, MSP, and DFW). Our ML approach is comprised of two different solutions: 1- using a subset of features that are known prior to the flight departure and do not change during flight (such as engine type, current temperature at departure & destination airports, aircraft type) and 2 - using a subset of temporal features of the flight trajectory (such as cruise altitude, Mach, airspeed, and rate of climb) in addition to the pre-departure features from the first solution. The labels or target variables are the APM parameters that were obtained by an optimized ordinary differential equations (ODE) fitting process (applied to individual flights). The ODE-fitting is very time intensive and is therefore performed offline. Thus, training an ML model to learn the relationship between the flight features and ODE-generated labels enables faster estimation of the APM parameters and is therefore amenable to real-time prediction. Various ML models including linear regression, random forest, XGBoost, and neural network were trained, and the results are compared. After model validation and hyperparameter-tuning, we observed that the Random Forest model outperformed the other three models by the overall mean square error (MSE) of 2% for the first solution and 1.5% for the second solution. Finally, the ML-derived parameters are compared against default BADA APM parameters using NASA’s Autonomy Development toolkit (ADK) simulation software. The simulation results for one of each aircraft type is shown and discussed.

Aida Sharif Rohani