Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “Experience replay”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 19 records

Stable Simulation of the Community Atmosphere Model Using Machine‐Learning Physical Parameterization Trained With Experience Replay

In recent years, machine learning (ML) models have been used to improve physical parameterizations of general circulation models (GCMs). A significant challenge of integrating ML models into GCMs is the online instability when they are coupled for long‐term simulation. We present a new strategy that demonstrates robust online stability when the physical parameterization package of an atmospheric GCM is replaced by a deep ML model. The method uses experience replay with a multistep training scheme of the ML model in which the model's own output at the previous time step is used in the training. Predicted physics tendencies in the replay buffer with the most recent errors in the training iterations are reused, making the ML model learn from its own errors. The training method reduces the gap between the offline and online environments of the ML model. The method is used to train the ML model as the physical parameterization of the Community Atmosphere Model (CAM5) with training data from the Multi‐scale Modeling Framework high resolution simulations. Three 6‐year online simulations of the CAM5 are carried out by using the ML physics package. The simulated spatial distributions of precipitation, surface temperature and zonally averaged atmospheric fields demonstrate overall better accuracy than that of the standard CAM5 and benchmark model even without the use of additional physical constraints or tuning. This work is the first to demonstrate a solution to address the online instability problem in climate modeling with ML physics by using experience replay.

54 ENVIRONMENTAL SCIENCES↗

In situ compression artifact removal in scientific data using deep transfer learning and experience replay

The massive amount of data produced during simulation on high-performance computers has grown exponentially over the past decade, exacerbating the need for streaming compression and decompression methods for efficient storage and transfer of this data---key to realizing the full potential of large-scale computational science. Lossy compression approaches such as JPEG when applied to scientific simulation data realized as a stream of images can achieve good compression rates but at the cost of introducing compression artifacts and loss of information. This paper develops a unified framework for in situ compression artifact removal in which the fully convolutional neural network architectures are combined with scalable training, transfer learning, and experience replay to achieve superior accuracy and efficiency while significantly decreasing the storage footprint as compared with the traditional optimization-based approaches. We demonstrate the proposed approach and compare it with compressed sensing postprocessing and other baseline deep learning models using climate simulations and nuclear reactor simulations, both of which are driven by hyperbolic partial differential equations. Our approach when applied to remove the compression artifacts on the JPEG-compressed nuclear reactor simulation data (using a transfer-trained model that was pretrained on the climate simulation data and updated incrementally as the nuclear reactor simulation progressed), achieved a significant improvement---mean peak signal-to-noise ratio of 42.438 as compared with 27.725 obtained with the compressed sensing approach.

97 MATHEMATICS AND COMPUTING↗

Miscanthus x giganteus yield in a REpeated PLAnting Year (REPLAY) experiment across three sites in Iowa, USA.

Research objectives were to study nitrogen (N) fertilization and age effect on Miscanthus x giganteus (mxg) yield and their interaction. In addition, we studied how establishment conditions affected mxg yield response to N fertilization. This is the first repeated planting year experiment conducted on perennial bioenergy grasses. Data showed that mxg yield response to N fertilization was highly variable. Establishment effects were an important driver of this variability and caused significant changes on the magnitude and direction of one-year-old stands’ response to N fertilization. With repeated planting we compared yield gain from first to second year of growth of different single starts. Yield gain of each single start was different within and between sites. This site-specific response suggests that important biases could be introduced when comparing results across studies if repeated planting is not utilized. Biomass from mxg plots was sampled annually in winter or early spring after multiple days with freezing temperatures (Table 1). At each harvest, two 1-m2 quadrats per plot were collected by hand at 0.10 m above the soil surface and weighed. In 2016 and 2018, we used a composite subsample to estimate dry matter content. In 2017, the dry weight subsamples were made of 15 randomly collected stems from each plot. Subsamples were weighed in the field and dried to constant mass at 60 °C in a forced air drier. The final mass was recorded and dry matter content was calculated for each plot.

Agricultural Science↗

Optimizing and Extending the Functionality of EXARL for Scalable Reinforcement Learning [Slides]

The main goal of the Co-Design Summer School 2021 is to provide algorithmic improvements to EXARL framework by improving performance and by adding functionalities. This presentation includes an introduction to reinforcement learning and to EXARL. The researchers expanded the capability of EXARL by including additional agents like (Asynchronized) Advantage Actor Critic (A2C/A3C) and Twin Delayed Deep Deterministic Policy Gradient (TD3). They also explored algorithmic improvements such as v-trace and Prioritized Experience Replay. They found that A2C/A3C performed best with v-trace and outperformed Deep Q-Network (DQN) on both the CartPole game and the ExaBooster scientific environment. Additionally, they found that TD3 performed as good as the existing Deep Deterministic Policy Gradient (DDPG) agent and that adding Prioritized Experience Replay to DDPG accelerated convergence.

97 MATHEMATICS AND COMPUTING↗

Efficient distributed continual learning for steering experiments in real-time

Deep learning has emerged as a powerful method for extracting valuable information from large volumes of data. However, when new training data arrives continuously (i.e., is not fully available from the beginning), incremental training suffers from catastrophic forgetting (i.e., new patterns are reinforced at the expense of previously acquired knowledge). Training from scratch each time new training data becomes available would result in extremely long training times and massive data accumulation. Rehearsal-based continual learning has shown promise for addressing the catastrophic forgetting challenge, but research to date has not addressed performance and scalability. To fill this gap, we propose an approach based on a distributed rehearsal buffer that efficiently complements data-parallel training on multiple GPUs to achieve high accuracy, short runtime, and scalability. It leverages a set of buffers (local to each GPU) and uses several asynchronous techniques for updating these local buffers in an embarrassingly parallel fashion, all while handling the communication overheads necessary to augment input minibatches using unbiased, global sampling. We further propose a generalization of rehearsal buffers to support both classification and generative learning tasks, as well as more advanced rehearsal strategies (notably Dark Experience Replay, leveraging knowledge distillation). We illustrate this approach with a real-life HPC streaming application from the domain of ptychographic image reconstruction. Furthermore, we run extensive experiments on up to 128 GPUs of the ThetaGPU supercomputer to compare our approach with baselines representative of training-from-scratch (the upper bound in terms of accuracy) and incremental training (the lower bound). Results show that rehearsal-based continual learning achieves a top-5 validation accuracy close to the upper bound, while simultaneously exhibiting a runtime close to the lower bound.

Asynchronous data management↗

The Impact of Ice Sheet Surface Representation on Surface Mass Balance in the Goddard Earth Observing System

Surface conditions and their impacts on surface mass balance (SMB) play important roles in ice sheet dynamics and ice sheet interactions with the overlying atmosphere and surrounding ocean. The NASA Global Modeling and Assimilation Office’s (GMAO) Goddard Earth Observing System (GEOS) – an ecosystem of models and reanalyses – represent ice sheet and glacier SMB components, including prognostic surface albedo evolution, fractional snow cover, and snowpack hydrology and meltwater retention. We find the successful representation of ice sheet surface mass balance in MERRA-2, and similar systems, is partly due to compensating biases in surface energy budget. In the atmospheric system, subsequent changes to cloud microphysics and in the longwave radiative transfer model are found to have reduced biases in surface net longwave radiation fluxes while exacerbating surface net shortwave radiation biases, producing erroneously high near-surface temperatures and surface melt in summer months. These issues can be exacerbated by poor experiment initialization and the use of two-moment cloud microphysics within the GEOS ocean-atmosphere coupled system. Here, we document the spatial and temporal extent and causes of these biases in the ice sheet surface energy budget across GEOS systems and implement a range of model improvements to mitigate these issues. We examine the combined impact on these changes on regional energy budget and SMB using both free running and replay experiments (to simulate the impact in reanalyses). Overall, these improvements increase modeled ice sheet surface realism and provide a strong basis for more accurate ice sheet SMB in GMAO’s future reanalyses and forecasting systems.

Lauren C Andrews↗

Microgrid energy scheduling under uncertain extreme weather: Adaptation from parallelized reinforcement learning agents

Microgrids are useful solutions for integrating renewable energy resources and providing seamless green electricity to minimize carbon footprint. In recent years, extreme weather events happened often worldwide and caused significant economic and societal losses. Such events bring uncertainties to the microgrid energy scheduling problems and increase the challenges of microgrid operation. Traditional optimization approaches suffer from the inaccuracy of the uncertain microgrid model and the unseen events. Existing reinforcement learning (RL) - based approaches are also hampered by the limited generalization and the increasing computational burden when stochastic formulations are required to accommodate the uncertainties. This paper proposes a new parallelized reinforcement learning (PRL) method based on the probabilistic events to handle the microgrid energy uncertainties. Specifically, several local learning agents are employed to interact with pertinent microgrid environments in a distributed manner and report outcomes to the global agent, which will optimize microgrid energy resources online during extreme events. The stochastic microgrid energy optimization problem is reformulated to include all possible scenarios with probabilities. The advantage estimate functions of learning agents are designed with a backward sweep to transfer the outcomes to the value function updating process. Two simulation studies, stochastic optimization and online testing, are performed to compare with several existing RL approaches. Results substantiate that the proposed PRL method can achieve up to 20% optimization performance improvement with 4 and 28 times less computation cost than Q-learning with experience replay and multi-agent Q-learning approaches, respectively.

24 POWER TRANSMISSION AND DISTRIBUTION↗

PowerModelsGAT-AI: Physics-Informed Graph Attention for Multi-System Power Flow With Continual Learning

Solving the alternating current power flow equations in real time is essential for secure grid operation, yet classical Newton–Raphson solvers can be slow under stressed conditions. Existing graph neural networks for power flow are typically trained on a single system and often degrade on different systems. We present PowerModelsGAT-AI, a physics-informed graph attention network that predicts bus voltages and generator injections. The model uses bus-type-aware masking to handle different bus types and balances multiple loss terms, including a power-mismatch penalty, using learned weights. We evaluate the model on 14 benchmark systems (4 to 6,470 buses) and train a unified model on 13 of these under contingency conditions with up to two branch outages, achieving an average normalized mean absolute error of 0.89% for voltage magnitudes and R 2 >0.99 for voltage angles. We also show continual learning: when adapting a base model to a new 1,354-bus system, standard fine-tuning causes severe forgetting with error increases exceeding 1000% on base systems, while our experience replay and elastic weight consolidation strategy keeps error increases below 2% and in some cases improves base-system performance. Interpretability analysis shows that learned attention weights correlate with physical branch parameters (susceptance: r=0.38 ; thermal limits: r=0.22 ), and feature importance analysis supports that the model captures established power flow relationships.

24 POWER TRANSMISSION AND DISTRIBUTION↗

The Artificial Scientist: in-Transit Machine Learning of Plasma Simulations

Large-scale simulations or scientific experiments produce petabytes of data per run. This poses massive challenges for I/O and storage when scientific analysis workflows are run manually offline. Unsupervised deep learning-based techniques to extract patterns and non-linear relations from these large amounts of data provide a way to build scientific understanding from raw data, reducing the need for manual pre-selection of analysis steps, but require exascale compute and memory to process the full dataset available. In this paper, we demonstrate a heterogeneous streaming workflow in which plasma simulation data is streamed directly to a Machine Learning (ML) application training a model on the simulation data in-transit, completely circumventing the capacity-constrained filesystem bottleneck. This workflow employs openPMD to provide a high level interface to describe scientific data and also uses ADIOS2, to transfer volumes of data that exceed the capabilities of the filesystem. We employ experience replay to avoid catastrophic forgetting in learning from this non-steady state process in a continual manner and adapt it to improve model convergence while learning in-transit. As a proof-of-concept, we approach the ill-posed inverse problem of predicting particle dynamics from radiation in a particle-incell (PIConGPU) simulation of the Kelvin-Helmholtz instability (KHI). We detail hardware-software co-design challenges as we scale PIConGPU to full Frontier, the Top-1 system as of June 2024 Top500 list.

Kelling, Jeffrey [Helmholtz-Zentrum Dresden Rossen↗

5G Open Core Hardware

This project aimed to develop an experimentation platform for 5G using commercially available software, firmware and hardware that is being used to standup experimental 5G Stand Alone (5G SA) networks. The commercially available systems are not natively set up for experimentation design and this work would develop a system capable of performing experimentation. This platform will need to be easily reconfigurable for each experiment, capture all configurations across key equipment before running an experiment, record and store the data during an experiment and be able to recall settings and data from prior experiments while being able to emulate and replay prior experiments. The project began in the process of creating this platform.

97 MATHEMATICS AND COMPUTING↗

Data-Driven Software Framework for Web-Based ISS Telescience

Software that enables authorized users to monitor and control scientific payloads aboard the International Space Station (ISS) from diverse terrestrial locations equipped with Internet connections is undergoing development. This software reflects a data-driven approach to distributed operations. A Web-based software framework leverages prior developments in Java and Extensible Markup Language (XML) to create portable code and portable data, to which one can gain access via Web-browser software on almost any common computer. Open-source software is used extensively to minimize cost; the framework also accommodates enterprise-class server software to satisfy needs for high performance and security. To accommodate the diversity of ISS experiments and users, the framework emphasizes openness and extensibility. Users can take advantage of available viewer software to create their own client programs according to their particular preferences, and can upload these programs for custom processing of data, generation of views, and planning of experiments. The same software system, possibly augmented with a subset of data and additional software tools, could be used for public outreach by enabling public users to replay telescience experiments, conduct their experiments with simulated payloads, and create their own client programs and other custom software.

Tso, Kam S.↗

Dynamic Adjustment of Model Parameters Using Assimilated Boundary Layer Height

The NASA Global Modeling and Assimilation Office is exploring the assimilation of boundary layer height (PBLH) derived from a variety of observations, including radiosondes, GNSS-RO, space- and ground-based lidar, and radar wind profilers. Previous work using the Goddard Earth Observing System (GEOS) has shown that within the lower troposphere, the representation of thermodynamic structure is strongly affected by model parameterizations. In assimilation experiments, parameterizations can compensate for analysis thermodynamic increments and reduce the observation information carried forward in time. To mitigate this effect, the present study explores a dynamic parameter adjustment based on the PBLH analysis field. The PBLH analysis increment is used to adjust model parameters that influence the depth of parameterized boundary layer mixing in both stable and convective regimes. Replay (nudging) experiments and cycled data assimilation experiments are used to evaluate the approach. The parameter adjustment is shown to reduce model compensation of the analysis thermodynamic tendencies and bring model parameterized mixing depths into closer agreement with observed PBLH.

Nathan Arnold↗

Continued Data Acquisition Development

This task focused on improving techniques for integrating data acquisition of secondary particles correlated in time with detected cosmic-ray muons. Scintillation detectors with Pulse Shape Discrimination (PSD) capability show the most promise as a detector technology based on work in FY13. Typically PSD parameters are determined prior to an experiment and the results are based on these parameters. By saving data in list mode, including the fully digitized waveform, any experiment can effectively be replayed to adjust PSD and other parameters for the best data capture. List mode requires time synchronization of two independent data acquisitions (DAQ) systems: the muon tracker and the particle detector system. Techniques to synchronize these systems were studied. Two basic techniques were identified: real time mode and sequential mode. Real time mode is the preferred system but has proven to be a significant challenge since two FPGA systems with different clocking parameters must be synchronized. Sequential processing is expected to work with virtually any DAQ but requires more post processing to extract the data.

46 INSTRUMENTATION RELATED TO NUCLEAR SCIENCE AND ↗

Real-Time Cyber-Physical False Data Attack Detection in Smart Grids Using Neural Networks

Attacks on cyber-physical systems have recently increased in frequency, impact, and publicity. In this paper, a cyber-physical false data attack detection mechanism is proposed to protect the operation of power transmission and distribution systems by automatically inferring underlying physical relationships using cross-sensor analytics in order to detect sensor failures, replay attacks, and other data integrity issues in real-time. We investigate a neural network based mechanism acting on voltage and current readings resulting from a wide variety of load conditions on the IEEE 30-bus power system standard and compare its performance with a support vector machine based mechanism. Experiments showed that 99% detection accuracy of replay attacks was achieved using the proposed neural network mechanism. More importantly, we showed that the best approach was to not create physics-based features using what we knew about the system, but rather to use a neural network to automatically learn the laws, and then use the outputs of that to build a classifier to identify whether and where data spoofing occurs. Thus, it is preferable to infer and exploit the physics using a single machine learning solution

Ferragut, Erik M.↗

NASA Downscaling Project: Final Report

A team of researchers from NASA Ames Research Center, Goddard Space Flight Center, the Jet Propulsion Laboratory, and Marshall Space Flight Center, along with university partners at UCLA, conducted an investigation to explore whether downscaling coarse resolution global climate model (GCM) predictions might provide valid insights into the regional impacts sought by decision makers. Since the computational cost of running global models at high spatial resolution for any useful climate scale period is prohibitive, the hope for downscaling is that a coarse resolution GCM provides sufficiently accurate synoptic scale information for a regional climate model (RCM) to accurately develop fine scale features that represent the regional impacts of a changing climate. As a proxy for a prognostic climate forecast model, and so that ground truth in the form of satellite and in-situ observations could be used for evaluation, the MERRA and MERRA - 2 reanalyses were used to drive the NU - WRF regional climate model and a GEOS - 5 replay. This was performed at various resolutions that were at factors of 2 to 10 higher than the reanalysis forcing. A number of experiments were conducted that varied resolution, model parameterizations, and intermediate scale nudging, for simulations over the continental US during the period from 2000 - 2010. The results of these experiments were compared to observational datasets to evaluate the output.

dynamical downsizing↗

NASA Downscaling Project

A team of researchers from NASA Ames Research Center, Goddard Space Flight Center, the Jet Propulsion Laboratory, and Marshall Space Flight Center, along with university partners at UCLA, conducted an investigation to explore whether downscaling coarse resolution global climate model (GCM) predictions might provide valid insights into the regional impacts sought by decision makers. Since the computational cost of running global models at high spatial resolution for any useful climate scale period is prohibitive, the hope for downscaling is that a coarse resolution GCM provides sufficiently accurate synoptic scale information for a regional climate model (RCM) to accurately develop fine scale features that represent the regional impacts of a changing climate. As a proxy for a prognostic climate forecast model, and so that ground truth in the form of satellite and in-situ observations could be used for evaluation, the MERRA and MERRA-2 reanalyses were used to drive the NU-WRF regional climate model and a GEOS-5 replay. This was performed at various resolutions that were at factors of 2 to 10 higher than the reanalysis forcing. A number of experiments were conducted that varied resolution, model parameterizations, and intermediate scale nudging, for simulations over the continental US during the period from 2000-2010. The results of these experiments were compared to observational datasets to evaluate the output.

Ferraro, Robert↗

Flight Test Results from Real-Time Relative Global Positioning System Flight Experiment on STS-69

A real-time global positioning system (GPS) Kalman filter has been developed to support automated rendezvous with the International Space Station (ISS). The filter is integrated with existing Shuttle rendezvous software running on a 486 laptop computer under Windows. In this work, we present real-time and postflight results achieved with the filter on STS-69. The experiment used GPS data from an Osborne/Jet propulsion Laboratory TurboRouge receiver carried on the Wake Shield Facility (WSF) free flyer and a Rockwell Collins 3M receiver carried on the Orbiter. Real time filter results, processed onboard the Shuttle and replayed in near-time on the ground, are based on single vehicle mode operation and on 5 to 20 minute snapshots of telemetry provided by WSF for dual-vehicle mode operation. The Orbiter and WSF state vectors calculated using our filter compare favorably with precise reference orbits determined by the University of Texas Center for Space Research. The lessons learned from this experiment will be used in conjunction with future experiments to mitigate the technology risk posed by automated rendezvous and docking to the ISS.

Park, Young W.↗