Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “distributional deep reinforcement learning”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

104 records · Page 6
All sourcesNASA NTRSDOE OSTI
DOE OSTI2025

A Novel LDPP-MADDPG Approach for Distributed Power Allocation in mmWave Cellular Networks

This paper considers the problem of distributed beam scheduling and power allocation problem in millimeter- Wave (mmWave) cellular networks, in which multiple Base Stations (BSs) operate as individual operators over a shared spectrum. We propose a novel learning-aided approach that integrates the Lyapunov Drift-Plus-Penalty (LDPP) framework and Multi-agent Deep Deterministic Policy Gradient (MADDPG) reinforcement learning algorithms. This offers a powerful approach to learning stable and constraint-aware policies, reaping the joint benefit of both LDPP and MADDPG, in complex multiagent environments. The major challenge for this approach is to integrate these two approaches in a meaningful and effective manner. The key idea to solve this problem is to introduce a novel feature of local observation that incorporates potential negative value of the reward function due to the stochastic constraints introduced by the LDPP framework. Empirical results demonstrate that our proposed scheme outperforms the baseline methods under various conditions.

99 - GENERAL AND MISCELLANEOUS↗
DOE OSTI2026

Optimization of the FRIB beam dump: a hybrid genetic algorithm and reinforcement learning approach

The operational envelope of high-power-density systems, such as particle accelerators and advanced nuclear energy systems, is critically constrained by the need to manage extreme thermal loads. To address this, we present a novel hybrid optimization framework combining a genetic algorithm (GA) with a soft actor-critic (SAC) deep reinforcement learning agent. This framework was applied to a practical high-heat-flux problem: redesigning the beam dump at the Facility for Rare Isotope Beams (FRIB) for a power upgrade from 20 kW to 50 kW. The resulting design, validated by three-dimensional conjugate heat transfer simulations, suppresses hazardous hot spots and yields a markedly more uniform temperature distribution. This provides a robust operating margin, increasing the average power-handling capability by 72% relative to the current design, demonstrating the framework’s potential to solve complex thermal management challenges in both accelerator technology and advanced nuclear systems.

Accelerator↗
DOE OSTI2022

Machine Learning Solutions for a Stable Grid Recovery

Grid operating security studies are typically employed to establish operating boundaries, ensuring secure and stable operation for a range of operation under NERC guidelines. However, if these boundaries are severely violated, existing system security margins will be largely unknown, as would be a secure incremental dispatch path to higher security margins while continuing to serve load. As an alternative to the use of complex optimizations over dynamic conditions, this work employs the use of machine learning to identify a sequence of secure state transitions which place the grid in a higher degree of operating security with greater static and dynamic stability margins. Several reinforcement learning solution methods were developed using deep learning neural networks, including Deep Q-learning, Mu-Zero, and the continuous algorithms Proximal Reinforcement Learning, and Advantage Actor Critic Learning. The work is demonstrated on a power grid with three control dimensions but can be scaled in size and dimensionality, which is the subject of ongoing research.

24 POWER TRANSMISSION AND DISTRIBUTION↗
DOE OSTI2026

Safe Deep Reinforcement Learning for Robust Frequency and Voltage-Constrained Networked Microgrid Restoration

Here, this paper proposes a safe soft actor-critic reinforcement learning (RL) algorithm–based controller for networked microgrid restoration. It formulates the post black-start start as a finite-horizon constrained Markov decision process. The RL agent co-optimizes real and reactive power set-points for both grid-forming and grid-following inverters under explicit voltage and frequency constraints, while enforcing proper power sharing via the Mean Active Power Sharing Index (MPSI) and Mean Reactive Power Sharing Index (MQSI). Numerical results obtained on the IEEE 123-bus distribution system show that the proposed method achieves a mean voltage build-up time of 0.01 s without breaching the 5% sharing-violation budget under various load scenarios, considering MPSI and MQSI indices. These findings demonstrate that the proposed method yields fast and safe black-start schedules without resorting to heuristic penalties.

Selim, Alaa [Dartmouth College, Hanover, NH (Unite↗
DOE OSTI2024

Integrating multi-modal remote sensing, deep learning, and attention mechanisms for yield prediction in plant breeding experiments

In both plant breeding and crop management, interpretability plays a crucial role in instilling trust in AI-driven approaches and enabling the provision of actionable insights. The primary objective of this research is to explore and evaluate the potential contributions of deep learning network architectures that employ stacked LSTM for end-of-season maize grain yield prediction. A secondary aim is to expand the capabilities of these networks by adapting them to better accommodate and leverage the multi-modality properties of remote sensing data. In this study, a multi-modal deep learning architecture that assimilates inputs from heterogeneous data streams, including high-resolution hyperspectral imagery, LiDAR point clouds, and environmental data, is proposed to forecast maize crop yields. The architecture includes attention mechanisms that assign varying levels of importance to different modalities and temporal features that, reflect the dynamics of plant growth and environmental interactions. The interpretability of the attention weights is investigated in multi-modal networks that seek to both improve predictions and attribute crop yield outcomes to genetic and environmental variables. This approach also contributes to increased interpretability of the model's predictions. The temporal attention weight distributions highlighted relevant factors and critical growth stages that contribute to the predictions. The results of this study affirm that the attention weights are consistent with recognized biological growth stages, thereby substantiating the network's capability to learn biologically interpretable features. Accuracies of the model's predictions of yield ranged from 0.82-0.93 R 2 ref in this genetics-focused study, further highlighting the potential of attention-based models. Further, this research facilitates understanding of how multi-modality remote sensing aligns with the physiological stages of maize. The proposed architecture shows promise in improving predictions and offering interpretable insights into the factors affecting maize crop yields, while demonstrating the impact of data collection by different modalities through the growing season. By identifying relevant factors and critical growth stages, the model's attention weights provide valuable information that can be used in both plant breeding and crop management. The consistency of attention weights with biological growth stages reinforces the potential of deep learning networks in agricultural applications, particularly in leveraging remote sensing data for yield prediction. To the best of our knowledge, this is the first study that investigates the use of hyperspectral and LiDAR UAV time series data for explaining/interpreting plant growth stages within deep learning networks and forecasting plot-level maize grain yield using late fusion modalities with attention mechanisms.

59 BASIC BIOLOGICAL SCIENCES↗
DOE OSTI2024

Design and optimization of a modular hydrogen-based integrated energy system to maximize revenue via nuclear-renewable sources

Here, this paper demonstrates a novel modular distributed framework that uses optimal energy-dispatching strategies to enable greater flexibility and profitability in nuclear-renewable integrated energy systems (NR-IES). Hydrogen is used as a commodity in this framework since its production can improve grid stability and system operational flexibility, decarbonize heavy industry, and create an additional revenue stream for electricity generators, particularly nuclear power plants with high operational expenses. The proposed solution addresses the challenges associated with merging multiple software and services from various domains by using functional mock-up units (FMU) to co-simulate diverse subsystems designed in various platforms. The tightly coupled integrated energy system (IES) is optimized to maximize revenue by utilizing the deep reinforcement learning (DRL) technique to make smart dispatching decisions based on variable electricity prices and the availability of renewable energy. Proximal policy optimization (PPO) algorithm is used in training and testing the DRL agent. Over a period of 120 days, the proposed hydrogen-based IES framework showed about 10% revenue boost compared to a non-hydrogen generating baseline IES while also providing an easily-adoptable framework which can help to improve the flexibility of future generation nuclear power plants.

29 ENERGY PLANNING, POLICY, AND ECONOMY↗
DOE OSTI2020

Chapter Nine - Automated Optimal Control in Energy Systems: The Reinforcement Learning Approach

With the development of smart grid technologies an increasing number of new devices and participants have joined modern energy systems and are inevitably making them more complicated and interdependent than ever. Optimally controlling such a complex energy system and maintaining its operation in a high-efficient, secure, and resilient manner are challenging tasks to the system operators. Fortunately, the revolution in deep learning and artificial intelligence (AI), both from hardware and algorithms perspectives, has provided new ideas and solutions to many previously intractable problems. As a result, this advance in computer science also sparked great research interests in utilizing AI in solving engineering problems related to the modern energy systems. Among many AI techniques, deep reinforcement learning (DRL) has demonstrated great potential for solving sequential optimization problems, which are very common in the engineering domains. Its ability to handle nonlinearity and stochasticity in controlled systems has out-competed many traditional optimal control algorithms. Therefore in this chapter, we focus on the state-of-the-art of DRL concepts and related algorithms, compare their pros and cons with traditional optimal control approaches and discuss the typical workflow for leveraging RL in solving complex problems in modern energy systems.

artificial intelligence↗
DOE OSTI2022

Learning and Fast Adaptation for Grid Emergency Control via Deep Meta Reinforcement Learning

As power systems are undergoing a significant transformation with more uncertainties, less inertia and closer to operation limits, there is increasing risk of large outages. Thus, there is an imperative need to enhance grid emergency control to maintain system reliability and security. Towards this end, great progress has been made in developing deep reinforcement learning (DRL) based grid control solutions in recent years. However, existing DRL-based solutions have two main limitations: 1) they cannot handle well with a wide range of grid operation conditions, system parameters, and contingencies; 2) they generally lack the ability to fast adapt to new grid operation conditions, system parameters, and contingencies, limiting their applicability for real-world applications. Here, in this paper, we mitigate these limitations by developing a novel deep meta-reinforcement learning (DMRL) algorithm. The DMRL combines the meta strategy optimization together with DRL, and trains policies modulated by a latent space that can quickly adapt to new scenarios. We test the developed DMRL algorithm on the IEEE 300-bus system. We demonstrate fast adaptation of the meta-trained DRL polices with latent variables to new operating conditions and scenarios using the proposed method, which achieves superior performance compared to the state-of-the-art DRL and model predictive control (MPC) methods.

24 POWER TRANSMISSION AND DISTRIBUTION↗
DOE OSTI2026

Demonstration of reconstruction-free static magnetic control of DIII-D plasma with deep reinforcement learning

This paper presents the development and experimental validation of a reinforcement learning (RL)-based magnetic controller on the DIII-D tokamak. The controller directly maps raw magnetic diagnostic signals to actuator commands, replacing the traditional isoflux control algorithm based on equilibrium reconstruction. Four RL controllers are trained using the Soft Actor–Critic algorithm with an asymmetric Actor–Critic architecture in the NSFsim simulator. All controllers are deployed in the DIII-D Plasma Control System and operated with a 4 kHz feedback loop. Two randomization strategies are evaluated during training: evolving kinetic profiles and fixed kinetic profiles within each episode. The latter approach is found to better capture experimental deviations in the current density profile and to provide overall improved control performance. Robust operation is demonstrated across heating power scans in both L- and H-mode plasmas, as well as during transient events such as L–H transitions and pellet injections. Control errors in plasma shape and radial position remained within 1.5–2.0 cm and 1 cm, respectively. A notable discrepancy was observed in the vertical X-point position, with errors of up to approximately 4 cm, attributed to the current density distribution mismatches between simulations and experiments.

DIII-D↗
DOE OSTI2023

High Performance Adaptive Deep-Reinforcement-Learning-based Real-time Emergency Control (HADREC) to Enhance Power Grid Resilience in Stochastic Environment (Final Report)

The HADREC technology developed in this project features real-time, validated, and predictive emergency control capabilities needed to effectively safeguard the grid against costly disturbances from extreme weather and other disruptive events. It has the potential to revolutionize the emergency control by providing faster and more effective solution for power grid emergency control. The technology can reduce the grid emergency control action time from minutes to seconds and mitigate the impact of outages and reduce system recovery time. The evaluation results using the models and data from PacifiCorp system have shown ~20% improvement in responding time and 70% improved in tripped generator output, comparing against today’s industrial application. This performance can help reduce preventive measurements and improve network utilization and market efficiency, and help lead to a more reliable and resilient power grid with renewable integration. The project has generated significant interest from industry, academic and commercial companies.

24 POWER TRANSMISSION AND DISTRIBUTION↗
DOE OSTI2020

Automated Control of Transactive HVACs in Energy Distribution Systems

Heating, Ventilation, and Air Conditioning (HVAC) systems contribute significantly to a building’s energy consumption. In the recent years, there is an increased interest in developing transactive approaches which could enable automated and flexible scheduling of HVAC systems based on the customer demand and the electricity prices decided by the suppliers. Flexible and automated scheduling of the HVAC systems make it a prime source for participation in residential demand response or transactive energy systems. Therefore, it is of significant interest to identify an optimal strategy to control the HVAC systems. Here, reducing the energy cost while keeping the comfort level acceptable to the users, we argue that such a control strategy should consider both the energy cost and user comfort simultaneously. Accordingly, we develop the control strategy through the solution of an optimization problem that balances between the energy cost and consumer’s dissatisfaction. This optimization enables us to solve a decision-making problem through first price prediction and then choosing HVAC temperature settings throughout the day based on the predicted price, history of the price and HVAC settings, and outside temperature. More specifically, we formulate the control design as a Markov decision process (MDP) using deep neural networks and use Deep Deterministic Policy Gradients (DDPG)-based deep reinforcement learning algorithm to find the optimal control strategy for HVAC systems that balances between electricity cost and user comfort.

32 ENERGY CONSERVATION, CONSUMPTION, AND UTILIZATI↗
DOE OSTI2024

Multi-Agent Graph-Attention Deep Reinforcement Learning for Post-Contingency Grid Emergency Voltage Control

Grid emergency voltage control (GEVC) is paramount in electric power systems to improve voltage stability and prevent cascading outages and blackouts in case of contingencies. While most deep reinforcement learning (DRL)-based paradigms perform single agents in a static environment, real-world agents for GEVC are expected to cooperate in a dynamically shifting grid. Moreover, due to high uncertainties from combinatory natures of various contingencies and load consumption, along with the complexity of dynamic grid operation, the data efficiency and control performance of the existing DRL-based methods are challenged. To address these limitations, we propose a multi-agent graph-attention (GATT)-based DRL algorithm for GEVC in multi-area power systems. Here, we develop graph convolutional network (GCN)-based agents for feature representation of the graph-structured voltages to improve the decision accuracy in a data-efficient manner. Furthermore, a cutting-edge attention mechanism concentrates on effective information sharing among multiple agents, synergizing different-sized subnetworks in the grid for cooperative learning. We address several key challenges in the existing DRL-based GEVC approaches, including low scalability and poor stability against high uncertainties. Test results in the IEEE benchmark system verify the advantages of the proposed method over several recent multi-agent DRL-based algorithms.

24 POWER TRANSMISSION AND DISTRIBUTION↗
DOE OSTI2020

DEEP Solar: Data DrivEn Modeling and Analytics for Enhanced System Layer ImPlementation

Realizing the SETO 2030 mission of reducing solar energy costs to 3-5 c/kWh will require innovative enabling research on effective, cost-efficient integration of local PV within distribution systems. However, the intermittent and variable nature of PVs compels operators to impose conservative hosting capacity constraints. Given the extremely high variability of (intermittent and unpredictable) solar energy generation, relaxing the capacity constraints (which are currently around 15%) and achieving 100% or greater integration of renewables will require a fundamental transformation of the power grid via the utilization of exponentially larger amounts of AMI enabled fine-grained data. To address the challenges in increasing the penetration of renewable energy based DERs, this project envisions an Enhanced System Layer (ESL) at the distribution network level that is reliable, cost-effective and scalable to millions of Distributed Energy Resources (DERs)/devices. This includes developing: 1) Transformative and highly scalable machine learning based predictive analytics tools that plug into distribution system planning and provide real-time situational awareness at the distribution level for short and long-term operational planning. The tools will be built using novel data-driven energy models of millions of active nodes with AMI, 2) Adaptive stochastic analysis and optimization algorithms for real-time grid operations, 3) Dynamic Scenario Analysis using parallel Cloudenabled implementations with < 1 minute computational cycle times.

14 SOLAR ENERGY↗
DOE OSTI2021

Stopping criteria for ending autonomous, single detector radiological source searches

While the localization of radiological sources has traditionally been handled with statistical algorithms, such a task can be augmented with advanced machine learning methodologies. The combination of deep and reinforcement learning has provided learning-based navigation to autonomous, single-detector, mobile systems. However, these approaches lacked the capacity to terminate a surveying/search task without outside influence of an operator or perfect knowledge of source location (defeating the purpose of such a system). Two stopping criteria are investigated in this work for a machine learning navigated system: one based upon Bayesian and maximum likelihood estimation (MLE) strategies commonly used in source localization, and a second providing the navigational machine learning network with a “stop search” action. A convolutional neural network was trained via reinforcement learning in a 10 m × 10 m simulated environment to navigate a randomly placed detector-agent to a randomly placed source of varied strength (stopping with perfect knowledge during training). The network agent could move in one of four directions (up, down, left, right) after taking a 1 s count measurement at the current location. During testing, the stopping criteria for this navigational algorithm was based upon a Bayesian likelihood estimation technique of source presence, updating this likelihood after each step, and terminating once the confidence of the source being in a single location exceeded 0.9. A second network was trained and tested with similar architecture as the previous but which contained a fifth action: for self-stopping. The accuracy and speed of localization with set detector and source initializations were compared over 50 trials of MLE-Bayesian approach and 1000 trials of the CNN with self-stopping. The statistical stopping condition yielded a median localization error of ~1.41 m and median localization speed of 12 steps. The machine learning stopping condition yielded a median localization error of 0 m and median localization speed of 17 steps. This work demonstrated two stopping criteria available to a machine learning guided, source localization system.

38 RADIATION CHEMISTRY, RADIOCHEMISTRY, AND NUCLEA↗