SEARCH · Engineering Papers
Results for “MARL”
Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.
Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.
One-Dimensional (1D) Reactive Transport with PFLOTRAN: Marl Leaching Case - Progress Status.
Abstract not provided.
One-Dimensional (1D) Reactive Transport with PFLOTRAN: Marl Leaching Case - Progress Status.
Abstract not provided.
Interpreting Primal-Dual Algorithms for Constrained Multiagent Reinforcement Learning: Preprint
We study multiagent reinforcement learning (MARL) with constraints. This setting is gaining importance as MARL algorithms find new applications in real-world systems ranging from power grids to drone swarms. Most constrained MARL (C-MARL) algorithms use a primal-dual approach to enforce constraints through a penalty function added to the reward. In this paper, we study the structural effects of the primal-dual approach on the constraints and value function. First, we show that using the constraint evaluation as the penalty leads to a weak notion of safety, but by making simple modifications to the penalty function, we can enforce meaningful probabilistic safety constraints. Second, we show that the penalty term changes the value function in a way that is easy to model, and demonstrate the consequences of not doing so. We conclude with simulations in a simple constrained multiagent environment to back up the theoretical results.
Interpreting Primal-Dual Algorithms for Constrained Multiagent Reinforcement Learning
Constrained multiagent reinforcement learning (C-MARL) is gaining importance as MARL algorithms find new applications in real-world systems ranging from energy systems to drone swarms. Most C-MARL algorithms use a primal-dual approach to enforce constraints through a penalty function added to the reward. In this paper, we study the structural effects of this penalty term on the MARL problem. First, we show that the standard practice of using the constraint function as the penalty leads to a weak notion of safety. However, by making simple modifications to the penalty term, we can enforce meaningful probabilistic (chance and conditional value at risk) constraints. Second, we quantify the effect of the penalty term on the value function, uncovering an improved value estimation procedure. We use these insights to propose a constrained multiagent advantage actor critic (C-MAA2C) algorithm. Simulations in a simple constrained multiagent environment affirm that our reinterpretation of the primal-dual method in terms of probabilistic constraints is effective, and that our proposed value estimate accelerates convergence to a safe joint policy.
PowerNet: Multi-agent Deep Reinforcement Learning for Scalable Powergrid Control
This paper develops an efficient multi-agent deep reinforcement learning algorithm for cooperative controls in powergrids. Specifically, we consider the decentralized inverter-based secondary voltage control problem in distributed generators (DGs), which is first formulated as a cooperative multi-agent reinforcement learning (MARL) problem. We then propose a novel on-policy MARL algorithm, PowerNet, in which each agent (DG) learns a control policy based on (sub-)global reward but local states and encoded communication messages from its neighbors. Motivated by the fact that a local control from one agent has limited impact on agents distant from it, we exploit a novel spatial discount factor to reduce the effect from remote agents, to expedite the training process and improve scalability. Furthermore, a differentiable, learning-based communication protocol is employed to foster the collaborations among neighboring agents. In addition, to mitigate the effects of system uncertainty and random noise introduced during on-policy learning, we utilize an action smoothing factor to stabilize the policy execution. To facilitate training and evaluation, we develop PGSim, an efficient, high-fidelity powergrid simulation platform. Here, experimental results in two microgrid setups show that the developed PowerNet outperforms the conventional model-based control method, as well as several state-of-the-art MARL algorithms. The decentralized learning scheme and high sample efficiency also make it viable to large-scale power grids.
Enhancing Autonomous Control of Microreactors Using Multi-Agent Reinforcement Learning
In order for microreactors to be economically competitive, operation costs will need to be minimized through some degree of autonomous control. Previous work has demonstrated the effectiveness of reinforcement learning (RL) for load-following control in a drum-controlled microreactor. This study extends that work by exploring the potential of RL to independently control each of the reactor’s drums. We compare a single-agent RL approach with a multi-agent RL (MARL) framework, testing them for generalization across different load-following power profiles and control timescales, and for robustness in cases of randomly disabled control drums. Since the point kinetics simulation environment used in this study cannot resolve spatial effects, we assume that in the absence of spatially localized disturbances, optimal drum movements should be symmetrical. We demonstrate that single-agent RL is able to achieve accurate performance only when symmetric actions are ignored; otherwise, it fails to train a useful controller. Meanwhile, the MARL framework performs symmetric actions by design and trains a robust, accurate agent, as evidenced by mean absolute errors in power matching of 0.41% for the training power profile, 0.68% for a profile with half the drums disabled, and 0.21% for a profile on a realistic load-following time horizon.
Nuclear microreactor transient and load-following control with deep reinforcement learning
The economic feasibility of nuclear microreactors will depend on minimizing operating costs through advancements in autonomous control, especially when these microreactors are operating alongside other types of energy systems (e.g., renewable energy). This study explores the application of deep reinforcement learning (RL) for real-time drum control in microreactors, exploring performance in regard to load-following scenarios. By leveraging a point kinetics model with thermal and xenon feedback, we first establish a baseline using a single-output RL agent, then compare it against a traditional proportional–integral–derivative (PID) controller. This study demonstrates that RL controllers, including both single- and multi-agent RL (MARL) frameworks, can achieve similar or even superior load-following performance as traditional PID control across a range of load-following scenarios. In short transients, the RL agent was able to reduce the tracking error rate in comparison to PID by one half to one third. Over extended 300-minute load-following scenarios in which xenon feedback becomes a dominant factor, PID maintained better accuracy, but RL still remained within a 1% error margin despite being trained only on short-duration scenarios. This highlights RL’s strong ability to generalize and extrapolate to longer, more complex transients, affording substantial reductions in training costs and reduced overfitting. Furthermore, when control was extended to multiple drums, MARL enabled independent drum control as well as maintained reactor symmetry constraints without sacrificing performance---an objective that standard single-agent RL could not learn. We also found that, as increasing levels of Gaussian noise were added to the power measurements, the RL controllers were able to maintain lower error rates than PID, and to do so with at least 10% and upwards of 150% less control effort. These findings illustrate RL's potential for autonomous nuclear reactor control, laying the groundwork for future integration into high-fidelity simulations and experimental validation efforts.
Non-Stationary Policy Learning for Multi-Timescale Multi-Agent Reinforcement Learning
In multi-timescale multi-agent reinforcement learning (MARL), agents interact across different timescales. In general, policies for time-dependent behaviors, such as those induced by multiple timescales, are non-stationary. Learning non-stationary policies is challenging and typically requires sophisticated or inefficient algorithms. Motivated by the prevalence of this control problem in real-world complex systems, we introduce a simple framework for learning non-stationary policies for multi-timescale MARL. Our approach uses available information about agent timescales to define and learn periodic multi-agent policies. In detail, we theoretically demonstrate that the effects of non-stationarity introduced by multiple timescales can be learned by a periodic multi-agent policy. To learn such policies, we propose a policy gradient algorithm that parameterizes the actor and critic with phase-functioned neural networks, which provide an inductive bias for periodicity. The framework's ability to effectively learn multi-timescale policies is validated on a gridworld and building energy management environment.
Deep Multi-Agent Reinforcement Learning for Real-World Signalized Traffic Corridor Control
Signalized traffic control problem has been addressed recently with deep Reinforcement Learning (RL) approaches involving diverse state, action, and reward structures. While significant progress has been noted in the literature, open challenges still remain in the areas of adaptive signal phase timing, coordination in a multi-intersection corridor setting, and consideration of real-world traffic conditions. In the context of deep RL-based problem framing, extensions are needed that enable adaptive signal phase timings in an intersection agent's action space, computationally efficient information sharing among neighboring signalized intersection agents along a corridor, and experimentation in realistic simulation environments. In this paper, we develop a deep Advantage Actor Critic (A2C) multi-agent RL (MARL) approach capturing the research extensions above and apply it within a real-world calibrated Aimsun Next traffic corridor simulation model based on traffic data from the City of Coral Gables, Florida. For a multi-intersection corridor control setting, our numerical simulation experiments with a decentralized A2C MARL algorithm applied at different time periods led to a total average corridor travel delay reduction (expressed in seconds/mile averaged over vehicles) from 4.9% to 19.9% compared to state-of-the-art actuated control.
Buyers Collusion in Incentivized Forwarding Networks: A Multi-Agent Reinforcement Learning Study
We present the issue of monetarily incentivized forwarding in a multi-hop mesh network architecture from an economic perspective. It is anticipated that credit-incentivized forwarding and relaying will be a simple method of exchanging transmission power and spectrum for connectivity. However, gateways and forwarding nodes, like any other free market, may create an oligopolistic market for the users they serve. In this study, a coalition scheme between buyers aims to address price control by gateways or nodes closer to gateways. In a Stackelberg competition game, buyer agents (users) and sellers (gateways) make decisions using reinforcement learning (RL), with decentralized Deep Q-Networks to buy and sell forwarding resources. We allow communication links between the buyers with a limited messaging space, without defining a collusion mechanism. The idea is to demonstrate that through messaging, and RL tacit collusion can emerge between agents in a decentralized setup. The multi-agent reinforcement learning (MARL) system is presented and analyzed from a machine-learning perspective. Moreover, MARL dynamics are discussed via mean field analysis to better understand divergence causes and make implementation recommendations for such systems. Finally, the simulation results show the results of coordination among the users.
PowerGridworld: A Framework for Multi-Agent Reinforcement Learning in Power Systems: Preprint
We present the PowerGridworld software package to provide users with a light-weight, modular, and customizable framework for creating power systems-focused, multi-agent gym environments that readily integrate with existing training frameworks for reinforcement learning (RL). While many frameworks exist for training multi-agent (MA) RL policies, none exist to rapidly prototype and develop the environments themselves, especially in the context of heterogeneous (composite, multi-device) power systems where power flow solutions are required to define grid-level variables and costs. PowerGridworld is an open-source software package that helps to fill this gap. To highlight PowerGridworld's key features, we present two case studies and demonstrate learning multi-agent RL policies using both OpenAI's MADDPG and RLLib's PPO algorithms where, in both cases, at least some subset of agents incorporate elements of the power flow solution at each time step as part of their reward (negative cost) structures.
Decentralized Voltage Control with Peer-to-peer Energy Trading in a Distribution Network
Utilizing distributed renewable and energy storage resources via peer-to-peer (P2P) energy trading has long been touted as a solution to improve energy system’s resilience and sustainability. Consumers and prosumers (those who have energy generation resources), however, do not have expertise to engage in repeated P2P trading, and the zero-marginal costs of renewables present challenges in determining fair market prices. To address these issues, we propose a multi-agent reinforcement learning (MARL) framework to help automate consumers’ bidding and management of their solar PV and energy storage resources, under a specific P2P clearing mechanism that utilizes the so-called supply-demand ratio. In addition, we show how the MARL framework can integrate physical network constraints to realize decentralized voltage control, hence ensuring physical feasibility of the P2P energy trading and paving ways for real-world implementations.
Integrating Aquatic Metabolism and Net Ecosystem CO2 Balance in Short- and Long-Hydroperiod Subtropical Freshwater Wetlands
Abstract How aquatic primary productivity influences the carbon (C) sequestering capacity of wetlands is uncertain. We evaluated the magnitude and variability in aquatic C dynamics and compared them to net ecosystem CO 2 exchange (NEE) and ecosystem respiration ( R eco ) rates within calcareous freshwater wetlands in Everglades National Park. We continuously recorded 30-min measurements of dissolved oxygen (DO), water level, water temperature ( T water ), and photosynthetically active radiation (PAR). These measurements were coupled with ecosystem CO 2 fluxes over 5 years (2012–2016) in a long-hydroperiod peat-rich, freshwater marsh and a short-hydroperiod, freshwater marl prairie. Daily net aquatic primary productivity (NAPP) rates indicated both wetlands were generally net heterotrophic. Gross aquatic primary productivity (GAPP) ranged from 0 to − 6.3 g C m −2 day −1 and aquatic respiration ( R Aq ) from 0 to 6.13 g C m −2 day −1 . Nonlinear interactions between water level, T water , and GAPP and R Aq resulted in high variability in NAPP that contributed to NEE. Net aquatic primary productivity accounted for 4–5% of the deviance explained in NEE rates. With respect to the flux magnitude, daily NAPP was a greater proportion of daily NEE at the long-hydroperiod site (mean = 95%) compared to the short-hydroperiod site (mean = 64%). Although we have confirmed the significant contribution of NAPP to NEE in both long- and short-hydroperiod freshwater wetlands, the decoupling of the aquatic and ecosystem fluxes could largely depend on emergent vegetation, the carbonate cycle, and the lateral C flux.
Multi-Agent Reinforcement Learning for Distribution System Critical Load Restoration
Grid resilience has become a critical topic recently because of the increasing occurrence of extreme events and the growing integration of intermittent renewable energy sources. To build a resilient distribution system, this paper develops a multiagent reinforcement learning-based (MARL) method to coordinate distribution energy resources (DERs) dispatch, load pickup, and network reconfiguration for load restoration after a system outage. With the help of two types of control agents, namely critical load restoration (CLR) and coordination (COR) agents, system loads can be restored efficiently, given available resources. The effectiveness and superiority of the proposed algorithm are demonstrated through simulations and comparative studies on a real distribution feeder in Western Colorado.
Signal Whisperers: Enhancing Wireless Reception Using DRL-Guided Reflector Arrays
This paper presents a multi-agent reinforcement learning (MARL) approach for controlling adjustable metallic reflector arrays to enhance wireless signal reception in non-line-of-sight (NLOS) scenarios. Unlike conventional reconfigurable intelligent surfaces (RIS) that require complex channel estimation, our system employs a centralized training with decentralized execution (CTDE) paradigm where individual agents corresponding to reflector segments autonomously optimize reflector element orientation in three-dimensional space using spatial intelligence based on user location information. Through extensive ray-tracing simulations with dynamic user mobility, the proposed multi-agent beam-focusing framework demonstrates substantial performance improvements over single-agent reinforcement learning baselines, while maintaining rapid adaptation to user movement within one simulation step. Comprehensive evaluation across varying user densities and reflector configurations validates system scalability and robustness. The results demonstrate the potential of learning-based approaches for adaptive wireless propagation control.
Final Technical Report: Center for Mechanistic Control of Unconventional Formations
The overarching mission of the Center for Mechanistic Control of Unconventional Formations (CMC-UF) was to garner cross-cutting, fundamental, geoscience knowledge to achieve mechanistic control over the strongly coupled nonequilibrium physical and geochemical processes in extreme geological environments including shale, mudstone, marls, and other tight rocks with nanoscale pores. Collectively, these are referred to as unconventional formations and they often play the role of seals for other subsurface storage formations. The fundamental knowledge garnered by CMC-UF enabled science-based management of US unconventional formations for subsurface storage of carbon dioxide and TWh quantities of renewable energy as hydrogen and/or compressed air over longer timeframes as well as for natural gas production, with reduced environmental impacts, in the short term.
Uranium Retardation Capacity of Lithologies from the Negev Desert, Israel—Rock Characterization and Sorption Experiments
A series of batch experiments were performed to assess the uranium sorption capacity of four mineralogically distinct lithologies from the Negev Desert, Israel, to evaluate the suitability of a potential site for subsurface radioactive waste disposal. The rock specimens consisted of an organic-rich phosphorite, a bituminous marl, a chalk, and a sandstone. The sorption data for each lithology were fitted using a general composite surface complexation model (GC SCM) implemented in PHREEQC. Sorption data were also fitted by a non-mechanistic Langmuir sorption isotherm, which can be used as an alternative to the GC SCM to provide a more computationally efficient method for uranium sorption. This is because all the rocks tested have high pH/alkalinity/calcium buffering capacities that restrict groundwater chemistry variations, so that the use of a GC SCM is not advantageous. The mineralogy of the rocks points to several dominant sorption phases for uranyl (UO 2 2+ ), including apatite, organic carbon, clays, and iron-bearing phases. The surface complexation parameters based on literature values for the minerals identified overestimate the uranium sorption capacities, so that for our application, an empirical approach that makes direct use of the experimental data to estimate mineral-specific sorption parameters appears to be more practical for predicting uranium sorption.