Fully Decentralized Reinforcement Learning-Based Control of Photovoltaics in Distribution Grids for Joint Provision of Real and Reactive Power
Not Available
El Helou, Rayan (ORCID:0000000259588019)↗
SEARCH · Engineering Papers
Results for “DECENTRALIZED CONTROL”
Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.
Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.
Machine Learning for Scalable and Optimal Load Shedding Under Power System Contingency
Prompt and effective corrective actions in response to unexpected contingencies are crucial for improving power system resilience and preventing cascading blackouts. The optimal load shedding (OLS) accounting for network limits has the potential to address the diverse system-wide impacts of contingency scenarios as compared to traditional local schemes. However, due to the fast cascading propagation of initial contingencies, real-time OLS solutions are challenging to attain in large systems with high computation and communication needs. In this paper, we propose a decentralized design that leverages offline training of a neural network (NN) model for individual load centers to autonomously construct the OLS solutions from locally available measurements. Our learning-for-OLS approach can greatly reduce the computation and communication needs during online emergency responses, thus preventing the cascading propagation of contingencies for enhanced power grid resilience. Numerical studies on both the IEEE 118-bus system and a synthetic Texas 2000-bus system have demonstrated the efficiency and effectiveness of our scalable OLS learning design for timely power system emergency operations.
24 POWER TRANSMISSION AND DISTRIBUTION↗
The Transactive Energy Network Template Metamodel
While transactive energy, which is defined as an allocation of electricity based on dynamically discovered values or prices, has been extensively studied, its uptake and use has been slow. This report describes a tool, the transactive network template, which should hasten the creation and uptake of transactive energy networks. Some basic principles of transactive energy are familiar from existing wholesale electricity markets. Locational prices are calculated today for zones within bulk electric transmission systems. Locational prices differ while accounting for the locational costs of electricity generation and the losses and constraints incurred when electricity is transmitted from generators and distributed to consumers. A transactive energy network might include these transmission zones. However, current research strives to apply transactive energy also in electricity distribution circuits, buildings, and even for individual generating and consuming devices. At the same time, researchers explore how to apply transactive energy in real time during increasingly shorter time intervals. Automated computational agents become necessary as transactive energy becomes applied to smaller circuit zones and at faster dynamic timescales. A transactive energy network is an example of a multi-agent system. Each zone in the network is represented by its transactive agent, which makes decisions for and acts on behalf of a business entity that is responsible for and manages one of the circuit regions. A transactive energy network is also an example of a decentralized, distributed control system. Control decisions and responsibilities are distributed among the network’s transactive agents. The transactive agents are independent; that is, there typically is no centralized authority or oversight function. Instead, transactive agents exchange transactive signals and thereby negotiate the prices and quantities of electricity that they will exchange. Initially, the circuit regions and responsibilities of transactive agents appear to be very dissimilar. Each circuit region may comprise transmission, distribution, or building-level circuits. Each has a unique position and electrical connectivity within the transactive energy network. Each possesses unique assets that either generate or consume electricity, and these (e.g., renewable energy generator, diesel generator, aggregate utility load, building load, space conditioning, refrigerator, etc.) may further differ in their price flexibility and in their strategies for responding to dynamic electricity prices. Given such diversity, an implementer’s first inclination might be to start from scratch to define all these devices and to engineer their seemingly unique interactions. Given that each implementer’s perspective may be narrow within a transactive energy network, it is unlikely that uniquely engineered systems would interact well. This is where the transactive network template is applicable. The transactive network template is a metamodel that has been developed to guide implementers as they configure their own transactive agent within a network of such agents. The object-oriented design of the transactive network template provides basic code object types that may be used and extended by implementers to represent each of the assets in their circuit region. These objects further facilitate the transactive agent’s necessary computations, which are divided among responsibilities to schedule power usage, balance electric supply and demand, and coordinate the exchange of electricity with the other transactive agents. This report addresses the conceptual transactive network template design. Implementers are directed to more formal design documents and reference implementations. A Python™-based1 reference implementation of the transactive network template has been coded, and three implementations have been configured to represent a national laboratory and two university campuses. Version 2 of the transactive node template generalizes the market class and its methods to facilitate multiple, and more diverse market coordination mechanisms than were facilitated by and demonstrated using Version 1. Version 3 includes new Appendix B, which addresses the designs of methods that would make dynamic prices track approved electricity rates. In the future, the author wishes to make the transactive network template more generally applicable to networks that require more accurate power flow. Development of the transactive network template is jointly funded by the U.S. Department of Energy (DOE) Energy Efficiency and Renewable Energy and the DOE Office of Electricity. In late 2015, one of the first projects to be funded by the DOE Grid Laboratory Modernization Laboratory Consortium was the Clean Energy and Transactive Campus project, led by Pacific Northwest National Laboratory. DOE funds were matched by an investment by the Washington Department of Commerce through its Clean Energy Fund. The transactive network template was developed to guide the implementation of transactive energy networks within this project’s scope.
24 POWER TRANSMISSION AND DISTRIBUTION↗
Correlation-Aided Robust Decentralized Dynamic State Estimation of Power Systems with Unknown Control Inputs
This paper proposes a correlation-aided robust adaptive unscented Kalman filter for power system decentralized dynamic state estimation with unknown inputs, termed as robust AUKF-UI. The temporal and spatial correlations among the unknown inputs are used to derive a vector auto-regressive (VAR) model in an adaptive manner. This VAR model is further integrated together with state transition and measurement models for joint state and unknown inputs estimation. This allows taking into account the implicit cross-correlations between the states and the unknown inputs. As a result, the rank requirement for unknown input vector estimation is relaxed and the local generator frequency measurement is not required. The temporal correlations of time series innovation vectors, predicted state and input vectors are also leveraged by the robust AUKFUI to detect, identify and process bad data. Without these correlations, it is very challenging to address bad data with unknown inputs. Simulation results carried out on the IEEE 39-bus system demonstrate that the proposed robust AUKF-UI achieves much better results than other methods in the presence of low measurement redundancy, strong nonlinearity, and bad data.
Zhao, Junbo↗
Near-Optimal Distributed Linear-Quadratic Regulator for Networked Systems
This paper studies the trade-off between the degree of decentralization and the performance of a distributed controller in a linear-quadratic control setting. We study a system of interconnected agents over a graph and a distributed controller, called k-distributed control, which lets the agents make control decisions based on the state information within distance k on the underlying graph. This controller can tune its degree of decentralization using the parameter k and thus allows a characterization of the relationship between decentralization and performance. We show that under mild assumptions, including stabilizability, detectability, and a subexponentially growing graph condition, the performance difference between k-distributed control and centralized optimal control becomes exponentially small in k. Finally, this result reveals that distributed control can achieve near-optimal performance with a moderate degree of decentralization, and thus it is an effective controller architecture for large-scale networked systems.
97 MATHEMATICS AND COMPUTING↗
Envisioning the Future Renewable and Resilient Energy Grids—A Power Grid Revolution Enabled by Renewables, Energy Storage, and Energy Electronics
Today’s power grids are facing tremendous challenges because of the ever-increasing power demand, system complexity, infrastructure cost, knowledge base, and policy and regulatory issues to achieve supply–demand power balance and resiliency with respect to more frequent extreme weather events and cyberattacks. It is particularly challenging when the transition toward 100% intermittent renewable energy sources is considered. Many countries are calling for building up more transmission and distribution lines to increase power delivery capacities. This article is an attempt to answer two urgent questions: Is more transmission and distribution infrastructure really needed to meet the increasing power demand? What kind of future grid infrastructure should we envision and build? This article attempts to answer these questions and proposes the concept of community-centric asynchronous renewable and resilient energy grids. By clearly differentiating the concepts of grid resilience and reliability, the importance of building resilient power electronics’ devices and robust system-level control algorithms to achieve 100% renewable energy integrated resilient grids is presented. To identify the shortcomings and propose advancements, power electronics’ technologies are categorized using the proposed concepts of natural source frequencies (NSf), energy storage, direct energy conversion/control and fault protection (DeCaFp), and high-efficiency energy consumption and buffering (heECaB) technology. The ability of networked microgrids to greatly reduce power outages and power system restoration time is demonstrated by leveraging robust decentralized and centralized control algorithms, identified through a comprehensive literature review. Future research areas are proposed to further enhance grid stability, controllability, cybersecurity, and protection against faults in the presence of 100% renewable sources by leveraging the advanced capabilities of NSf, DeCaFp, and heECaB devices and system-level control algorithms.
14 SOLAR ENERGY↗
A Fully Decentralized Modulation Scheme for Modular Multilevel Converters
As the number of submodules rapidly increase, control architecture of Modular Multilevel Converters (MMCs) is transforming from centralized to distributed in order to address the challenges of tremendous computational burden and massive wiring. In this digest, we propose a modulation scheme for the MMCs with distributed controls, where each submodule is able to handle both controls and modulation in a decentralized manner. It enables the distributed controllers to synchronize in terms of fast-switching pulse width modulation (PWM). Aiming at this key technical challenge, this research develops a physics-informed PWM control strategy of mirroring the spontaneous synchronization feature of coupled oscillator’s behavior at switching frequency. Thanks to less reliance on wirings and communications, the proposed approach has the potential of i) improving reliability of MMC systems, and ii) reducing cost and space requirements for MMCs. The proposed approach has been preliminarily verified through a simulation tool for power electronics systems.
Lu, Minghui [BATTELLE (PACIFIC NW LAB)]↗
PowerNet: Multi-agent Deep Reinforcement Learning for Scalable Powergrid Control
This paper develops an efficient multi-agent deep reinforcement learning algorithm for cooperative controls in powergrids. Specifically, we consider the decentralized inverter-based secondary voltage control problem in distributed generators (DGs), which is first formulated as a cooperative multi-agent reinforcement learning (MARL) problem. We then propose a novel on-policy MARL algorithm, PowerNet, in which each agent (DG) learns a control policy based on (sub-)global reward but local states and encoded communication messages from its neighbors. Motivated by the fact that a local control from one agent has limited impact on agents distant from it, we exploit a novel spatial discount factor to reduce the effect from remote agents, to expedite the training process and improve scalability. Furthermore, a differentiable, learning-based communication protocol is employed to foster the collaborations among neighboring agents. In addition, to mitigate the effects of system uncertainty and random noise introduced during on-policy learning, we utilize an action smoothing factor to stabilize the policy execution. To facilitate training and evaluation, we develop PGSim, an efficient, high-fidelity powergrid simulation platform. Here, experimental results in two microgrid setups show that the developed PowerNet outperforms the conventional model-based control method, as well as several state-of-the-art MARL algorithms. The decentralized learning scheme and high sample efficiency also make it viable to large-scale power grids.
24 POWER TRANSMISSION AND DISTRIBUTION↗
A Hybrid Framework Combining Model-Based and Data-Driven Methods for Hierarchical Decentralized Robust Dynamic State Estimation
This paper combines model-based and data-driven methods to develop a hierarchical, decentralized, robust dynamic state estimator (DSE). A two-level hierarchy is proposed, where the lower level consists of robust, model-based, decentralized DSEs. The state estimates sent from the lower level are received at the upper level, where they are filtered by a robust data-driven DSE after a principled sparse selection. This selection allows us to shrink the dimension of the problem at the upper level and hence significantly speed up the computational time. The proposed hybrid framework does not depend on the centralized infrastructure of the control centers; thus it can be completely embedded into the wide-area measurement systems. This feature will ultimately facilitate the placement of hierarchical decentralized control schemes at the phasor data concentrator locations. Also, the network model is not necessary; thus, a topology processor is not required. Finally, there is no assumption on the dynamics of the electric loads. The proposed framework is tested on the 2,000-bus synthetic Texas system, and shown to be capable of reconstructing the dynamic states of the generators with high accuracy, and of forecasting in the advent of missing data.
24 POWER TRANSMISSION AND DISTRIBUTION↗
Solid-State Transformer and Hybrid Transformer With Integrated Energy Storage in Active Distribution Grids: Technical and Economic Comparison, Dispatch, and Control
Solid-state transformer (SST) and hybrid transformer (HT) are promising alternatives to the line-frequency transformer (LFT) in smart grids. The SST features medium-frequency isolation, full controllability for voltage regulation, reactive power compensation, and the capability of battery energy storage system (BESS) integration with multiport configuration. The HT has a partially-rated converter for fractional controllability and can integrate a small BESS. Fast grid-edge voltage fluctuations from increased solar photovoltaic (PV) and electric vehicle (EV) penetration are difficult to manage for mechanical load tap changers. Hence, along with the trend towards more BESS in the grid, the controllability and the storage integration capability of the SST and HT are of strong interest. However, a review of literature shows existing SST and HT research is mostly at converter level, while system-level assessments are scarce. Assessing technical and economic impacts is critical to understanding the benefits and role of the SST and HT to guide future research, which is presented for the first time in this article. Experimental results from medium-voltage (MV) SST and MV HT prototypes are shown to confirm equipment-level feasibility, where the voltage controllability waveforms of a MV HT prototype are reported for the first time. Comparative simulations are performed on a modified IEEE 34-bus system. Here, a grid-model-less decentralized grid-edge voltage control method and a day-ahead BESS dispatch method are proposed for the SST and HT. The simulations show that the SST and HT with integrated storage can host more PV, achieve peak shaving, mitigate voltage fluctuation and reverse power flow, and support energy arbitrage for operational cost reduction, as compared to the LFT. Moreover, comprehensive analyses of net present value (NPV) and internal rate of return (IRR) are performed under different installed PV capacities, HT’s partial converter ratings, and BESS capacities. Sensitivities to future cost reductions of the PV and BESS are studied. Although the NPV and IRR are currently negative, 60% capital cost reduction or 150% revenue increase will make the SST and HT economically viable in the use case studied.
14 SOLAR ENERGY↗
Federated Deep Reinforcement Learning for Decentralized VVO of BTM DERs
The future of grid control requires a hybrid approach combining centralized and decentralized methods to fully utilize the potential of smart edge devices with artificial intelligence (AI) capabilities. This paper aims to develop and evaluate a federated deep reinforcement learning (FDRL) framework for decentralized adaptive volt-var optimization (VVO) of behind-the-meter (BTM) distributed energy resources (DERs). First, this paper models a single deep reinforcement learning (DRL) agent using the Markov Decision Process (MDP) framework for decentralized adaptive VVO of BTM DERs. Two DRL algorithms, soft actor-critic (SAC) and twin-delayed deep deterministic policy gradient (TD3), are compared for their effectiveness in optimizing VVO. Results show that TD3 outperforms SAC, achieving a 71.3% improvement in mean reward. Finally, the DRL agent is deployed within the FDRL framework, using the Flower platform, to enhance learning, provide adaptive control, and ensure data privacy for BTM DERs.
Ravi, Abhijith↗
HumanLight: Incentivizing ridesharing via human-centric deep reinforcement learning in traffic signal control
Not Available
33 ADVANCED PROPULSION SYSTEMS↗
Mitigating Phase Unbalance for Distribution Systems with High Penetrations of Solar PV (Final Technical Report)
Distribution system operators have traditionally limited unbalance among phases by maintaining similar loadings on each phase. High penetrations of distributed solar PV continually change the net loading on each phase, resulting in time-varying phase unbalances that can damage three-phase devices such as three-phase motors, violate grid codes, and increase technical losses. This project has developed several control strategies for the reactive power outputs of solar PV inverters in order to mitigate power quality issues related to phase unbalance. These control strategies include a decentralized approach that is solely based on local measurements, distributed and grouped approaches that consider subsets of loads and PV generators, and a centralized approach that leverages measurements from a variety of locations in order to compute optimal reactive power setpoints for each inverter. Variants of the controllers handle challenges relevant to practical implementations, including noisy measurements, delayed communications, and reactive power limits. Moreover, the project developed theory that provides convergence guarantees for systems with multiple interacting controllers as well as “balanceability” certificates that ensure satisfaction of phase unbalance requirements with variable loading. The controllers were integrated with NRECA’s Open Modeling Framework (omf.coop) and evaluated using actual distribution system models obtained from several NRECA member utilities. Application of the controllers results in significant improvements to phase unbalance in these test cases with decreases from base case levels of over 3% to under 0.5%, which is within the 2% IEC phase unbalance standard.
14 SOLAR ENERGY↗
Communication-Constrained Robust Control and Learning of Grid-Connected
The electric grid of things (EGoT) promises great potential for innovative grid services by tapping into vast load flexibility. However, the unique characteristics of EGoT, being a part of the cyber-physical electric power system, present both opportunities and challenges, especially concerning supply-demand balancing, stability, and communication constraints. Traditionally, centralized control was employed to ensure balance and stability in power systems. However, with the massive influx of EGoT devices, new strategies are needed to efficiently coordinate and control these distributed devices for optimal grid operations. While some studies have explored efficiency and economic models, there remains a gap in ensuring reliability under everyday operations and resilience during extreme conditions. Addressing this gap, this project develops the technology for an Energy Service Interface (ESI) that includes novel pricing, control, learning, and distributed optimization algorithms, which will enable utilities to recruit EGoT assets for crucial grid services such as load flexibility, voltage regulation, and situation-awareness. The key novelty of the proposed technology is the careful distribution of learning and control functions across utility and EGoT asset owners such that provably efficient and resilient grid operations are attained while respecting communication and information-exchange constraints. Specifically, the project team develops machine-learning enhanced load modeling methods to allow EGoT asset owners to learn their load capability and flexibility, and develops pricing-based and decentralized learning-based control so that asset owners can coordinate to meet system-wide demand-supply balance and reliability goals. For extreme situations involving high-impact, low-probability catastrophic events (termed the “black-sky” events), the team also develops (1) a “Feeder-Operating Center-on-a-Laptop” (FOCAL) software that can assist utility personnel in leveraging EGoT assets to accelerate the service recovery of damaged feeders, and (2) distributed optimization algorithms that can coordinate the operation points of EGoT devices under severe communication constraints. The proposed technology has been extensively tested and evaluated through simulations and on a testbed. In summary, as we transition into a more interconnected and digital power grid era, our project’s findings and developments offer a pivotal step toward guaranteeing both efficiency and resilience in the face of both everyday operations and rare “black-sky” events.
24 POWER TRANSMISSION AND DISTRIBUTION↗
MPC-Based Local Voltage Control Strategy of DGs in Active Distribution Networks
The increasing penetration of distributed generators (DGs) exacerbates the voltage violations in active distribution networks (ADNs). The commonly used centralized voltage control may aggravate the burden of computation and communication. Local voltage control can realize fast response to frequent voltage fluctuations. However, effective local control for DGs still needs to be further investigated. In this paper, a model predictive control (MPC)-based local voltage control strategy of DGs is proposed to cope with voltage violation problems and enhance the adaptation to the DG volatility. DGs are regulated with local Q-V control curves to respond to voltage fluctuations rapidly. Based on the dynamic network partition, a unified control curve for DGs is set in the same area to realize efficient operation. To further improve the performance of voltage control, the parameter tuning of control curves is regularly conducted based on decentralized inter-area coordination with the rolling of the control horizon. Finally, case study is verified on the modified IEEE 33-node distribution system. The proposed local control strategy of DGs can effectively improve the voltage profile and reduce power losses of the whole network without high computation and communication burden.
active distribution network (ADN)↗
Constellation: The autonomous control and data acquisition system for dynamic experimental setups
The operation of instruments and detectors in laboratory or beamline environments presents a complex challenge, requiring stable operation of multiple concurrent devices, often controlled by separate hardware and software solutions. These environments frequently undergo modifications, such as the inclusion of different auxiliary devices depending on the experiment or facility, adding further complexity. The successful management of such dynamic configurations demands a flexible and robust system capable of controlling data acquisition, monitoring experimental setups, enabling seamless reconfiguration, and integrating new devices with limited effort. This paper presents Constellation, a flexible and network-distributed control and data acquisition software framework tailored to laboratory and beamline environments, that addresses the limitations of existing solutions. The framework is designed with a focus on extensibility, providing a streamlined interface for instrument integration. It supports efficient system setup via network discovery mechanisms, promotes stability through autonomous operational features, and provides comprehensive documentation and supporting tools for operators and application developers such as controllers and logging interfaces. At the core of the architectural design is the autonomy of the individual components, called satellites, which can make independent decisions about their operation and communicate these decisions to other components. This paper introduces the design principles and framework architecture of Constellation, presents the available graphical user interfaces, shares insights from initial successful deployments, and provides an outlook on future developments and applications.
Autonomy↗
Solar Energy CommUnity Resiliency (Final Technical Report)
Final Technical Report for DE-EE0009336 Solar Energy CommUnity Resiliency (SECURE) The Solar Energy CommUnity Resiliency (SECURE) project aimed to support development of resilient community microgrids to improve grid reliability and maintain power during times of crisis. The SECURE project approached this objective by addressing key technical and business challenges impeding implementation of resilient community microgrids. To achieve the goals of this project, we pursued several specific objectives through comprehensive design and requirements development.
24 POWER TRANSMISSION AND DISTRIBUTION↗