Learning and Adapting Model Predictive Controllers with Reinforcement Learning for Time-Varying Systems
Presented by NETL at 2019 AIChE Annual Meeting, 11/19
Zitney, Steve↗
SEARCH · Engineering Papers
Results for “Adaptive Control”
Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.
Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.
Adaptive Protection and Control for High Penetration PV and Grid Resilience.
Abstract not provided.
Reno, Matthew J.↗
Adaptive strong-field control of vibrational population in NO 2+
An adaptive closed-loop system employing coincidence time-of-flight feedback is used to determine the optimal pulse shapes for manipulating the branching ratio of NO dications following double ionization by an intense laser pulse. Selection between the long-lived NO 2+ and the dissociative N + + O + final states requires control of the vibrational population distribution in the transient NO 2+ . The ability to both suppress and enhance NO 2+ relative to N + + O + is observed, with the effectiveness of shaped pulses surpassing near Fourier transform-limited pulses by about an order of magnitude in each direction, depending on the pulse energy. The control is subsequently investigated using velocity map imaging, identifying plausible dissociation pathways leading to N + + O + . Combining the information about the N + + O + dissociation with a well-defined control objective supports the conclusion that the primary control mechanism involves selectively populating long-lived NO 2+ vibrational states.
37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗
ARMAX-Based Transfer Function Model Identification Using Wide-Area Measurement for Adaptive and Coordinated Damping Control
One of the main drawbacks of the existing oscillation damping controllers that are designed based on offline dynamic models is adaptivity to the power system operating condition. With the increasing availability of wide-area measurements and the rapid development of system identification techniques, it is possible to identify a measurement-based transfer function model online that can be used to tune the oscillation damping controller. Such a model could capture all dominant oscillation modes for adaptive and coordinated oscillation damping control. our paper describes a comprehensive approach to identify a low-order transfer function model of a power system using a multi-input multi-output (MIMO) autoregressive moving average exogenous (ARMAX) model. This methodology consists of five steps: 1) input selection; 2) output selection; 3) identification trigger; 4) model estimation; and 5) model validation. The proposed method is validated by using ambient data and ring-down data in the 16-machine 68-bus Northeast Power Coordinating Council system. Our results demonstrate that the measurement-based model using MIMO ARMAX can capture all the dominant oscillation modes. Compared with the MIMO subspace state space model, the MIMO ARMAX model has equivalent accuracy but lower order and improved computational efficiency. The proposed model can be applied for adaptive and coordinated oscillation damping control.
24 POWER TRANSMISSION AND DISTRIBUTION↗
Adaptive Power System Emergency Control using Deep Reinforcement Learning
Power system emergency control is generally regarded as the final safety net for grid security and resiliency. Existing emergency control schemes are usually designed off-line based on either the conceived “worst” case scenarios or a few typical operation scenarios. These schemes are facing significant adaptiveness and robustness issues as increasing uncertainties and variations occur in modern electrical grids. To address these challenges, for the first time, this paper proposes a novel adaptive emergency control scheme using deep reinforcement learning (DRL), by leveraging the high-dimensional feature extraction and non-linear generalization capabilities DRL has for complex systems with high-dimensional variations. Furthermore, an open-source platform named DeepGrid has been designed for the first time to assist the DRL development and benchmarking processes in power system emergency control. Details of the platform, DRL, and emergency control schemes that use dynamic braking or under-voltage load shedding are presented. Extensive case studies performed in both two-area four-machine system and IEEE 39-Bus system have demonstrated the excellent performance and robustness of the proposed schemes.
Deep reinforcement learning, emergency control, lo↗
Adaptive Reinforcement Learning (ARL) Control of a Multi-port Resonant Converter in UAV Systems
This study presents an adaptive reinforcement learning (ARL) control framework for a multi-port resonant converter used in hybrid unmanned aerial vehicle (UAV) power systems. The converter integrates high-frequency half-bridge input ports connected to a rectified engine–generator set and a battery energy storage system, along with a semi-bridgeless active rectifier supplying the propulsion load. A deep RL agent is trained to dynamically regulate inter-port phase-shift commands in real time based on flight conditions and load power demand. The ARL controller autonomously identifies phase-shift combinations that maximize conversion efficiency while maintaining stable and coordinated power flow, even under rapidly varying operating scenarios. This data-driven approach eliminates the need for explicit system modeling or extensive manual tuning and enables coordinated control among multiple power ports without inter-port communication. Experimental results validate that the ARL based strategy achieves reliable power sharing and consistently high-efficiency operation across diverse UAV operating conditions.
Asa, Erdem [ORNL] (ORCID:0000000190884812)↗
Adaptive fuzzy dynamic surface control for the chaotic permanent magnet synchronous motor using Nussbaum gain
This paper is concerned with the problem of adaptive fuzzy dynamic surface control (DSC) for the permanent magnet synchronous motor (PMSM) system with chaotic behavior, disturbance and unknown control gain and parameters. Nussbaum gain is adopted to cope with the situation that the control gain is unknown. And the unknown items can be estimated by fuzzy logic system. The proposed controller guarantees that all the signals in the closed-loop system are bounded and the system output eventually converges to a small neighborhood of the desired reference signal. Finally, the numerical simulations indicate that the proposed scheme can suppress the chaos of PMSM and show the effectiveness and robustness of the proposed method.
71 CLASSICAL AND QUANTUM MECHANICS, GENERAL PHYSIC↗
Developing an Energy-Conscious Traffic Signal Control System for Optimized Fuel Consumption in Connected Vehicle Environments
The project titled “Developing an Energy-Conscious Traffic Signal Control System for Optimized Fuel Consumption in Connected Vehicle Environments” addresses energy-related challenges associated with adaptive traffic control systems by integrating connected vehicles (CV) and connected infrastructure (CI). The system developed in this project, a CV-based adaptive traffic control system, aims to improve fuel consumption in mixed traffic environments by capitalizing on emerging CV and CI communication technologies, as well as leveraging recent advances in Artificial Intelligence (AI), optimization, and edge computing. The system was tested at the MLK Smart Corridor, an urban testbed managed by the University of Tennessee at Chattanooga (UTC) and the City of Chattanooga. The system was validated through extensive simulations, both Software-in-the-Loop (SILS) and Hardware-in-the-Loop (HILS), and was further implemented and tested in real-world conditions at several intersections along the corridor. The Fuel Consumption Performance Index (FC-PI) and the Ecological Performance Index (Eco-PI) were developed as the key components for evaluating the system’s impact on fuel consumption and emissions. These metrics provided a comprehensive means of understanding the impact of traffic signal control optimization in mixed traffic environments. The report presents an in-depth analysis of the Eco-PI, FC-PI, adaptive traffic control system integration, and the testing and field implementation of the system. The results demonstrate significant reductions in fuel consumption and emissions, showcasing the system’s capability to contribute to more sustainable urban traffic management. The report also documents the challenges encountered and recommendations for scaling and further improving the system.
32 ENERGY CONSERVATION, CONSUMPTION, AND UTILIZATI↗
Adaptive neural network backstepping control for a class of uncertain fractional-order chaotic systems with unknown backlash-like hysteresis
In this paper, we consider the control problem of a class of uncertain fractional-order chaotic systems preceded by unknown backlash-like hysteresis nonlinearities based on backstepping control algorithm. We model the hysteresis by using a differential equation. Based on the fractional Lyapunov stability criterion and the backstepping algorithm procedures, an adaptive neural network controller is driven. No knowledge of the upper bound of the disturbance and system uncertainty is required in our controller, and the asymptotical convergence of the tracking error can be guaranteed. Finally, we give two simulation examples to confirm our theoretical results.
71 CLASSICAL AND QUANTUM MECHANICS, GENERAL PHYSIC↗
Secondary Frequency Control for Reconfigurable Interconnecting Microgrids
Microgrids and microgrid technologies offer an operations and control framework for realizing the needed aggregation of a growing share of distributed energy resources. As the number of microgrids increases, so will the incentive to interconnect them, leading to a combinatorial growth of potential operating states. At the same time, their growing complexity nudges interconnected microgrids conceptually closer to balancing authorities in the bulk power system. This paper focuses on secondary frequency control and leverages the conceptual similarity between balancing authorities and interconnected microgrids to adopt an automatic generation control design with some adaptations. Frequency control and restoration is demonstrated via several use-cases with switching and reconfiguration events.
Schweitzer, Eran↗
Integrated Framework of Vehicle Dynamics, Instabilities, Energy Models, and Sparse Flow Smoothing Controllers
This work presents an integrated framework of: vehicle dynamics models, with a particular attention to instabilities and traffic waves; vehicle energy models, with particular attention to accurate energy values for strongly unsteady driving profiles; and sparse Lagrangian controls via automated vehicles, with a focus on controls that can be executed via existing technology such as adaptive cruise control systems. This framework serves as a key building block in developing control strategies for human-in-the-loop traffic flow smoothing on real highways. In this contribution, we outline the fundamental merits of integrating vehicle dynamics and energy modeling into a single framework, and we demonstrate the energy impact of sparse flow smoothing controllers via simulation results.
Lee, Jonathan W.↗
NEXT GENERATION INTELLIGENT TRAFFIC SIGNALS FOR THE MULTIMODAL, SHARED, AND AUTOMATED FUTURE
Explore the source record for details and available documents.
32 ENERGY CONSERVATION, CONSUMPTION, AND UTILIZATI↗
Nonlinear burn control in ITER using adaptive allocation of actuators with uncertain dynamics
Abstract ITER will be the first tokamak to sustain a fusion-producing, or burning, plasma. If the plasma temperature were to inadvertently rise in this burning regime, the positive correlation between temperature and the fusion reaction rate would establish a destabilizing positive feedback loop. Careful regulation of the plasma’s temperature and density, or burn control, is required to prevent these potentially reactor-damaging thermal excursions, neutralize disturbances and improve performance. In this work, a Lyapunov-based burn controller is designed using a full zero-dimensional nonlinear model. An adaptive estimator manages destabilizing uncertainties in the plasma confinement properties and the particle recycling conditions (caused by plasma–wall interactions). The controller regulates the plasma density with requests for deuterium and tritium particle injections. In ITER-like plasmas, the fusion-born alpha particles will primarily heat the plasma electrons, resulting in different electron and ion temperatures in the core. By considering separate response models for the electron and ion energies, the proposed controller can independently regulate the electron and ion temperatures by requesting that different amounts of auxiliary power be delivered to the electrons and ions. These two commands for a specific control effort (electron and ion heating) are sent to an actuator allocation module that optimally maps them to the heating actuators available to ITER: an electron cyclotron heating system (20 MW), an ion cyclotron heating system (20 MW), and two neutral beam injectors (16.5 MW each). Two different actuator allocators are presented in this work. The first actuator allocator finds the optimal mapping by solving a convex quadratic program that includes actuator saturation and rate limits. It is nonadaptive and assumes that the mapping between the commanded control efforts and the allocated actuators (i.e. the effector model) contains no uncertainties. The second actuator allocation module has an adaptive estimator to handle uncertainties in the effector model. This uncertainty includes actuator efficiencies, the fractions of neutral beam heating that are deposited into the plasma electrons and ions, and the tritium concentration of the fueling pellets. Furthermore, the adaptive allocator considers actuator dynamics (actuation lag) that contain uncertainty. This adaptive allocation algorithm is more computationally efficient than the aforementioned nonadaptive allocator because it is computed using dynamic update laws so that finding the solution to a static optimization problem is not required at every time step. A simulation study assesses the performance of the proposed adaptive burn controller augmented with each of the actuator allocation modules.
Physics↗
Off-policy deep reinforcement learning with automatic entropy adjustment for adaptive online grid emergency control
Electric overloading conditions and contingencies put modern power systems at risk of voltage collapse and blackouts. Load shedding is crucial to maintain voltage stability for grid emergency control. However, the rule- or model-based schemes rely on accurate dynamic system models and face considerable challenges in adapting to various operating conditions and uncertain event occurrences. Here, to address these issues, this paper proposes a novel deep reinforcement learning (DRL)-based voltage stability control algorithm with automatic entropy adjustment (AEA) for grid emergency control. Various dynamic network components for complex system operations are modeled to construct the DRL environment. An off-policy soft actor-critic architecture is developed to maximize the expected reward and policy entropy simultaneously. The AEA mechanism is proposed to facilitate the policy maximum entropy procedure, and the proposed method can automatically provide effective discrete and continuous actions against various fault scenarios. Our approach accomplishes high sampling efficiency, scalability, and auto-adaptivity of the control policies under high uncertainties. Comparative studies with the existing DRL-based control methods in IEEE benchmarks indicate salient performance improvement of the proposed method for dynamic system emergency control.
24 POWER TRANSMISSION AND DISTRIBUTION↗
Connected Traffic Signal Coordination Optimization Framework through Network-Wide Adaptive Linear Quadratic Regulator–Based Control Strategy
Traffic congestion in metropolitan areas causes several significant challenges, such as longer travel times, decreased productivity, increased fuel consumption and vehicle emissions, and even severe injuries during crashes. Traffic signal control is a management approach to reduce traffic congestion and allocate the appropriate right of way for safety and mobility efficiency, both in temporal and spatial domains. Here, this study proposes a network-wide adaptive signal control coordination optimization framework based on the linear quadratic regulator algorithm. The traffic flow conditions driven by signal control inputs are formulated based on their network-wide state-space representation. After modeling traffic control regulation constraints, an adaptive linear quadratic regulator algorithm is designed to maximize the network-wide total throughput under the current conditions. Optimal signal control split time durations for multiple intersections in the network are derived by solving the algebraic Riccati equation. Furthermore, the recursive least square parameter estimation method is employed to quantify dynamic traffic condition changes. To verify the effectiveness of this proposed signal control framework, both simulation and real-world experimental tests are conducted for multiple intersections in downtown Chattanooga, Tennessee, United States. In preparation for real-world experimental tests, pipelines for real-time data processing implementation and historical traffic flow data analysis are conducted. The test results demonstrate that the proposed control framework achieves a decrease in travel time by up to 19.4%, total time spent (TTS) by up to 11.9%, and relative queue balance (RQB) by up to 15.6%. The research findings indicate that the proposed signal control framework can be generalized to handle large scale signal control optimization network-wide.
97 MATHEMATICS AND COMPUTING↗
Reinforcement learning for online adaptation of model predictive controllers: Application to a selective catalytic reduction unit
Here we present a novel application of reinforcement learning (RL) for online dynamic tuning of model predictive controllers (MPC). Applying a state-action-reward-state-action (SARSA) algorithm for temporal difference learning with a control-specific reward function improves the error tracking performance of a standard MPC formulation. The proposed RL approach is also readily adaptable to other MPCs, or entirely different control approaches. Practical details for the implementation of the RL-MPC algorithm are also presented. The proposed algorithm is applied to a case study of controlling nitrogen oxide (NO x ) emissions in an industrial selective catalytic reduction (SCR) unit, a control problem characterized by significant nonlinearity and time delay. Along with an RL-MPC formulation for NOx control, another MPC is proposed to mitigate ammonia slip and decrease ammonia consumption in the SCR. Results showing the efficacy of the RL-MPC for NO x control through learning and implementation on the nonlinear SCR dynamic model are presented.
37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗
Traffic Control via Connected and Automated Vehicles (CAVs): An Open-Road Field Experiment with 100 CAVs
The CIRCLES project aims to reduce instabilities in traffic flow, which are naturally occurring phenomena due to human driving behavior. Also called “phantom jams” or “stop-and-go waves,” these instabilities are a significant source of wasted energy. Toward this goal, the CIRCLES project designed a control system, referred to as the MegaController by the CIRCLES team, that could be deployed in real traffic. Our field experiment, the MegaVanderTest (MVT), leveraged a heterogeneous fleet of 100 longitudinally controlled vehicles as Lagrangian traffic actuators, each of which ran a controller with the architecture described in this article. The MegaController is a hierarchical control architecture that consists of two main layers. The upper layer is called the Speed Planner and is a centralized optimal control algorithm. It assigns speed targets to the vehicles, conveyed through the LTE cellular network. The lower layer is a control layer, running on each vehicle. It performs local actuation by overriding the stock adaptive cruise controller, using the stock onboard sensors. The Speed Planner ingests live data feeds provided by third parties as well as data from our own control vehicles and uses both to perform the speed assignment. The architecture of the Speed Planner allows for the modular use of standard control techniques, such as optimal control, model predictive control (MPC), kernel methods, and others. The architecture of the local controller allows for the flexible implementation of local controllers. Corresponding techniques include deep reinforcement learning (RL), MPC, and explicit controllers. Depending on the vehicle architecture, all onboard sensing data can be accessed by the local controllers or only some. Likewise, control inputs vary across different automakers, with inputs ranging from torque or acceleration requests for some cars to electronic selection of adaptive cruise control (ACC) setpoints in others. The proposed architecture technically allows for the combination of all possible settings proposed previously, that is {Speed Planner algorithms} × {local Vehicle Controller algorithms} × {full or partial sensing} × {torque or speed control}. As a result, most configurations were tested throughout the ramp up to the MegaVandertest (MVT).
32 ENERGY CONSERVATION, CONSUMPTION, AND UTILIZATI↗
Advancing Platooning with ADAS (Advanced Driver-Assistance Systems) Control Integration and Assessment
Application of Cooperative Adaptive Cruise Control (CACC) to heavy duty trucks known as truck platooning has shown fuel economy improvements on the test track under ideal driving conditions. However, limited test data is available to assess the truck platooning under real-world driving conditions. Under this Cummins-led project that was funded by the U.S. Department of Energy, truck platooning with CACC has been tested on a real-world interstate highway and the results of the project are reviewed in this report. At first, the real-world driving conditions were characterized using National Renewable Energy Laboratory (NREL) Fleet DNA database to define test factors, including route, terrain, and highway traffic. Afterward, both test track and on-highway testing guided by SAE J1321 procedures were conducted to assess truck platooning under controlled and real-world driving conditions. On-highway testing was done on a highway route in Indiana, consisting of low, medium, and high road grade segments. The highway test results of 2-truck and 3-truck platooning showed considerably reduced fuel savings compared to the controlled test track data, which mainly stems from the traffic or high-grade portions of the route. However, integration of Cummins powertrain and vehicle eco-driving features such as predictive cruise control and neutral coasting called ADEPT™ on the lead truck showed an improvement of fuel saving for the trucks in CACC operation. Furthermore, the importance of tire connectivity in efficient and safe operation of the trucks in platooning is characterized.
32 ENERGY CONSERVATION, CONSUMPTION, AND UTILIZATI↗