SEARCH · Engineering Papers

Results for “graph reinforcement learning”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

SourceResearch connectionFrom yearThrough year

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 19 records

All sources NASA NTRS DOE OSTI

DOE OSTI2021

Learning Sequential Distribution System Restoration via Graph-Reinforcement Learning

We report a distribution service restoration algorithm as a fundamental resilient paradigm for system operators provides an optimally coordinated, resilient solution to enhance the restoration performance. The restoration problem is formulated to coordinate distribution generators and controllable switches optimally. A model-based control scheme is usually designed to solve this problem, relying on a precise model and resulting in low scalability. To tackle these limitations, this work proposes a graph-reinforcement learning framework for the restoration problem. We link the power system topology with a graph convolutional network, which captures the complex mechanism of network restoration in power networks and understands the mutual interactions among controllable devices. Latent features over graphical power networks produced by graph convolutional layers are exploited to learn the control policy for network restoration using deep reinforcement learning. The solution scalability is guaranteed by modeling distributed generators as agents in a multi-agent environment and a proper pre-training paradigm. Comparative studies on IEEE 123-node and 8500-node test systems demonstrate the performance of the proposed solution.

24 POWER TRANSMISSION AND DISTRIBUTION↗

DOE OSTI2025

Graph reinforcement learning for exploring model spaces beyond the standard model

We present a methodology for performing scans of beyond the standard model (BSM) parameter spaces with reinforcement learning. We identify a novel procedure using graph neural networks that is capable of exploring spaces of models without the user specifying a fixed particle content, allowing broad classes of BSM models to be explored—in theory, the technique is applicable to nearly any model space with a prespecified gauge group. We provide a generic procedure by which a suitable graph grammar can be developed for any BSM model that features user-specified symmetry groups and a finite number of different possible particle species, the use of which is applicable to a variety of machine learning tasks over the actions of BSM theories beyond our particular reinforcement learning use case. As a proof of concept, we construct the graph grammar for theories with vectorlike leptons that may or may not be charged under a dark U ( 1 ) group, inspired by portal matter extensions of the sub-GeV vector portal/kinetic mixing simplified dark matter models. We then use this graph grammar to create a reinforcement learning environment tasked with creating models with these vectorlike leptons that are consistent with a list of a variety of precision observables. The reinforcement learning agent succeeds in developing models that can address the observed muon anomalous magnetic moment discrepancy while remaining consistent with flavor violation and electroweak precision observables, including both constructions that have previously been studied as well as new models that have not, to our knowledge, previously been identified. By inspecting the resulting ensembles of models that the agent produces and experimenting with different configurations for our reinforcement learning environment and graph grammar, we also infer various lessons about the development of these environments that can be transferable to reinforcement learning scans of more complicated model spaces and comment on future directions for the development of this technique into a more mature tool. Published by the American Physical Society 2025

Wojcik, George N.↗

DOE OSTI2022

Graph Partitioning and Sparse Matrix Ordering using Reinforcement Learning and Graph Neural Networks

We present a novel method for graph partitioning, based on reinforcement learning and graph convolutional neural networks. Our approach is to recursively partition coarser representations of a given graph. The neural network is implemented using SAGE graph convolution layers, and trained using an advantage actor critic (A2C) agent. We present two variants, one for finding an edge separator that minimizes the normalized cut or quotient cut, and one that finds a small vertex separator. The vertex separators are then used to construct a nested dissection ordering to permute a sparse matrix so that its triangular factorization will incur less fill-in. The partitioning quality is compared with partitions obtained using METIS and SCOTCH, and the nested dissection ordering is evaluated in the sparse solver SuperLU. Our results show that the proposed method achieves similar partitioning quality as METIS and SCOTCH. Furthermore, the method generalizes across different classes of graphs, and works well on a variety of graphs from the SuiteSparse sparse matrix collection.

97 MATHEMATICS AND COMPUTING↗

DOE OSTI2023

Challenges and Opportunities in Deep Reinforcement Learning With Graph Neural Networks: A Comprehensive Review of Algorithms and Applications

Deep reinforcement learning (DRL) has empowered a variety of artificial intelligence fields, including pattern recognition, robotics, recommendation-systems, and gaming. Similarly, graph neural networks (GNN) have also demonstrated their superior performance in supervised learning for graph-structured data. In recent times, the fusion of GNN with DRL for graph-structured environments has attracted a lot of attention. Here, this paper provides a comprehensive review of these hybrid works. These works can be classified into two categories: (1) algorithmic enhancement, where DRL and GNN complement each other for better utility; (2) application-specific enhancement, where DRL and GNN support each other. This fusion effectively addresses various complex problems in engineering and life sciences. Based on the review, we further analyze the applicability and benefits of fusing these two domains, especially in terms of increasing generalizability and reducing computational complexity. Finally, the key challenges in integrating DRL and GNN, and potential future research directions are highlighted, which will be of interest to the broader machine learning community.

97 MATHEMATICS AND COMPUTING↗

DOE OSTI2022

graphenv: a Python library for reinforcement learning on graph search spaces

Many important and challenging problems in combinatorial optimization (CO) can be expressed as graph search problems, in which graph vertices represent full or partial solutions and edges represent decisions that connect them. Graph structure not only introduces strong relational inductive biases for learning (Battaglia et al., 2018) - in this context, by providing a way to explicitly model the value of transitioning (along edges) between one search state (vertex) and the next - but lends itself to problems both with and without clearly defined algebraic structure. For example, classic CO problems on graphs such as the Traveling Salesman Problem (TSP) can be expressed as either pure graph search or integer programs. Other problems, however, such as molecular optimization, do no have concise algebraic formulations and yet are readily implemented as a graph search (V. et al., 2022; Zhou et al., 2019). Such "model-free" problems constitute a large fraction of modern reinforcement learning (RL) research owing to the fact that it is often much easier to write a forward simulation that expresses all of the state transitions and rewards, than to write down the precise mathematical expression of the full optimization problem. In the case of molecular optimization, for example, one can use domain knowledge alongside existing software libraries to model the effect of adding a single bond or atom to an existing but incomplete molecule, and let the RL algorithm build a model of how good a given decision is by "experiencing" the simulated environment many times through. In contrast, a model-based mathematical formulation that fully expresses all the chemical and physical constraints is intractable. In recent years, RL has emerged as an effective paradigm for optimizing searches over graphs and led to state-of-the-art heuristics for games like Go and chess, as well as for classical CO problems such as the TSP. This combination of graph search and RL, while powerful, requires non-trivial software to execute, especially when combining advanced state representations such as Graph Neural Networks (GNN) with scalable RL algorithms.

97 MATHEMATICS AND COMPUTING↗

DOE OSTI2024

Artificial Intelligence/Machine Learning Technology in Power System Applications

The primary purpose of this report is to provide an overview of the advancement in artificial intelligence and machine learning (AI/ML) technologies and their applications in power systems. It offers a foundation for understanding the transformative role of AI/ML in power systems and aims to stimulate further research and development in this area. This report begins with a historical perspective of AI/ML technologies, then explores their advancement to today’s prominence. The document highlights key contributors to the success of AI/ML technologies, including increased computational power, greater data availability, innovative algorithms, and advanced tools. It further introduces various AI/ML techniques, including supervised, unsupervised and reinforcement learning, graph neural networks, and generative AI. It also emphasizes the critical importance of ensuring the safety, security, and trustworthiness of these AI/ML techniques within this sector. The report reviews the recent representative advancements in various power system applications enhanced by AI/ML techniques, underscoring key developments and their transformative impact as evidenced by numerous studies. It also explores both the opportunities and challenges associated with the application of AI/ML technologies to improve power system applications. While the report extensively covers AI/ML applications in power systems, focusing primarily on the technical and operational aspects, it may not thoroughly explore the sociopolitical, economic, and broader regulatory implications of AI/ML integration in power systems. AI/ML techniques hold significant potential for enhancing power system applications; however, they are not omnipotent. It is crucial to acknowledge their limitations and understand that they may not be able to address all challenges in the power system domain. Various factors must be considered that influence the implementation, adoption, and effectiveness of AI/ML solutions, including but not limited to safety, security, transparency, and trustworthiness. Additionally, the incorporation of advanced human–machine interfaces is essential, as it enables humans to validate the effectiveness of AI/ML solutions while remaining actively engaged, fostering trust in AI/ML deployment. Finally, the report summarizes AI/ML research activities supported by the Department of Energy (DOE) Office of Electricity (OE) through the Advanced Grid Modeling (AGM) program. The work aligns with the interests and mission of DOE-OE AGM, with the report serving as a resource for identifying existing progress and for pinpointing future applications within AI/ML that need further exploration and support.

24 POWER TRANSMISSION AND DISTRIBUTION↗

DOE OSTI2024

Multi-Agent Graph-Attention Deep Reinforcement Learning for Post-Contingency Grid Emergency Voltage Control

Grid emergency voltage control (GEVC) is paramount in electric power systems to improve voltage stability and prevent cascading outages and blackouts in case of contingencies. While most deep reinforcement learning (DRL)-based paradigms perform single agents in a static environment, real-world agents for GEVC are expected to cooperate in a dynamically shifting grid. Moreover, due to high uncertainties from combinatory natures of various contingencies and load consumption, along with the complexity of dynamic grid operation, the data efficiency and control performance of the existing DRL-based methods are challenged. To address these limitations, we propose a multi-agent graph-attention (GATT)-based DRL algorithm for GEVC in multi-area power systems. Here, we develop graph convolutional network (GCN)-based agents for feature representation of the graph-structured voltages to improve the decision accuracy in a data-efficient manner. Furthermore, a cutting-edge attention mechanism concentrates on effective information sharing among multiple agents, synergizing different-sized subnetworks in the grid for cooperative learning. We address several key challenges in the existing DRL-based GEVC approaches, including low scalability and poor stability against high uncertainties. Test results in the IEEE benchmark system verify the advantages of the proposed method over several recent multi-agent DRL-based algorithms.

24 POWER TRANSMISSION AND DISTRIBUTION↗

DOE OSTI2022

End-to-end optimization for battery materials and molecules by combining graph neural networks and reinforcement learning

The National Renewable Energy Laboratory (NREL), together with the Colorado School of Mines (CSM) and Colorado State University (CSU), has developed a machine learning-enhanced approach to design new battery materials. Currently, such materials are designed in part via numerous expensive high-fidelity computational simulations that predict the performance of a given composition. Even with computational screening tools, the vast landscape of possible molecular or crystal structures exceeds current and future computational capacity. Improving the efficiency by which new materials can be optimized will therefore disrupt the cost, risk, and time required to bring new energy solutions to the marketplace. Predicting the properties of an organic molecule or periodic crystalline material given its structure has grown increasingly common. These approaches leverage large-scale computational and experimental databases and ML approaches such as graph neural networks. The inverse design problem of finding a material that possesses desired properties is substantially more challenging, since enumerating all valid material structures is not feasible. In this project, we leveraged recent success in reinforcement learning to efficiently navigate this high-dimensional search space. Just as algorithms can find the optimal chess moves from nearly limitless options, we train an approach to evolve a simple starting structure into a complex structure that possess the desired properties. Our solution has been demonstrated by applying it to two related design application tasks for short- and long-term energy storage, respectively: (1) the design of solid-state ion conductors and (2) the design of organic redox-active materials. The project has resulted an open-source software library for material design, documented examples of applying the library to both organic and inorganic material optimization, and peer-reviewed publications detailing the data, computational models, and resulting candidate materials.

25 ENERGY STORAGE↗

DOE OSTI2022

End-to-End Optimization for Battery Materials and Molecules by Combining Graph Neural Networks and Reinforcement Learning

The National Renewable Energy Laboratory (NREL), together with the Colorado School of Mines (CSM) and Colorado State University (CSU), has developed a machine learning-enhanced approach to the design of new battery materials. Currently, such materials are designed in part via numerous expensive high-fidelity computational simulations that predict the performance of a given composition. Even with computational screening tools, the vast landscape of possible molecular or crystal structures exceeds current and future computational capacity. Improving the efficiency by which new materials can be optimized will therefore disrupt the cost, risk, and time required to bring new energy solutions to the marketplace. Predicting the properties of an organic molecule or periodic crystalline material given its structure has grown increasingly common. These approaches leverage large-scale computational and experimental databases and ML approaches such as graph neural networks. The inverse design problem of finding a material that possesses desired properties is substantially more challenging, since enumerating all valid material structures is not feasible. In this project, we leveraged recent success in reinforcement learning to efficiently navigate this high-dimensional search space. Just as algorithms can find the optimal chess moves from nearly limitless options, we train an approach to evolve a simple starting structure into a complex structure that possess the desired properties. Our solution has been demonstrated by applying it to two related design application tasks for short- and long-term energy storage, respectively: (1) the design of solid-state ion conductors and (2) the design of organic redox-active materials. The project has resulted an open-source software library for material design, documented examples of applying the library to both organic and inorganic material optimization, and peer-reviewed publications detailing the data, computational models, and resulting candidate materials.

25 ENERGY STORAGE↗

DOE OSTI2024

Integration of Decentralized Graph-Based Multi-Agent Reinforcement Learning with Digital Twin for Traffic Signal Optimization

Machine learning (ML) methods, particularly Reinforcement Learning (RL), have gained widespread attention for optimizing traffic signal control in intelligent transportation systems. However, existing ML approaches often exhibit limitations in scalability and adaptability, particularly within large traffic networks. This paper introduces an innovative solution by integrating decentralized graph-based multi-agent reinforcement learning (DGMARL) with a Digital Twin to enhance traffic signal optimization, targeting the reduction of traffic congestion and network-wide fuel consumption associated with vehicle stops and stop delays. In this approach, DGMARL agents are employed to learn traffic state patterns and make informed decisions regarding traffic signal control. The integration with a Digital Twin module further facilitates this process by simulating and replicating the real-time asymmetric traffic behaviors of a complex traffic network. The evaluation of this proposed methodology utilized PTV-Vissim, a traffic simulation software, which also serves as the simulation engine for the Digital Twin. The study focused on the Martin Luther King (MLK) Smart Corridor in Chattanooga, Tennessee, USA, by considering symmetric and asymmetric road layouts and traffic conditions. Comparative analysis against an actuated signal control baseline approach revealed significant improvements. Experiment results demonstrate a remarkable 55.38% reduction in Eco_PI, a developed performance measure capturing the cumulative impact of stops and penalized stop delays on fuel consumption, over a 24 h scenario. In a PM-peak-hour scenario, the average reduction in Eco_PI reached 38.94%, indicating the substantial improvement achieved in optimizing traffic flow and reducing fuel consumption during high-demand periods. These findings underscore the effectiveness of the integrated DGMARL and Digital Twin approach in optimizing traffic signals, contributing to a more sustainable and efficient traffic management system.

actuated signal control↗

DOE OSTI2024

Integration of Decentralized Graph-Based Multi-Agent Reinforcement Learning with Digital Twin for Traffic Signal Optimization

42 ENGINEERING↗

DOE OSTI2024

Pedestrian-Involved Traffic Signal Optimization Using Decentralized Graph-based Multi-Agent Reinforcement Learning

Incorporating pedestrian movements in traffic signal timing optimization is essential for ensuring smooth and safe urban transportation systems. To develop an effective signal timing plan, it is crucial to comprehend how various pedestrian accommodation strategies impact both vehicle and pedestrian flow. This study explores the application of the Decentralized Graph-based Multi-Agent Reinforcement Learning (DGMARL) method for signal timing optimization. It considers both pedestrian and vehicle traffic state and assesses the effects of fixed and adaptive pedestrian request response on traffic and signal timing. The evaluation of DGMARL encompasses vehicles Eco_PI which is fuel consumption impact related to vehicle stops and stop delay, and pedestrian waiting time to ensure overall system efficiency. The proposed approach was evaluated using a Digital Twin microscopic traffic simulation model of MLK Smart Corridor in Chattanooga, Tennessee. The evaluation outcomes of vehicles Eco_PI, pedestrian waiting time and serving time are compared with the actuated and DGMARL signal timing with fixed pedestrian recall and minimum recall to determine the most suitable approaches for various traffic conditions. The results indicated that, on average, the strategy of Signal timing optimization with adaptive pedestrian request response improved the Eco_PI by 32.43%, and Signal timing optimization with both automated pedestrian traffic demand and adaptive pedestrian request response improved the Eco_PI by 31.62% compared to the actuated and DGMARL signal timing with fixed pedestrian recall and minimum recall.

K Kumarasamy, Vijayalakshmi↗

DOE OSTI2022

Graph-Env

The Graph-Env library provides a framework for adapting graph search problems into OpenAI Gym environments for reinforcement learning. In other words, Graph-Env enables reinforcement learning algorithms to be applied to graph search problems. Graph search problems include molecule and crystal structure design problems; route planning problems, including traveling salesperson problem; puzzles and games, such as chess and go; shortest path problems; minimum spanning tree; vehicle path generation; and others.

Tripp, Charles↗

DOE OSTI2024

GraMeR: Gra ph Me ta R einforcement learning for multi-objective influence maximization

Influence maximization (IM) is a combinatorial problem of identifying a subset of seed nodes in a network (graph), which when activated, provide a maximal spread of influence in the network for a given diffusion model and a budget for seed set size. IM has numerous applications such as viral marketing, epidemic control, sensor placement and other network-related tasks. However, its practical uses are limited due to the computational complexity of current algorithms. Recently, deep reinforcement learning has been leveraged to solve IM in order to ease the computational burden. However, there are serious limitations in current approaches, including narrow IM formulation that only consider influence via spread and ignore self-activation, low scalability to large graphs, and lack of generalizability across graph families leading to a large running time for every test network. In this work, we address these limitations through a unique approach that involves: (1) Formulating a generic IM problem as a Markov decision process that handles both intrinsic and influence activations; (2)incorporating generalizability via meta-learning across graph families. There are previous works that combine deep reinforcement learning with graph neural network, but this work solves a more realistic IM problem and incorporates generalizability across graphs via meta reinforcement learning. Extensive experiments are carried out in various standard networks to validate performance of the proposed Graph Meta Reinforcement learning (GraMeR) framework. Finally, the results indicate that GraMeR is multiple orders faster and generic than conventional approaches when applied on small to medium scale graphs.

97 MATHEMATICS AND COMPUTING↗

DOE OSTI2021

Graph Convolutional Network-Based Topology Embedded Deep Reinforcement Learning for Voltage Stability Control

Topological variations in power system is a common phenomenon and can impose significant challenges to traditional controllers of power system. Recent study revealed the strength of deep reinforcement learning (DRL) based approaches in power system preventive and corrective control. But topological variations are difficult to capture using classical fully connected neural network (FCN) model and has not been explicitly modeled in previous work. Hence, we develop a Graph Convolutional Network (GCN) based DRL framework to tackle topology changes in control design of power system. The GCN model exploits the graph structure of the power network and helps the DRL agent to embed the topology information during learning process. Our GCN based approach is evaluated using the IEEE-39 bus system and it outperforms the FCN-based DRL scheme in terms of training convergence and control performance considering grid topology changes.

Hossain, Ramij Raja↗

DOE OSTI2022

Spatial Graph Attention and Curiosity-driven Policy for Antiviral Drug Discovery

We developed Distilled Graph Attention Policy Network (DGAPN), a reinforcement learning model to generate novel graph-structured chemical representations that optimize user-defined objectives by efficiently navigating a physically constrained domain. The framework is examined on the task of generating molecules that are designed to bind, noncovalently, to functional sites of SARS-CoV-2 proteins. We present a spatial Graph Attention (sGAT) mechanism that leverages self-attention over both node and edge attributes as well as encoding the spatial structure --- this capability is of considerable interest in synthetic biology and drug discovery. An attentional policy network is introduced to learn the decision rules for a dynamic, fragment-based chemical environment, and state-of-the-art policy gradient techniques are employed to train the network with stability. Exploration is driven by the stochasticity of the action space design and the innovation reward bonuses learned and proposed by random network distillation. In experiments, our framework achieved outstanding results compared to state-of-the-art algorithms, while reducing the complexity of paths to chemical synthesis.

Wu, Yulun↗

DOE OSTI2022

SPATIAL GRAPH ATTENTION AND CURIOSITY-DRIVEN POLICY FOR ANTIVIRAL DRUG DISCOVERY

We developed Distilled Graph Attention Policy Network (DGAPN), a reinforcement learning model to generate novel graph-structured chemical representations that optimize user-defined objectives by efficiently navigating a physically constrained domain. The framework is examined on the task of generating molecules that are designed to bind, noncovalently, to functional sites of SARS-CoV-2 proteins. We present a spatial Graph Attention (sGAT) mechanism that leverages self-attention over both node and edge attributes as well as encoding the spatial structure - this capability is of considerable interest in synthetic biology and drug discovery. An attentional policy network is introduced to learn the decision rules for a dynamic, fragment-based chemical environment, and state-of-the-art policy gradient techniques are employed to train the network with stability. Exploration is driven by the stochasticity of the action space design and the innovation reward bonuses learned and proposed by random network distillation. In experiments, our framework achieved outstanding results compared to state-of-the-art algorithms, while reducing the complexity of paths to chemical synthesis.

Wu, Y↗

DOE OSTI2023

Traffic Signal Optimization by Integrating Reinforcement Learning and Digital Twins

Machine learning (ML) methods, especially reinforcement learning (RL), have been widely considered for traffic signal optimization in intelligent transportation systems. Most of these ML methods are centralized, lacking in scalability and adaptability in large traffic networks. Further, it is challenging to train such ML models due to the lack of training platforms and/or the cost of deploying and training in a real traffic networks. This paper presents an approach for the integration of decentralized graph-based multi-agent reinforcement learning (DGMARL) with a Digital Twin (DT) to optimize traffic signals for the reduction of traffic congestion and network-wide fuel consumption related to stopping. Specifically, the DGMARL agents learn traffic state patterns and make decisions regarding traffic signal control with assistance from a Digital Twin module, which simulates and replicates the traffic behaviors of a real traffic network. The proposed approach was evaluated using PTV-Vissim [1], a microscopic traffic simulation platform. PTV-Vissim is also the simulation engine of the DT, enabling emulation and optimization of the traffic signals on the MLK Smart Corridor in Chattanooga, Tennessee. Compared to an actuated signal control baseline approach, experiment results show that Eco_PI, a developed performance measure capturing the impact of stops on fuel consumption, was reduced by 44.27% in a 24-hour and an average of 29.88% in a PM-peak-hour scenario.

Multi-Agent Reinforcement Learning, Digital Twin, ↗