Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “REINFORCEMENT”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 739 records · Page 41
All sourcesNASA NTRSDOE OSTI
DOE OSTI2021

Deep reinforcement learning for predicting kinetic pathways to surface reconstruction in a ternary alloy

Abstract The majority of computational catalyst design focuses on the screening of material components and alloy composition to optimize selectivity and activity for a given reaction. However, predicting the metastability of the alloy catalyst surface at realistic operating conditions requires an extensive sampling of possible surface reconstructions and their associated kinetic pathways. We present CatGym, a deep reinforcement learning (DRL) environment for predicting the thermal surface reconstruction pathways and their associated kinetic barriers in crystalline solids under reaction conditions. The DRL agent iteratively changes the positions of atoms in the near-surface region to generate kinetic pathways to accessible local minima involving changes in the surface compositions. We showcase our agent by predicting the surface reconstruction pathways of a ternary Ni 3 Pd 3 Au 2 (111) alloy catalyst. Our results show that the DRL agent can not only explore more diverse surface compositions than the conventional minima hopping method, but also generate the kinetic surface reconstruction pathways. We further demonstrate that the kinetic pathway to a global minimum energy surface composition and its associated transition state predicted by our agent is in good agreement with the minimum energy path predicted by nudged elastic band calculations.

36 MATERIALS SCIENCE↗
DOE OSTI2022

Variational quantum reinforcement learning via evolutionary optimization

Abstract Recent advances in classical reinforcement learning (RL) and quantum computation point to a promising direction for performing RL on a quantum computer. However, potential applications in quantum RL are limited by the number of qubits available in modern quantum devices. Here, we present two frameworks for deep quantum RL tasks using gradient-free evolutionary optimization. First, we apply the amplitude encoding scheme to the Cart-Pole problem, where we demonstrate the quantum advantage of parameter saving using amplitude encoding. Second, we propose a hybrid framework where the quantum RL agents are equipped with a hybrid tensor network-variational quantum circuit (TN-VQC) architecture to handle inputs of dimensions exceeding the number of qubits. This allows us to perform quantum RL in the MiniGrid environment with 147-dimensional inputs. The hybrid TN-VQC architecture provides a natural way to perform efficient compression of the input dimension, enabling further quantum RL applications on noisy intermediate-scale quantum devices.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2022

Discovering mechanisms for materials microstructure optimization via reinforcement learning of a generative model

Abstract The design of materials structure for optimizing functional properties and potentially, the discovery of novel behaviors is a keystone problem in materials science. In many cases microstructural models underpinning materials functionality are available and well understood. However, optimization of average properties via microstructural engineering often leads to combinatorically intractable problems. Here, we explore the use of the reinforcement learning (RL) for microstructure optimization targeting the discovery of the physical mechanisms behind enhanced functionalities. We illustrate that RL can provide insights into the mechanisms driving properties of interest in a 2D discrete Landau ferroelectrics simulator. Intriguingly, we find that non-trivial phenomena emerge if the rewards are assigned to favor physically impossible tasks, which we illustrate through rewarding RL agents to rotate polarization vectors to energetically unfavorable positions. We further find that strategies to induce polarization curl can be non-intuitive, based on analysis of learned agent policies. This study suggests that RL is a promising machine learning method for material design optimization tasks, and for better understanding the dynamics of microstructural simulations.

36 MATERIALS SCIENCE↗
DOE OSTI2023

Probing reaction channels via reinforcement learning

Abstract Chemical reactions are dynamical processes involving the correlated reorganization of atomic configurations, driving the conversion of an initial reactant into a result product. By virtue of the metastability of both the reactants and products, chemical reactions are rare events, proceeding fleetingly. Reaction pathways can be modelled probabilistically by using the notion of reactive density in the phase space of the molecular system. Such density is related to a function known as the committor function, which describes the likelihood of a configuration evolving to one of the nearby metastable regions. In theory, the committor function can be obtained by solving the backward Kolmogorov equation (BKE), which is a partial differential equation (PDE) defined in the full dimensional phase space. However, using traditional methods to solve this problem is not practical for high dimensional systems. In this work, we propose a reinforcement learning based method to identify important configurations that connect reactant and product states along chemical reaction paths. By shooting multiple trajectories from these configurations, we can generate an ensemble of states that concentrate on the transition path ensemble. This configuration ensemble can be effectively employed in a neural network-based PDE solver to obtain an approximation solution of a restricted BKE, even when the dimension of the problem is very high. The resulting solution provides an approximation for the committor function that encodes mechanistic information for the reaction, paving a new way for understanding of complex chemical reactions and evaluation of reaction rates.

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗
DOE OSTI2024

Reinforcement learning pulses for transmon qubit entangling gates

The utility of a quantum computer is highly dependent on the ability to reliably perform accurate quantum logic operations. For finding optimal control solutions, it is of particular interest to explore model-free approaches, since their quality is not constrained by the limited accuracy of theoretical models for the quantum processor—in contrast to many established gate implementation strategies. In this work, we utilize a continuous control reinforcement learning algorithm to design entangling two-qubit gates for superconducting qubits; specifically, our agent constructs cross-resonance and CNOT gates without any prior information about the physical system. Using a simulated environment of fixed-frequency fixed-coupling transmon qubits, we demonstrate the capability to generate novel pulse sequences that outperform the standard cross-resonance gates in both fidelity and gate duration, while maintaining a comparable susceptibility to stochastic unitary noise. We further showcase an augmentation in training and input information that allows our agent to adapt its pulse design abilities to drifting hardware characteristics, importantly, with little to no additional optimization. Our results exhibit clearly the advantages of unbiased adaptive-feedback learning-based optimization methods for transmon gate design.

71 CLASSICAL AND QUANTUM MECHANICS, GENERAL PHYSIC↗
DOE OSTI2024

Controlling optical-cavity locking using reinforcement learning

Abstract This study applies an effective methodology based on Reinforcement Learning to a control system. Using the Pound–Drever–Hall locking scheme, we match the wavelength of a controlled laser to the length of a Fabry-Pérot cavity such that the cavity length is an exact integer multiple of the laser wavelength. Typically, long-term drift of the cavity length and laser wavelength exceeds the dynamic range of this control if only the laser’s piezoelectric transducer is actuated, so the same error signal also controls the temperature of the laser crystal. In this work, we instead implement this feedback control grounded on Q-Learning. Our system learns in real-time, eschewing reliance on historical data, and exhibits adaptability to system variations post-training. This adaptive quality ensures continuous updates to the learning agent. This innovative approach maintains lock for eight days on average.

47 OTHER INSTRUMENTATION↗
DOE OSTI2026

Explainable physics-based constraints on reinforcement learning for accelerator optimization

We present a reinforcement learning (RL) framework for optimizing particle accelerator experiments that builds explainable physics-based constraints on agent behavior. The goal is to increase transparency and trust by letting users verify that the agent’s decision-making process incorporates suitable physics. Our algorithm uses a learnable surrogate function for physical observables, such as energy, and uses them to fine-tune how actions are chosen. This surrogate can be represented by a neural network or by an interpretable sparse dictionary model. We test our algorithm on a range of particle accelerator optimization environments designed to emulate the Continuous Electron Beam Accelerator Facility at Jefferson Lab. By examining the mathematical form of the learned constraint function, we are able to confirm the agent has learned to use the established physics of each environment. In addition, we find that the introduction of a physics-based surrogate enables our RL algorithms to reliably converge for difficult high-dimensional accelerator optimization environments.

explainability↗
DOE OSTI2026

Microsecond-latency feedback at a particle accelerator by online reinforcement learning on hardware

The commissioning and operation of future large-scale scientific experiments will challenge current tuning and control methods. Reinforcement learning (RL) algorithms are a promising solution due to their ability to dynamically adapt to changing environments and consider delayed consequences. In many real-world applications, RL policies must produce actions in real time, often within microseconds to milliseconds, imposing significant constraints on system latency and computational overhead that conventional machine learning libraries are not designed to handle. To control phenomena in real time at these timescales, RL needs to be deployed on-the-edge, namely on dedicated hardware located near the system it controls, without relying on a host CPU or cloud-based inference. In this work we present the design and deployment of an experience accumulator system in a particle accelerator. In this system, deep-RL algorithms run using hardware acceleration and act within a few microseconds, enabling the use of RL for control of phenomena like beam instabilities. The training uses the collected data offline to reduce the number of operations carried out on the acceleration hardware. The proposed architecture was tested in real experimental conditions at the Karlsruhe research accelerator, a synchrotron light source, where the system was used to control artificially induced horizontal betatron oscillations in real-time, with a control loop period of just 2.7 μs. The results showed a performance comparable to the commercial feedback system available at the accelerator, demonstrating the viability and potential of this approach. Due to the self-learning and reconfiguration capability of this implementation, a seamless application to other control problems is possible. Applications range from particle accelerators to large-scale research and industrial facilities.

FPGA↗
DOE OSTI2019

Deep Reinforcement Learning and Simulation as a Path Toward Precision Medicine

In the past, precision medicine involves classifying patients to identify subpopulations that respond favorably to specific therapeutics. We discuss precision medicine as a dynamic feedback control problem, where treatment administered to a patient is guided by measurements taken during the course of treatment. We consider sepsis, a life-threatening condition in which dysregulation of the immune system causes tissue damage. We leverage an existing simulation of the innate immune response to infection and apply deep reinforcement learning (DRL) to discover an adaptive personalized treatment policy that specifies effective multicytokine therapy to simulated sepsis patients based on systemic measurements. The learned policy achieves a dramatic reduction in mortality rate over a set of 500 simulated patients relative to standalone antibiotic therapy. Advantages of our approach are threefold: (1) the use of simulation allows exploring therapeutic strategies beyond clinical practice and available data, (2) advances in DRL accommodate learning complex therapeutic strategies for complex biological systems, and (3) optimized treatments respond to a patient's individual disease progression over time, therefore, capturing both differences across patients and the inherent randomness of disease progression within a single patient. We believe that this work motivates both considering adaptive personalized multicytokine mediation therapy for sepsis and exploiting simulation with DRL for precision medicine more broadly.

59 BASIC BIOLOGICAL SCIENCES↗
DOE OSTI2025

Enhancing quantum memory lifetime with measurement-free local error correction and reinforcement learning

Reliable quantum computation requires systematic identification and correction of errors that occur and accumulate in quantum hardware. To diagnose and correct such errors, standard quantum error-correcting protocols utilize global error information across the system obtained by mid-circuit readout of ancillary qubits. We investigate circuit-level error-correcting protocols that are measurement-free and based on local error information. Such a local error correction (LEC) circuit consists of faulty multi-qubit gates to perform both syndrome extraction and ancilla-controlled error removal. We develop and implement a reinforcement learning framework that takes a fixed set of faulty gates as inputs and outputs an optimized LEC circuit. To evaluate this approach, we quantitatively characterize an extension of logical qubit lifetime by a noisy LEC circuit. For the two-dimensional (2D) classical Ising model and four-dimensional toric code, our optimized LEC circuit performs better at extending a memory lifetime compared with a conventional LEC circuit based on Toom's rule in a subthreshold gate error regime. We further show that such circuits can be used to reduce the rate of mid-circuit readouts to preserve a 2D toric code memory. Lastly, we discuss the application of the LEC protocol on dissipative preparation of quantum states with topological phases.

74 ATOMIC AND MOLECULAR PHYSICS↗
DOE OSTI2026

Adaptive X-ray imaging with reinforcement learning

X-ray imaging is a powerful technique to scan samples in a variety of contexts including biological, environmental and materials science, but commonly requires a synchrotron light source to produce X-rays at sufficient intensity. As these facilities are expensive to operate, the available beam time is limited and always in high demand. Particularly if the illuminated samples are sparse, standard raster scanning methods can be time-consuming, with a majority of that time being spent on areas of the image that carry little information. To increase the efficiency and maximize the information gain for a given time budget, we split the scanning process into a series of steps where previous measurements are used to inform the decision making and adapt the exposure distribution at later stages of the sequence. We formulate this task as a reinforcement learning problem where the goal is to produce a sequence of exposure maps that maximize a predefined scalar metric. We demonstrate the potential of this approach in simulations where the adaptive illumination can accelerate the measurement process by up to an order of magnitude compared with standard raster scanning. Finally, we present the first results from deploying the trained agents on an X-ray fluorescence beamline at the Stanford Synchrotron Radiation Lightsource.

Reinforcement Learning↗
DOE OSTI2024

Non-Stationary Policy Learning for Multi-Timescale Multi-Agent Reinforcement Learning

In multi-timescale multi-agent reinforcement learning (MARL), agents interact across different timescales. In general, policies for time-dependent behaviors, such as those induced by multiple timescales, are non-stationary. Learning non-stationary policies is challenging and typically requires sophisticated or inefficient algorithms. Motivated by the prevalence of this control problem in real-world complex systems, we introduce a simple framework for learning non-stationary policies for multi-timescale MARL. Our approach uses available information about agent timescales to define and learn periodic multi-agent policies. In detail, we theoretically demonstrate that the effects of non-stationarity introduced by multiple timescales can be learned by a periodic multi-agent policy. To learn such policies, we propose a policy gradient algorithm that parameterizes the actor and critic with phase-functioned neural networks, which provide an inductive bias for periodicity. The framework's ability to effectively learn multi-timescale policies is validated on a gridworld and building energy management environment.

control↗
DOE OSTI2023

Tuning Phase Lock Loop Controller of Grid Following Inverters by Reinforcement Learning to Support Networked Microgrid Operations

The dynamic operation of networked microgrids leads to varying topological configurations and generator commitments and dispatches. These variations correspond to systems with different electrical characteristics. The fixed control gains of high-speed power electronic devices may result in undesirable system performance when the electrical characteristics change significantly. As such, it is necessary to tune the control gains of power electronics devices to adapt to the changing system characteristics. This paper uses observer-based reinforcement learning to automatically tune the proportional-integral (PI) gains of phase lock loop (PLL) controller of grid-following (GFL) inverters to adapt to the changing system strengths, that would be seen in networked microgrid operations. Simulation results using an operational electric distribution system, modeled as networked microgrids, are presented to demonstrate the need and effectiveness of the proposed adaptive controls.

networked microgrids, reinforcement learning, grid↗
DOE OSTI2023

Deep Reinforcement Learning for Distribution System Cyber Attack Defense with DERs

The use of smart inverter capabilities of distributed energy resources (DERs) enhances the grid reliability but in the meanwhile exhibits more vulnerabilities to cyber-attacks. This paper proposes a deep reinforcement learning (DRL)-based defense approach. The defense problem is reformulated as a Markov decision making process to control DERs and minimizing load shedding to address the voltage violations caused by cyber-attacks. The original soft actor-critic (SAC) method for continuous actions has been extended to handle discrete and continuous actions for controlling DERs' setpoints and loadshedding scenarios. Numerical comparison results with other control approaches, such as Volt-VAR and Volt-Watt on the modified IEEE 33-node, show that the proposed method can achieve better voltage regulation and have less power losses in the presence of cyber-attacks.

active distribution systems↗
DOE OSTI2022

Data-Driven Distribution System Coordinated PV Inverter Control Using Deep Reinforcement Learning

The deployment of distributed solar photovoltaic (PV) systems has increased consistently over the past decades. High penetrations of PVs could cause a series of adverse grid impacts, such as voltage violations. The recent development of smart inverter technologies rises the incentives of developing PV control solutions that regulate the inverter output power and seeking the optimization on system operational objectives. This paper proposes a data-driven control solution based on deep reinforcement learning (DRL) to optimize PV inverters for voltage regulation. The proposed solution can minimize PV real power curtailment while maintaining network voltage at an acceptable range. Comparison results between the proposed DRL control algorithms with deep deterministic policy gradient (DDPG) and volt-var control on a real feeder in west Colorado highlight the advantage of the proposed framework in controlling the system voltage while minimizing the PV real power curtailment.

deep reinforcement learning↗
DOE OSTI2018

Minimizing Energy Consumption from Connected Signalized Intersections by Reinforcement Learning

Explicit energy minimization objectives are often discouraged in signal optimization algorithms due to its negative impact on mobility performance. One potential direction to solve this problem is to provide a balanced objective function to achieve desired mobility with minimized energy consumption. This research developed a reinforcement learning (RL) based control with reward functions considering energy and mobility in a joint manner-a penalty function is introduced for number of stops. Further, we proposed a clustering-based technique to make the state-space finite which is critical for a tractable implementation of the RL algorithm. We implemented the algorithm in a calibrated NG-SIM network within a traffic micro-simulator-PTV VISSIM. With sole focus on energy, we report 47% reduction in energy consumption when compared with existing signal control schemes, however causing a 65.6% increase in system travel time. In contrast, the control strategy focusing on energy minimization with penalty for stops yields 6.7% reduction in energy consumption with 27% increase in system travel time. The developed RL algorithm with a flexible penalty function in the reward will achieve desired energy goals for a network of signalized intersections without compromising on the mobility performance.

32 ENERGY CONSERVATION, CONSUMPTION, AND UTILIZATI↗
DOE OSTI2021

A Cyber-Physical System for Freeway Ramp Meter Signal Control Using Deep Reinforcement Learning in a Connected Environment

Freeway bottlenecks such as on-ramp merging areas account for about 40% of recurring freeway congestion. It is generally agreed that building more roads and adding more lanes to existing infrastructure does not solve the congestion problem, and so dynamic traffic control measures offer a more cost-effective alternative. Ramp meters, traffic signal devices that regulate traffic flow entering freeways, are among the most effective measures to mitigate congestion at on-ramp merging areas on freeways. The confluence of deep reinforcement learning (RL) and connectivity provides a possible solution to advance ramp meter signal control. Deep RL is a group of machine-learning methods that enables an agent learning from the environment to improve its performance. In this study, three deep RL methods-proximal policy optimization (PPO), Ape-X deep Q-network (DQN), and asynchronous advantage actor-critic agents (A3C)-are explored for ramp meter signal control to maximize vehicle speed and traffic throughput, as well as to minimize energy consumption and emissions at freeway on-ramp merging areas in a connected environment. The low computational requirement and scalability of deep RL for deployment make it a powerful optimization tool for time-sensitive applications such as ramp meter signal control. The results of this study show that deep RL methods yield superior performance to both a fixed-time controller and ALINE A, a state-of-the-art feedback controller.

ADVANCED PROPULSION SYSTEMS,MATHEMATICS AND COMPUT↗
DOE OSTI2023

AutoTG: Reinforcement Learning-Based Symbolic Optimization for AI-Assisted Power Converter Design

Power converters are pervasive in modern electronic component design. They can be found in all electronic devices from household appliances and cellphone chargers to vehicles. Currently, designing new circuit topologies is hard because it requires human expertise based on experience and is difficult to automate. However, artificial-intelligence-assisted design can significantly facilitate the development of new power converters and/or improve the final result. Intelligently designed highly efficient power converters can have a significant effect on many important attributes, such as power efficiency, layout size, cost, heat dissemination, energy requirements, etc. We propose Autonomous Topology Generator (AutoTG), a reinforcement-learning-based framework that generates power converter topology candidates based on user specifications, optimized for user preferences. By modeling power converter design as a symbolic optimization problem, we sequentially sample components in an autoregressive manner until new topologies are formed, providing both the topology specification and the sizing (magnitude of each component parameter) of the proposed power converter. Here, we provide an empirical evaluation and show that AutoTG is able to generate varied high-efficiency topologies within component restrictions based on user input and show that previously unknown topologies can be found for further evaluation.

(AI)-based design↗