Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “REINFORCEMENT”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 469 records · Page 26
All sourcesNASA NTRSDOE OSTI
DOE OSTI2020

Deep reinforcement learning for complex evaluation of one-loop diagrams in quantum field theory

In this paper we present a technique based on deep reinforcement learning that allows for numerical analytic continuation of integrals that are often encountered in one-loop diagrams in quantum field theory. To extract certain quantities of two-point functions, such as spectral densities, mass poles or multiparticle thresholds, it is necessary to perform an analytic continuation of the correlator in question. At one-loop level in Euclidean space, this results in the necessity to deform the integration contour of the loop integral in the complex plane of the square of the loop momentum, to avoid nonanalyticities in the integration plane. Using a toy model for which an exact solution is known, we train a reinforcement learning agent to perform the required contour deformations. We report our study shows great promise for an agent to be deployed in iterative numerical approaches used to compute nonperturbative two-point functions, such as the quark propagator Dyson-Schwinger equation, or more generally, Fredholm equations of the second kind, in the complex domain.

72 PHYSICS OF ELEMENTARY PARTICLES AND FIELDS↗
DOE OSTI2021

Methodology for interpretable reinforcement learning model for HVAC energy control

Deep reinforcement learning (DRL) approaches have been used in various application areas to improve efficiency, optimization, or automation. However, very little is known about how the DRL algorithms make decisions and what features affect their performance. Using a case study of a DRL based Heating, Ventilation and Air Conditioning (HVAC) optimization methodology, we demonstrate how we can address these challenges by applying interpretability tools and systematically exploring the model inputs for better understanding the DRL behaviour and decision making process. We developed a methodology for interpretable reinforcement learning and evaluated our approach in real-world house located in Knoxville, TN. Our findings explain the reasoning behind DRL-based optimization decisions under different circumstances which has been discussed and confirmed by the experts in the field.

Kotevska, Olivera↗
DOE OSTI2023

Fast Frequency Response using Reinforcement Learning-Controlled Wind Turbines

To fulfill the auxiliary grid services such as load regulation, spin and non-spin reserve, and frequency support during emergencies, power system operators often require certain wind farms to operate in de-loaded modes. By leveraging the fast response capability of wind farms, the reserved power in deloaded modes can significantly enhance the stability and reliability of power grids. This paper presents a novel methodology that incorporates wind turbines into reinforcement learning-based solutions for frequency response. The proposed approach employs the state-of-the-art reinforcement learning algorithm, surrogategradient- based evolution strategy (GSES), for continuous control of the wind farm output. Our methodology is tested on a modified IEEE-39 bus system, and simulation outcomes demonstrate that the proposed approach can reliably support the frequency of the power system and prevent unnecessary load shedding.

Gao, Wei↗
DOE OSTI2022

MARS: Malleable Actor-Critic Reinforcement Learning Scheduler

In this paper, we introduce MARS, a new scheduling system for HPC-cloud infrastructures based on a cost-aware, flexible reinforcement learning approach, which serves as an intermediate layer for next generation HPC-cloud resource manager. MARS ensembles the pre-trained models from heuristic workloads and decides on the most cost-effective strategy for optimization. A whole workflow application would be split into several optimizable dependent sub-tasks, then based on the pre- defined resource management plan, a reward will be generated after executing a scheduled task. Lastly, MARS updates the Deep Neural Network (DNN) model based on the reward. MARS is designed to optimize the existing models through reinforcement mechanisms. MARS adapts to the dynamics of workflow applications, selects the most cost-effective scheduling solution among pre-built scheduling strategies (backfilling, SJF, etc.) and self- learning deep neural network model at run-time. We evaluate MARS with different real-world workflow traces. MARS can achieve 5%-60% increased performance compare to state-of-the- art approaches.

Baheri, Betis↗
DOE OSTI2025

Integrated Routing and Traffic Signal Control for CAVs via Reinforcement Learning Approach

Incorporating Connected and Automated Vehicles (CAVs) into urban traffic networks presents opportunities and challenges for traffic management systems. This paper aims to develop an integrated routing and traffic signal control system designed explicitly for CAVs, utilizing a Reinforcement Learning (RL) approach. The objective is to enhance traffic flow and improve overall transportation efficiency in the controlled areas. We propose an innovative framework that employs the Deep Reinforcement Learning (DRL) algorithm, especially the Deep Q-network (DQN), to dynamically adjust the number of vehicles in the routes and the duration of traffic signals. Our simulation results demonstrate that a DQN agent successfully optimizes the number of vehicles in the routes and traffic signal timings of traffic signal controllers, eventually reducing total travel time. The study illustrates the potential usage of RL-based systems in managing routing and traffic signals for CAVs, offering a promising opportunity for future urban traffic management strategies.

Park, Jiho [New York University]↗
DOE OSTI2024

Federated Deep Reinforcement Learning for Decentralized VVO of BTM DERs

The future of grid control requires a hybrid approach combining centralized and decentralized methods to fully utilize the potential of smart edge devices with artificial intelligence (AI) capabilities. This paper aims to develop and evaluate a federated deep reinforcement learning (FDRL) framework for decentralized adaptive volt-var optimization (VVO) of behind-the-meter (BTM) distributed energy resources (DERs). First, this paper models a single deep reinforcement learning (DRL) agent using the Markov Decision Process (MDP) framework for decentralized adaptive VVO of BTM DERs. Two DRL algorithms, soft actor-critic (SAC) and twin-delayed deep deterministic policy gradient (TD3), are compared for their effectiveness in optimizing VVO. Results show that TD3 outperforms SAC, achieving a 71.3% improvement in mean reward. Finally, the DRL agent is deployed within the FDRL framework, using the Flower platform, to enhance learning, provide adaptive control, and ensure data privacy for BTM DERs.

Ravi, Abhijith↗
DOE OSTI2025

Quantum Reinforcement Learning for Volt-VAR Control in Power Distribution Systems

Volt-VAR control (VVC) is crucial in active distribution networks for optimizing voltage profiles and minimizing network losses. While traditional deep reinforcement learning (DRL) algorithms exhibit promise for VVC, they often require extensive computational resources to handle such a high-dimensional problem. As a potential solution, quantum reinforcement learning (QRL) algorithms integrate the computational capabilities of quantum computing into the DRL framework. However, existing QRL algorithms struggle with complex VVC problems due to the limitations of current quantum hardware. To bridge this gap, this paper proposes an innovative QRL algorithm featuring an end-to-end architecture that integrates a classical autoencoder, variational quantum circuits (VQCs), and classical post-processing layers. This design efficiently compresses high-dimensional grid states, enabling VQCs to leverage quantum advantages while producing multiple control device outputs tailored for VVC tasks. Numerical studies on three representative distribution systems verify the effectiveness and scalability of the proposed QRL algorithm, and demonstrate its enhanced performance over classical approaches with only approximately 1% of the parameters. Additionally, the robustness of our developed algorithm is validated through noisy quantum environments.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2019

Resilient Reinforcement in Secure State Estimation against Sensor Attacks with a priori Information

This paper deals with the state estimation problem in the presence of malicious sensor attacks. Unlike the existing work, in this paper, we address the problem with a priori information of the state to be estimated. Specifically, we address three prior knowledge: the sparsity information, (alpha, n_0)-sparsity information, and side information, and in each scenario, we show that the state can be reconstructed even if more sensors are compromised. This implies that the prior information reinforces the system resilience against sensor attacks. Then, an estimator under sensor attacks considering the information is developed and, under a certain condition, the estimator can be relaxed into a tractable convex optimization problem. Further, we extend this analysis to systems in presence of measurement noises, and it is shown that the prior information reduces the state-estimation error caused by the noise. Here, the numerical simulations in a diffusion process finally illustrate the reinforcement and error-reduction results with the information.

24 POWER TRANSMISSION AND DISTRIBUTION↗
DOE OSTI2025

Deep Reinforcement Learning Based Control of Wind Turbines for Fast Frequency Response

In order to fulfill vital auxiliary grid services, such as load regulation, spin and non-spin reserve provision, and frequency support during emergencies, there is often a requirement for certain wind farms to operate in de-loaded modes. Leveraging the swift response capabilities of wind farms, this study demonstrates that reserving power in de-loaded modes can significantly enhance power grid stability and reliability during system contingencies. Controlling wind farms optimally for frequency support is intricate due to the nonlinearity of models and controllers and the complexity of wind farm interactions with power systems. Here, to address this challenge, this paper introduces a novel approach that integrates wind turbines into reinforcement learning-based solutions for frequency response. This innovative methodology utilizes the state-of-the-art reinforcement learning algorithm known as the surrogate-gradient-based evolutionary strategy. The proposed learning-based algorithm provides continuous control of wind farm output to rapidly stabilize system frequency and prevent unnecessary trips of under-frequency load shedding relays. To facilitate efficient training, parallel computing techniques are employed. The proposed methodology is evaluated on a modified IEEE-39 bus system, and simulation results reveal its efficacy in reliably supporting power system frequency and preventing the need for unnecessary load shedding.

Gao, Wei [Argonne National Laboratory (ANL), Argon↗
DOE OSTI2025

A Two-Stage Quantum Reinforcement Learning Method for Multi-Objective Transmission Switching

Multi-objective transmission switching (MO-TS) problems involve the strategic reconfiguration of network topology to simultaneously optimize multiple objectives. As the system scale increases, finding feasible solutions becomes increasingly challenging due to the problem's nonlinearity and high computational complexity. To address these challenges, this paper proposes a two-stage quantum reinforcement learning method that leverages potential quantum advantages for MO-TS. In the first stage, candidate switching lines are identified using a graph-theoretical approach to reduce the problem's dimensionality. The second stage introduces a quantum-classical reinforcement learning framework, where a learnable measurement-based CNN-ResVQC architecture is developed to effectively reduce the input dimension for quantum processing, mitigate vanishing gradients, and enhance trainability while improving the quantum circuit's flexibility in modeling complex decision policies for MO-TS. Numerical studies on IEEE 14-bus, 57-bus, and 118-bus systems demonstrate that the proposed algorithm achieves superior training stability and faster convergence with approximately 1% of the network parameters required by classical algorithms, highlighting its effectiveness, efficiency, and scalability. Furthermore, the practicality is validated through its stable convergence under three common quantum noise channels.

99 GENERAL AND MISCELLANEOUS↗
DOE OSTI2020

Preparation, mechanical properties, and toughening mechanisms of SiCw/SiCp‐reinforced zirconia‐toughened alumina ceramics

ABSTRACT Zirconia‐toughened alumina (ZTA) ceramics with high mechanical properties were sintered by hot‐pressing method using SiC particles (SiCp) and SiC whiskers (SiCw) as the reinforcing agents simultaneously. The influences of sintering temperature, SiCp, and SiCw contents on the microstructure and mechanical properties of ZTA ceramics were investigated. It was found that both SiCp and SiCw could contribute to grain refinement significantly and promote the mechanical properties of the ceramics. However, the excess addition of SiCp or SiCw led to the formation of pores with large sizes and degraded the mechanical properties instead. When 13 wt% SiCp was introduced, the maximum flexural strength of 1180.0 MPa and fracture toughness of 15.9 MPa·m 1/2 were obtained, whereas the maximum flexural strength of 1314.0 MPa and fracture toughness of 14.7 MPa·m 1/2 were achieved at 20 wt% SiCw. Interestingly, the simultaneous addition of SiCp and SiCw could further improve the mechanical properties, and the highest flexural strength of 1334.0 MPa and fracture toughness of 16.0 MPa·m 1/2 were achieved at a SiCw/SiCp ratio of 16/4. The reinforcement mechanisms in the ceramics mainly included the phase transformation toughening of ZrO 2 , the crack deflection and bridging of SiCp and SiCw, and the pull‐out of SiCw.

Liu, Longyang↗
DOE OSTI2023

Tailoring the Toughening Effects in Two-Dimensional Nanomaterial-Reinforced Ceramic Matrix Composites

Abstract Ceramic matrix composites (CMCs) reinforced by two-dimensional (2D) nanomaterials have shown extraordinary load-carrying capacities, even in the harsh environments required by emerging applications. Their exceptional mechanical performance, especially fracture toughness, primarily arises from their heterogeneous microstructures. The deliberate dispersion of 2D reinforcements enables toughening mechanisms that are extrinsic to the matrix and thus endows the composites with substantial resistance to catastrophic failure. However, the incomplete understanding of the fracture behavior of such nanocomposites, especially the complex energy dissipation process of the matrix/reinforcement interface, limits the development of stronger and tougher CMCs. To overcome these limitations, we investigate crack deflection and energy dissipation in nanocomposites using an extended cohesive shear-lag model. This new model accounts for interfacial debonding and friction, which critically control the toughening of nanocomposites. Our analysis provides mechanistic insights for optimizing the toughening effects of CMCs.

Mechanics↗
DOE OSTI2021

Passive sensing of a microparticle modified hybrid, fiber-reinforced composite

The widespread commercial adoption of high-performance, fiber-reinforced composites has pushed research interests toward the next generation of composites. These new composites are tasked with integrating additional functionalities into the structures without causing a trade-off in mechanical performance. One such functionality that has received significant interest is sensing. This is especially important for composites using high-performance fibers (e.g., carbon fiber) because their strain-to-failure is relatively low, resulting in brittle fracture. Besides, fiber damage can be hidden within the composite, potentially leading to premature catastrophic failure if not detected. In prior research, we demonstrated continuous feed-through deposition of ceramic nanoparticles on carbon fiber’s surface that simultaneously enhanced both the piezoresistive response and interlaminar shear strength. In this work, a similar continuous feed-through deposition process was used to demonstrate passive sensing and energy harvesting by integrating ferroelectric microparticles on the surface of electrically nonconductive fibers. The sensing and energy harvesting capabilities were characterized by mechanically straining composite beams and measuring the power generated. The improvements in mechanical properties are shown through interlaminar shear strength tests. Therefore, this research aims to demonstrate a high throughput, commercially scalable approach to coat fibers with ferroelectric microparticles that enable passive sensing as well as improved mechanical performance when fabricated into a fiber-reinforced composite.

Bowland, Christopher↗
DOE OSTI2020

Controlling colloidal crystals via morphing energy landscapes and reinforcement learning

We report a feedback control method to remove grain boundaries and produce circular shaped colloidal crystals using morphing energy landscapes and reinforcement learning–based policies. We demonstrate this approach in optical microscopy and computer simulation experiments for colloidal particles in ac electric fields. First, we discover how tunable energy landscape shapes and orientations enhance grain boundary motion and crystal morphology relaxation. Next, reinforcement learning is used to develop an optimized control policy to actuate morphing energy landscapes to produce defect-free crystals orders of magnitude faster than natural relaxation times. Morphing energy landscapes mechanistically enable rapid crystal repair via anisotropic stresses to control defect and shape relaxation without melting. This method is scalable for up to at least N = 10 3 particles with mean process times scaling as N 0.5 . Further scalability is possible by controlling parallel local energy landscapes (e.g., periodic landscapes) to generate large-scale global defect-free hierarchical structures.

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗
DOE OSTI2023

Top-down design of protein architectures with reinforcement learning

As a result of evolutionary selection, the subunits of naturally occurring protein assemblies often fit together with substantial shape complementarity to generate architectures optimal for function in a manner not achievable by current design approaches. We describe a “top-down” reinforcement learning–based design approach that solves this problem using Monte Carlo tree search to sample protein conformers in the context of an overall architecture and specified functional constraints. Cryo–electron microscopy structures of the designed disk-shaped nanopores and ultracompact icosahedra are very close to the computational models. The icosohedra enable very-high-density display of immunogens and signaling molecules, which potentiates vaccine response and angiogenesis induction. Our approach enables the top-down design of complex protein nanomaterials with desired system properties and demonstrates the power of reinforcement learning in protein design.

Science & Technology - Other Topics↗
DOE OSTI2022

Deep reinforcement learning with online data augmentation to improve sample efficiency for intelligent HVAC control

Deep Reinforcement Learning (DRL) has started showing success in real-world applications such as building energy optimization. Much of the research in this space utilized simulated environments to train RL-agent in an offline mode. Very few research have used DRL-based control in real-world systems due to two main reasons: 1) sample efficiency challenge---DRL approaches need to perform a lot of interactions with the environment to collect sufficient experiences to learn from, which is difficult in real systems, and 2) comfort or safety related constraints---user's comfort must never or at least rarely be violated. In this work, we propose a novel deep Reinforcement Learning framework with online Data Augmentation (RLDA) to address the sample efficiency challenge of real-world RL. We used a time series Generative Adversarial Network (TimeGAN) architecture as a data generator. We further evaluated the proposed RLDA framework using a case study of an intelligent HVAC control. With a ≈28% improvement in the sample efficiency, RLDA framework lays the way towards increased adoption of DRL-based intelligent control in real-world building energy management systems.

Kurte, Kuldeep↗
DOE OSTI2020

rlmolecule: A library for general-purpose material and molecular optimization using AlphaZero-style reinforcement learning

This library includes in-progress code for the optimization of materials and molecules using reinforcement learning. This project seeks to develop a generalized machine learning approach for optimizing targeted, and often complex, functional properties over the space of possible material structures, allowing faster identification of promising candidates. Electronic and transport properties of materials are determined by their molecular structure (for molecules) or their crystal structure (for in-organic crystalline materials). The atomic composition, bonding, and 3-D-spatial arrangement in these systems therefore represent a vast, discontinuous search space for optimal candidates. This library implements AlphaZero style reinforcement learning applied to two worked examples: the optimization of stable organic radicals for redox flow batteries, and the determination of stable electrolyte species for solid state batteries. The current code contains proof-of-concept results run on the Eagle HPC system, while the completed package will be a flexible tool for material optimization across of number of energy-relevant fields, which leverages cloud compute resources for the majority of the computational heavy lifting.

Biagioni, David↗
DOE OSTI2017

Structural Evaluation of Complex Subcomponents Manufactured by Large Scale Extrusion Deposition of Carbon Fiber Reinforced ABS

The large scale extrusion deposition process has recently been commercialized in the form of additive manufacturing machines such as the BAAM from Cincinnati Inc., which enables layered production of chopped fiber reinforced thermoplastic polymer systems. The application of this process to the automotive market is being pursued by Local Motors to enable rapid design to market cycles and build on-demand capability for a wide range of vehicle styles from a distributed network of micro-factories. This endeavor will require an understanding of the structural performance of components with complex geometries, where the properties are known to vary as a function of printing conditions, as well as orientation and shape. In order to rapidly build a database of structural performance, a beam specimen with outer dimensions of 914 x 305 x 64 mm3 was selected to be subjected to bending, torsion, and shear loading with full field displacement measurement by digital image correlation (DIC). A total of 25 different specimen configurations were designed including variation of print geometry, reinforcement by foam filling, and overwrapping with advanced composites. This work documents the results of the study and presents the relative performance of the structures in addition to the approach to simulation that is being initiated to serve as a framework for efficient design of structures manufactured in this way. The results are presented as a table of structural efficiency including stiffness and strength as functions of mass for each design. Additional results will also be provided in the form of surface displacements from DIC; future work will utilize this data to better optimize simulations for predicting structural performance of articles manufactured by this process.

Bedsole, Robert↗