Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “REINFORCEMENT”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 379 records · Page 21
All sourcesNASA NTRSDOE OSTI
DOE OSTI2022

Learning and Fast Adaptation for Grid Emergency Control via Deep Meta Reinforcement Learning

As power systems are undergoing a significant transformation with more uncertainties, less inertia and closer to operation limits, there is increasing risk of large outages. Thus, there is an imperative need to enhance grid emergency control to maintain system reliability and security. Towards this end, great progress has been made in developing deep reinforcement learning (DRL) based grid control solutions in recent years. However, existing DRL-based solutions have two main limitations: 1) they cannot handle well with a wide range of grid operation conditions, system parameters, and contingencies; 2) they generally lack the ability to fast adapt to new grid operation conditions, system parameters, and contingencies, limiting their applicability for real-world applications. Here, in this paper, we mitigate these limitations by developing a novel deep meta-reinforcement learning (DMRL) algorithm. The DMRL combines the meta strategy optimization together with DRL, and trains policies modulated by a latent space that can quickly adapt to new scenarios. We test the developed DMRL algorithm on the IEEE 300-bus system. We demonstrate fast adaptation of the meta-trained DRL polices with latent variables to new operating conditions and scenarios using the proposed method, which achieves superior performance compared to the state-of-the-art DRL and model predictive control (MPC) methods.

24 POWER TRANSMISSION AND DISTRIBUTION↗
DOE OSTI2020

Deep Reinforcement Learning Based Volt-VAR Optimization in Smart Distribution Systems

This paper develops a model-free volt-VAR optimization (VVO) algorithm via multi-agent deep reinforcement learning (DRL) in unbalanced distribution systems. This method is novel since we cast the VVO problem in distribution networks to an intelligent deep Q-network (DQN) framework, which avoids solving a specific optimization model directly when facing time-varying operating conditions in the systems. We consider statuses/ratios of switchable capacitors, voltage regulators, and smart inverters installed at distributed generators as the action variables of the agents. A delicately designed reward function guides these agents to interact with the distribution system, in the direction of reinforcing voltage regulation and power loss reduction simultaneously. The forward-backward sweep method for radial three-phase distribution systems provides accurate power flow results within a few iterations to the DRL environment. The proposed method realizes the dual goals for VVO. We test this algorithm on the unbalanced IEEE 13-bus and 123-bus systems. Numerical simulations validate the excellent performance of this method in voltage regulation and power loss reduction.

41 EE - Solar Energy Technologies Office (EE-4S)↗
DOE OSTI2015

Reinforced concrete membranes under shear: Global behaviour

The main purpose of this article is to study compressed reinforced concrete membranes under perpendicular tensile stress. The state-of-the-art is presented and it shows that not much research has been done on this topic, probably because it needed very expensive laboratory equipment. In the current investigation, the authors have tried a new type of equipment, less expensive than that of previous investigations, and it proved to be adequate for the type of tests needed for studying this topic. The authors tested six reinforced concrete panels under pure shear up to failure. A parameter that was studied in these tests was the thickness of the membranes. This is of particular importance in webs or walls of beams under shear or torsion forces. From the experimental measurements, the authors have computed the principal directions and the values of the stresses and strains for the tested beams. In this article, these values are presented and the principal strains and stresses are compared with those at 45° and the two fairly agree with each other, particularly for the case of the stresses.

42 ENGINEERING↗
DOE OSTI2022

Adapting Reinforcement Learning Treatment Policies Using Limited Data to Personalize Critical Care

We report reinforcement learning (RL) demonstrates promise for developing effective treatment policies in critical care settings. However, existing RL methods often require large and comprehensive patient data sets and do not readily lend themselves to settings in which certain patient subpopulations are severely underrepresented. In this study, we develop a new method, noisy Bayesian policy updates (NBPU), for selecting high-performing reinforcement learning–based treatment policies for underrepresented patient subpopulations using limited observations. Our method uses variational inference to learn a probability distribution over treatment policies based on a reference patient subpopulation for which sufficient data are available. It then exploits limited data from an underrepresented patient subpopulation to update this probability distribution and adapts its recommendations to this subpopulation. We demonstrate our method’s utility on a data set of ICU patients receiving intravenous blood anticoagulant medication. Our results show that NBPU outperforms state-of-the-art methods in terms of both selecting effective treatment policies for patients with nontypical clinical characteristics and predicting the corresponding policies’ performance for these patients.

60 APPLIED LIFE SCIENCES↗
DOE OSTI2023

Discovery of False Data Injection Attacks on Power Grid Frequency Controllers with Reinforcement Learning [Poster]

While inverter-based DER (distributed energy resources) are instrumental to integrating renewable energy into the power grid, they reduce the grid's mechanical inertia, thereby increasing the risk of frequency instabilities. To compensate for frequency instability risks, the grid must also undergo a transformation to include digital technologies that allow for two-way communication between the utility and customers. The current and future state of the power grid allows for building a cleaner energy landscape. However, the grid may also become vulnerable to novel cyber threats. To preemptively protect the power grid against elaborate cyber-attacks, we propose to discover potential threats via reinforcement learning. In this work, the focus is on studying false data injection attacks that target the control logic of frequency controllers. We show that a reinforcement learning agent can successfully discover how to best inject false data into linear droop controllers.

24 POWER TRANSMISSION AND DISTRIBUTION↗
DOE OSTI2025

Reducing Mass of Steel Auto Bodies using Thin Advanced High Strength Steel with Carbon-Fiber Reinforced Epoxy

Diversitak, a company based in Detroit, MI, has developed a proprietary, low specific gravity, carbon fiber-reinforced epoxy (CFRE) under U.S. patent number 9,963,58832. Preliminary testing on this new material conducted in collaboration with ArcelorMittal Steel Company proved out the CFRE concept. A thin layer of this CFRE was applied to a stamped sheet of steel with residual stamping oils from a mill, in a time corresponding to automotive processing (e.g., ~15 seconds), and processed following automotive e-coat procedures (phosphating + 175–200°C heating), to complete the curing. No problems with adherence or performance were noted. While the CFRE does add weight to a thin gauge steel panel, it weighs much less than what is displaced by using thicker conventional mild steel gauges. The application of the coating showed a significant increased dent resistance, oil canning resistance, and part stiffness.This current two-year project was designed to mature this new technology to near manufacturing readiness to reduce the weight of a vehicle and lower the cost of weight reduction. The process involves the use of thinner gauge steels than are currently used. The collaborative development team included two industrial manufacturers: Diversitak and ArcelorMittal Steel Company; LightMAT; and two National Laboratories: Oak Ridge National Laboratory (ORNL) and Idaho National Laboratory (INL). The team developed a new manufacturing process to reduce the weight of a vehicle and lower the cost of weight reduction, as well as a better understanding of how to apply the coating so that it will perform to a high standard in-service. The team also performed an in-depth study to determine the long-term durability of the materials manufactured using this technology and well-known automotive industry standard tests.The overall process involved stiffening the thinner gauge steel by applying the CFRE on only one side. To accomplish this goal, the optimal reinforcement fiber length and fiber concentration was first determined. This was followed by measuring the coefficient of thermal expansion (CTE) in all three directions, so it could be fed into manufacturing models and methods for rapidly and inexpensively applying the coating. This was followed by panel level evaluations of the coating and steel combination, and then by full part demonstration of the technology on door panels. The final step was corrosion testing of the parts.ArcelorMittal characterized the advanced high strength steel (AHSS) (e.g., metallurgy-heat treatment for required AHSS properties as a function of the sheet thickness, state of internal stress) and quantified CFRE adhesion to the steel as a function of sheet preparation (e.g., rolling and stamping).ORNL optimized the fiber length, fiber concentration, and coating thickness for best vehicle function and performance at the least cost. Along with the suppliers, ORNL developed a durable CFRE application process (e.g., gun material, design, robotic dispensing process) and identified the adhesion stability of the CFRE during process holding. An approach to ensure that application/curing timing conforms to conventional assembly line speed and plant cycle times was determined. ORNL also determined the CTE of the material in all three directions and performed material scanning electron microscopy (SEM) analyses.INL characterized the corrosion properties of the steel panels coated with CFRE. The panels were investigated for corrosion resistance and stability as replacement materials used in automotive body panels to reduce mass. The coupons tested at INL were supplied by Diversitak after an optimized CFRE formulation was achieved in the already coated form for corrosion testing.

99 GENERAL AND MISCELLANEOUS↗
DOE OSTI2019

Distributed Reinforcement Learning with ADMM-RL

This paper presents a new algorithm for distributed Reinforcement Learning (RL). RL is an artificial intelligence (AI) control strategy such that controls for highly nonlinear systems over multi-step time horizons may be learned by experience, rather than directly computed on the fly by optimization. Here we introduce ADMM-RL, a combination of the Alternating Direction Method of Multipliers (ADMM) and reinforcement learning that allows for integrating learned controllers as subsystems in generally convergent distributed control applications. ADMM has become the workhorse algorithm for distributed control, combining the advantages of dual decomposition (namely, enabling decoupled, parallel, distributed solution) with the advantages of the method of multipliers (namely, convexification/stability). Our ADMM-RL algorithm replaces one or more of the subproblems in ADMM with several steps of RL. When the nested iterations converge, we are left with a pretrained subsolver that can potentially increase the efficiency of the deployed distributed controller by orders of magnitude. We illustrate ADMM-RL in both distributed wind farm yaw control and distributed grid-aware demand aggregation for water heaters.

ADMM↗
DOE OSTI2022

Physics Informed Reinforcement Learning for Power Grid Control using Augmented Random Search

Wide adoption of deep reinforcement learning need to overcome several challenges in energy system domain, including scalability, learning from limited samples, and high-dimensional continuous state and action spaces. In this paper, we integrated physics-based information from the normal generator operation state formula in the reinforcement learning agent's neural network loss function, and applied an augmented random search agent to optimize the generator control under dynamic contingency. Simulation results demonstrated the reliability performance improvements in training speed, reward convergence, sampling efficiency, scalability, and transferability.

physics informed ML, Physics Informed Neural Netwo↗
DOE OSTI2018

Reinforcement of Wood–Polymer Composites by Prestressed Elements

The reinforcement of wood–polymer composite beams and other long structural components by means of strong continuous prestressed elements is considered, in theoretical and practical terms. The use of plunger extrusion to introduce such reinforcement during the shaping process is proposed.

36 MATERIALS SCIENCE↗
DOE OSTI2023

A Novel Deep Reinforcement Learning Approach to Traffic Signal Control with Connected Vehicles

The advent of connected vehicle (CV) technology offers new possibilities for a revolution in future transportation systems. With the availability of real-time traffic data from CVs, it is possible to more effectively optimize traffic signals to reduce congestion, increase fuel efficiency, and enhance road safety. The success of CV-based signal control depends on an accurate and computationally efficient model that accounts for the stochastic and nonlinear nature of the traffic flow. Without the necessity of prior knowledge of the traffic system’s model architecture, reinforcement learning (RL) is a promising tool to acquire the control policy through observing the transition of the traffic states. In this paper, we propose a novel data-driven traffic signal control method that leverages the latest in deep learning and reinforcement learning techniques. By incorporating a compressed representation of the traffic states, the proposed method overcomes the limitations of the existing methods in defining the action space to include more practical and flexible signal phases. The simulation results demonstrate the convergence and robust performance of the proposed method against several existing benchmark methods in terms of average vehicle speeds, queue length, wait time, and traffic density.

42 ENGINEERING↗
DOE OSTI2023

Deep Reinforcement Learning for Resilient Power and Energy Systems: Progress, Prospects, and Future Avenues

In recent years, deep reinforcement learning (DRL) has garnered substantial attention in the context of enhancing resilience in power and energy systems. Resilience, characterized by the ability to withstand, absorb, and quickly recover from natural disasters and human-induced disruptions, has become paramount in ensuring the stability and dependability of critical infrastructure. This comprehensive review delves into the latest advancements and applications of DRL in enhancing the resilience of power and energy systems, highlighting significant contributions and key insights. The exploration commences with a concise elucidation of the fundamental principles of DRL, highlighting the intricate interplay among reinforcement learning (RL), deep learning, and the emergence of DRL. Furthermore, it categorizes and describes various DRL algorithms, laying a robust foundation for comprehending the applicability of DRL. The linkage between DRL and power system resilience is forged through a systematic classification of DRL applications into five pivotal dimensions: dynamic response, recovery and restoration, energy management and control, communications and cybersecurity, and resilience planning and metrics development. This structured categorization facilitates a methodical exploration of how DRL methodologies can effectively tackle critical challenges within the domain of power and energy system resilience. The review meticulously examines the inherent challenges and limitations entailed in integrating DRL into power and energy system resilience, shedding light on practical challenges and potential pitfalls. Additionally, it offers insights into promising avenues for future research, with the aim of inspiring innovative solutions and further progress in this vital domain.

24 POWER TRANSMISSION AND DISTRIBUTION↗
DOE OSTI2024

Integration of Decentralized Graph-Based Multi-Agent Reinforcement Learning with Digital Twin for Traffic Signal Optimization

Machine learning (ML) methods, particularly Reinforcement Learning (RL), have gained widespread attention for optimizing traffic signal control in intelligent transportation systems. However, existing ML approaches often exhibit limitations in scalability and adaptability, particularly within large traffic networks. This paper introduces an innovative solution by integrating decentralized graph-based multi-agent reinforcement learning (DGMARL) with a Digital Twin to enhance traffic signal optimization, targeting the reduction of traffic congestion and network-wide fuel consumption associated with vehicle stops and stop delays. In this approach, DGMARL agents are employed to learn traffic state patterns and make informed decisions regarding traffic signal control. The integration with a Digital Twin module further facilitates this process by simulating and replicating the real-time asymmetric traffic behaviors of a complex traffic network. The evaluation of this proposed methodology utilized PTV-Vissim, a traffic simulation software, which also serves as the simulation engine for the Digital Twin. The study focused on the Martin Luther King (MLK) Smart Corridor in Chattanooga, Tennessee, USA, by considering symmetric and asymmetric road layouts and traffic conditions. Comparative analysis against an actuated signal control baseline approach revealed significant improvements. Experiment results demonstrate a remarkable 55.38% reduction in Eco_PI, a developed performance measure capturing the cumulative impact of stops and penalized stop delays on fuel consumption, over a 24 h scenario. In a PM-peak-hour scenario, the average reduction in Eco_PI reached 38.94%, indicating the substantial improvement achieved in optimizing traffic flow and reducing fuel consumption during high-demand periods. These findings underscore the effectiveness of the integrated DGMARL and Digital Twin approach in optimizing traffic signals, contributing to a more sustainable and efficient traffic management system.

42 ENGINEERING↗
NASA NTRS2016

Time-Dependent Stress Rupture Strength Degradation of Hi-Nicalon Fiber-Reinforced Silicon Carbide Composites at Intermediate Temperatures

The stress rupture strength of silicon carbide fiber-reinforced silicon carbide composites with a boron nitride fiber coating decreases with time within the intermediate temperature range of 700 to 950 degree Celsius. Various theories have been proposed to explain the cause of the time-dependent stress rupture strength. The objective of this paper is to investigate the relative significance of the various theories for the time-dependent strength of silicon carbide fiber-reinforced silicon carbide composites. This is achieved through the development of a numerically based progressive failure analysis routine and through the application of the routine to simulate the composite stress rupture tests. The progressive failure routine is a time-marching routine with an iterative loop between a probability of fiber survival equation and a force equilibrium equation within each time step. Failure of the composite is assumed to initiate near a matrix crack and the progression of fiber failures occurs by global load sharing. The probability of survival equation is derived from consideration of the strength of ceramic fibers with randomly occurring and slow growing flaws as well as the mechanical interaction between the fibers and matrix near a matrix crack. The force equilibrium equation follows from the global load sharing presumption. The results of progressive failure analyses of the composite tests suggest that the relationship between time and stress-rupture strength is attributed almost entirely to the slow flaw growth within the fibers. Although other mechanisms may be present, they appear to have only a minor influence on the observed time-dependent behavior.

Progressive Failure Analysis↗
NASA NTRS2017

Multi-Objective Reinforcement Learning-Based Deep Neural Networks for Cognitive Space Communications

Future communication subsystems of space exploration missions can potentially benefit from software-defined radios (SDRs) controlled by machine learning algorithms. In this paper, we propose a novel hybrid radio resource allocation management control algorithm that integrates multi-objective reinforcement learning and deep artificial neural networks. The objective is to efficiently manage communications system resources by monitoring performance functions with common dependent variables that result in conflicting goals. The uncertainty in the performance of thousands of different possible combinations of radio parameters makes the trade-off between exploration and exploitation in reinforcement learning (RL) much more challenging for future critical space-based missions. Thus, the system should spend as little time as possible on exploring actions, and whenever it explores an action, it should perform at acceptable levels most of the time. The proposed approach enables on-line learning by interactions with the environment and restricts poor resource allocation performance through virtual environment exploration. Improvements in the multiobjective performance can be achieved via transmitter parameter adaptation on a packet-basis, with poorly predicted performance promptly resulting in rejected decisions. Simulations presented in this work considered the DVB-S2 standard adaptive transmitter parameters and additional ones expected to be present in future adaptive radio systems. Performance results are provided by analysis of the proposed hybrid algorithm when operating across a satellite communication channel from Earth to GEO orbit during clear sky conditions. The proposed approach constitutes part of the core cognitive engine proof-of-concept to be delivered to the NASA Glenn Research Center SCaN Testbed located onboard the International Space Station.

space archtiecture↗
NASA NTRS2017

Multi-Objective Reinforcement Learning-based Deep Neural Networks for Cognitive Space Communications

Future communication subsystems of space exploration missions can potentially benefit from software-defined radios (SDRs) controlled by machine learning algorithms. In this paper, we propose a novel hybrid radio resource allocation management control algorithm that integrates multi-objective reinforcement learning and deep artificial neural networks. The objective is to efficiently manage communications system resources by monitoring performance functions with common dependent variables that result in conflicting goals. The uncertainty in the performance of thousands of different possible combinations of radio parameters makes the trade-off between exploration and exploitation in reinforcement learning (RL) much more challenging for future critical space-based missions. Thus, the system should spend as little time as possible on exploring actions, and whenever it explores an action, it should perform at acceptable levels most of the time. The proposed approach enables on-line learning by interactions with the environment and restricts poor resource allocation performance through virtual environment exploration. Improvements in the multiobjective performance can be achieved via transmitter parameter adaptation on a packet-basis, with poorly predicted performance promptly resulting in rejected decisions. Simulations presented in this work considered the DVB-S2 standard adaptive transmitter parameters and additional ones expected to be present in future adaptive radio systems. Performance results are provided by analysis of the proposed hybrid algorithm when operating across a satellite communication channel from Earth to GEO orbit during clear sky conditions. The proposed approach constitutes part of the core cognitive engine proof-of-concept to be delivered to the NASA Glenn Research Center SCaN Testbed located onboard the International Space Station.

space archtiecture↗