Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “REINFORCEMENT”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 1,063 records · Page 59
All sourcesNASA NTRSDOE OSTI
DOE OSTI2019

Gnu-RL: A Precocial Reinforcement Learning Solution for Building HVAC Control Using a Differentiable MPC Policy

Reinforcement learning (RL) was first demonstrated to be a feasible approach to controlling heating, ventilation, and air conditioning (HVAC) systems more than a decade ago. However, there has been limited progress towards a practical and scalable RL solution for HVAC control. While one can train an RL agent in simulation, it is not cost-effective to create a model for each thermal zone or building. Likewise, existing RL agents generally take a long time to learn and are opaque to expert interrogation, making them unattractive for real-world deployment. To tackle these challenges, we propose Gnu-RL: a novel approach that enables practical deployment of RL for HVAC control and re- quires no prior information other than historical data from exist- ing HVAC controllers. To achieve this, Gnu-RL adopts a recently- developed Differentiable Model Predictive Control (MPC) policy, which encodes domain knowledge on planning and system dynam- ics, making it both sample-efficient and interpretable. Prior to any interaction with the environment, a Gnu-RL agent is pre-trained on historical data using imitation learning, which enables it to match the behavior of the existing controller. Once it is put in charge of controlling the environment, the agent continues to improve its policy end-to-end, using a policy gradient algorithm. We evaluate Gnu-RL on both an EnergyPlus model and a real- world testbed. In both experiments, our agents were directly de- ployed in the environment after offline pre-training on expert demonstration. In the simulation experiment, our approach saved 6.6% energy compared to the best published RL result for the same environment, while maintaining a higher level of occupant comfort. Next, Gnu-RL was deployed to control the HVAC of a real-world conference room for a three-week period. Our results show that Gnu-RL saved 16.7% of cooling demand compared to the existing controller and tracked temperature set-point better.

32 ENERGY CONSERVATION, CONSUMPTION, AND UTILIZATI↗
DOE OSTI2020

Electricity Pricing aware Deep Reinforcement Learning based Intelligent HVAC Control

Recently, deep reinforcement learning (DRL) based intelligent control of Heating, Ventilation, and Air Conditioning (HVAC) has gained a lot of attention due to DRL's ability to optimally control HVAC for minimizing operational cost while maintaining resident's comfort. The success of such DRL-based techniques largely depends on the articulation of the problem in terms of states, actions, and reward function. Inclusion of the electricity pricing information in the problem formulation can play an important role in saving the cost of HVAC operation. However, less attention has been given in the literature on formulating well-crafted state features based on electricity pricing. In this work, we propose an approach for training the DRL model with a specific focus on feature engineering based on electricity pricing. During training, we generate random but sufficiently realistic electricity price signals so that the pre-trained DRL model is robust and adaptive to the dynamic and variable electricity prices. The validation results are encouraging and show the potential of ≈12%-15% savings in the one day cost of HVAC operation, proving the usefulness of including electricity pricing related features as state features.

Kurte, Kuldeep↗
DOE OSTI2020

Deep Reinforcement Learning in Buildings: Implicit Assumptions and their Impact

As deep reinforcement learning (DRL) continues to gain interest in the smart building research community, there is a transition from simulation-based evaluations to deploying DRL control strategies in actual buildings. While the efficacy of a solution could depend on a particular implementation, there are common obstacles that developers have to overcome to deliver an effective controller. Additionally, a deployment in a physical building can invalidate some of the assumptions made during the controller development. Assumptions on the sensor placement or on the equipment behavior can quickly come undone. This paper presents some of the significant assumptions made during the development of DRL based controllers that could affect their operations in a physical building. Furthermore, a preliminary evaluation revealed that controllers developed with some of these assumptions can incur twice the expected costs when they are deployed in a building.

Prakash, Anand Krishnan↗
DOE OSTI2022

SchedInspector: A Batch Job Scheduling Inspector Using Reinforcement Learning

Improving the performance of job executions is an important goal of HPC batch job schedulers, such as minimizing job waiting time, slowdown, or completion time. Such a goal is often accomplished using carefully designed heuristics based on job features, such as job size and job duration. However, these heuristics overlook important runtime factors (e.g., cluster availability and waiting job patterns), which may vary across time and make a previously sound scheduling decision not hold any longer. In this study, we propose a new approach to incorporate runtime factors into batch job scheduling for better job execution performance. The key idea is to add a scheduling inspector on top of the base job scheduler to scrutinize its scheduling decisions. The inspector will take the runtime factors into consideration and accordingly determine the fitness of the scheduled job. It then either accepts the scheduled job or rejects it and asks the base schedulers to try again later. We realize such an inspector, namely SchedInspector, by leveraging the intelligence of reinforcement learning. Through extensive experiments, we show SchedInspector can intelligently integrate the runtime factors into various batch job scheduling policies, including the state-of-the-art one, to gain better job execution performance, such as smaller average bounded job slowdown (up to 69% better) or average job waiting time (up to 52% better), across various real-world workloads. We also show that although rejecting scheduling decisions may leave the resources idle hence affect the system utilization, SchedInspector is able to achieve the job execution performance improvement with marginal impact on the system utilization (typically less than 1%). We consider one key advantage of SchedInspector is it automatically learns to work with and improve existing job scheduling policies without changing them, which makes it promising to serve as a generic enhancer for various batch job scheduling policies.

Zhang, Di↗
DOE OSTI2021

PowerGridworld: A Framework for Multi-Agent Reinforcement Learning in Power Systems [SWR-22-07]

NREL's PowerGridworld provides a modular simulation environment for training heterogenous, grid-aware, multi-agent reinforcement learning (RL) policies at scale. The package enables the user to create component gym environments that can be composed into more complex agents. For example, a grid interactive building environment can be created by composing together component environments each encapsulating the building, PV, and battery physics. These multi-component environments can then be combined into multi-agent simulation where each agent's power consumption/injection becomes an input for solving the optimal power flow on a distribution feeder modeled in OpenDSS. Information from OpenDSS, such as bus voltages and line flows, can be included in the agents' observation spaces to enable grid-aware rewards. The default API for the PowerGridworld simulator conforms to RLLib's MultiAgent API and thus enables distributed training using HPC and cloud resources.

Biagioni, David↗
DOE OSTI2022

MPRL (Multi-Pulse Reinforcement Learning)

MPRL contains code for defining and deploying reinforcement learning control agents for multi-pulse, multi-fuel advanced compression ignition engines.

Wimer, Nicholas↗
DOE OSTI2022

Hybrid-RL-MPC4CLR (Hybird-Reinforcement-Learning-Model-Predictive-Control-for-Reserve-Policy-Assisted-Critical-Load-Restoration-in-Distribution-Grids)

Hybrid-RL-MPC4CLR was developed as a hybrid controller for active distribution grid critical load restoration, combining deep reinforcement learning (RL) and model predictive control (MPC) aiming at maximizing total restored load following an extreme event. The RL determines a policy for quantifying operating reserve requirements, thereby hedging against uncertainty, while the MPC models grid operations incorporating the RL policy actions (i.e., reserve requirements), renewable (wind and solar) power predictions, and load demand forecasts. The developers formulated the reserve requirement determination problem as a sequential decision-making problem based on the Markov Decision Process (MDP) and design an RL learning environment based on the OpenAI Gym framework and MPC simulation. The RL agent reward and MPC objective function aim to maximize and monotonically increase total restored load and minimize load shedding and renewable power curtailment. The software is developed using various software packages in Python. The MPC's optimal power flow (OPF) model is implemented using the Pyomo package, the RL simulation environment is implemented using the MPC simulation with various scenarios of renewable energy and load demand profiles and power outage beginning times, based on the OpenAI Gym framework. The RL agent training is performed using the RLlib Ray package. The RL algorithm is trained offline using historical forecasts of renewable generation and load demand profiles. Simulation analysis and performance tests are conducted using a modified IEEE 13-bus distribution test feeder containing wind turbine, photovoltaic, microturbine, and battery.

Eseye, Abinet Tesfaye↗
DOE OSTI2022

RLC4CLR (Reinforcement Learning Controller for Critical Load Restoration Problems)

RLC4CLR demonstrates using a reinforcement learning controller (RLC) to solve a critical load restoration (CLR) problem, which improves the grid resilience after a substation outage event. RLC4CLR consists of two parts. (1) RL environment: This environment encapsulates the CLR problem to be solved and provides interfacing functions to follow the standard OpenAI Gym format. A power system simulator, i.e., OpenDSS, is included to provide the power flow solution. Controller inputs and outputs (RL state and action) as well as the reward are defined in this environment as well. In summary, the RL environment is the problem formulation from which the RL agent can learn. (2) RL training script: The training script enables the RL agent to learn its control policy by interacting with the RL environment. For RL training, an open-sourced RL library, i.e., RLlib, is leveraged which is based on a distributed computing framework (Ray). The training script is designed to be able to be run on both local machine or the NREL HPC system. Other components of RLC4CLR include input data, e.g., grid model (standard IEEE test feeders), and other files used for results analysis.

Zhang, Xiangyu↗
DOE OSTI2021

Reinforcement Learning for Distribution Grid Optimization (PyCIGAR) v0.1

PyCIGAR is a python software package that merges off-the-shelf reinforcement learning libraries (RLLib and Ray) with electric power distribution system simulation tools (OpenDSS and a custom power flow solver built by LBL). PyCIGAR enables the training of neural networks to optimize the behavior of different components in the electric distribution grid, such as control systems in photovoltaic rooftop solar inverters and electric battery storage systems. The software package has been used to train neural networks to update settings in photovoltaic rooftop solar inverter control systems to mitigate cyber attacks on other solar photovoltaic rooftop devices.

Arnold, Daniel↗
DOE OSTI2025

CyRRL (Cyber Resilient Reinforcement Learning for grid voltage control) [SWR-24-115]

This codebase contains a multi-agent, actor-critic reinforcement learning implementation for cyber-resilient grid voltage control. It uses a 123-bus OpenDSS system as the environment, with three-phase power flow translating nodal power injections into solved nodal voltages. The reward function penalizes deviations from nominal voltage as well as reactive power dispatch, while encouraging agents to take actions that result in fast convergence to nominal conditions. The codebase models false data injection attacks and includes functionality for training, testing, hyper-parameter tuning, and visualization.

Murphy, Sinnott [National Renewable Energy Laborat↗
DOE OSTI2012

Effects of Surface Treatments on Mechanical Properties and Water Resistance of Kenaf Fiber-Reinforced Unsaturated Polyester Composites

Effects of surface treatments on the strength and water resistance of kenaf fiber-reinforced unsaturated polyester (UPE) composites were investigated. A new coupling agent that consists of 1,6-diisocyanato-hexane (DIH) and 2-hydroxylethyl acrylate (HEA) was investigated for surface treatments of kenaf fibers. The surface treatments were found to significantly enhance the tensile strength, modulus of rupture, modulus of elasticity, and water resistance of the resulting kenaf UPE composites. Fourier transform infrared spectroscopy (FTIR) confirmed that DIH-HEA was covalently bonded onto kenaf fibers. Scanning electron microscopy (SEM) images of the composites revealed that chemical treatment of kenaf fibers with a combination of DIH and HEA improved the interfacial adhesion between kenaf fibers and UPE resin in the DIHHEA-treated kenafUPE composites. The mechanisms by which the chemical treatment of kenaf fiber surfaces improved strength and water resistance of the resulting kenaf UPE composites were discussed.

co-polymerization↗
DOE OSTI2024

Vertical z-axis discontinuous carbon fibers for improved lightning strike performance of continuous fiber-reinforced polymer composites

Effective lightning strike protection for critical aerospace and wind applications requires high electrical conductivity to dissipate current efficiently. However, polymer matrix composites face a challenge due to their inherently insulating nature. While conventional carbon fiber-reinforced composites (CFRP) exhibit electrical conductivity in the planar direction, achieving through-thickness conductivity remains an ongoing challenge. In this work, we have undertaken the fabrication of CFRP interleaved with vertically oriented carbon fibers (Z-fiber) to impart higher electrical conductivity along the thickness direction. Two Z-fiber composite variations are prepared: Z-1 with a single layer of Z-fiber and Z-5 with five interleaved layers and compared with no Z-fiber layer (Z-0) composite. The composite panels were subjected to lab-scale lightning strike tests with a current magnitude of 100 kA. To emulate real-world service conditions, an aerospace-grade paint coating was applied to the composite laminates. Comparative analysis shows Z-1 reduces damage diameter to ∼22 mm compared to Z-0 (∼26 mm), while Z-5 exhibits the least damage (∼16.7 mm), confirmed by optical microscopy. Z-5 demonstrates nine times higher through-thickness electrical conductivity than Z-0, reducing electrical anisotropy substantially. Thermal-electric finite element damage modeling predicts surface damage within 6% of experimental values for both Z-0 and Z-5 composites. Flexural tests post-lightning reveal Z-5 retains 66% flexural strength and 86% modulus, significantly better than Z-0, which retains less than 40% for both properties. This study highlights the efficacy of Z-fiber composites in lightning strike protection, offering improved through-thickness conductivity and mechanical property retention.

36 MATERIALS SCIENCE↗
DOE OSTI2023

Carbon Nanotube and Graphene Oxide Reinforced Composites for Lightning Strike Protection of Composite Structures

Carbon fiber reinforced plastics (CFRP) composites have become the primary materials in the aerospace industry. Future transportation means, such as urban air mobility (UAM), will be employing CFRP structures. However, lightning strike damage on composite structures is a serious concern. CFRP composites possess low through-thickness electrical conductivity (≈ 1 S/m), which can cause extreme resistive heat in the event of lightning strikes. Research on polymeric lightning strike protection has gained momentum in the past few years. Researchers have studied various conductive fillers to improve the CFRP performance against lightning strikes. Carbon nanotubes (CNTs) and graphene oxide (GO) have been studied as future lightning Strike Protection (LSP) materials the most. The use of CNT and GO-based composites has shown improved mechanical properties and improved electrical conductivity of the composites. Researchers investigated CNTs and GO as the fillers to the matrix system, sizing the carbon fibers and the surface coating on top of the CFRP surface. This chapter highlights the remarkable improvement in the lightning damage suppression of CNT and GO-based composites and their structural applications. Special attention is given to their potential use as LSP material in future aircraft.

Kumar, Vipin↗
DOE OSTI2023

Enhanced lightning strike protection using vertically oriented carbon fiber melded with conventional carbon fiber-reinforced composite and its validation through damage analysis

Lightning strike protection (LSP) is one of the prime factors in aerospace and wind industries concerning safety, enhanced service life, and reduced downtime. To ensure better LSP, high electrical conductivity is required to dissipate the current, which is always a challenge for polymer composites due to the inherent insulating nature of the polymer matrix. Conventional carbon fiber-reinforced composites (CFRP) offer electrical conductivity in the planar direction while the vertical i.e. through-thickness conductivity still remains a challenge. Having been motivated by this fact, we have fabricated CFRP interleaved with vertically oriented CF (Z-fiber) for the lightning strike test (100 kA). A paint was applied to mimic the actual service condition of the composite laminates. We have prepared two Z-fiber composites: (a) Z-1 containing one Z-fiber layer on the top along with conventional CF layers and (b) Z-5 containing five interleaved Z-fiber layers and compared the properties with conventional CFRP (Z-0). Interestingly, even a single Z-fiber layer (Z-1) showed lower damage (visual damage diameter 22 mm) than Z-0 (visual damage diameter 26 mm). Z-5 displayed the lowest damage (visual damage diameter 16.7 mm), which was further confirmed by cross-sectional optical microscopy. Improved LSP was attributed to the higher through-thickness electrical conductivity in the case of Z-5 (9 times with respect to Z-0) reflecting a lower electrical anisotropy for Z-fiber composites. The residual mechanical property after the lightning test was analyzed through the flexural test, and the retention of flexural strength and modulus was 66% and 86%, respectively for Z-5 showing a significant improvement in comparison to Z-0 (>40% for both).

Saha, Subhabrata↗
DOE OSTI2022

PWR loading pattern optimization with reinforcement learning

The core loading pattern optimization problem belongs to the class of combinatorial optimization problem and has been studied since the dawn of commercial nuclear energy industry. It is characterized by multiple objectives and constraints, with a very high number of candidate patterns, which makes it impossible to solve explicitly. Stochastic optimization methodologies including Genetic Algorithms and Simulated Annealing are used by different nuclear utilities and vendors to perform fuel cycle reload design. Nevertheless, hand-designed solutions continue to be the prevalent method in the industry. To improve the state-of-the-art core reload patterns, we aim to create a method as scalable as possible, that agrees with the designer's goal of performance and safety. To help in this task Deep Reinforcement Learning (DRL), in particular Proximal Policy Optimization is leveraged. DRL has recently experienced a strong impetus from its successes applied to games, sometimes even reaching 'super-human' performances. This paper lays out the foundation of this method and proposes to study the behavior of several hyper-parameters that influence the DRL algorithm. The algorithm is highly dependent on multiple factors such as an exploration/exploitation trade-off that manifests through different parameters such as the number of loading patterns seen and the number of samples collected before a policy update, but also the shape of the objective function derived for the core design. Experimental results also demonstrate the effectiveness of the method in finding high-quality solutions from scratch within a reasonable amount of time. (authors)

21 SPECIFIC NUCLEAR REACTORS AND ASSOCIATED PLANTS↗
DOE OSTI2013

Tetramerization Reinforces the Dimer Interface of MnSOD

Two yeast manganese superoxide dismutases (MnSOD), one from Saccharomyces cerevisiae mitochondria (ScMnSOD) and the other from Candida albicans cytosol (CaMnSODc), have most biochemical and biophysical properties in common, yet ScMnSOD is a tetramer and CaMnSODc is a dimer or ‘‘loose tetramer’’ in solution. Although CaMnSODc was found to crystallize as a tetramer, there is no indication from the solution properties that the functionality of CaMnSODc in vivo depends upon the formation of the tetrameric structure. To elucidate further the functional significance of MnSOD quaternary structure, wild-type and mutant forms of ScMnSOD (K182R, A183P mutant) and CaMnSODc (K184R, L185P mutant) with the substitutions at dimer interfaces were analyzed with respect to their oligomeric states and resistance to pH, heat, and denaturant. Dimeric CaMnSODc was found to be significantly more subject to thermal or denaturant-induced unfolding than tetrameric ScMnSOD. The residue substitutions at dimer interfaces caused dimeric CaMnSODc but not tetrameric ScMnSOD to dissociate into monomers. We conclude that the tetrameric assembly strongly reinforces the dimer interface, which is critical for MnSOD activity.

59 BASIC BIOLOGICAL SCIENCES↗
DOE OSTI2024

Reinforcement Learning as a Parsimonious Alternative to Prediction Cascades: A Case Study on Image Segmentation

Deep learning architectures have achieved state-of-the-art (SOTA) performance on computer vision tasks such as object detection and image segmentation. This may be attributed to the use of over-parameterized, monolithic deep learning architectures executed on large datasets. Although such large architectures lead to increased accuracy, this is usually accompanied by a larger increase in computation and memory requirements during inference. While this is a non-issue in traditional machine learning (ML) pipelines, the recent confluence of machine learning and fields like the Internet of Things (IoT) has rendered such large architectures infeasible for execution in low-resource settings. For some datasets, large monolithic pipelines may be overkill for simpler inputs. To address this problem, previous efforts have proposed decision cascades where inputs are passed through models of increasing complexity until the desired performance is achieved. However, we argue that cascaded prediction leads to sub-optimal throughput and increased computational cost due to wasteful intermediate computations. To address this, we propose PaSeR (Parsimonious Segmentation with Reinforcement Learning) a non-cascading, cost-aware learning pipeline as an efficient alternative to cascaded decision architectures. Through experimental evaluation on both real-world and standard datasets, we demonstrate that PaSeR achieves better accuracy while minimizing computational cost relative to cascaded models. Further, we introduce a new metric IoU/GigaFlop to evaluate the balance between cost and performance. On the real-world task of battery material phase segmentation, PaSeR yields 179% improvement over SOTA MatPhase model and a 196% improvement over IDK Cascades under the IoU/GigaFlop metric. We also demonstrate PaSeR’s adaptability to complementary models trained on a noisy MNIST dataset, where it outperforms all baselines on IoU/GigaFlop by an average of 44%.

Srikshan, Bharat↗