Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “reinforcement Learning”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 883 records · Page 49
All sourcesNASA NTRSDOE OSTI
NASA NTRS1992

Learning and tuning fuzzy logic controllers through reinforcements

A new method for learning and tuning a fuzzy logic controller based on reinforcements from a dynamic system is presented. In particular, our Generalized Approximate Reasoning-based Intelligent Control (GARIC) architecture: (1) learns and tunes a fuzzy logic controller even when only weak reinforcements, such as a binary failure signal, is available; (2) introduces a new conjunction operator in computing the rule strengths of fuzzy control rules; (3) introduces a new localized mean of maximum (LMOM) method in combining the conclusions of several firing control rules; and (4) learns to produce real-valued control actions. Learning is achieved by integrating fuzzy inference into a feedforward network, which can then adaptively improve performance by using gradient descent methods. We extend the AHC algorithm of Barto, Sutton, and Anderson to include the prior control knowledge of human operators. The GARIC architecture is applied to a cart-pole balancing system and has demonstrated significant improvements in terms of the speed of learning and robustness to changes in the dynamic system's parameters over previous schemes for cart-pole balancing.

Berenji, Hamid R.↗
DOE OSTI2024

Learning-Based Building Flexibility Estimation and Control to Improve Microgrid Economics and Resilience: Preprint

This paper proposes a learning-based building flexibility estimation and control framework to improve system economics and resilience. A data-driven building load flexibility model consisting of weather forecasting and estimating load consumption is proposed to quantify building heating, ventilation, and air conditioning (HVAC) load flexibility. A reinforcement learning-based microgrid controller is proposed to dispatch distributed generators, distributed energy resources, and build HVAC loads while taking flexibility information as one of the inputs. Simulation analysis is conducted on the model of a real microgrid in California. The effectiveness of the proposed learning-based building flexibility estimation and control in reducing microgrid energy costs and improving the sustainability of critical loads is demonstrated.

building load flexibility↗
DOE OSTI2024

Accelerated Sequence Design of Star Block Copolymers: An Unbiased Exploration Strategy via Fusion of Molecular Dynamics Simulations and Machine Learning

Star block copolymers (s-BCPs) have potential applications as novel surfactants or amphiphiles for emulsification, compatibilization, chemical transformations, and separations. s-BCPs have chain architectures where three or more linear diblock copolymer arms comprised of two chemically distinct linear polymers, e.g., solvophobic and solvophilic chains, are covalently joined at one point. The chemical composition of each of the subunit polymer chains comprising the arms, their molecular weights, and the number of arms can be varied to tailor the surface and interfacial activity of these architecturally unique molecules. Further, this makes identification of the optimal s-BCP design nontrivial as the total number of plausible s-BCP architectures is experimentally or computationally intractable. In this work, we use molecular dynamics (MD) simulations coupled with a reinforcement learning-based Monte Carlo tree search (MCTS) to identify s-BCP designs that minimize the interfacial tension between polar and nonpolar solvents. We first validate the MCTS approach for the design of small- and medium-sized s-BCPs and then use it to efficiently identify sequences of copolymer blocks for large-sized s-BCPs. The structural origins of interfacial tension in these systems are also identified by using the configurations obtained from MD simulations. Chemical insights into the arrangement of copolymer blocks that promote lower interfacial tension were mined using machine learning (ML) techniques. Overall, this work provides an efficient approach to solve design problems via fusion of simulations and ML and provides important groundwork for future experimental investigation of s-BCPs for various applications.

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗
DOE OSTI2023

PV inverter experimental data

The increase in power electronic based generation sources require accurate modeling of inverters. Accurate modeling requires experimental data over wider operation range. We used 20 kW off-the-shelf grid following PV inverter in the experiments. We used controllable AC supply and controllable DC supply to emulate AC and DC side characteristics. The experiments were performed at NREL's Energy Systems Integration Facility. Due to the limitations of the DC supply used, inverter is tested under 75%, 50%, 25% load conditions (This dataset does not contain 100% load condition). In the first dataset, for each operating condition, controllable AC source voltage is varied from 0.88 to 1.09 per unit (p.u) with a step value of 0.025 p.u while keeping the frequency at 60 Hz. In the second dataset, under similar load conditions (75%, 50%, 25% ), the frequency of the controllable AC source voltage was varied from 59.4 Hz to 60.45 Hz with a step value of 0.2 Hz. Voltage and frequency range is chosen based on inverter protection. Voltages and currents on DC and AC side are included in the dataset.

24 POWER TRANSMISSION AND DISTRIBUTION↗
DOE OSTI2023

Split phase inverter data

The increase in power electronic based generation sources require accurate modeling of inverters. Accurate modeling requires experimental data over wider operation range. We used 8.35 kW off-the-shelf grid following split phase PV inverter in the experiments. We used controllable AC supply and controllable DC supply to emulate AC and DC side characteristics. The experiments were performed at NREL's Energy Systems Integration Facility. Inverter is tested under 100%, 75%, 50%, 25% load conditions. In the first dataset, for each operating condition, controllable AC source voltage is varied from 0.9 to 1.1 per unit (p.u) with a step value of 0.025 p.u while keeping the frequency at 60 Hz. In the second dataset, under similar load conditions (100%, 75%, 50%, 25% ), the frequency of the controllable AC source voltage was varied from 59 Hz to 61 Hz with a step value of 0.2 Hz. Voltage and frequency range is chosen based on inverter protection. Voltages and currents on DC and AC side are included in the dataset.

24 POWER TRANSMISSION AND DISTRIBUTION↗
DOE OSTI2023

PV Inverter Experimental Dataset Version 2 with 100 Percent Power

The increase in power electronic based generation sources require accurate modeling of inverters. Accurate modeling requires experimental data over wider operation range. We used 20 kW off-the-shelf grid following PV inverter in the experiments. We used controllable AC supply and controllable DC supply to emulate AC and DC side characteristics. The experiments were performed at NREL's Energy Systems Integration Facility. The PV inverter is tested under 100%, 75%, 50%, 25% load conditions. In the first dataset, for each operating condition, controllable AC source voltage is varied from 0.88 to 1.09 per unit (p.u) with a step value of 0.025 p.u while keeping the frequency at 60 Hz. In the second dataset, under similar load conditions (100%, 75%, 50%, 25% ), the frequency of the controllable AC source voltage was varied from 59.4 Hz to 60.45 Hz with a step value of 0.2 Hz. Voltage and frequency range is chosen based on inverter protection. Voltages and currents on DC and AC side are included in the dataset.

14 SOLAR ENERGY↗
DOE OSTI2022

Transforming Energy Through Computational Excellence: Artificial Intelligence and Machine Learning

The National Renewable Energy Laboratory's (NREL's) expertise accelerates a decarbonized energy economy through the most advanced computational techniques, including artificial intelligence (AI) and Machine Learning (ML). AI is the simulation of human intelligence processes by machines, while ML encompasses the study of computer algorithms to imitate the way humans learn.

97 MATHEMATICS AND COMPUTING↗
NASA NTRS1992

Learning and tuning fuzzy logic controllers through reinforcements

This paper presents a new method for learning and tuning a fuzzy logic controller based on reinforcements from a dynamic system. In particular, our generalized approximate reasoning-based intelligent control (GARIC) architecture (1) learns and tunes a fuzzy logic controller even when only weak reinforcement, such as a binary failure signal, is available; (2) introduces a new conjunction operator in computing the rule strengths of fuzzy control rules; (3) introduces a new localized mean of maximum (LMOM) method in combining the conclusions of several firing control rules; and (4) learns to produce real-valued control actions. Learning is achieved by integrating fuzzy inference into a feedforward neural network, which can then adaptively improve performance by using gradient descent methods. We extend the AHC algorithm of Barto et al. (1983) to include the prior control knowledge of human operators. The GARIC architecture is applied to a cart-pole balancing system and demonstrates significant improvements in terms of the speed of learning and robustness to changes in the dynamic system's parameters over previous schemes for cart-pole balancing.

Berenji, Hamid R.↗
DOE OSTI2020

Reinforcement-based Program Induction in a Neural Virtual Machine

We present a neural virtual machine that can be trained to perform algorithmic tasks. Rather than combining a neural controller with non-neural memory storage as has been done in the past, this architecture is purely neural and emulates tape-based memory via fast associative weights (onestep learning). Here we formally define the architecture, and then extend the system to learn programs using recurrent policy gradient reinforcement learning based on examples of program inputs labeled with corresponding output targets, which are compared against actual output to generate a sparse reward signal. We describe the policy gradient training procedure used, and report its empirical performance on a number of smallscale list processing tasks, such as finding the maximum list element, filtering out certain elements, and reversing the order of the elements. These results show that program induction via reinforcement learning is possible using sparse rewards and solely neural computations.

Katz, Garrett E.↗
NASA NTRS2014

CLEANing the Reward: Counterfactual Actions to Remove Exploratory Action Noise in Multiagent Learning

Learning in multiagent systems can be slow because agents must learn both how to behave in a complex environment and how to account for the actions of other agents. The inability of an agent to distinguish between the true environmental dynamics and those caused by the stochastic exploratory actions of other agents creates noise in each agent's reward signal. This learning noise can have unforeseen and often undesirable effects on the resultant system performance. We define such noise as exploratory action noise, demonstrate the critical impact it can have on the learning process in multiagent settings, and introduce a reward structure to effectively remove such noise from each agent's reward signal. In particular, we introduce Coordinated Learning without Exploratory Action Noise (CLEAN) rewards and empirically demonstrate their benefits

Reinforcement Learning↗
DOE OSTI2023

Language model-accelerated deep symbolic optimization

Symbolic optimization methods have been used to solve varied challenging and relevant problems such as symbolic regression and neural architecture search. However, the current state of the art typically learns each problem from scratch and is unable to leverage pre-existing knowledge and datasets that are available for many applications. Here, inspired by the similarity between sequence representations learned in natural language processing and the formulation of symbolic optimization as a discrete sequence optimization problem, we propose language model-accelerated deep symbolic optimization (LA-DSO), a method that leverages language models to learn symbolic optimization solutions more efficiently. We demonstrate LA-DSO in two tasks: symbolic regression, which allows us to perform extensive experimentation due to its low computation requirements, and computational antibody optimization, which shows that our proposal accelerates learning in challenging real-world problems.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2021

Human-in-the-loop Sensing and Control for Commercial Building Energy Efficiency and Occupant Comfort

Most of the existing heating, ventilation and air conditioning (HVAC) systems in commercial buildings operate in a conservative manner by assuming maximum occupancy in each room during pre-specified periods of the week, leading to significant energy being wasted as rooms are over-conditioned compared to the actual requirements of the occupants. Though critical, our understanding of occupancy patterns and thermal comfort needs of the occupants in commercial buildings is lacking and it is well known that both of these quantities are stochastic and time-varying, thus requiring sensing solutions to estimate them. This project had the goal of designing, implementing and evaluating a hardware and software solution to ameliorate this challenge. In particular, a depth camera (one whose pixels reveal distance from the camera as opposed to color values) placed on doorways is used to detect entrance and exit events from thermal zones in the building, and thereby estimate their occupancy levels. This information is then fed to a novel control algorithm that can, through interactions with the HVAC system, learn how to provide control inputs that maximize comfort and minimize energy waste. The resulting system represents a significant improvement over existing controllers for commercial HVAC systems and allowed us to improve our understanding of the design of future human-in-the-loop control solutions. For this solution to be feasible, the project had target metrics for its performance and cost. In particular, entrance and exit events for occupants moving about the building would need to be detected with an accuracy higher than 97%; and the resulting control inputs derived from this information would need to lead to approximately 10% energy savings compared to a schedule-based controller. Furthermore, regarding the final hardware design, the project had a target bill of materials (BOM) cost for the sensing solution of less than US$200 per unit while using less than 25W of power on average. All of these target metrics were met or exceeded by our final proposed solution. We performed evaluations by deploying the system in over 20 rooms of different types across 6 commercial buildings in Pittsburgh, PA over the course of three years, and performing targeted controlled experiments to test its performance along the different metrics. The human-in-the-loop control solutions (both hardware and software) developed through this project are expected to lead to significant improvements in the comfort and energy efficiency of HVAC systems used in commercial buildings. The insights we developed through the project pave the way to HVAC systems that can condition interior spaces according to their real-time utilization and the thermal comfort needs of the occupants, thereby reducing energy use. They also open up a new learning-based way of configuring HVAC controllers without having to manually fine-tune them for each building. These innovations can significantly increase the adoption of novel control solutions by the industry and thereby save resources and reduce costs of operation.

32 ENERGY CONSERVATION, CONSUMPTION, AND UTILIZATI↗
NASA NTRSDate not supplied

Monte Carlo Tree Search for Integrated Planning, Learning, and Execution in Nondeterministic Python

We present a novel use of Monte Carlo Tree Search (MCTS),adapted to explore a search space produced by the choice points embedded in Python code. The choice points are non-deterministic assignment statements and subroutine calls. We present MCTS extensions required for doing tree search in this context which includes control constructs like hierarchical decomposition (subroutine calls), iterative while loops and conditional statements. We demonstrate how the system works in a simulated rideshare scenario in an urban setting, and present preliminary experiments as a proof of concept.

Automatic planning↗
DOE OSTI2025

Integrating Crack Detection and Pipe Shape Optimization for Enhanced Sewage System Durability

Crack detection in underground reinforced concrete pipes has been essential in determining the state of stormwater infrastructure. Detection models have been implemented for detecting cracks and other defects in pipes using CCTV footage for stormwater drainage systems. In addition, Finite element models have been used to determine optimum shapes and pipe thickness for different boundary conditions such as header pipes in power plants. The concept of shape optimization emerges as a crucial factor in power plant design and operation, with the potential to maximize performance while minimizing the use of materials. Shape optimization not only enhances efficiency but also contributes to reducing the environmental footprint. This paper discusses the integration of both topics by using the cracks detected in underground pipes as boundary conditions for shape optimization of the pipes. A machine learning model has been developed which uses limited data for training and outlines the location of detected cracks. A shape optimization methodology is proposed in which ANSYS modules are used to analyze fluid flow and then optimize the shape of the pipe. The crack detection model developed has been applied to a crack detected in lab setting and machine learning model used has an accuracy of 98% using a random forest algorithm.

20 FOSSIL-FUELED POWER PLANTS↗