Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “REINFORCEMENT”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 1,099 records · Page 61
All sourcesNASA NTRSDOE OSTI
DOE OSTI2022

Predicting Catalyst Surface Stability Under Reaction Conditions Using Deep Reinforcement Learning and Machine Learning Potentials

Catalysts are critical for most large-scale energy intensive chemical transformation processes, such as energy storage, liquid fuel production, and the formation of chemical building blocks. The catalyst composition, structure, and morphology impact the performance under reaction conditions and influences properties like activity and selectivity. Many industrial catalysts offer imperfect activity/selectivity, or contain expensive metals. Further, catalysts can deactivate over time as the morphology changes or harsh reactive environments alter the surface structure. Methods to automatically model and predict the kinetics of how catalyst surfaces will restructure would enable engineers to design around these challenges, improve performance, increase longevity. This project investigated a specific type of machine learning model, deep reinforcement learning, machine learning models to act as surrogates for the physical system, and compared the results of these approaches with a specialized high-throughput experimental synthesis and measurement platform.

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗
DOE OSTI2024

Exploration with Scalable Gaussian Process Reinforcement Learning

Exploration is a challenging problem in reinforcement learning (RL), especially in environments with sparse rewards. Quantifying and utilizing the parametric uncertainty has been shown to be paramount for successful exploration [Osband et al., 2018]. Bayesian, or approximately Bayesian, methods present a principled means of estimating the parametric uncertainty in RL problems. Gaussian processes, nonparametric Bayesian models, are often impractical due to poor scalability and computational bottlenecks. We introduce a scalable Gaussian process RL (GPRL) method which directly induces sparsity in the covariance matrix to facilitate faster computation. This is a departure from previous GPRL methods which instead rely on data reduction and subsampling. We compare various covariance-based exploration techniques (Thompson sampling, upper confidence bound, and probabilistic maximum variance) which leverage our scalable GP framework in sparse reward environments. Finally, we show favorable comparison against the bootstrapped deep Q-Network.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2023

Reinforcement Learning Approach to Cybersecurity in Space (RELACSS)

Securing satellite groundstations against cyber-attacks is vital to national security missions. However, these cyber threats are constantly evolving. As vulnerabilities are discovered and patched, new vulnerabilities are discovered and exploited. In order to automate the process of discovering existing vulnerabilities and the means to exploit them, a reinforcement learning framework is presented in this report. We demonstrate that this framework can learn to successfully navigate an unknown network and detect nodes of interest despite the presence of a moving target defense. The agent then exfiltrates a file of interest from the node as quickly as possible. This framework also incorporates a defensive software agent that learns to impede the attacking agents progress. This setup allows for the agents to work against each other and improve their abilities. We anticipate that this capability will help uncover unforeseen vulnerabilities and the means to mitigate them. The modular nature of the framework enables users to swap out learning algorithms and modify the reward functions in order to adapt the learning tasks to various use cases and environments. Several algorithms, viz., tabular Q learning, deep Q networks, proximal policy optimization, advantage actor-critic, generative adversarial imitation learning, are explored for the agents and the results highlighted. The agent learns to solve the tasks in a light-weight abstract environment. Once the agent learns to perform sufficiently well, it can be deployed in a minimega virtual machine environment (or a real network) with wrappers that map abstract actions to software commands. The agent also uses a local representation of the actions called a ‘slot-mechanism’. This allows the agent to learn in a certain network and generalize it to different networks. The defensive agent learns to predict the actions taken by an offensive agent and uses that information to anticipate the threat. This information can then either be used to raise an alarm or to take actions to thwart the attack. We believe that with the appropriate reward design, a representative environment, and action set, this framework can be generalized to tackle other cybersecurity tasks. By sufficiently training these agents, we can anticipate vulnerabilities leading to robust future designs. We can also deploy automated defensive agents that can help secure satellite groundstation and their vital national security missions.

45 MILITARY TECHNOLOGY, WEAPONRY, AND NATIONAL DEF↗
DOE OSTI2024

Pedestrian-Involved Traffic Signal Optimization Using Decentralized Graph-based Multi-Agent Reinforcement Learning

Incorporating pedestrian movements in traffic signal timing optimization is essential for ensuring smooth and safe urban transportation systems. To develop an effective signal timing plan, it is crucial to comprehend how various pedestrian accommodation strategies impact both vehicle and pedestrian flow. This study explores the application of the Decentralized Graph-based Multi-Agent Reinforcement Learning (DGMARL) method for signal timing optimization. It considers both pedestrian and vehicle traffic state and assesses the effects of fixed and adaptive pedestrian request response on traffic and signal timing. The evaluation of DGMARL encompasses vehicles Eco_PI which is fuel consumption impact related to vehicle stops and stop delay, and pedestrian waiting time to ensure overall system efficiency. The proposed approach was evaluated using a Digital Twin microscopic traffic simulation model of MLK Smart Corridor in Chattanooga, Tennessee. The evaluation outcomes of vehicles Eco_PI, pedestrian waiting time and serving time are compared with the actuated and DGMARL signal timing with fixed pedestrian recall and minimum recall to determine the most suitable approaches for various traffic conditions. The results indicated that, on average, the strategy of Signal timing optimization with adaptive pedestrian request response improved the Eco_PI by 32.43%, and Signal timing optimization with both automated pedestrian traffic demand and adaptive pedestrian request response improved the Eco_PI by 31.62% compared to the actuated and DGMARL signal timing with fixed pedestrian recall and minimum recall.

K Kumarasamy, Vijayalakshmi↗
DOE OSTI2024

ReLIC: Full-Scale Realization of Reinforcement Learning for Infrastructure Control

Prior efforts have shown that deep reinforcement learning (DRL) may provide a new method for controlling networked power systems. Though successful, prior approaches have not yet demonstrated their behavior on systems of realistic scale. This effort examined multiple theoretical and technical approaches to allow a DRL model to operate over a system of 2,000 buses or more. We find that allowing the DRL models to run training episodes in parallel provides near limitless efficiency gains, allowing us to train successful agents to behave on our Kuramoto transmission model of up to 4,000 buses. We further show that we can expand our PowerWorld DRL implementation to systems of up to 25 buses but struggle to go beyond this limit due to PowerWorld’s inability to run multiple instances at once. Finally, we examine a multi-agent approach and find that it performs as well if not better than our existing centralized approach.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2022

End-to-End Optimization for Battery Materials and Molecules by Combining Graph Neural Networks and Reinforcement Learning

The National Renewable Energy Laboratory (NREL), together with the Colorado School of Mines (CSM) and Colorado State University (CSU), has developed a machine learning-enhanced approach to the design of new battery materials. Currently, such materials are designed in part via numerous expensive high-fidelity computational simulations that predict the performance of a given composition. Even with computational screening tools, the vast landscape of possible molecular or crystal structures exceeds current and future computational capacity. Improving the efficiency by which new materials can be optimized will therefore disrupt the cost, risk, and time required to bring new energy solutions to the marketplace. Predicting the properties of an organic molecule or periodic crystalline material given its structure has grown increasingly common. These approaches leverage large-scale computational and experimental databases and ML approaches such as graph neural networks. The inverse design problem of finding a material that possesses desired properties is substantially more challenging, since enumerating all valid material structures is not feasible. In this project, we leveraged recent success in reinforcement learning to efficiently navigate this high-dimensional search space. Just as algorithms can find the optimal chess moves from nearly limitless options, we train an approach to evolve a simple starting structure into a complex structure that possess the desired properties. Our solution has been demonstrated by applying it to two related design application tasks for short- and long-term energy storage, respectively: (1) the design of solid-state ion conductors and (2) the design of organic redox-active materials. The project has resulted an open-source software library for material design, documented examples of applying the library to both organic and inorganic material optimization, and peer-reviewed publications detailing the data, computational models, and resulting candidate materials.

25 ENERGY STORAGE↗
DOE OSTI2025

Reducing Mass of Steel Auto Bodies using Thin Advanced High Strength Steel with Carbon-Fiber Reinforced Epoxy

Diversitak, a company based in Detroit, MI, has developed a proprietary, low specific gravity, carbon fiber reinforced epoxy (CFRE) under U.S. patent number 9,963,58832. Preliminary testing on this new material conducted in collaboration with ArcelorMittal Steel Company proved out the CFRE concept. A thin layer of this CFRE was applied to a stamped sheet of steel with residual stamping oils from a mill, in a time corresponding to automotive processing (e.g., ~15 seconds), and processed following automotive e-coat procedures (phosphating + 175–200°C heating), to complete the curing. No problems with adherence or performance were noted. While the CFRE does add weight to a thin gauge steel panel, it weighs much less than what is displaced by using thicker conventional mild steel gauges. The application of the coating showed a significant increased dent resistance, oil canning resistance, and part stiffness.

36 MATERIALS SCIENCE↗
DOE OSTI2025

A Novel Manufacturing Process of Lightweight Automotive Seats (Integration of Additive Manufacturing and Reinforced Polymer Composite)

Lightweight automotive seats offer multiple benefits to original equipment manufacturers in terms of cost savings from various aspects, including less material usage, more integrated processes, and compliance with Corporate Average Fuel Economy Standards. Original equipment manufacturers have been focusing on innovative ways to produce light weight automotive seats. The commercially available automotive seats are currently made of multiple metal components combined through welding and fasteners. The use of additive manufacturing and composite structures is particularly useful for light weighting the automotive components. Additive manufacturing (AM) offers multiple advantages over traditional manufacturing processes such as freedom of design thereby enabling complex structural geometries, mass customization and waste minimization, and control over the fiber alignment through deposition in a predetermined pattern. Combining metal inserts with polymer composites through a novel manufacturing process allows design of lightweight and high-performance materials for automotive components. However, fabricating these metal polymer composite structures through traditional manufacturing processes limits their mechanical properties due to limited design freedom, lack of control over fiber orientation in composite parts, and poor interfacial bonding between the constituent materials. It is essential to develop a novel manufacturing process to enable high throughput production of lightweight automotive seats using metal and polymer composites. As such it is important to design the automotive seat suitable for manufacturing via this process and perform mechanical characterization on various subcomponents of the seat to ensure that the design and performance requirements provided by the auto manufacturer are met. The aim of this project is to develop a novel manufacturing technique to produce lightweight automotive seat by combining AM with conventional manufacturing processes. The car seat back panel will be designed via topology optimization and numerical simulations to minimize the overall weight while ensuring it meets all the performance requirements. The optimization of the seat back structure will be based on computational stress analysis to maximize the stiffness and minimize the weight. Materials currently used by Ford Motor Company will be adopted for a few subcomponents while the in-house composite materials will be used for the rest of the seat back. The composite and metallic materials will be tested to determine their mechanical properties as these are necessary for simulations. A novel manufacturing process will be developed to integrate AM metal inserts with discontinuous reinforced composite through large scale additive manufacturing and compression overmolding processes. The developed manufacturing technique will be used to fabricated various subcomponents suitable for the seat back design and mechanically tested to determine their properties. The manufacturing of the lightweight seat back design through this process involves integrated AM metal inserts with the composite structure for recliner connection. The manufacturing of the entire seat back which is lightweight through the novel manufacturing process will be discussed. The performance of the designed seat back will be investigated through numerical simulations and shown to meet all the requirements provided by the auto manufacturer. The final goal of developing a novel manufacturing process for lightweight automotive seats is met through design optimization of seat back, manufacturing of subcomponents, mechanical characterization, and validation through numerical simulations. The routes to achieve the final goal of the project and the depth in which they were investigated changed throughout the project due to personnel changes and the COVID-19 pandemic. The project resulted in the development of a novel manufacturing process to integrate metal inserts with tailored polymer composite preforms through overmolding. Leveraging this proven manufacturing process, a lightweight seat back was designed through topology optimization and numerical simulations. The designed seat back uses AM metal inserts and compression overmolding of tailored polymer composite preforms obtained via large scale additive manufacturing. The metal polymer composite structures fabricated through this process exhibited enhancement in stiffness and improved ductility upon testing. Overall, the project provided an alternative design and manufacturing technique for automotive seat back that enables weight saving while meeting the safety and performance requirements.

32 ENERGY CONSERVATION, CONSUMPTION, AND UTILIZATI↗
DOE OSTI2026

Nondestructive Evaluation of Carbon Fiber Reinforced Polymers

The American Society of Mechanical Engineers Boiler and Pressure Vessel Code requires repair and replacement of safety-related piping materials to meet the original Construction Code; however, no construction criteria currently exist for carbon fiber reinforced polymer (CFRP) materials in the nuclear industry. Although nondestructive examination (NDE) techniques for cast and wrought ferritic and austenitic steels are well established, licensees are increasingly deploying novel materials such as CFRP for which assessment of the use of NDE is required, as the inspectability of such materials and the influence of manufacturing processes on inspectability remain insufficiently understood. To address these gaps, Pacific Northwest National Laboratory (PNNL) evaluated the fabrication of several CFRP repair mockups and the effectiveness of NDE methods to support regulatory review of CFRP repairs in nuclear power plant applications. Representative flat-plate mockups containing realistic fabrication defects were manufactured and inspected using manual and automated tap testing, dynamic response spectroscopy (DRS), and ultrasonic testing (UT). The study identified significant fabrication variability, particularly in controlling defect size and epoxy thickness, which strongly influenced defect detectability by NDE methods. Tap testing was effective for shallow defects in thin epoxy configurations but unreliable for thicker repairs and deeper flaws. DRS demonstrated higher sensitivity to dry spots but produced unconfirmed indications for thicker plates, requiring further validation. Conventional UT, particularly at 1.0 MHz, showed the strongest overall capability for detecting a range of defects, although performance degraded with increasing repair thickness and complexity. The results highlight the need to better define critical defect sizes, develop reliable mockup fabrication methods, and validate NDE methods needed to support CFRP repairs.

36 MATERIALS SCIENCE↗
DOE OSTI2026

Adaptive Cybersecurity for Distributed Energy Resources (AdCyDER): Online Reinforcement Learning with Stackelberg-Optimized Defenses — Pipeline Architecture, Evaluation Methodology, and Findings from a Synthetic-Data Evaluation

This report documents the design and evaluation of an integrated online-learning pipeline developed within the AdCyDER project for Distributed Energy Resource (DER) cybersecurity. The pipeline couples a Reinforcement Learning (RL) attack classifier — which produces an attack-type probability distribution — with a Stackelberg game-theoretic (GT) defense selector that consumes those distributions alongside SME-encoded priors over (defense, attack) effectiveness pairings and perdefense costs to choose grid-health-preserving defenses. The objective is not attack classification per se but production of distributions that drive effective defense selection through the Stackelberg layer, learned from delayed grid-health feedback rather than labeled attack data. AdCyDER as a whole is broader than the work presented here; this report covers the specific RL/GT loop integration and its evaluation. We present the integrated pipeline (SCADA telemetry with Fronius inverter physics, Suricata IDS, time-windowed aggregation, per-facility LSTM classifier, Stackelberg optimizer, OpenC2 actuators), an experimental campaign of 28 eight-hour iterations across three baseline modes, and a pipeline-ordered diagnostic protocol. The protocol identifies two distinct failure modes within the loop: paired supervised ceilings on the same features establish that the deployed online RL classifier (macro F1 ≈ 0.07) sits at least 4.7× below a same-architecture supervised LSTM (≈ 0.34) and 10–11× below a linear feature-signal ceiling (≈ 0.70–0.79 depending on per-facility isolation), localizing the dominant failure to the training procedure; and the reward signal driving online updates carries weak directional coupling with classifier correctness in the methodology-expected direction (multi-lens convergent: top-decile P(true) records produce more frequent state changes and slightly larger improvements, top-vs-bot Cohen’s 𝑑 ≈ −0.19), but at effect magnitudes too small to drive gradient-based learning at the campaign sample size. The original learning hypothesis is not supported by the data. The primary contributions are the diagnostic methodology — proposed as a transferable falsification protocol for online RL/GT defense pipelines learning from delayed environmental reward — and the open, reproducible experimental infrastructure. We outline reward reformulation as the highest-priority aspirational next step given the underpowered-but-aligned Q6 reading, with hardware-in-the-loop evaluation as the broadest scope-expansion option.

Blakely, Benjamin [Argonne National Laboratory (AN↗
DOE OSTI2024

Reinforcement Learning Control for Enhancing Marine Hydrokinetic Turbine Energy Generation

This paper proposes a reinforcement learning-based method to maximize power generation for a direct-drive marine hydrokinetic turbine. A high levelized cost of energy (LCOE) is preventative in the widespread adoption of many marine energy conversion technologies. A straightforward way to reduce LCOE is to increase conversion efficiency and ensure maximum energy generation. The proposed method utilizes a damping control methodology, varying applied generator torque via a linear relationship between the applied damping coefficient and rotor speed. A state-action-reward-state-action (SARSA) algorithm has been used to learn the optimal control action for a given flow velocity. The proposed SARSA methodology uses Gaussian radial basis functions to create a three-dimensional surface to estimate the relationship between damping coefficient, incoming flow velocity, and coefficient of power (C p ). Here, the SARSA algorithm was compared against a baseline optimal tip speed ratio controller over a year-long flow velocity case profile while considering the effects of biofouling on the turbine system, where the proposed RL method generated 0.92% more energy than the baseline.

Damp↗
DOE OSTI2022

On the Verification of Deep Reinforcement Learning Solution for Intelligent Operation of Distribution Grids

Capabilities of deep reinforcement learning (DRL) in obtaining fast decision policies in high dimensional and stochastic environments have led to its extensive use in operational research, including the operation of distribution grids with high penetration of distributed energy resources (DER). However, the feasibility and robustness of DRL solutions are not guaranteed for the system operator, and hence, those solutions may be of limited practical value. This paper proposes an analytical method to find feasibility ellipsoids that represent the range of multi-dimensional system states in which the DRL solution is guaranteed to be feasible. Empirical studies and stochastic sampling determine the ratio of the discovered to the actual feasible space as a function of the sample size. In addition, the performance of logarithmic, linear, and exponential penalization of infeasibility during the DRL training are studied and compared in order to reduce the number of infeasible solutions

Hosseini, Mohammad Mehdi↗
DOE OSTI2022

Utilizing Reinforcement Learning to Continuously Improve a Primitive-Based Motion Planner

We report in this paper describes how the performance of motion primitive-based planning algorithms can be improved using reinforcement learning. Specifically, we describe and evaluate a framework that autonomously improves the performance of a primitive-based motion planner. The improvement process consists of three phases: exploration, extraction, and reward updates. This process can be iterated continuously to provide successive improvement. The exploration step generates new trajectories, and the extraction step identifies new primitives from these trajectories. These primitives are then used to update rewards for continued exploration. This framework required novel shaping rewards, development of a primitive extraction algorithm, and modification of the Hybrid A* algorithm. The framework is tested on a navigation task using a nonlinear F-16 model. The framework autonomously added 91 motion primitives to the primitive library and reduced average path cost by 21.6 seconds, or 35.75% of the original cost. The learned primitives are applied to an obstacle field navigation task, which was not used in training, and reduced path cost by 16.3 seconds, or 24.1%. Additionally, two heuristics for the modified Hybrid A* algorithm are designed to improve effective branching factor.

42 ENGINEERING↗
DOE OSTI2024

Design, Selection, and Evaluation of Reinforcement Learning Single Agents for Ground Target Tracking

Previous approaches for small fixed-wing unmanned air systems that carry strapdown rather than gimbaled cameras achieved satisfactory ground target tracking performance using both standard and deep reinforcement learning algorithms. However, these approaches have significant restrictions and abstractions to the dynamics of the vehicle, such as constant airspeed and constant altitude, because the number of states and actions was necessarily limited. Thus, extensive tuning was required to obtain good tracking performance. The expansion from 4 state–action degrees of freedom to 15 enabled the agent to exploit previous reward functions that produced novel yet undesirable emergent behavior. This paper investigates the causes of and various potential solutions to undesirable emergent behavior in the ground target tracking problem. A combination of changes to the environment, reward structure, action space simplification, command rate, and controller implementation provides insight into obtaining stable tracking results. Consideration is given to reward structure selection and refinement to mitigate undesirable emergent behavior. Results presented in the paper for a simulated environment of a single unmanned air system tracking a randomly moving single ground target show that a soft actor–critic algorithm can produce feasible tracking trajectories without limiting the state space and action space, provided that the environment is properly posed.

Engineering↗
DOE OSTI2026

A User-Friendly GUI Tool for Automated Microstructural Analysis of Fiber-Reinforced Composites and Porous Structures

Understanding and quantifying microstructural features such as fiber orientation and porosity is critical for predicting the mechanical behavior and performance of fiber-reinforced polymer composites. Traditional manual analysis is time-consuming, subjective, and unsuitable for high-throughput datasets. We present a graphical user interface (GUI) application that automates the analysis of microscopy images to extract key microstructural metrics, including fiber orientation tensors, fiber orientation distribution, porosity and pore size distribution. The app integrates multiple image segmentation techniques including global and local thresholding, clustering, and region-based approaches, offering flexibility for different types of image qualities and features. Users can load microstructural images, select regions of interest and segmentation techniques tailored to their image dataset. It also addresses a critical challenge in fiber orientation analysis: the ambiguities caused by touching, overlapping, or partially cut fibers. It supports autorun examples for standardized workflows, enabling reproducible analysis and facilitating training and benchmarking. This tool significantly reduces manual intervention, enhances consistency, and accelerates data generation for structure–property modeling, process optimization, and digital materials research. The tool is intended for use by materials scientists, engineers, and researchers engaged in composite characterization, quality control, and machine learning-based microstructural studies.

Chawla, Komal [ORNL] (ORCID:0000000190327565)↗
DOE OSTI2018

A Novel CAE Method for Compression Molding Simulation of Carbon Fiber-Reinforced Thermoplastic Composite Sheet Materials

Its high-specific strength and stiffness with lower cost make discontinuous fiber-reinforced thermoplastic (FRT) materials an ideal choice for lightweight applications in the automotive industry. Compression molding is one of the preferred manufacturing processes for such materials as it offers the opportunity to maintain a longer fiber length and higher volume production. In the past, we have demonstrated that compression molding of FRT in bulk form can be simulated by treating melt flow as a continuum using the conservation of mass and momentum equations. However, the compression molding of such materials in sheet form using a similar approach does not work well. The assumption of melt flow as a continuum does not hold for such deformation processes. To address this challenge, we have developed a novel simulation approach. First, the draping of the sheet was simulated as a structural deformation using the explicit finite element approach. Next, the draped shape was compressed using fluid mechanics equations. The proposed method was verified by building a physical part and comparing the predicted fiber orientation and warpage measurements performed on the physical parts. The developed method and tools are expected to help in expediting the development of FRT parts, which will help achieve lightweight targets in the automotive industry.

36 MATERIALS SCIENCE↗
DOE OSTI2022

Mechanical Properties of High-Temperature Fiber-Reinforced Thermoset Composites with Plain Weave and Unidirectional Carbon Fiber Fillers

Fiber-reinforced thermoset composites are a class of materials that address the arising needs from the aerospace and hypersonic industries for high specific strength, temperature-resistant structural materials. Among the high-temperature resistant thermoset categories, phenolic triazine (PT) cyanate esters stand out thanks to their inherent high degradation temperature, glass transition temperature, and mechanical strength. Despite the outstanding properties of these thermosets, the performance of carbon fiber composites using PT cyanate esters as matrices has not been thoroughly characterized. This work evaluated PT and carbon fiber composites’ compressive properties and failure mechanisms with different fiber arrangements. A PT resin with both plain weave (PW) and non-crimped unidirectional (UD) carbon fiber mats was analyzed in this research. Highly loaded thermoset composites were obtained using process temperatures not exceeding 260 °C, and the composites proved to retain compressive strength at temperatures beyond 300 °C. Compressive testing revealed that PT composites retained compressive strength values of 50.4% of room temperature for UD composites and 61.4% for PW composites. Post-compressive failure observations of the gage section revealed that the mechanisms for failure evolved with temperature from brittle, delamination-dominant failure to shear-like failure promoted by the plastic failure of the matrix. This study demonstrated that PT composites are a good candidate for structural applications in harsh environments.

Hall, Samuel Ernesto↗