Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “REINFORCEMENT”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 1,153 records · Page 64

An Advanced Meso-Scale Peridynamic Modeling Technology using High-Performance Computing for Cost-Effective Product Design and Testing of Carbon Fiber Reinforced Polymer Composites in Light-weight Vehicles

We study a peridynamic composite modeling technology based on the discontinuous Galerkin finite element method, implemented in the commercial LS-DYNA software, for modeling and prediction of failure in carbon fiber reinforced polymer composites. The proposed technology is developed for the material failure analysis at the meso-scale, which provides the prevailing fiber-matrix interaction mechanism, without adoption of the representative volume element method and thus avoiding complicated numerical calibration procedures. Three types of experimental tests—in-plane coupon test, out-of-plane coupon test, and a component crash test—are simulated in a high-performance computing environment to assess the performance of the proposed peridynamic composite modeling technology.

36 MATERIALS SCIENCE↗

Functionally Designed Ultra-Lightweight Carbon Fiber Reinforced Thermoplastic Composites Door Assembly (Final Scientific/Technical Report)

This is the final technical report for a project that developed an ultra-lightweight carbon fiber reinforced thermoplastics composite door assembly for a mid-size SUV. The project was able achieve a weight reduction of at least 42.5% while maintaining or exceeding the fit, function, safety, stiffness, and crash performance requirements of the current baseline door. Additionally, the target cost was set at less than $5 per pound of weight saved. To accomplish these goals, the project team employed a systems approach that involves redesigning the door assembly, exploring novel materials, optimizing the design, and utilizing advanced manufacturing technologies.

42 ENGINEERING↗

Predicting Catalyst Surface Stability Under Reaction Conditions Using Deep Reinforcement Learning and Machine Learning Potentials

Catalysts are critical for most large-scale energy intensive chemical transformation processes, such as energy storage, liquid fuel production, and the formation of chemical building blocks. The catalyst composition, structure, and morphology impact the performance under reaction conditions and influences properties like activity and selectivity. Many industrial catalysts offer imperfect activity/selectivity, or contain expensive metals. Further, catalysts can deactivate over time as the morphology changes or harsh reactive environments alter the surface structure. Methods to automatically model and predict the kinetics of how catalyst surfaces will restructure would enable engineers to design around these challenges, improve performance, increase longevity. This project investigated a specific type of machine learning model, deep reinforcement learning, machine learning models to act as surrogates for the physical system, and compared the results of these approaches with a specialized high-throughput experimental synthesis and measurement platform.

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗

Exploration with Scalable Gaussian Process Reinforcement Learning

Exploration is a challenging problem in reinforcement learning (RL), especially in environments with sparse rewards. Quantifying and utilizing the parametric uncertainty has been shown to be paramount for successful exploration [Osband et al., 2018]. Bayesian, or approximately Bayesian, methods present a principled means of estimating the parametric uncertainty in RL problems. Gaussian processes, nonparametric Bayesian models, are often impractical due to poor scalability and computational bottlenecks. We introduce a scalable Gaussian process RL (GPRL) method which directly induces sparsity in the covariance matrix to facilitate faster computation. This is a departure from previous GPRL methods which instead rely on data reduction and subsampling. We compare various covariance-based exploration techniques (Thompson sampling, upper confidence bound, and probabilistic maximum variance) which leverage our scalable GP framework in sparse reward environments. Finally, we show favorable comparison against the bootstrapped deep Q-Network.

97 MATHEMATICS AND COMPUTING↗

Reinforcement Learning Approach to Cybersecurity in Space (RELACSS)

Securing satellite groundstations against cyber-attacks is vital to national security missions. However, these cyber threats are constantly evolving. As vulnerabilities are discovered and patched, new vulnerabilities are discovered and exploited. In order to automate the process of discovering existing vulnerabilities and the means to exploit them, a reinforcement learning framework is presented in this report. We demonstrate that this framework can learn to successfully navigate an unknown network and detect nodes of interest despite the presence of a moving target defense. The agent then exfiltrates a file of interest from the node as quickly as possible. This framework also incorporates a defensive software agent that learns to impede the attacking agents progress. This setup allows for the agents to work against each other and improve their abilities. We anticipate that this capability will help uncover unforeseen vulnerabilities and the means to mitigate them. The modular nature of the framework enables users to swap out learning algorithms and modify the reward functions in order to adapt the learning tasks to various use cases and environments. Several algorithms, viz., tabular Q learning, deep Q networks, proximal policy optimization, advantage actor-critic, generative adversarial imitation learning, are explored for the agents and the results highlighted. The agent learns to solve the tasks in a light-weight abstract environment. Once the agent learns to perform sufficiently well, it can be deployed in a minimega virtual machine environment (or a real network) with wrappers that map abstract actions to software commands. The agent also uses a local representation of the actions called a ‘slot-mechanism’. This allows the agent to learn in a certain network and generalize it to different networks. The defensive agent learns to predict the actions taken by an offensive agent and uses that information to anticipate the threat. This information can then either be used to raise an alarm or to take actions to thwart the attack. We believe that with the appropriate reward design, a representative environment, and action set, this framework can be generalized to tackle other cybersecurity tasks. By sufficiently training these agents, we can anticipate vulnerabilities leading to robust future designs. We can also deploy automated defensive agents that can help secure satellite groundstation and their vital national security missions.

45 MILITARY TECHNOLOGY, WEAPONRY, AND NATIONAL DEF↗

Pedestrian-Involved Traffic Signal Optimization Using Decentralized Graph-based Multi-Agent Reinforcement Learning

Incorporating pedestrian movements in traffic signal timing optimization is essential for ensuring smooth and safe urban transportation systems. To develop an effective signal timing plan, it is crucial to comprehend how various pedestrian accommodation strategies impact both vehicle and pedestrian flow. This study explores the application of the Decentralized Graph-based Multi-Agent Reinforcement Learning (DGMARL) method for signal timing optimization. It considers both pedestrian and vehicle traffic state and assesses the effects of fixed and adaptive pedestrian request response on traffic and signal timing. The evaluation of DGMARL encompasses vehicles Eco_PI which is fuel consumption impact related to vehicle stops and stop delay, and pedestrian waiting time to ensure overall system efficiency. The proposed approach was evaluated using a Digital Twin microscopic traffic simulation model of MLK Smart Corridor in Chattanooga, Tennessee. The evaluation outcomes of vehicles Eco_PI, pedestrian waiting time and serving time are compared with the actuated and DGMARL signal timing with fixed pedestrian recall and minimum recall to determine the most suitable approaches for various traffic conditions. The results indicated that, on average, the strategy of Signal timing optimization with adaptive pedestrian request response improved the Eco_PI by 32.43%, and Signal timing optimization with both automated pedestrian traffic demand and adaptive pedestrian request response improved the Eco_PI by 31.62% compared to the actuated and DGMARL signal timing with fixed pedestrian recall and minimum recall.

K Kumarasamy, Vijayalakshmi↗

ReLIC: Full-Scale Realization of Reinforcement Learning for Infrastructure Control

Prior efforts have shown that deep reinforcement learning (DRL) may provide a new method for controlling networked power systems. Though successful, prior approaches have not yet demonstrated their behavior on systems of realistic scale. This effort examined multiple theoretical and technical approaches to allow a DRL model to operate over a system of 2,000 buses or more. We find that allowing the DRL models to run training episodes in parallel provides near limitless efficiency gains, allowing us to train successful agents to behave on our Kuramoto transmission model of up to 4,000 buses. We further show that we can expand our PowerWorld DRL implementation to systems of up to 25 buses but struggle to go beyond this limit due to PowerWorld’s inability to run multiple instances at once. Finally, we examine a multi-agent approach and find that it performs as well if not better than our existing centralized approach.

97 MATHEMATICS AND COMPUTING↗

End-to-End Optimization for Battery Materials and Molecules by Combining Graph Neural Networks and Reinforcement Learning

The National Renewable Energy Laboratory (NREL), together with the Colorado School of Mines (CSM) and Colorado State University (CSU), has developed a machine learning-enhanced approach to the design of new battery materials. Currently, such materials are designed in part via numerous expensive high-fidelity computational simulations that predict the performance of a given composition. Even with computational screening tools, the vast landscape of possible molecular or crystal structures exceeds current and future computational capacity. Improving the efficiency by which new materials can be optimized will therefore disrupt the cost, risk, and time required to bring new energy solutions to the marketplace. Predicting the properties of an organic molecule or periodic crystalline material given its structure has grown increasingly common. These approaches leverage large-scale computational and experimental databases and ML approaches such as graph neural networks. The inverse design problem of finding a material that possesses desired properties is substantially more challenging, since enumerating all valid material structures is not feasible. In this project, we leveraged recent success in reinforcement learning to efficiently navigate this high-dimensional search space. Just as algorithms can find the optimal chess moves from nearly limitless options, we train an approach to evolve a simple starting structure into a complex structure that possess the desired properties. Our solution has been demonstrated by applying it to two related design application tasks for short- and long-term energy storage, respectively: (1) the design of solid-state ion conductors and (2) the design of organic redox-active materials. The project has resulted an open-source software library for material design, documented examples of applying the library to both organic and inorganic material optimization, and peer-reviewed publications detailing the data, computational models, and resulting candidate materials.

25 ENERGY STORAGE↗

Reducing Mass of Steel Auto Bodies using Thin Advanced High Strength Steel with Carbon-Fiber Reinforced Epoxy

Diversitak, a company based in Detroit, MI, has developed a proprietary, low specific gravity, carbon fiber reinforced epoxy (CFRE) under U.S. patent number 9,963,58832. Preliminary testing on this new material conducted in collaboration with ArcelorMittal Steel Company proved out the CFRE concept. A thin layer of this CFRE was applied to a stamped sheet of steel with residual stamping oils from a mill, in a time corresponding to automotive processing (e.g., ~15 seconds), and processed following automotive e-coat procedures (phosphating + 175–200°C heating), to complete the curing. No problems with adherence or performance were noted. While the CFRE does add weight to a thin gauge steel panel, it weighs much less than what is displaced by using thicker conventional mild steel gauges. The application of the coating showed a significant increased dent resistance, oil canning resistance, and part stiffness.

36 MATERIALS SCIENCE↗

A Novel Manufacturing Process of Lightweight Automotive Seats (Integration of Additive Manufacturing and Reinforced Polymer Composite)

Lightweight automotive seats offer multiple benefits to original equipment manufacturers in terms of cost savings from various aspects, including less material usage, more integrated processes, and compliance with Corporate Average Fuel Economy Standards. Original equipment manufacturers have been focusing on innovative ways to produce light weight automotive seats. The commercially available automotive seats are currently made of multiple metal components combined through welding and fasteners. The use of additive manufacturing and composite structures is particularly useful for light weighting the automotive components. Additive manufacturing (AM) offers multiple advantages over traditional manufacturing processes such as freedom of design thereby enabling complex structural geometries, mass customization and waste minimization, and control over the fiber alignment through deposition in a predetermined pattern. Combining metal inserts with polymer composites through a novel manufacturing process allows design of lightweight and high-performance materials for automotive components. However, fabricating these metal polymer composite structures through traditional manufacturing processes limits their mechanical properties due to limited design freedom, lack of control over fiber orientation in composite parts, and poor interfacial bonding between the constituent materials. It is essential to develop a novel manufacturing process to enable high throughput production of lightweight automotive seats using metal and polymer composites. As such it is important to design the automotive seat suitable for manufacturing via this process and perform mechanical characterization on various subcomponents of the seat to ensure that the design and performance requirements provided by the auto manufacturer are met. The aim of this project is to develop a novel manufacturing technique to produce lightweight automotive seat by combining AM with conventional manufacturing processes. The car seat back panel will be designed via topology optimization and numerical simulations to minimize the overall weight while ensuring it meets all the performance requirements. The optimization of the seat back structure will be based on computational stress analysis to maximize the stiffness and minimize the weight. Materials currently used by Ford Motor Company will be adopted for a few subcomponents while the in-house composite materials will be used for the rest of the seat back. The composite and metallic materials will be tested to determine their mechanical properties as these are necessary for simulations. A novel manufacturing process will be developed to integrate AM metal inserts with discontinuous reinforced composite through large scale additive manufacturing and compression overmolding processes. The developed manufacturing technique will be used to fabricated various subcomponents suitable for the seat back design and mechanically tested to determine their properties. The manufacturing of the lightweight seat back design through this process involves integrated AM metal inserts with the composite structure for recliner connection. The manufacturing of the entire seat back which is lightweight through the novel manufacturing process will be discussed. The performance of the designed seat back will be investigated through numerical simulations and shown to meet all the requirements provided by the auto manufacturer. The final goal of developing a novel manufacturing process for lightweight automotive seats is met through design optimization of seat back, manufacturing of subcomponents, mechanical characterization, and validation through numerical simulations. The routes to achieve the final goal of the project and the depth in which they were investigated changed throughout the project due to personnel changes and the COVID-19 pandemic. The project resulted in the development of a novel manufacturing process to integrate metal inserts with tailored polymer composite preforms through overmolding. Leveraging this proven manufacturing process, a lightweight seat back was designed through topology optimization and numerical simulations. The designed seat back uses AM metal inserts and compression overmolding of tailored polymer composite preforms obtained via large scale additive manufacturing. The metal polymer composite structures fabricated through this process exhibited enhancement in stiffness and improved ductility upon testing. Overall, the project provided an alternative design and manufacturing technique for automotive seat back that enables weight saving while meeting the safety and performance requirements.

32 ENERGY CONSERVATION, CONSUMPTION, AND UTILIZATI↗

Nondestructive Evaluation of Carbon Fiber Reinforced Polymers

The American Society of Mechanical Engineers Boiler and Pressure Vessel Code requires repair and replacement of safety-related piping materials to meet the original Construction Code; however, no construction criteria currently exist for carbon fiber reinforced polymer (CFRP) materials in the nuclear industry. Although nondestructive examination (NDE) techniques for cast and wrought ferritic and austenitic steels are well established, licensees are increasingly deploying novel materials such as CFRP for which assessment of the use of NDE is required, as the inspectability of such materials and the influence of manufacturing processes on inspectability remain insufficiently understood. To address these gaps, Pacific Northwest National Laboratory (PNNL) evaluated the fabrication of several CFRP repair mockups and the effectiveness of NDE methods to support regulatory review of CFRP repairs in nuclear power plant applications. Representative flat-plate mockups containing realistic fabrication defects were manufactured and inspected using manual and automated tap testing, dynamic response spectroscopy (DRS), and ultrasonic testing (UT). The study identified significant fabrication variability, particularly in controlling defect size and epoxy thickness, which strongly influenced defect detectability by NDE methods. Tap testing was effective for shallow defects in thin epoxy configurations but unreliable for thicker repairs and deeper flaws. DRS demonstrated higher sensitivity to dry spots but produced unconfirmed indications for thicker plates, requiring further validation. Conventional UT, particularly at 1.0 MHz, showed the strongest overall capability for detecting a range of defects, although performance degraded with increasing repair thickness and complexity. The results highlight the need to better define critical defect sizes, develop reliable mockup fabrication methods, and validate NDE methods needed to support CFRP repairs.

36 MATERIALS SCIENCE↗

Adaptive Cybersecurity for Distributed Energy Resources (AdCyDER): Online Reinforcement Learning with Stackelberg-Optimized Defenses — Pipeline Architecture, Evaluation Methodology, and Findings from a Synthetic-Data Evaluation

This report documents the design and evaluation of an integrated online-learning pipeline developed within the AdCyDER project for Distributed Energy Resource (DER) cybersecurity. The pipeline couples a Reinforcement Learning (RL) attack classifier — which produces an attack-type probability distribution — with a Stackelberg game-theoretic (GT) defense selector that consumes those distributions alongside SME-encoded priors over (defense, attack) effectiveness pairings and perdefense costs to choose grid-health-preserving defenses. The objective is not attack classification per se but production of distributions that drive effective defense selection through the Stackelberg layer, learned from delayed grid-health feedback rather than labeled attack data. AdCyDER as a whole is broader than the work presented here; this report covers the specific RL/GT loop integration and its evaluation. We present the integrated pipeline (SCADA telemetry with Fronius inverter physics, Suricata IDS, time-windowed aggregation, per-facility LSTM classifier, Stackelberg optimizer, OpenC2 actuators), an experimental campaign of 28 eight-hour iterations across three baseline modes, and a pipeline-ordered diagnostic protocol. The protocol identifies two distinct failure modes within the loop: paired supervised ceilings on the same features establish that the deployed online RL classifier (macro F1 ≈ 0.07) sits at least 4.7× below a same-architecture supervised LSTM (≈ 0.34) and 10–11× below a linear feature-signal ceiling (≈ 0.70–0.79 depending on per-facility isolation), localizing the dominant failure to the training procedure; and the reward signal driving online updates carries weak directional coupling with classifier correctness in the methodology-expected direction (multi-lens convergent: top-decile P(true) records produce more frequent state changes and slightly larger improvements, top-vs-bot Cohen’s 𝑑 ≈ −0.19), but at effect magnitudes too small to drive gradient-based learning at the campaign sample size. The original learning hypothesis is not supported by the data. The primary contributions are the diagnostic methodology — proposed as a transferable falsification protocol for online RL/GT defense pipelines learning from delayed environmental reward — and the open, reproducible experimental infrastructure. We outline reward reformulation as the highest-priority aspirational next step given the underpowered-but-aligned Q6 reading, with hardware-in-the-loop evaluation as the broadest scope-expansion option.

Blakely, Benjamin [Argonne National Laboratory (AN↗

Sustainable Energy Solutions Task 4.2: UV Degradation Prevention on Fiber-Reinforced Composite Blades. Final report

Use of wind energy has expanded very quickly because of the energy prices, environmental concerns and improved efficiency of wind generators. Rather than using metal and alloy based wind turbine blades, larger size fiber (glass and carbon) reinforced composite blades have been recently utilized to increase the efficiency of the wind energy in both high and low wind potential areas. In the current composite manufacturing, pre-preg and vacuum-assisted/heat sensitive resin transfer molding and resin infusion methods are employed. However, these lighter, stiffer and stronger composite blades experience ultraviolet (UV) light degradation where polymers (epoxies and hardeners) used for the blades manufacturing absorb solar UV lights, and cause photolytic, thermo-oxidative and photo-oxidative reactions resulting in breaking of carbon-hydrogen bonds, polymer degradation and internal and external stresses. One of the main reasons is the weak protective coatings/paints on the composite blades. This process accelerates the aging and fatigue cracks, and reduces the overall mechanical properties of the blades. Thus, the lack of technology on coatings for blade manufacturing is forcing many government agencies and private companies (local and national windmill companies) to find a better solution for the composite wind blades. Kansas has a great wind potential for the future energy demand, so efficient wind generators can be an option for continuous energy production. The research goal of the present project was to develop nanocomposite coatings using various inclusions against UV degradation and corrosion, and advance the fundamental understanding of degradation (i.e., physical, chemical and physiochemical property changes) on those coatings. In pursuit of the research goal, the research objective of the present program was to investigate the effects of UV light and duration on various nanocomposites made mainly of carbon nanotubes and graphene nanoflakes, contribute the valuable information to this emerging field of advanced materials and manufacturing and advance the Kansas economy through creation of engineering knowledge and products in the wind energy. The proposed work was involved in a multidisciplinary research program that incorporates nanocomposite fabrication, advanced coating, characterization, surface and colloidal chemistry, physicochemistry, corrosion science, and analysis with a simple and effective testing methodology. The findings were closely related to our hypothesis and approaches that we proposed in this proposal. The data produced in the study offered to advance the physical understanding of the behavior of nanostructured materials for the prevention of UV light at different exposure time and salt fogging. Founding of this proposal enabled the first UV resistive nanocomposite corrosion coating effort in Kansas to impact the local and national wind mill industry. Results of this program provided valuable opportunities for the multidisciplinary training of undergraduate and graduate students at Wichita State University (WSU), as well as a number of aircraft companies (e.g., Cessna, Hawker Beechcraft, Spirit, Boeing and Bombardier/Learjet) and other local and regional industries.

17 WIND ENERGY↗

Sustainable Energy Solutions Task 4.1 Intelligent Manufacturing of Hybrid Carbon-Glass Fiber-Reinforced Composite Wind Turbine Blades

In this subtask, the manufacturability of hybrid carbon-glass fiber-reinforced composite wind turbine blades using Vacuum-Assisted Resin Transfer Molding (VARTM) was investigated. The objective of this investigation was to study the VARTM process and its parameters to manufacture cost-effective wind turbine blades with no defects (mainly eliminate dry spots and reduce manufacturing time). A 2.5-dimensional model and a 3-dimensional model were developed to simulate mold filling and part curing under different conditions. These conditions included isothermal and non-isothermal filling, curing of the part during and after filling, and placement of injection gates at different locations. Results from this investigation reveal that the process can be simulated and also that manufacturing parameters can be optimized to eliminate dry spot formation and reduce the manufacturing time. Using computer-based models is a cost-effective way to simulate manufacturing of wind turbine blades. The approach taken herein allows the design of the wind blade manufacturing processes without physically running trial-and-error experiments that are expensive and time-consuming; especially for larger blades needed for more demanding environmental conditions. This will benefit the wind energy industry by reducing initial design and manufacturing costs which can later be passed down to consumers and consequently make the wind energy industry more competitive.

17 WIND ENERGY↗

Reinforcement Learning Control for Enhancing Marine Hydrokinetic Turbine Energy Generation

This paper proposes a reinforcement learning-based method to maximize power generation for a direct-drive marine hydrokinetic turbine. A high levelized cost of energy (LCOE) is preventative in the widespread adoption of many marine energy conversion technologies. A straightforward way to reduce LCOE is to increase conversion efficiency and ensure maximum energy generation. The proposed method utilizes a damping control methodology, varying applied generator torque via a linear relationship between the applied damping coefficient and rotor speed. A state-action-reward-state-action (SARSA) algorithm has been used to learn the optimal control action for a given flow velocity. The proposed SARSA methodology uses Gaussian radial basis functions to create a three-dimensional surface to estimate the relationship between damping coefficient, incoming flow velocity, and coefficient of power (C p ). Here, the SARSA algorithm was compared against a baseline optimal tip speed ratio controller over a year-long flow velocity case profile while considering the effects of biofouling on the turbine system, where the proposed RL method generated 0.92% more energy than the baseline.

Damp↗

On the Verification of Deep Reinforcement Learning Solution for Intelligent Operation of Distribution Grids

Capabilities of deep reinforcement learning (DRL) in obtaining fast decision policies in high dimensional and stochastic environments have led to its extensive use in operational research, including the operation of distribution grids with high penetration of distributed energy resources (DER). However, the feasibility and robustness of DRL solutions are not guaranteed for the system operator, and hence, those solutions may be of limited practical value. This paper proposes an analytical method to find feasibility ellipsoids that represent the range of multi-dimensional system states in which the DRL solution is guaranteed to be feasible. Empirical studies and stochastic sampling determine the ratio of the discovered to the actual feasible space as a function of the sample size. In addition, the performance of logarithmic, linear, and exponential penalization of infeasibility during the DRL training are studied and compared in order to reduce the number of infeasible solutions

Hosseini, Mohammad Mehdi↗

Utilizing Reinforcement Learning to Continuously Improve a Primitive-Based Motion Planner

We report in this paper describes how the performance of motion primitive-based planning algorithms can be improved using reinforcement learning. Specifically, we describe and evaluate a framework that autonomously improves the performance of a primitive-based motion planner. The improvement process consists of three phases: exploration, extraction, and reward updates. This process can be iterated continuously to provide successive improvement. The exploration step generates new trajectories, and the extraction step identifies new primitives from these trajectories. These primitives are then used to update rewards for continued exploration. This framework required novel shaping rewards, development of a primitive extraction algorithm, and modification of the Hybrid A* algorithm. The framework is tested on a navigation task using a nonlinear F-16 model. The framework autonomously added 91 motion primitives to the primitive library and reduced average path cost by 21.6 seconds, or 35.75% of the original cost. The learned primitives are applied to an obstacle field navigation task, which was not used in training, and reduced path cost by 16.3 seconds, or 24.1%. Additionally, two heuristics for the modified Hybrid A* algorithm are designed to improve effective branching factor.

42 ENGINEERING↗