Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “agent based simulation”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 271 records · Page 15
All sourcesNASA NTRSDOE OSTI
DOE OSTI2022

Diffuse Radiation Forcing Constraints on Gross Primary Productivity and Global Terrestrial Evapotranspiration

Abstract The diffuse radiation fertilization effect—the increase in plant productivity in the presence of higher diffuse radiation ( K ↓,d )—is an important yet understudied aspect of atmosphere‐biosphere interactions and can modify the terrestrial carbon, energy, and water budgets. The K ↓,d fertilization effect links the carbon cycle with clouds and aerosols, all of which are large sources of uncertainties for our current understanding of the Earth system and for future climate projections. Here we establish to what extent observational and modeling uncertainty in sunlight's diffuse fraction ( k d ) affects simulated gross primary productivity (GPP) and terrestrial evapotranspiration ( λE ). We find only 48 eddy covariance sites with simultaneous sufficient measurements of K ↓,d with none in the tropical climate zone, making it difficult to constrain this mechanism globally using observations. Using a land modeling framework based on the latest version of the Community Land Model, we find that global GPP ranges from 114 Pg C year −1 when using k d forcing from the Modern‐Era Retrospective analysis for Research and Applications, version 2 reanalysis to a ∼7% higher value of 122 Pg C year −1 when using the Clouds and the Earth's Radiant Energy System satellite product, with especially strong differences apparent over the tropical region (mean increase ∼9%). The differences in λE , although smaller (−0.4%) due to competing changes in shaded and sunlit leaf transpiration, can be greater than regional impacts of individual forcing agents like aerosols. Our results demonstrate the importance of comprehensively and systematically validating the simulated k d by atmosphere modules as well as the response to differences in k d within land modules across Earth System Models.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2023

Optimizing Patient-Specific Medication Regimen Policies Using Wearable Sensors in Parkinson’s Disease

Effective treatment of Parkinson’s disease (PD) is a continual challenge for healthcare providers, and providers can benefit from leveraging emerging technologies to supplement traditional clinic care. We develop a data-driven reinforcement learning (RL) framework to optimize PD medication regimens through wearable sensors. We leverage a data set of n = 26 PD patients who wore wrist-mounted movement trackers for two separate six-day periods. Using these data, we first build and validate a simulation model of how individual patients’ movement symptoms respond to medication administration. We then pair this simulation model with an on-policy RL algorithm that recommends optimal medication types, timing, and dosages during the day while incorporating human-in-the-loop considerations on medication administration. The results show that the RL-prescribed medication regimens outperform physicians’ medication regimens, despite physicians having access to the same data as the RL agent. To validate our results, we assess our wearable-based RL medication regimens using n = 399 PD patients from the Parkinson’s Progression Markers Initiative data set. We show that the wearable-based RL medication regimens would lead to significant symptom improvement for these patients, even more so than training RL policies directly from this data set. In doing so, we show that RL models from even small data sets of wearable data can offer novel, generalizable clinical insights and medication strategies, which may outperform those derived from larger data sets without wearable data.

60 APPLIED LIFE SCIENCES↗
DOE OSTI2024

Integration of Decentralized Graph-Based Multi-Agent Reinforcement Learning with Digital Twin for Traffic Signal Optimization

Machine learning (ML) methods, particularly Reinforcement Learning (RL), have gained widespread attention for optimizing traffic signal control in intelligent transportation systems. However, existing ML approaches often exhibit limitations in scalability and adaptability, particularly within large traffic networks. This paper introduces an innovative solution by integrating decentralized graph-based multi-agent reinforcement learning (DGMARL) with a Digital Twin to enhance traffic signal optimization, targeting the reduction of traffic congestion and network-wide fuel consumption associated with vehicle stops and stop delays. In this approach, DGMARL agents are employed to learn traffic state patterns and make informed decisions regarding traffic signal control. The integration with a Digital Twin module further facilitates this process by simulating and replicating the real-time asymmetric traffic behaviors of a complex traffic network. The evaluation of this proposed methodology utilized PTV-Vissim, a traffic simulation software, which also serves as the simulation engine for the Digital Twin. The study focused on the Martin Luther King (MLK) Smart Corridor in Chattanooga, Tennessee, USA, by considering symmetric and asymmetric road layouts and traffic conditions. Comparative analysis against an actuated signal control baseline approach revealed significant improvements. Experiment results demonstrate a remarkable 55.38% reduction in Eco_PI, a developed performance measure capturing the cumulative impact of stops and penalized stop delays on fuel consumption, over a 24 h scenario. In a PM-peak-hour scenario, the average reduction in Eco_PI reached 38.94%, indicating the substantial improvement achieved in optimizing traffic flow and reducing fuel consumption during high-demand periods. These findings underscore the effectiveness of the integrated DGMARL and Digital Twin approach in optimizing traffic signals, contributing to a more sustainable and efficient traffic management system.

42 ENGINEERING↗
DOE OSTI2024

Integration of Decentralized Graph-Based Multi-Agent Reinforcement Learning with Digital Twin for Traffic Signal Optimization

Machine learning (ML) methods, particularly Reinforcement Learning (RL), have gained widespread attention for optimizing traffic signal control in intelligent transportation systems. However, existing ML approaches often exhibit limitations in scalability and adaptability, particularly within large traffic networks. This paper introduces an innovative solution by integrating decentralized graph-based multi-agent reinforcement learning (DGMARL) with a Digital Twin to enhance traffic signal optimization, targeting the reduction of traffic congestion and network-wide fuel consumption associated with vehicle stops and stop delays. In this approach, DGMARL agents are employed to learn traffic state patterns and make informed decisions regarding traffic signal control. The integration with a Digital Twin module further facilitates this process by simulating and replicating the real-time asymmetric traffic behaviors of a complex traffic network. The evaluation of this proposed methodology utilized PTV-Vissim, a traffic simulation software, which also serves as the simulation engine for the Digital Twin. The study focused on the Martin Luther King (MLK) Smart Corridor in Chattanooga, Tennessee, USA, by considering symmetric and asymmetric road layouts and traffic conditions. Comparative analysis against an actuated signal control baseline approach revealed significant improvements. Experiment results demonstrate a remarkable 55.38% reduction in Eco_PI, a developed performance measure capturing the cumulative impact of stops and penalized stop delays on fuel consumption, over a 24 h scenario. In a PM-peak-hour scenario, the average reduction in Eco_PI reached 38.94%, indicating the substantial improvement achieved in optimizing traffic flow and reducing fuel consumption during high-demand periods. These findings underscore the effectiveness of the integrated DGMARL and Digital Twin approach in optimizing traffic signals, contributing to a more sustainable and efficient traffic management system.

actuated signal control↗
DOE OSTI2022

Multi-objective goal-directed optimization of de novo stable organic radicals for aqueous redox flow batteries

Abstract Advances in the field of goal-directed molecular optimization offer the promise of finding feasible candidates for even the most challenging molecular design applications. One example of a fundamental design challenge is the search for novel stable radical scaffolds for an aqueous redox flow battery that simultaneously satisfy redox requirements at the anode and cathode, as relatively few stable organic radicals are known to exist. To meet this challenge, we develop a new open-source molecular optimization framework based on AlphaZero coupled with a fast, machine-learning-derived surrogate objective trained with nearly 100,000 quantum chemistry simulations. The objective function comprises two graph neural networks: one that predicts adiabatic oxidation and reduction potentials and a second that predicts electron density and local three-dimensional environment, previously shown to be correlated with radical persistence and stability. With no hard-coded knowledge of organic chemistry, the reinforcement learning agent finds molecule candidates that satisfy a precise combination of redox, stability and synthesizability requirements defined at the quantum chemistry level, many of which have reasonable predicted retrosynthetic pathways. The optimized molecules show that alternative stable radical scaffolds may offer a unique profile of stability and redox potentials to enable low-cost symmetric aqueous redox flow batteries.

25 ENERGY STORAGE↗
DOE OSTI2025

High-pressure reduction of carbon dioxide in reactive liquid mixtures

Density-functional theory based molecular-dynamics simulations were used to investigate high-pressure chemical reactions in liquid mixtures of CO 2 with several elements (Si, Mn, and Fe) at high temperatures of 2000-3000 K. Our ab initio simulations indicate that these reactant elements can reduce CO 2 to C at high pressures (20 GPa) leading to the formation of C-C chains, with Si by far the most effective carbon-reducing agent. A combined chemical analysis using Bader charge analysis and Crystal Orbital Hamilton Population (COHP) on simulation snapshots shows that significant charge transfer from the reducing element to the C atoms creates instability in the C-O covalent bonds. COHP analysis further shows that Mn/Fe-O and Mn/Fe-C bonding interactions are weaker compared to the Si counterparts. These results further our understanding of the redox chemistry of CO 2 at conditions relevant to planetary mantle interiors and demonstrate the effectiveness of high pressure in the reduction of CO 2 directly to solid carbon.

Ab initio molecular dynamics↗
DOE OSTI2020

Multi-agent motion planning with sporadic communications for collision avoidance

Here, a novel multi-vehicle motion planning and collision avoidance algorithm is proposed and analyzed. The algorithm aims to reduce the amount of onboard calculations and inter-agent communications needed for each vehicle to successfully navigate through an environment with static obstacles and reach their goals. To this end, each agent first calculates a path to the goal by means of an asymptotically optimal rapidly-exploring random tree (RRT*) with respect to the static obstacles. Then, other agents are treated as dynamic obstacles and potential collisions are determined by means of collision cones. Collision cones depend on the position and velocity from other agents and are grown conservatively between inter-agent communications. Based on the available information, each agent determines if a deconfliction maneuver is needed, if it can continue along its current path, or if communication is needed to make a decision about a conflict. With probability one, our algorithm guarantees that the agents keep from colliding with each other. Under an assumption on the existence of a solution for a vehicle to its goal, this algorithm also solves the planning problem with probability one. Simulations illustrate a group of agents successfully reaching their goal configurations and examine how the uncertainty affects the communication frequency of the multi-agent system.

33 ADVANCED PROPULSION SYSTEMS↗
DOE OSTI2024

Dyn$\mathrm{AMO}$: Multi-agent reinforcement learning for dynamic anticipatory mesh optimization with applications to hyperbolic conservation laws

Here we introduce DynAMO, a reinforcement learning paradigm for Dynamic Anticipatory Mesh Optimization. Adaptive mesh refinement is an effective tool for optimizing computational cost and solution accuracy in numerical methods for partial differential equations. However, traditional adaptive mesh refinement approaches for time-dependent problems typically rely only on instantaneous error indicators to guide adaptivity. As a result, standard strategies often require frequent remeshing to maintain accuracy. In the DynAMO approach, multi-agent reinforcement learning is used to discover new local refinement policies that can anticipate and respond to future solution states by producing meshes that deliver more accurate solutions for longer time intervals. By applying DynAMO to discontinuous Galerkin methods for the linear advection and compressible Euler equations in two dimensions, we demonstrate that this new mesh refinement paradigm can outperform conventional threshold-based strategies while also generalizing to different mesh sizes, remeshing and simulation times, and initial conditions.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2024

Pedestrian-Involved Traffic Signal Optimization Using Decentralized Graph-based Multi-Agent Reinforcement Learning

Incorporating pedestrian movements in traffic signal timing optimization is essential for ensuring smooth and safe urban transportation systems. To develop an effective signal timing plan, it is crucial to comprehend how various pedestrian accommodation strategies impact both vehicle and pedestrian flow. This study explores the application of the Decentralized Graph-based Multi-Agent Reinforcement Learning (DGMARL) method for signal timing optimization. It considers both pedestrian and vehicle traffic state and assesses the effects of fixed and adaptive pedestrian request response on traffic and signal timing. The evaluation of DGMARL encompasses vehicles Eco_PI which is fuel consumption impact related to vehicle stops and stop delay, and pedestrian waiting time to ensure overall system efficiency. The proposed approach was evaluated using a Digital Twin microscopic traffic simulation model of MLK Smart Corridor in Chattanooga, Tennessee. The evaluation outcomes of vehicles Eco_PI, pedestrian waiting time and serving time are compared with the actuated and DGMARL signal timing with fixed pedestrian recall and minimum recall to determine the most suitable approaches for various traffic conditions. The results indicated that, on average, the strategy of Signal timing optimization with adaptive pedestrian request response improved the Eco_PI by 32.43%, and Signal timing optimization with both automated pedestrian traffic demand and adaptive pedestrian request response improved the Eco_PI by 31.62% compared to the actuated and DGMARL signal timing with fixed pedestrian recall and minimum recall.

K Kumarasamy, Vijayalakshmi↗
DOE OSTI2020

Flexible Reinforcement Learning Framework for Building Control using EnergyPlus-Modelica Energy Models

In recent years, reinforcement learning (RL) methods have been greatly enhanced by leveraging deep learning approaches. RL methods applied to building control have shown potential in many applications due to their ability to complement or replace conventional methods such as model-based or rule-based controls. However, RL-based building control software is likely tailored either to one target building system or to a specific RL method so that significant additional effort would be required to customize the RL-based controller for use in other building systems or with other RL approaches. Also, RL-based building controls usually depend on building energy simulations to train controllers, so emulating building dynamics (i.e., thermal dynamics and control dynamics) and capturing sub-hourly dynamic profiles are crucial to further the development of effective RL-based building control methods. To address these challenges, we present an open source RL-based control software employing a high-fidelity hybrid EnergyPlus-Modelica building energy model which emulates building dynamics at 1-minute resolution. This software consists of decoupled components (environment, building emulator, control agent, and RL algorithm), which allows for quick prototyping and benchmarking of standard RL algorithms in different systems; for example, a single component can be replaced without revising all of the software. To demonstrate this software framework, we conducted a benchmark study using an EnergyPlus-Modelica building energy model for a Chicago office building with an RL-based controller to dynamically control the chilled water temperature setpoint and the air handling unit supply air temperature setpoint on selected floors.

Lee, Joon-Yong↗
DOE OSTI2025

HEPTAPOD: Orchestrating High Energy Physics Workflows Towards Autonomous Agency

Many workflows in high-energy-physics (HEP) stand to benefit from recent advances in transformer-based large language models (LLMs). While early applications of LLMs focused on text generation and code completion, modern LLMs now support orchestrated agency: the coordinated execution of complex, multi-step tasks through tool use, structured context, and iterative reasoning. We introduce the HEP Toolkit for Agentic Planning, Orchestration, and Deployment (HEPTAPOD), an orchestration framework designed to bring this emerging paradigm to HEP pipelines. The framework enables LLMs to interface with domain-specific tools, construct and manage simulation workflows, and assist in common utility and data analysis tasks through schema-validated operations and run-card-driven configuration. To demonstrate these capabilities, we consider a representative Beyond the Standard Model (BSM) Monte Carlo validation pipeline that spans model generation, event simulation, and downstream analysis within a unified, reproducible workflow. HEPTAPOD provides a structured and auditable layer between human researchers, LLMs, and computational infrastructure, establishing a foundation for transparent, human-in-the-loop systems.

Menzo, Tony [Alabama U.; Fermilab] (ORCID:00000002↗
DOE OSTI2022

Agent-Based Modeling for the Circular Economy: Lessons Learned From Three Case Studies

The circular economy (CE) aims at decoupling human activities from resource use, creating wealth in the process. Recently, many scholars have questioned the link between increased circularity and sustainability, resulting in many methodological approaches being developed for that purpose. This presentation summarizes the insights gained from the application of agent-based modeling (ABM) to study the techno-economic and social conditions promoting circularity and sustainability of three technologies: photovoltaic (PV) modules, hard disk drives (HDDs), and wind blades. Four main categories of agents are defined in the ABM: asset owners, service providers (e.g., refurbishers), recyclers, and manufacturers. Two main CE strategies are represented: lifetime extension (through repair or reuse) and recycling. The developed models start by projecting installed capacities and end-of-life (EOL) quantities. Then the theory of planned behavior - a social psychology model explaining behavior adoption based on attitude, peer influence, and costs - is used to model the asset owners' EOL decision (i.e., landfill, recycle or extend the lifetime of the asset). Then, the quantities of assets flowing to the recycler and service provider agents and quantities of materials flowing to manufacturers are computed. Recyclers' economies of scale are dynamically modeled, and the value generated by the CE strategies for the recyclers, service providers, and manufacturers is computed within the model. When data are available, avoided greenhouse gas emissions resulting from the CE strategies adoption are calculated exogenously from the ABM simulations. Results show that with improved used PV modules warranties, the reuse CE strategy adoption increases from 1% to 23% between 2020 and 2050. Similarly, improved standards could enhance HDDs end-users trust in data-wiping - a prerequisite to reuse - leading to a 3-fold increase in the reuse rate and avoid about 5 million tons of CO2 eq by 2050. Regarding wind blades, 5-15 years lifetime extension could reduce EOL blade quantities by 13%. High costs and logistic issues prevent blades from being recycled in greater quantities. One insight from the case studies is the necessity to have mature secondary markets for reuse to be a viable option. Interestingly, PV reuse is limited by the willingness of PV owners to purchase used modules (on the demand side), while HDDs reuse is constrained by the lack of trust toward data-wiping (limiting the supply of used HDDs). The six limits of the CE concepts presented by Korhonen et al. (2018) are finally used to interpret the results. The HDD case study is an exemplary lock-in, where the first accepted practice (shredding) retains most of the market. The PV results illustrate the technical limitation to reuse, as the growing demand cannot be supplied entirely with used PV modules. The wind case study shows the relevance of clearly defining physical flows - what type of waste should wind blades be considered, how should they be transported and landfilled? - a crucial consideration that also applies to PV modules. Finally, the three case studies highlight the relevance of studying a technology's technical, economic, and market material efficiency potentials altogether and the potential benefit of coupling ABM to life cycle assessment.

agent-based modeling↗
DOE OSTI2021

Agent-Based Distributed Energy Resources for Supporting Intelligence at the Grid Edge

This article proposes a novel multi-agent framework that can link various forms of resources and power electronic systems into distributed energy resources (DER). The proposed multiagent architecture can also integrate DERs to a central controller for optimization and control to support the grid. To demonstrate the flexibility of this novel framework, the developed agent system is applied to a set of end-use systems. Furthermore, the agent framework is validated in hardware using controller-hardware-in-the-loop simulation platform.

24 POWER TRANSMISSION AND DISTRIBUTION↗
DOE OSTI2020

The interplay between chromophore and protein determines the extended excited state dynamics in a single-domain phytochrome

Phytochromes are a diverse family of bilin-binding photoreceptors that regulate a wide range of physiological processes. Their photochemical properties make them attractive for applications in optogenetics and superresolution microscopy. Phytochromes undergo reversible photoconversion triggered by the Z ⇌ E photoisomerization about the double bond in the bilin chromophore. However, it is not fully understood at the molecular level how the protein framework facilitates the complex photoisomerization dynamics. In this work, we have studied a single-domain bilin-binding photoreceptor All2699g1 ( Nostoc sp. PCC 7120) that exhibits photoconversion between the red light-absorbing (P r ) and far red-absorbing (P fr ) states just like canonical phytochromes. We present the crystal structure and examine the photoisomerization mechanism of the P r form as well as the formation of the primary photoproduct Lumi-R using time-resolved spectroscopy and hybrid quantum mechanics/molecular mechanics simulations. We show that the unusually long excited state lifetime (broad lifetime distribution centered at ~300 picoseconds) is due to the interactions between the isomerizing pyrrole ring D and an adjacent conserved Tyr142. The decay kinetics shows a strongly distributed character which is imposed by the nonexponential protein dynamics. Our findings offer a mechanistic insight into how the quantum efficiency of the bilin photoisomerization is tuned by the protein environment, thereby providing a structural framework for engineering bilin-based optical agents for imaging and optogenetics applications.

59 BASIC BIOLOGICAL SCIENCES↗
DOE OSTI2024

Reinforcement learning for block decomposition of planar CAD models

Abstract The problem of hexahedral mesh generation of general CAD models has vexed researchers for over 3 decades and analysts often spend more than 50% of the design-analysis cycle time decomposing complex models into simpler blocks meshable by existing techniques. The decomposed blocks are required for generating good quality meshes (tilings of quadrilaterals or hexahedra) suitable for numerical simulations of physical systems governed by conservation laws. We present a novel AI-assisted method for decomposing (segmenting) planar CAD (computer-aided design) models into well shaped rectangular blocks. Even though the simple examples presented here can also be meshed using many conventional methods, we believe this work is proof-of-principle of a AI-based decomposition method that can eventually be generalized to complex 2D and 3D CAD models. Our method uses reinforcement learning to train an agent to perform a series of optimal cuts on the CAD model that result in a good quality block decomposition. We show that the agent quickly learns an effective strategy for picking the location and direction of the cuts and maximizing its rewards. This paper is the first successful demonstration of an agent autonomously learning how to perform this block decomposition task effectively, thereby holding the promise of a viable method to automate this challenging process for more complex cases.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2023

Microgrid energy scheduling under uncertain extreme weather: Adaptation from parallelized reinforcement learning agents

Microgrids are useful solutions for integrating renewable energy resources and providing seamless green electricity to minimize carbon footprint. In recent years, extreme weather events happened often worldwide and caused significant economic and societal losses. Such events bring uncertainties to the microgrid energy scheduling problems and increase the challenges of microgrid operation. Traditional optimization approaches suffer from the inaccuracy of the uncertain microgrid model and the unseen events. Existing reinforcement learning (RL) - based approaches are also hampered by the limited generalization and the increasing computational burden when stochastic formulations are required to accommodate the uncertainties. This paper proposes a new parallelized reinforcement learning (PRL) method based on the probabilistic events to handle the microgrid energy uncertainties. Specifically, several local learning agents are employed to interact with pertinent microgrid environments in a distributed manner and report outcomes to the global agent, which will optimize microgrid energy resources online during extreme events. The stochastic microgrid energy optimization problem is reformulated to include all possible scenarios with probabilities. The advantage estimate functions of learning agents are designed with a backward sweep to transfer the outcomes to the value function updating process. Two simulation studies, stochastic optimization and online testing, are performed to compare with several existing RL approaches. Results substantiate that the proposed PRL method can achieve up to 20% optimization performance improvement with 4 and 28 times less computation cost than Q-learning with experience replay and multi-agent Q-learning approaches, respectively.

24 POWER TRANSMISSION AND DISTRIBUTION↗
DOE OSTI2023

Reinforcement Learning for Load-balanced Parallel Particle Tracing

We explore an online reinforcement learning (RL) paradigm to dynamically optimize parallel particle tracing performance in distributed-memory systems. Our method combines three novel components: (1) a work donation algorithm, (2) a high-order workload estimation model, and (3) a communication cost model. First, we design an RL-based work donation algorithm. Our algorithm monitors workloads of processes and creates RL agents to donate data blocks and particles from high-workload processes to low-workload processes to minimize program execution time. The agents learn the donation strategy on the fly based on reward and cost functions designed to consider processes' workload changes and data transfer costs of donation actions. Second, we propose a workload estimation model, helping RL agents estimate the workload distribution of processes in future computations. Third, we design a communication cost model that considers both block and particle data exchange costs, helping RL agents make effective decisions with minimized communication costs. We demonstrate that our algorithm adapts to different flow behaviors in large-scale fluid dynamics, ocean, and weather simulation data. Our algorithm improves parallel particle tracing performance in terms of parallel efficiency, load balance, and costs of I/O and communication for evaluations with up to 16,384 processors.

Distributed and parallel particle tracing↗
DOE OSTI2020

Formal methods approach to the charging facility location problem for battery electric vehicles

Battery electric vehicles (BEVs) are becoming more prevalent as improvements in battery technology and energy management continue to be made. As the number of electric vehicles grows, the demand for fast-charging stations is expected to increase dramatically. Thus, building new charging station infrastructure efficiently will be key to reducing upfront costs while meeting consumer demands. In this work, we propose a method for choosing a set of charging station locations that are optimized based on a set of given common vehicle demand points. As part of this solution, we also offer a novel abstraction of the road network on which energy-efficient paths that account for charge-time delays may be found. The current algorithm chooses the optimal charging locations for a single agent which has a route objective specified using temporal logic. To demonstrate the proposed method, the running example shows how a charging station could be chosen for an electric delivery vehicle. Simulations were run on sample road networks with a given set of demand points to service and potential charging station locations to compare. The method is shown to successfully rank potential charging stations in terms of their expected average charging time cost.

29 ENERGY PLANNING, POLICY, AND ECONOMY↗