Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “MARL”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 19 records
All sourcesNASA NTRSDOE OSTI
NASA NTRS1995

Molecular indicators for palaeoenvironmental change in a Messinian evaporitic sequence (Vena del Gesso, Italy). II: High-resolution variations in abundances and 13C contents of free and sulphur-bound carbon skeletons in a single marl bed

The extractable organic matter of 10 immature samples from a marl bed of one evaporitic cycle of the Vena del Gesso sediments (Gessoso-solfifera Fm., Messinian, Italy) was analyzed quantitatively for free hydrocarbons and organic sulphur compounds. Nickel boride was used as a desulphurizing agent to recover sulphur-bound lipids from the polar and asphaltene fractions. Carbon isotopic compositions (delta vs PDB) of free hydrocarbons and of S-bound hydrocarbons were also measured. Relationships between these carbon skeletons, precursor biolipids, and the organisms producing them could then be examined. Concentrations of S-bound lipids and free hydrocarbons and their delta values were plotted vs depth in the marl bed and the profiles were interpreted in terms of variations in source organisms, 13 C contents of the carbon source, and environmentally induced changes in isotopic fractionation. The overall range of delta values measured was 24.7%, from -11.6% for a component derived from green sulphur bacteria (Chlorobiaceae) to -36.3% for a lipid derived from purple sulphur bacteria (Chromatiaceae). Deconvolution of mixtures of components deriving from multiple sources (green and purple sulphur bacteria, coccolithophorids, microalgae and higher plants) was sometimes possible because both quantitative and isotopic data were available and because either the free or S-bound pool sometimes appeared to contain material from a single source. Several free n-alkanes and S-bound lipids appeared to be specific products of upper-water-column primary producers (i.e. algae and cyanobacteria). Others derived from anaerobic photoautotrophs and from heterotrophic protozoa (ciliates), which apparently fed partly on Chlorobiaceae. Four groups of n-alkanes produced by algae or cyanobacteria were also recognized based on systematic variations of abundance and isotopic composition with depth. For hydrocarbons probably derived from microalgae, isotopic variations are well correlated with those of total organic carbon. A resistant aliphatic biomacromolecule produced by microalgae is, therefore, probably an important component of the kerogen. These variations reflect changes in the depositional environment and early diagenetic transformations. Changes in the concentrations of S-bound lipids induced by variations in conditions favourable for sulphurization were discriminated from those related to variations in primary producer assemblages. The water column of the lagoonal basin was stratified and photic zone anoxia occurred during the early and middle stages of marl deposition. During the last stage of the marl deposition the stratification collapsed due to a significant shallowing of the water column. Contributions from anaerobic photoautotrophs were apparently associated with variations in depth of the chemocline.

Non-NASA Center↗
NASA NTRS1995

Climatic forcing of carbon-oxygen isotopic covariance in temperate-region marl lakes

Carbon and oxygen stable isotopic compositions of lacustrine carbonate from a southeastern Michigan marl lake display linear covariance over a range of 4.0% Peedee belemnite (PDB) in oxygen and 3.9% (PDB) in carbon. Mechanisms of delta 13 C-delta 18 O coupling conventionally attributed to lake closure in arid-region basins are inapplicable to hydrologically open lake systems. Thus, an alternative explanation of isotopic covariance in temperate region dimictic marl lakes is required. We propose that isotopic covariance is a direct record of change in regional climate. In short-residence-time temperate-region lake basins, summer meteoric precipitation is enriched in 18O relative to winter values, and summer organic productivity enriches epilimnic dissolved inorganic carbon in 13C. Thus, climate change toward longer summers and/or shorter winters could result in greater proportions of warm-month meteoric precipitation, longer durations of warm-month productivity, and net long-term enrichment in carbonate 18O and 13C. Isotopic covariance observed in the Michigan marl lake cores is interpreted to reflect postglacial warming from 10 to 3 ka followed by cooler mean annual temperature, a shift toward greater proportions of seasonal summer precipitation, a shortening of the winter season, or some combination of these three factors.

NASA Discipline Number 52-30↗
NASA NTRS1993

Molecular and bulk isotopic analyses of organic matter in marls of the Mulhouse Basin (Tertiary, Alsace, France)

Contents of 13C in kerogens and carbonates in 21 samples from a core of the MAX borehole, Mulhouse Evaporite Basin, range from -27.3 to -23.5 and -3.7 to -1.8% vs PDB, respectively. Organic nitrogen in the same samples is enriched in 15N relative to atmospheric N2 by 12.2-15.7%. Hydrogen indices and delta values for kerogens vary systematically with facies, averaging 493 mg HC/g Corg and -25.7% in the most saline facies (dominated by inputs from aquatic sources) and 267 mg HC/g Corg and -23.7% in the least saline facies (50/50 aquatic/terrigenous). Values of delta were measured for individual aliphatic hydrocarbons from three samples representing three different organic facies. For all samples, terrigenous inputs were unusually rich in 13C, the estimated delta value for bulk terrigenous debris, apparently derived partly from CAM plants, being -22.5%. In the most saline facies, isotopic evidence indicates the mixing of 13C-depleted products of photosynthetic bacteria with 13C-enriched products of halotolerant eukaryotic algae. At lower salinities, a change in the producer community is marked by a decrease in the 13C content of algal lipids. The content of 13C in algal lipids increases in the least saline facies, due either to succession of different organisms or to decreased concentrations of dissolved CO2.

NASA Discipline Exobiology↗
NASA NTRS1988

Faunal and erosional events in the Eastern Tethyan Sea across the K/T boundary

A regional pattern of three closely spaced erosional events at and above the K/T boundary was determined from six Cretaceous/Tertiary boundary sections in the Negev of Israel. The sections were collected from locations throughout the central and northern Negev. All sections are lithologically similar. The Maastrichtian consists of a sequence of limestone beds intercalated with thin marly beds. In some sections, the last limestone bed is followed by 1 to 2 m of calcareous marls grading upwards into several meters of grey shale. In other sections the limestone bed is followed directly by grey shale with the contact containing particles of limestone and marl. A 5 to 20 cm thick dark grey organic-rich clay layer is present about 1.5 to 2.5 m above the base of the grey shale. The grey shale grades upwards into increasingly carbonate rich marls. No unconformities are apparent in field outcrops. During field collection the dark grey clay layer was believed to represent the K/T boundary clay. Microfossil analysis however identified the boundary at the base of the grey shale. The black shale represents a low productivity anoxic event similar to, but younger than, the K/T boundary clay in other K/T boundary sections. High resolution planktic foraminiferal and carbonate analysis of these sections (at 5 to 10 cm intervals) yield surprising results. The K/T boundary is marked by an erosional event which removed part or all of the uppermost Maastrichtian marls above the last limestone bed. Percent carbonate data for four Negev sections are illustrated and show the regional similarities in carbonate sedimentation. Faunal and carbonate data from the Negev sections thus show three closely spaced short erosional events at the K/T boundary and within the first 50,000 to 100,000 years of the Danian. These K/T boundary erosional events may represent global climatic or paleoceanographic events.

Keller, G.↗
DOE OSTI2023

Interpreting Primal-Dual Algorithms for Constrained Multiagent Reinforcement Learning: Preprint

We study multiagent reinforcement learning (MARL) with constraints. This setting is gaining importance as MARL algorithms find new applications in real-world systems ranging from power grids to drone swarms. Most constrained MARL (C-MARL) algorithms use a primal-dual approach to enforce constraints through a penalty function added to the reward. In this paper, we study the structural effects of the primal-dual approach on the constraints and value function. First, we show that using the constraint evaluation as the penalty leads to a weak notion of safety, but by making simple modifications to the penalty function, we can enforce meaningful probabilistic safety constraints. Second, we show that the penalty term changes the value function in a way that is easy to model, and demonstrate the consequences of not doing so. We conclude with simulations in a simple constrained multiagent environment to back up the theoretical results.

data-driven control↗
DOE OSTI2023

Interpreting Primal-Dual Algorithms for Constrained Multiagent Reinforcement Learning

Constrained multiagent reinforcement learning (C-MARL) is gaining importance as MARL algorithms find new applications in real-world systems ranging from energy systems to drone swarms. Most C-MARL algorithms use a primal-dual approach to enforce constraints through a penalty function added to the reward. In this paper, we study the structural effects of this penalty term on the MARL problem. First, we show that the standard practice of using the constraint function as the penalty leads to a weak notion of safety. However, by making simple modifications to the penalty term, we can enforce meaningful probabilistic (chance and conditional value at risk) constraints. Second, we quantify the effect of the penalty term on the value function, uncovering an improved value estimation procedure. We use these insights to propose a constrained multiagent advantage actor critic (C-MAA2C) algorithm. Simulations in a simple constrained multiagent environment affirm that our reinterpretation of the primal-dual method in terms of probabilistic constraints is effective, and that our proposed value estimate accelerates convergence to a safe joint policy.

chance constraints↗
DOE OSTI2022

PowerNet: Multi-agent Deep Reinforcement Learning for Scalable Powergrid Control

This paper develops an efficient multi-agent deep reinforcement learning algorithm for cooperative controls in powergrids. Specifically, we consider the decentralized inverter-based secondary voltage control problem in distributed generators (DGs), which is first formulated as a cooperative multi-agent reinforcement learning (MARL) problem. We then propose a novel on-policy MARL algorithm, PowerNet, in which each agent (DG) learns a control policy based on (sub-)global reward but local states and encoded communication messages from its neighbors. Motivated by the fact that a local control from one agent has limited impact on agents distant from it, we exploit a novel spatial discount factor to reduce the effect from remote agents, to expedite the training process and improve scalability. Furthermore, a differentiable, learning-based communication protocol is employed to foster the collaborations among neighboring agents. In addition, to mitigate the effects of system uncertainty and random noise introduced during on-policy learning, we utilize an action smoothing factor to stabilize the policy execution. To facilitate training and evaluation, we develop PGSim, an efficient, high-fidelity powergrid simulation platform. Here, experimental results in two microgrid setups show that the developed PowerNet outperforms the conventional model-based control method, as well as several state-of-the-art MARL algorithms. The decentralized learning scheme and high sample efficiency also make it viable to large-scale power grids.

24 POWER TRANSMISSION AND DISTRIBUTION↗
DOE OSTI2025

Enhancing Autonomous Control of Microreactors Using Multi-Agent Reinforcement Learning

In order for microreactors to be economically competitive, operation costs will need to be minimized through some degree of autonomous control. Previous work has demonstrated the effectiveness of reinforcement learning (RL) for load-following control in a drum-controlled microreactor. This study extends that work by exploring the potential of RL to independently control each of the reactor’s drums. We compare a single-agent RL approach with a multi-agent RL (MARL) framework, testing them for generalization across different load-following power profiles and control timescales, and for robustness in cases of randomly disabled control drums. Since the point kinetics simulation environment used in this study cannot resolve spatial effects, we assume that in the absence of spatially localized disturbances, optimal drum movements should be symmetrical. We demonstrate that single-agent RL is able to achieve accurate performance only when symmetric actions are ignored; otherwise, it fails to train a useful controller. Meanwhile, the MARL framework performs symmetric actions by design and trains a robust, accurate agent, as evidenced by mean absolute errors in power matching of 0.41% for the training power profile, 0.68% for a profile with half the drums disabled, and 0.21% for a profile on a realistic load-following time horizon.

22 - GENERAL STUDIES OF NUCLEAR REACTORS↗
DOE OSTI2025

Nuclear microreactor transient and load-following control with deep reinforcement learning

The economic feasibility of nuclear microreactors will depend on minimizing operating costs through advancements in autonomous control, especially when these microreactors are operating alongside other types of energy systems (e.g., renewable energy). This study explores the application of deep reinforcement learning (RL) for real-time drum control in microreactors, exploring performance in regard to load-following scenarios. By leveraging a point kinetics model with thermal and xenon feedback, we first establish a baseline using a single-output RL agent, then compare it against a traditional proportional–integral–derivative (PID) controller. This study demonstrates that RL controllers, including both single- and multi-agent RL (MARL) frameworks, can achieve similar or even superior load-following performance as traditional PID control across a range of load-following scenarios. In short transients, the RL agent was able to reduce the tracking error rate in comparison to PID by one half to one third. Over extended 300-minute load-following scenarios in which xenon feedback becomes a dominant factor, PID maintained better accuracy, but RL still remained within a 1% error margin despite being trained only on short-duration scenarios. This highlights RL’s strong ability to generalize and extrapolate to longer, more complex transients, affording substantial reductions in training costs and reduced overfitting. Furthermore, when control was extended to multiple drums, MARL enabled independent drum control as well as maintained reactor symmetry constraints without sacrificing performance---an objective that standard single-agent RL could not learn. We also found that, as increasing levels of Gaussian noise were added to the power measurements, the RL controllers were able to maintain lower error rates than PID, and to do so with at least 10% and upwards of 150% less control effort. These findings illustrate RL's potential for autonomous nuclear reactor control, laying the groundwork for future integration into high-fidelity simulations and experimental validation efforts.

22 - GENERAL STUDIES OF NUCLEAR REACTORS↗
DOE OSTI2024

Non-Stationary Policy Learning for Multi-Timescale Multi-Agent Reinforcement Learning

In multi-timescale multi-agent reinforcement learning (MARL), agents interact across different timescales. In general, policies for time-dependent behaviors, such as those induced by multiple timescales, are non-stationary. Learning non-stationary policies is challenging and typically requires sophisticated or inefficient algorithms. Motivated by the prevalence of this control problem in real-world complex systems, we introduce a simple framework for learning non-stationary policies for multi-timescale MARL. Our approach uses available information about agent timescales to define and learn periodic multi-agent policies. In detail, we theoretically demonstrate that the effects of non-stationarity introduced by multiple timescales can be learned by a periodic multi-agent policy. To learn such policies, we propose a policy gradient algorithm that parameterizes the actor and critic with phase-functioned neural networks, which provide an inductive bias for periodicity. The framework's ability to effectively learn multi-timescale policies is validated on a gridworld and building energy management environment.

control↗
DOE OSTI2024

Deep Multi-Agent Reinforcement Learning for Real-World Signalized Traffic Corridor Control

Signalized traffic control problem has been addressed recently with deep Reinforcement Learning (RL) approaches involving diverse state, action, and reward structures. While significant progress has been noted in the literature, open challenges still remain in the areas of adaptive signal phase timing, coordination in a multi-intersection corridor setting, and consideration of real-world traffic conditions. In the context of deep RL-based problem framing, extensions are needed that enable adaptive signal phase timings in an intersection agent's action space, computationally efficient information sharing among neighboring signalized intersection agents along a corridor, and experimentation in realistic simulation environments. In this paper, we develop a deep Advantage Actor Critic (A2C) multi-agent RL (MARL) approach capturing the research extensions above and apply it within a real-world calibrated Aimsun Next traffic corridor simulation model based on traffic data from the City of Coral Gables, Florida. For a multi-intersection corridor control setting, our numerical simulation experiments with a decentralized A2C MARL algorithm applied at different time periods led to a total average corridor travel delay reduction (expressed in seconds/mile averaged over vehicles) from 4.9% to 19.9% compared to state-of-the-art actuated control.

Shuvo, Salman S. [BATTELLE (PACIFIC NW LAB)]↗
DOE OSTI2024

Buyers Collusion in Incentivized Forwarding Networks: A Multi-Agent Reinforcement Learning Study

We present the issue of monetarily incentivized forwarding in a multi-hop mesh network architecture from an economic perspective. It is anticipated that credit-incentivized forwarding and relaying will be a simple method of exchanging transmission power and spectrum for connectivity. However, gateways and forwarding nodes, like any other free market, may create an oligopolistic market for the users they serve. In this study, a coalition scheme between buyers aims to address price control by gateways or nodes closer to gateways. In a Stackelberg competition game, buyer agents (users) and sellers (gateways) make decisions using reinforcement learning (RL), with decentralized Deep Q-Networks to buy and sell forwarding resources. We allow communication links between the buyers with a limited messaging space, without defining a collusion mechanism. The idea is to demonstrate that through messaging, and RL tacit collusion can emerge between agents in a decentralized setup. The multi-agent reinforcement learning (MARL) system is presented and analyzed from a machine-learning perspective. Moreover, MARL dynamics are discussed via mean field analysis to better understand divergence causes and make implementation recommendations for such systems. Finally, the simulation results show the results of coordination among the users.

42 ENGINEERING↗
DOE OSTI2022

PowerGridworld: A Framework for Multi-Agent Reinforcement Learning in Power Systems: Preprint

We present the PowerGridworld software package to provide users with a light-weight, modular, and customizable framework for creating power systems-focused, multi-agent gym environments that readily integrate with existing training frameworks for reinforcement learning (RL). While many frameworks exist for training multi-agent (MA) RL policies, none exist to rapidly prototype and develop the environments themselves, especially in the context of heterogeneous (composite, multi-device) power systems where power flow solutions are required to define grid-level variables and costs. PowerGridworld is an open-source software package that helps to fill this gap. To highlight PowerGridworld's key features, we present two case studies and demonstrate learning multi-agent RL policies using both OpenAI's MADDPG and RLLib's PPO algorithms where, in both cases, at least some subset of agents incorporate elements of the power flow solution at each time step as part of their reward (negative cost) structures.

MATHEMATICS AND COMPUTING↗