Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “Markov process”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 829 records · Page 46

Matter-Wave Emission in Optical Lattices: Single Particle and Collective Effects

We introduce a simple setup corresponding to the matter-wave analogue of impurity atoms embedded in a photonic crystal and interacting with the radiation field. Atoms in a given internal level are trapped in an optical lattice, and play the role of the impurities. Atoms in an untrapped level play the role of the radiation field. The interaction is mediated by means of lasers that couple those levels. By tuning the lasers parameters, it is possible to drive the system through different regimes, and observe phenomena such as matter-wave superradiance, non-Markovian atom emission, and the appearance of bound atomic states.

71 CLASSICAL AND QUANTUM MECHANICS, GENERAL PHYSIC↗

Prediction of Spatiotemporal Patterns of Neural Activity from Pairwise Correlations

We designed a model-based analysis to predict the occurrence of population patterns in distributed spiking activity. Using a maximum entropy principle with a Markovian assumption, we obtain a model that accounts for both spatial and temporal pairwise correlations among neurons. This model is tested on data generated with a Glauber spin-glass system and is shown to correctly predict the occurrence probabilities of spatiotemporal patterns significantly better than Ising models only based on spatial correlations. This increase of predictability was also observed on experimental data recorded in parietal cortex during slow-wave sleep. This approach can also be used to generate surrogates that reproduce the spatial and temporal correlations of a given data set.

71 CLASSICAL AND QUANTUM MECHANICS, GENERAL PHYSIC↗

Solid-State Spin-Photon Quantum Interface without Spin-Orbit Coupling

We show that coherent optical manipulation of a single confined spin is possible even in the absence of spin-orbit coupling. To this end, we consider the non-Markovian dynamics of a single valence orbital hole spin that has optically induced spin-exchange coupling to a low-temperature partially polarized electron gas. We show that the fermionic nature of the reservoir induces a coherent component to the hole spin dynamics that does not generate entanglement with the reservoir modes. We analyze in detail the competition of this reservoir-assisted coherent contribution with dissipative components displaying markedly different behavior at different time scales and determine the fidelity of optically controlled spin rotations.

71 CLASSICAL AND QUANTUM MECHANICS, GENERAL PHYSIC↗

Kinetic Theory for Distribution Functions of Wave-Particle Interactions in Plasmas

The evolution of a charged particle distribution function under the influence of coherent electromagnetic waves in a plasma is determined from kinetic theory. For coherent waves, the dynamical phase space of particles is an inhomogeneous mix of chaotic and regular orbits. The persistence of long time correlations between the particle motion and the phase of the waves invalidates any simplifying Markovian or statistical assumptions--the basis for usual quasilinear theories. The generalized formalism in this Letter leads to a hierarchy of evolution equations for the reduced distribution function. The evolution operators, in contrast to the quasilinear theories, are time dependent and nonsingular and include the rich phase space dynamics of particles interacting with coherent waves.

70 PLASMA PHYSICS AND FUSION TECHNOLOGY↗

Autonomous Cyber Defense Against Dynamic Multi-strategy Infrastructural DDoS Attacks

Dynamic Infrastructural Distributed Denial of Service (I-DDoS) attacks constantly change attack vectors to congest core backhaul links and disrupt critical network availability while evading end-system defenses. To effectively counter these highly dynamic attacks, defense mechanisms need to exhibit adaptive decision strategies for real-time mitigation. This paper presents a novel Autonomous DDoS Defense framework that employs model-based reinforcement agents. The framework continuously learns attack strategies, predicts attack actions, and dynamically determines the optimal composition of defense tactics such as filtering, limiting, and rerouting for flow diversion. Our contributions include extending the underlying formulation of the Markov Decision Process (MDP) to address simultaneous DDoS attack and defense behavior, and accounting for environmental uncertainties. We also propose a fine-grained action mitigation approach robust to classification inaccuracies in Intrusion Detection Systems (IDS). Additionally, our reinforcement learning model demonstrates resilience against evasion and deceptive attacks. Evaluation experiments using real-world and simulated DDoS traces demonstrate that our autonomous defense framework ensures the delivery of approximately 96 - 98% of benign traffic despite the diverse range of attack strategies.

Dutta, Ashutosh↗

An Online Approach to Solve the Dynamic Vehicle Routing Problem with Stochastic Trip Requests for Paratransit Services

Many transit agencies operating paratransit and microtransit services have to respond to trip requests that arrive in real-time, which entails solving hard combinatorial and sequential decision-making problems under uncertainty. To avoid decisions that lead to significant inefficiency in the long term, vehicles should be allocated to requests by optimizing a non-myopic utility function or by batching requests together and optimizing a myopic utility function. While the former approach is typically offline, the latter can be performed online. We point out two major issues with such approaches when applied to paratransit services in practice. First, it is difficult to batch paratransit requests together as they are temporally sparse. Second, the environment in which transit agencies operate changes dynamically (e.g., traffic conditions can change over time), causing the estimates that are learned offline to become stale. To address these challenges, we propose a fully online approach to solve the dynamic vehicle routing problem (DVRP) with time windows and stochastic trip requests that is robust to changing environmental dynamics by construction. We focus on scenarios where requests are relatively sparse—our problem is motivated by applications to paratransit services. We formulate DVRP as a Markov decision process and use Monte Carlo tree search to evaluate actions for any given state. Accounting for stochastic requests while optimizing a non-myopic utility function is computationally challenging; indeed, the action space for such a problem is intractably large in practice. To tackle the large action space, we leverage the structure of the problem to design heuristics that can sample promising actions for the tree search. Our experiments using real-world data from our partner agency show that the proposed approach outperforms existing state-of-the-art approaches both in terms of performance and robustness.

32 ENERGY CONSERVATION, CONSUMPTION, AND UTILIZATI↗

Constrained Deep Reinforcement Learning for Energy Sustainable Multi-UAV Based Random Access IoT Networks With NOMA

In this paper, we apply the Non-Orthogonal Multiple Access (NOMA) technique to improve the massive channel access of a wireless IoT network where solar-powered Unmanned Aerial Vehicles (UAVs) relay data from IoT devices to remote servers. Specifically, IoT devices contend for accessing the shared wireless channel using an adaptive p-persistent slotted Aloha protocol; and the solar-powered UAVs adopt Successive Interference Cancellation (SIC) to decode multiple received data from IoT devices to improve access efficiency. To enable an energy-sustainable capacity-optimal network, we study the joint problem of dynamic multi-UAV altitude control and multi-cell wireless channel access management of IoT devices as a stochastic control problem with multiple energy constraints. We first formulate this problem as a Constrained Markov Decision Process (CMDP), and propose an online model-free Constrained Deep Reinforcement Learning (CDRL) algorithm based on Lagrangian primal-dual policy optimization to solve the CMDP. Extensive simulations demonstrate that our proposed algorithm learns a cooperative policy in which the altitude of UAVs and channel access probability of IoT devices are dynamically controlled to attain the maximal long-term network capacity while ensuring energy sustainability of UAVs, outperforming baseline schemes. The proposed CDRL agent can be trained on a small network, yet the learned policy can efficiently manage networks with a massive number of IoT devices and varying initial states, which can amortize the cost of training the CDRL agent.

42 ENGINEERING↗

A General Framework for Bounding Approximate Dynamic Programming Schemes

For years, there has been interest in approximation methods for solving dynamic programming problems, because of the inherent complexity in computing optimal solutions characterized by Bellman’s principle of optimality. A wide range of approximate dynamic programming (ADP) methods now exists. It is of great interest to guarantee that the performance of an ADP scheme be at least some known fraction, say ß , of optimal. This letter introduces a general approach to bounding the performance of ADP methods, in this sense, in the stochastic setting. The approach is based on new results for bounding greedy solutions in string optimization problems, where one has to choose a string (ordered set) of actions to maximize an objective function. This bounding technique is inspired by submodularity theory, but submodularity is not required for establishing bounds. Instead, the bounding is based on quantifying certain notions of curvature of string functions; the smaller the curvatures the better the bound. The key insight is that any ADP scheme is a greedy scheme for some surrogate string objective function that coincides in its optimal solution and value with those of the original optimal control problem. The ADP scheme then yields to the bounding technique mentioned above, and the curvatures of the surrogate objective determine the value ß of the bound. The surrogate objective and its curvatures depend on the specific ADP.

discrete event systems↗

Federated Deep Reinforcement Learning for Decentralized VVO of BTM DERs

The future of grid control requires a hybrid approach combining centralized and decentralized methods to fully utilize the potential of smart edge devices with artificial intelligence (AI) capabilities. This paper aims to develop and evaluate a federated deep reinforcement learning (FDRL) framework for decentralized adaptive volt-var optimization (VVO) of behind-the-meter (BTM) distributed energy resources (DERs). First, this paper models a single deep reinforcement learning (DRL) agent using the Markov Decision Process (MDP) framework for decentralized adaptive VVO of BTM DERs. Two DRL algorithms, soft actor-critic (SAC) and twin-delayed deep deterministic policy gradient (TD3), are compared for their effectiveness in optimizing VVO. Results show that TD3 outperforms SAC, achieving a 71.3% improvement in mean reward. Finally, the DRL agent is deployed within the FDRL framework, using the Flower platform, to enhance learning, provide adaptive control, and ensure data privacy for BTM DERs.

Ravi, Abhijith↗

Releasing Correlated Trajectories: Towards High Utility and Optimal Differential Privacy

A mutual correlation between trajectories of two users is very helpful to real-life applications such as product recommendation and social media. While providing tremendous benefits, the releasing of correlated trajectories may leak sensitive social relations, due to potential links between mutual correlations and social relations. To the best of our knowledge, here we take the first step to propose a mathematically rigorous n-body Laplace framework, satisfying epsilon-differential privacy, which efficiently prevents a social relation inference through the mutual correlation between n-node trajectories of two users. The problem is mathematically formulated by defining a trajectory correlation score to measure the social relation between two users. Then, under the n-body Laplace framework, we propose two Lagrange Multiplier-based Differentially Private (LMDP) approaches to optimize the privacy budgets, for the data utility measured by location distances and the data utility measured by location correlations, i.e., UD-LMDP and UC-LMDP. Also, we present detailed analyses of privacy, data utility, adversary knowledge and the constrained optimizations. Finally, we perform experimental studies with real-life data. Our experimental results show that our proposed approaches achieve better privacy and data utility than the existing approaches.

97 MATHEMATICS AND COMPUTING↗

Agent-Centric Approach for Cybersecurity Decision-Support with Partial Observability

Generating automated cyber resilience policies for real-world settings is a challenging research problem that must account for uncertainties in system state over time and dynamics between attackers and defenders. In addition to understanding attacker and defender motives and tools, and identifying “relevant” system and attack data, it is also critical to develop rigorous mathematical formulations representing the defender’s decision-support problem under uncertainty. Game-theoretic approaches involving cyber resource allocation optimization with Markov decision processes (MDP) have been previously proposed in the literature. Moreover, advancements in reinforcement learning approaches have motivated the development of partially observable stochastic games (POSGs) in various multi-agent problem domains with partial information. Recent advances in cyber-system state space modeling have also generated interest in potential applicability of POSGs for cybersecurity. However, as is the case in strategic card games such as poker, research challenges using game-theoretic approaches for practical cyber defense applications include: 1) solving for equilibrium and designing efficient algorithms for large-scale, general problems; 2) establishing mathematical guarantees that equilibrium exists; 3) handling possible existence of multiple equilibria; and 4) exploitation of opponent weaknesses. Inspired by advances in solving strategic card games while acknowledging practical challenges associated with the use of game-theoretic approaches in cyber settings, this paper proposes an agent-centric approach for cybersecurity decision-support with partial system state observability.

Tipireddy, Ramakrishna↗

Safe Deep Reinforcement Learning for Robust Frequency and Voltage-Constrained Networked Microgrid Restoration

Here, this paper proposes a safe soft actor-critic reinforcement learning (RL) algorithm–based controller for networked microgrid restoration. It formulates the post black-start start as a finite-horizon constrained Markov decision process. The RL agent co-optimizes real and reactive power set-points for both grid-forming and grid-following inverters under explicit voltage and frequency constraints, while enforcing proper power sharing via the Mean Active Power Sharing Index (MPSI) and Mean Reactive Power Sharing Index (MQSI). Numerical results obtained on the IEEE 123-bus distribution system show that the proposed method achieves a mean voltage build-up time of 0.01 s without breaching the 5% sharing-violation budget under various load scenarios, considering MPSI and MQSI indices. These findings demonstrate that the proposed method yields fast and safe black-start schedules without resorting to heuristic penalties.

Selim, Alaa [Dartmouth College, Hanover, NH (Unite↗

Application of Markov Chain Monte Carlo Methods for Uncertainty Quantification in Inverse Transport Problems

Determination of the components of a radioactive source/shield system using the system’s radiation signature is of great importance in homeland security, material safeguards, and waste management. Although significant progress has been made toward solving this inverse transport problem in recent years, work remains to be done to quantify the uncertainty in reconstructed results. In this article we apply two Markov chain Monte Carlo (MCMC) approaches, the delayed rejection adaptive metropolis (DRAM) and differential evolution adaptive metropolis (DREAM) methods, to solve inverse problems and quantify uncertainty. The DRAM method uses delayed rejection combined with global adaptation of the proposal covariance matrix. Furthermore, the DREAM method hybridizes MCMC sampling with the differential evolution (DE) algorithm. In numerical test cases, the DRAM and DREAM methods are shown to be superior to a first-order inverse Hessian approach for problems with noisy data and multiple unknown quantities, with DREAM converging to the posterior distribution more quickly than DRAM. The DREAM and DRAM results indicate that a full posterior distribution is required to quantify uncertainty in many inverse transport problems.

98 NUCLEAR DISARMAMENT, SAFEGUARDS, AND PHYSICAL P↗

Distributional Deep Reinforcement Learning-Based Emergency Frequency Control

Emergency frequency control is one of the most critical approaches to maintain power system stability after major disturbances. With the increasing number of grid-connected renewable energy sources, existing model-based methods of frequency control are facing up with challenges of computational speed and scalability for large-scale systems. In this paper, the emergency frequency control problem is formulated as a Markov Decision Process (MDP) and solved through a novel Distributional Deep Reinforcement Learning (DDRL) method, namely the distributional soft actor critic (DSAC) method. Compared with other RL methods that only estimate the mean value, the proposed DSAC model estimates the distribution of value function over returns. This advancement can lead to more insights and knowledge for the agent, with the benefit of a much faster and more stable learning process, and the improved frequency control performance. Here, the simulation results on IEEE 39-bus and IEEE 118-bus systems demonstrate the effectiveness and robustness of proposed models, as well as the advantage compared to other state-of-the-art DRL algorithms.

24 POWER TRANSMISSION AND DISTRIBUTION↗

Optimal Load Ensemble Control in Chance-Constrained Optimal Power Flow

Distribution system operators (DSOs) world-wide foresee a rapid roll-out of distributed energy resources. From the system perspective, their reliable and cost effective integration requires accounting for their physical properties in operating tools used by the DSO. This work describes an decomposable approach to leverage the dispatch flexibility of thermostatically controlled loads (TCLs) for operating distribution systems with a high penetration level of photovoltaic resources. Each TCL ensemble is modeled using the Markov Decision Process (MDP). The MDP model is then integrated with a chance constrained optimal power flow that accounts for the uncertainty of PV resources. Since the integrated optimization model cannot be solved efficiently by existing dynamic programming methods or off-the-shelf solvers, this paper proposes an iterative Spatio-Temporal Dual Decomposition algorithm (ST-D2). Finally, we demonstrate the merits of the proposed integrated optimization and ST-D2 algorithm on the IEEE 33-bus test system.

24 POWER TRANSMISSION AND DISTRIBUTION↗

Automated Control of Transactive HVACs in Energy Distribution Systems

Heating, Ventilation, and Air Conditioning (HVAC) systems contribute significantly to a building’s energy consumption. In the recent years, there is an increased interest in developing transactive approaches which could enable automated and flexible scheduling of HVAC systems based on the customer demand and the electricity prices decided by the suppliers. Flexible and automated scheduling of the HVAC systems make it a prime source for participation in residential demand response or transactive energy systems. Therefore, it is of significant interest to identify an optimal strategy to control the HVAC systems. Here, reducing the energy cost while keeping the comfort level acceptable to the users, we argue that such a control strategy should consider both the energy cost and user comfort simultaneously. Accordingly, we develop the control strategy through the solution of an optimization problem that balances between the energy cost and consumer’s dissatisfaction. This optimization enables us to solve a decision-making problem through first price prediction and then choosing HVAC temperature settings throughout the day based on the predicted price, history of the price and HVAC settings, and outside temperature. More specifically, we formulate the control design as a Markov decision process (MDP) using deep neural networks and use Deep Deterministic Policy Gradients (DDPG)-based deep reinforcement learning algorithm to find the optimal control strategy for HVAC systems that balances between electricity cost and user comfort.

32 ENERGY CONSERVATION, CONSUMPTION, AND UTILIZATI↗

Deep Reinforcement Learning based Model-free On-line Dynamic Multi-Microgrid Formation to Enhance Resilience

Multi-microgrid formation (MMGF) is a promising solution for enhancing power system resilience. This paper proposes a new deep reinforcement learning (RL) based model-free on-line dynamic MMGF scheme. Additionally, the dynamic MMGF problem is formulated as a Markov decision process, and a complete deep RL framework is specially designed for the topologytransformable micro-grids. In order to reduce the large action space caused by flexible switch operations, a topology transformation method is proposed and an action-decoupling Q-value is applied. Then, a convolutional neural network (CNN) based multi-buffer double deep Q-network (CM-DDQN) is developed to further improve the learning ability of the original DQN method. The proposed deep RL method provides real-time computing to support the on-line dynamic MMGF scheme, and the scheme handles a long-term resilience enhancement problem using an adaptive on-line MMGF to defend changeable conditions. The effectiveness of the proposed method is validated using a 7-bus system and the IEEE 123-bus system. The results show strong learning ability, timely response for varying system conditions and convincing resilience enhancement.

24 POWER TRANSMISSION AND DISTRIBUTION↗