Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “Markov process”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 37 records · Page 2

Projected metastable Markov processes and their estimation with observable operator models

The determination of kinetics of high-dimensional dynamical systems, such as macromolecules, polymers, or spin systems, is a difficult and generally unsolved problem — both in simulation, where the optimal reaction coordinate(s) are generally unknown and are difficult to compute, and in experimental measurements, where only specific coordinates are observable. Markov models, or Markov state models, are widely used but suffer from the fact that the dynamics on a coarsely discretized state spaced are no longer Markovian, even if the dynamics in the full phase space are. The recently proposed projected Markov models (PMMs) are a formulation that provides a description of the kinetics on a low-dimensional projection without making the Markovianity assumption. However, as yet no general way of estimating PMMs from data has been available. Here, we show that the observed dynamics of a PMM can be exactly described by an observable operator model (OOM) and derive a PMM estimator based on the OOM learning.

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗

Distribution of chirality in the quantum walk: Markov process and entanglement

The asymptotic behavior of the quantum walk on the line is investigated, focusing on the probability distribution of chirality independently of position. It is shown analytically that this distribution has a longtime limit that is stationary and depends on the initial conditions. This result is unexpected in the context of the unitary evolution of the quantum walk as it is usually linked to a Markovian process. The asymptotic value of the entanglement between the coin and the position is determined by the chirality distribution. For given asymptotic values of both the entanglement and the chirality distribution, it is possible to find the corresponding initial conditions within a particular class of spatially extended Gaussian distributions.

71 CLASSICAL AND QUANTUM MECHANICS, GENERAL PHYSIC↗

Linearization Techniques for Controlled Piecewise Deterministic Markov Processes; Application to Zubov's Method

We aim at characterizing domains of attraction for controlled piecewise deterministic processes using an occupational measure formulation and Zubov's approach. Firstly, we provide linear programming (primal and dual) formulations of discounted, infinite horizon control problems for PDMPs. These formulations involve an infinite-dimensional set of probability measures and are obtained using viscosity solutions theory. Secondly, these tools allow to construct stabilizing measures and to avoid the assumption of stability under concatenation for controls. The domain of controllability is then characterized as some level set of a convenient solution of the associated Hamilton-Jacobi integral-differential equation. The theoretical results are applied to PDMPs associated to stochastic gene networks. Explicit computations are given for Cook's model for gene expression.

97 MATHEMATICS AND COMPUTING↗

Towards a Theory of Sampled-Data Piecewise-Deterministic Markov Processes

The analysis and design of practical control systems requires that stochastic models be employed. Analysis and design tools have been developed, for example, for Markovian jump linear continuous and discrete-time systems, piecewise-deterministic processes (PDP's), and general stochastic hybrid systems (GSHS's). These model classes have been used in many applications, including fault tolerant control and networked control systems. This paper presents initial results on the analysis of a sampled-data PDP representation of a nonlinear sampled-data system with a jump linear controller. In particular, it is shown that the state of the sampled-data PDP satisfies the strong Markov property. In addition, a relation between the invariant measures of a sampled-data system driven by a stochastic process and its associated discrete-time representation are presented. As an application, when the plant is linear with no external input, a sufficient testable condition for the convergence in distribution to the invariant delta Dirac measure is given.

Herencia-Zapana, Heber↗

K-Spin Hamiltonian for Quantum-Resolvable Markov Decision Processes

The Markov decision process is the mathematical formalization underlying the modern field of reinforcement learning when transition and reward functions are unknown. We derive a pseudo-Boolean cost function that is equivalent to a K-spin Hamiltonian representation of the discrete, finite, discounted Markov decision process with infinite horizon. This K-spin Hamiltonian furnishes a starting point from which to solve for an optimal policy using heuristic quantum algorithms such as adiabatic quantum annealing and the quantum approximate optimization algorithm on near-term quantum hardware. In arguing that the variational minimization of our Hamiltonian is approximately equivalent to the Bellman optimality condition for a prevalent class of environments we establish an interesting analogy with classical field theory. Along with proof-of-concept calculations to corroborate our formulation by simulated and quantum annealing against classical Q-Learning, we analyze the scaling of physical resources required to solve our Hamiltonian on quantum hardware.

Hamiltonian↗

The exit-time problem for a Markov jump process

The purpose of this paper is to consider the exit-time problem for a finite-range Markov jump process, i.e, the distance the particle can jump is bounded independent of its location. Additionally, such jump diffusions are expedient models for anomalous transport exhibiting super-diffusion or nonstandard normal diffusion. We refer to the associated deterministic equation as a volume-constrained nonlocal diffusion equation. The volume constraint is the nonlocal analogue of a boundary condition necessary to demonstrate that the nonlocal diffusion equation is well-posed and is consistent with the jump process. A critical aspect of the analysis is a variational formulation and a recently developed nonlocal vector calculus. Finally, this calculus allows us to pose nonlocal backward and forward Kolmogorov equations, the former equation granting the various moments of the exit-time distribution.

71 CLASSICAL AND QUANTUM MECHANICS, GENERAL PHYSIC↗

Markov Jump Processes Approximating a Non-Symmetric Generalized Diffusion

Consider a non-symmetric generalized diffusion X( Dot-Operator ) in Double-Struck-Capital-R {sup d} determined by the differential operator A(x) = -{Sigma}{sub ij} {partial_derivative}{sub i}a{sub ij}(x){partial_derivative}{sub j} + {Sigma}{sub i} b{sub i}(x){partial_derivative}{sub i}. In this paper the diffusion process is approximated by Markov jump processes X{sub n}( Dot-Operator ), in homogeneous and isotropic grids G{sub n} Subset-Of Double-Struck-Capital-R {sup d}, which converge in distribution in the Skorokhod space D([0,{infinity}), Double-Struck-Capital-R {sup d}) to the diffusion X( Dot-Operator ). The generators of X{sub n}( Dot-Operator ) are constructed explicitly. Due to the homogeneity and isotropy of grids, the proposed method for d{>=}3 can be applied to processes for which the diffusion tensor {l_brace}a{sub ij}(x){r_brace}{sub 11}{sup dd} fulfills an additional condition. The proposed construction offers a simple method for simulation of sample paths of non-symmetric generalized diffusion. Simulations are carried out in terms of jump processes X{sub n}( Dot-Operator ). For piece-wise constant functions a{sub ij} on Double-Struck-Capital-R {sup d} and piece-wise continuous functions a{sub ij} on Double-Struck-Capital-R {sup 2} the construction and principal algorithm are described enabling an easy implementation into a computer code.

97 MATHEMATICS AND COMPUTING↗

Hybrid Discrete-Continuous Markov Decision Processes

This paper proposes a Markov decision process (MDP) model that features both discrete and continuous state variables. We extend previous work by Boyan and Littman on the mono-dimensional time-dependent MDP to multiple dimensions. We present the principle of lazy discretization, and piecewise constant and linear approximations of the model. Having to deal with several continuous dimensions raises several new problems that require new solutions. In the (piecewise) linear case, we use techniques from partially- observable MDPs (POMDPS) to represent value functions as sets of linear functions attached to different partitions of the state space.

Feng, Zhengzhu↗

Spiking Markov Reward Process v.0.1

SAND2024-11150O The Spiking Markov Reward Process software is a spiking neural network that streams binary arithmetic and computes the state value function of a Markov reward process. The software will be released to the SpiNNcloud group for development of neuromorphic acceleration. Sandia National Laboratories is a multimission laboratory managed and operated by National Technology & Engineering Solutions of Sandia, LLC, a wholly owned subsidiary of Honeywell International Inc., for the U.S. Department of Energy’s National Nuclear Security Administration under contract DE-NA0003525.

Wang, Felix↗

'PSA-SPN' - A Parameter Sensitivity Analysis Method Using Stochastic Petri Nets: Application to a Production Line System

The dynamic behavior of a discrete event dynamic system can be significantly affected for some uncertain changes in its decision parameters. So, parameter sensitivity analysis would be a useful way in studying the effects of these changes on the system performance. In the past, the sensitivity analysis approaches are frequently based on simulation models. In recent years, formal methods based on stochastic process including Markov process are proposed in the literature. In this paper, we are interested in the parameter sensitivity analysis of discrete event dynamic systems by using stochastic Petri nets models as a tool for modelling and performance evaluation. A sensitivity analysis approach based on stochastic Petri nets, called PSA-SPN method, will be proposed with an application to a production line system.

71 CLASSICAL AND QUANTUM MECHANICS, GENERAL PHYSIC↗

Conditions for the Solvability of the Linear Programming Formulation for Constrained Discounted Markov Decision Processes

We consider a discrete-time constrained discounted Markov decision process (MDP) with Borel state and action spaces, compact action sets, and lower semi-continuous cost functions. We introduce a set of hypotheses related to a positive weight function which allow us to consider cost functions that might not be bounded below by a constant, and which imply the solvability of the linear programming formulation of the constrained MDP. In particular, we establish the existence of a constrained optimal stationary policy. Our results are illustrated with an application to a fishery management problem.

97 MATHEMATICS AND COMPUTING↗

A fast exact simulation method for a class of Markov jump processes

A new method of the stochastic simulation algorithm (SSA), named the Hashing-Leaping method (HLM), for exact simulations of a class of Markov jump processes, is presented in this paper. The HLM has a conditional constant computational cost per event, which is independent of the number of exponential clocks in the Markov process. The main idea of the HLM is to repeatedly implement a hash-table-like bucket sort algorithm for all times of occurrence covered by a time step with length τ. This paper serves as an introduction to this new SSA method. We introduce the method, demonstrate its implementation, analyze its properties, and compare its performance with three other commonly used SSA methods in four examples. Our performance tests and CPU operation statistics show certain advantages of the HLM for large scale problems.

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗

On Integral of a Semi-Markov Diffusion Process

Let (X(t)) (t ≥ 0) be a semi-Markov diffusion process. The process (J(T )) (T ≥ 0) equal to the integral of (X(t)) on interval [0, T ) is studied. The relation between one-dimensional differential equation of the second order of elliptical type and asymptotics of a solution to Dirichlet problem on an interval with length tending to zero is established. This relation is used to derive a differential equation for the Laplace transform of the semi-Markov generating function of the process (J(t)).

97 MATHEMATICS AND COMPUTING↗

On spatial mutation-selection models

We discuss the selection procedure in the framework of mutation models. We study the regulation for stochastically developing systems based on a transformation of the initial Markov process which includes a cost functional. The transformation of initial Markov process by cost functional has an analytic realization in terms of a Kimura-Maruyama type equation for the time evolution of states or in terms of the corresponding Feynman-Kac formula on the path space. The state evolution of the system including the limiting behavior is studied for two types of mutation-selection models.

71 CLASSICAL AND QUANTUM MECHANICS, GENERAL PHYSIC↗