Spiking Neural Algorithms for Markov Process Random Walk.
Abstract not provided.
Severa, William Mark↗
SEARCH · Engineering Papers
Results for “MARKOV PROCESS”
Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.
Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.
Series of semi-Markov processes to model infrastructure resilience under multihazards
Civil infrastructure systems are subjected to multiple hazards, including natural and anthropogenic, that disrupt their function or the level of service offered. Estimating the function recovery of these systems (or how soon normalcy of operations will be restored) when subjected to repeated hazard events by considering the inter-event dependencies is an important problem in multihazard infrastructure resilience. However, this problem has been less addressed in the field. This paper proposes a series of semi-Markov processes model to capture the inter-event dependencies in infrastructure recovery when subjected to successive hazard events. Recovery after each new hazard event is represented by a unique semi-Markov process that models the reduced recovery rates and the increased recovery times caused by the system’s incomplete recovery from the preceding event. Two novel formulations of the inter-event dependency modeling, namely Maximal Effects Dependency (considers the worst impact of two successive hazard events) and Cumulative Effects Dependency (considers the aggregated impacts of two successive hazard events), are proposed and discussed. The model is demonstrated by considering the following applications: Three-state system subjected to deterministic and random occurrences of identical hazard events; and Multihazard resilience of a building in Charleston, SC, considering earthquake and hurricane hazards. Results indicate that considering inter-event dependencies in recovery modeling can lead to lesser-predicted resilience, thereby affecting resilience-based decision-making.
42 ENGINEERING↗
Markov and semi-Markov processes as a failure rate
In this paper the reliability function is defined by the stochastic failure rate process with a non negative and right continuous trajectories. Equations for the conditional reliability functions of an object, under assumption that the failure rate is a semi-Markov process with an at most countable state space are derived. A proper theorem is presented. The linear systems of equations for the appropriate Laplace transforms allow to find the reliability functions for the alternating, the Poisson and the Furry-Yule failure rate processes.
97 MATHEMATICS AND COMPUTING↗
Markov Process to Evaluate the Value Proposition of a Risk-Informed Predictive Maintenance Strategy
To achieve high-capacity factors, the nuclear fleet has relied on labor-intensive and time-consuming operation and preventive maintenance programs for plant systems. Manually performed inspection, calibration, testing, and maintenance of plant assets at periodic frequencies, along with time-based replacement of assets irrespective of condition, have resulted in a costly, labor-centric business model. Fortunately, there are technologies that can eliminate unnecessary preventive maintenance activities by deploying risk-informed predictive maintenance, enabling the transition to a technology-centric business model. The technology-centric business model will enable plants to optimize and automate maintenance activities, leading to cost reductions since labor is a rising cost and technology is a declining cost. The implementation of scalable technologies and methodologies across plant systems and across the nuclear fleet is critical for successful deployment of a risk-informed predictive maintenance strategy at commercial nuclear power plants. The work presented in the report is being developed as part of a collaborative research effort between Idaho National Laboratory and Public Service Enterprise Group Nuclear, LLC. This report describes the technical basis using the Markov Process to evaluate the value proposition for the risk-informed predictive maintenance strategy for the circulating water system. The plant process data from the Salem nuclear plant’s circulating water system is utilized to develop a Markov chain risk models and formulation to estimate the loss and gain in revenue based on plant availability. The outcomes presented in this report provides the technical basis for extensive quantitative evaluation of a scalable risk-informed predictive maintenance strategy as part of future research.
46 INSTRUMENTATION RELATED TO NUCLEAR SCIENCE AND ↗
Efficient analysis of stochastic gene dynamics in the non-adiabatic regime using piecewise deterministic Markov processes
Single-cell experiments show that gene expression is stochastic and bursty, a feature that can emerge from slow switching between promoter states with different activities. In addition to slow chromatin and/or DNA looping dynamics, one source of long-lived promoter states is the slow binding and unbinding kinetics of transcription factors to promoters, i.e. the non-adiabatic binding regime. Here, we introduce a simple analytical framework, known as a piecewise deterministic Markov process (PDMP), that accurately describes the stochastic dynamics of gene expression in the non-adiabatic regime. We illustrate the utility of the PDMP on a non-trivial dynamical system by analysing the properties of a titration-based oscillator in the non-adiabatic limit. We first show how to transform the underlying chemical master equation into a PDMP where the slow transitions between promoter states are stochastic, but whose rates depend upon the faster deterministic dynamics of the transcription factors regulated by these promoters. We show that the PDMP accurately describes the observed periods of stochastic cycles in activator and repressor-based titration oscillators. We then generalize our PDMP analysis to more complicated versions of titration-based oscillators to explain how multiple binding sites lengthen the period and improve coherence. Finally, we show how noise-induced oscillation previously observed in a titration-based oscillator arises from non-adiabatic and discrete binding events at the promoter site.
59 BASIC BIOLOGICAL SCIENCES↗
K-Spin Hamiltonian for Quantum-Resolvable Markov Decision Processes
The Markov decision process is the mathematical formalization underlying the modern field of reinforcement learning when transition and reward functions are unknown. We derive a pseudo-Boolean cost function that is equivalent to a K-spin Hamiltonian representation of the discrete, finite, discounted Markov decision process with infinite horizon. This K-spin Hamiltonian furnishes a starting point from which to solve for an optimal policy using heuristic quantum algorithms such as adiabatic quantum annealing and the quantum approximate optimization algorithm on near-term quantum hardware. In arguing that the variational minimization of our Hamiltonian is approximately equivalent to the Bellman optimality condition for a prevalent class of environments we establish an interesting analogy with classical field theory. Along with proof-of-concept calculations to corroborate our formulation by simulated and quantum annealing against classical Q-Learning, we analyze the scaling of physical resources required to solve our Hamiltonian on quantum hardware.
Hamiltonian↗
Analysis of surface and seismic sources in dense array data with match field processing and Markov chain Monte Carlo sampling
Not provided.
Geochemistry & Geophysics↗
Spiking Markov Reward Process v.0.1
SAND2024-11150O The Spiking Markov Reward Process software is a spiking neural network that streams binary arithmetic and computes the state value function of a Markov reward process. The software will be released to the SpiNNcloud group for development of neuromorphic acceleration. Sandia National Laboratories is a multimission laboratory managed and operated by National Technology & Engineering Solutions of Sandia, LLC, a wholly owned subsidiary of Honeywell International Inc., for the U.S. Department of Energy’s National Nuclear Security Administration under contract DE-NA0003525.
Wang, Felix↗
Conditions for the Solvability of the Linear Programming Formulation for Constrained Discounted Markov Decision Processes
We consider a discrete-time constrained discounted Markov decision process (MDP) with Borel state and action spaces, compact action sets, and lower semi-continuous cost functions. We introduce a set of hypotheses related to a positive weight function which allow us to consider cost functions that might not be bounded below by a constant, and which imply the solvability of the linear programming formulation of the constrained MDP. In particular, we establish the existence of a constrained optimal stationary policy. Our results are illustrated with an application to a fishery management problem.
97 MATHEMATICS AND COMPUTING↗
On Integral of a Semi-Markov Diffusion Process
Let (X(t)) (t ≥ 0) be a semi-Markov diffusion process. The process (J(T )) (T ≥ 0) equal to the integral of (X(t)) on interval [0, T ) is studied. The relation between one-dimensional differential equation of the second order of elliptical type and asymptotics of a solution to Dirichlet problem on an interval with length tending to zero is established. This relation is used to derive a differential equation for the Laplace transform of the semi-Markov generating function of the process (J(t)).
97 MATHEMATICS AND COMPUTING↗
Impulsive Control for Continuous-Time Markov Decision Processes: A Linear Programming Approach
In this paper, we investigate an optimization problem for continuous-time Markov decision processes with both impulsive and continuous controls. We consider the so-called constrained problem where the objective of the controller is to minimize a total expected discounted optimality criterion associated with a cost rate function while keeping other performance criteria of the same form, but associated with different cost rate functions, below some given bounds. Our model allows multiple impulses at the same time moment. The main objective of this work is to study the associated linear program defined on a space of measures including the occupation measures of the controlled process and to provide sufficient conditions to ensure the existence of an optimal control.
97 MATHEMATICS AND COMPUTING↗
Random field optimization
Herein we present a new modeling paradigm for optimization that we call random field optimization. Random fields are a powerful modeling abstraction that aims to capture the behavior of random variables that live on infinite-dimensional spaces (e.g., space and time) such as stochastic processes (e.g., time series, Gaussian processes, and Markov processes), random matrices, and random spatial fields. This paradigm involves sophisticated mathematical objects (e.g., stochastic differential equations and space-time kernel functions) and has been widely used in neuroscience, geoscience, physics, civil engineering, and computer graphics. Despite of this, however, random fields have seen limited use in optimization; specifically, existing optimization paradigms that involve uncertainty (e.g., stochastic programming and robust optimization) mostly focus on the use of finite random variables. This trend is rapidly changing with the advent of statistical optimization (e.g., Bayesian optimization) and multi-scale optimization (e.g., integration of molecular sciences and process engineering). Our work extends a recently-proposed abstraction for infinite-dimensional optimization problems by capturing more general uncertainty representations. Moreover, we discuss solution paradigms for this new class of problems based on finite transformations and sampling, and identify open questions and challenges.
97 MATHEMATICS AND COMPUTING↗
Distributionally Robust Partially Observable Markov Decision Process with Moment-Based Ambiguity
In this paper, we consider a distributionally robust partially observable Markov decision process (DR-POMDP), where the distribution of the transition-observation probabilities is unknown at the beginning of each decision period, but their realizations can be inferred using side information at the end of each period after an action being taken. We build an ambiguity set of the joint distribution using bounded moments via conic constraints and seek an optimal policy to maximize the worst-case (minimum) reward for any distribution in the set. We show that the value function of DR-POMDP is piecewise linear convex with respect to the belief state and propose a heuristic search value iteration method for obtaining lower and upper bounds of the value function. We conduct numerical studies and demonstrate the computational performance of our approach via testing instances of a dynamic epidemic control problem. Our results show that DR-POMDP can produce more robust policies under misspecified distributions of transition-observation probabilities as compared to POMDP but has less costly solutions than robust POMDP. The DR-POMDP policies are also insensitive to varying parameter in the ambiguity set and to noise added to the true transition-observation probability values obtained at the end of each decision period.
97 MATHEMATICS AND COMPUTING↗
Decision-making based on Markov decision process in integrated artificial reasoning framework—Part I: Theory
This paper presents a decision-making framework based on an integrated artificial reasoning framework and Markov decision process (MDP). The integrated artificial reasoning framework provides a physics-based approach that converts system information into state transition models, and the analysis result will be represented by the transition probabilities that can be used with an MDP to find a traceable and explainable optimal pathway. A dynamic Bayesian network (DBN) is well suited for representing the structure of an MDP. The causality information among process variables (or among subsystems) is mathematically represented in a DBN by the conditional probabilities of the node’s states provided different probabilities of the parent node’s states. To define node states in a physically understandable manner, we used multilevel flow modeling (MFM). An MFM follows the fundamental energy and mass conservation laws and supports the selection of process variables that represent the system of interest so that causal relations among process variables are properly captured. An MFM-based DBN supports developing state transition models in an MDP to capture the effect of process variables of system having physical relations. The operators of the target system can capture stochastic system dynamics as multiple subsystem state transitions based on their physical relations and uncertainties coming from component degradation or random failures. We analyzed a simplified exemplary system to illustrate an optimal operational policy using the suggested approach.
Markov decision process↗
Quantum logic gate synthesis as a Markov decision process
Reinforcement learning has witnessed recent applications to a variety of tasks in quantum programming. The underlying assumption is that those tasks could be modeled as Markov decision processes (MDPs). Here, we investigate the feasibility of this assumption by exploring its consequences for single-qubit quantum state preparation and gate compilation. By forming discrete MDPs, we solve for the optimal policy exactly through policy iteration. We find optimal paths that correspond to the shortest possible sequence of gates to prepare a state or compile a gate, up to some target accuracy. Our method works in both the absence and presence of noise and compares favorably to other quantum compilation methods, such as the Ross–Selinger algorithm. This work provides theoretical insight into why reinforcement learning may be successfully used to find optimally short gate sequences in quantum programming.
71 CLASSICAL AND QUANTUM MECHANICS, GENERAL PHYSIC↗
Markov Decision Processes for Intelligent, Risk-Informed Asset-Management Decision-Making
Advanced nuclear reactors are a promising option for aiding the world in achieving its net-zero carbon emission goals, however, there are significant challenges to attaining and maintaining economic competitiveness with other sources of electricity. To improve the economic competitiveness of advanced reactor designs, a project was initiated to explore the use of Markov Decision Processes (MDPs) to guide asset-management decision-making during advanced reactor operation. MDPs are a powerful tool for optimizing decision-making in complex environments and their application to advanced reactors can aid in planning maintenance and repair activities to minimize downtime and maximize generation. The described approach expands on previous work regarding the use of MDPs for operational decision-making through the direct incorporation of real-time plant information. The integral MDP analysis includes information from online component diagnostic tools and the plant’s real-time generation risk assessment (GRA) and probabilistic risk assessment (PRA), which evaluate plant risk from both an economic and safety perspective. The result is an asset-management optimization framework that is based on real-time data regarding plant component status and the current best-estimate of plant risk. The paper presents an overview of the theoretical framework to incorporate the different information pathways into an integral MDP analysis, along with example analyses.
Grabaskas, David↗
Arbitrarily accurate, nonparametric coarse graining with Markov renewal processes and the Mori–Zwanzig formulation
Stochastic dynamics, such as molecular dynamics, are important in many scientific applications. However, summarizing and analyzing the results of such simulations is often challenging due to the high dimension in which simulations are carried out and, consequently, due to the very large amount of data that are typically generated. Coarse graining is a popular technique for addressing this problem by providing compact and expressive representations. Coarse graining, however, potentially comes at the cost of accuracy, as dynamical information is, in general, lost when projecting the problem in a lower-dimensional space. This article shows how to eliminate coarse-graining error using two key ideas. First, we represent coarse-grained dynamics as a Markov renewal process. Second, we outline a data-driven, non-parametric Mori–Zwanzig approach for computing jump times of the renewal process. Numerical tests on a small protein illustrate the method.
36 MATERIALS SCIENCE↗
Numerical Simulation of Polynomial-Speed Convergence Phenomenon
We provide a hybrid method that captures the polynomial speed of convergence and polynomial speed of mixing for Markov processes. The hybrid method that we introduce is based on the coupling technique and renewal theory. We propose to replace some estimates in classical results about the ergodicity of Markov processes by numerical simulations when the corresponding analytical proof is difficult. After that, all remaining conclusions can be derived from rigorous analysis. Then we apply our results to seek numerical justification for the ergodicity of two 1D microscopic heat conduction models. The mixing rate of these two models are expected to be polynomial but very difficult to prove. In both examples, our numerical results match the expected polynomial mixing rate well.
71 CLASSICAL AND QUANTUM MECHANICS, GENERAL PHYSIC↗