Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “Markov processes”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 19 records
All sourcesNASA NTRSDOE OSTI
DOE OSTI2020

Series of semi-Markov processes to model infrastructure resilience under multihazards

Civil infrastructure systems are subjected to multiple hazards, including natural and anthropogenic, that disrupt their function or the level of service offered. Estimating the function recovery of these systems (or how soon normalcy of operations will be restored) when subjected to repeated hazard events by considering the inter-event dependencies is an important problem in multihazard infrastructure resilience. However, this problem has been less addressed in the field. This paper proposes a series of semi-Markov processes model to capture the inter-event dependencies in infrastructure recovery when subjected to successive hazard events. Recovery after each new hazard event is represented by a unique semi-Markov process that models the reduced recovery rates and the increased recovery times caused by the system’s incomplete recovery from the preceding event. Two novel formulations of the inter-event dependency modeling, namely Maximal Effects Dependency (considers the worst impact of two successive hazard events) and Cumulative Effects Dependency (considers the aggregated impacts of two successive hazard events), are proposed and discussed. The model is demonstrated by considering the following applications: Three-state system subjected to deterministic and random occurrences of identical hazard events; and Multihazard resilience of a building in Charleston, SC, considering earthquake and hurricane hazards. Results indicate that considering inter-event dependencies in recovery modeling can lead to lesser-predicted resilience, thereby affecting resilience-based decision-making.

42 ENGINEERING↗
NASA NTRS2005

Monte Carlo Simulation of Markov, Semi-Markov, and Generalized Semi- Markov Processes in Probabilistic Risk Assessment

A standard tool of reliability analysis used at NASA-JSC is the event tree. An event tree is simply a probability tree, with the probabilities determining the next step through the tree specified at each node. The nodal probabilities are determined by a reliability study of the physical system at work for a particular node. The reliability study performed at a node is typically referred to as a fault tree analysis, with the potential of a fault tree existing.for each node on the event tree. When examining an event tree it is obvious why the event tree/fault tree approach has been adopted. Typical event trees are quite complex in nature, and the event tree/fault tree approach provides a systematic and organized approach to reliability analysis. The purpose of this study was two fold. Firstly, we wanted to explore the possibility that a semi-Markov process can create dependencies between sojourn times (the times it takes to transition from one state to the next) that can decrease the uncertainty when estimating time to failures. Using a generalized semi-Markov model, we studied a four element reliability model and were able to demonstrate such sojourn time dependencies. Secondly, we wanted to study the use of semi-Markov processes to introduce a time variable into the event tree diagrams that are commonly developed in PRA (Probabilistic Risk Assessment) analyses. Event tree end states which change with time are more representative of failure scenarios than are the usual static probability-derived end states.

English, Thomas↗
DOE OSTI2020

Markov Process to Evaluate the Value Proposition of a Risk-Informed Predictive Maintenance Strategy

To achieve high-capacity factors, the nuclear fleet has relied on labor-intensive and time-consuming operation and preventive maintenance programs for plant systems. Manually performed inspection, calibration, testing, and maintenance of plant assets at periodic frequencies, along with time-based replacement of assets irrespective of condition, have resulted in a costly, labor-centric business model. Fortunately, there are technologies that can eliminate unnecessary preventive maintenance activities by deploying risk-informed predictive maintenance, enabling the transition to a technology-centric business model. The technology-centric business model will enable plants to optimize and automate maintenance activities, leading to cost reductions since labor is a rising cost and technology is a declining cost. The implementation of scalable technologies and methodologies across plant systems and across the nuclear fleet is critical for successful deployment of a risk-informed predictive maintenance strategy at commercial nuclear power plants. The work presented in the report is being developed as part of a collaborative research effort between Idaho National Laboratory and Public Service Enterprise Group Nuclear, LLC. This report describes the technical basis using the Markov Process to evaluate the value proposition for the risk-informed predictive maintenance strategy for the circulating water system. The plant process data from the Salem nuclear plant’s circulating water system is utilized to develop a Markov chain risk models and formulation to estimate the loss and gain in revenue based on plant availability. The outcomes presented in this report provides the technical basis for extensive quantitative evaluation of a scalable risk-informed predictive maintenance strategy as part of future research.

46 INSTRUMENTATION RELATED TO NUCLEAR SCIENCE AND ↗
NASA NTRS2008

A Stable Clock Error Model Using Coupled First and Second Order Gauss-Markov Processes

Long data outages may occur in applications of global navigation satellite system technology to orbit determination for missions that spend significant fractions of their orbits above the navigation satellite constellation(s). Current clock error models based on the random walk idealization may not be suitable in these circumstances, since the covariance of the clock errors may become large enough to overflow flight computer arithmetic. A model that is stable, but which approximates the existing models over short time horizons is desirable. A coupled first- and second-order Gauss-Markov process is such a model.

Carpenter, Russell↗
NASA NTRS2006

Towards a Theory of Sampled-Data Piecewise-Deterministic Markov Processes

The analysis and design of practical control systems requires that stochastic models be employed. Analysis and design tools have been developed, for example, for Markovian jump linear continuous and discrete-time systems, piecewise-deterministic processes (PDP's), and general stochastic hybrid systems (GSHS's). These model classes have been used in many applications, including fault tolerant control and networked control systems. This paper presents initial results on the analysis of a sampled-data PDP representation of a nonlinear sampled-data system with a jump linear controller. In particular, it is shown that the state of the sampled-data PDP satisfies the strong Markov property. In addition, a relation between the invariant measures of a sampled-data system driven by a stochastic process and its associated discrete-time representation are presented. As an application, when the plant is linear with no external input, a sufficient testable condition for the convergence in distribution to the invariant delta Dirac measure is given.

Herencia-Zapana, Heber↗
DOE OSTI2020

K-Spin Hamiltonian for Quantum-Resolvable Markov Decision Processes

The Markov decision process is the mathematical formalization underlying the modern field of reinforcement learning when transition and reward functions are unknown. We derive a pseudo-Boolean cost function that is equivalent to a K-spin Hamiltonian representation of the discrete, finite, discounted Markov decision process with infinite horizon. This K-spin Hamiltonian furnishes a starting point from which to solve for an optimal policy using heuristic quantum algorithms such as adiabatic quantum annealing and the quantum approximate optimization algorithm on near-term quantum hardware. In arguing that the variational minimization of our Hamiltonian is approximately equivalent to the Bellman optimality condition for a prevalent class of environments we establish an interesting analogy with classical field theory. Along with proof-of-concept calculations to corroborate our formulation by simulated and quantum annealing against classical Q-Learning, we analyze the scaling of physical resources required to solve our Hamiltonian on quantum hardware.

Hamiltonian↗
NASA NTRS2003

Hybrid Discrete-Continuous Markov Decision Processes

This paper proposes a Markov decision process (MDP) model that features both discrete and continuous state variables. We extend previous work by Boyan and Littman on the mono-dimensional time-dependent MDP to multiple dimensions. We present the principle of lazy discretization, and piecewise constant and linear approximations of the model. Having to deal with several continuous dimensions raises several new problems that require new solutions. In the (piecewise) linear case, we use techniques from partially- observable MDPs (POMDPS) to represent value functions as sets of linear functions attached to different partitions of the state space.

Feng, Zhengzhu↗
DOE OSTI2024

Spiking Markov Reward Process v.0.1

SAND2024-11150O The Spiking Markov Reward Process software is a spiking neural network that streams binary arithmetic and computes the state value function of a Markov reward process. The software will be released to the SpiNNcloud group for development of neuromorphic acceleration. Sandia National Laboratories is a multimission laboratory managed and operated by National Technology & Engineering Solutions of Sandia, LLC, a wholly owned subsidiary of Honeywell International Inc., for the U.S. Department of Energy’s National Nuclear Security Administration under contract DE-NA0003525.

Wang, Felix↗
NASA NTRS1990

Performability analysis using semi-Markov reward processes

Beaudry (1978) proposed a simple method of computing the distribution of performability in a Markov reward process. Two extensions of Beaudry's approach are presented. The method is generalized to a semi-Markov reward process by removing the restriction requiring the association of zero reward to absorbing states only. The algorithm proceeds by replacing zero-reward nonabsorbing states by a probabilistic switch; it is therefore related to the elimination of vanishing states from the reachability graph of a generalized stochastic Petri net and to the elimination of fast transient states in a decomposition approach to stiff Markov chains. The use of the approach is illustrated with three applications.

Ciardo, Gianfranco↗
NASA NTRSDate not supplied

Markov Decision Process based Trajectory Planning for UAVs under Uncertain Wind Conditions

In this paper we propose a Markov Decision Process (MDP) algorithm for path-planning of Unmanned Aviation Vehicles (UAVs) under varying wind conditions. Solutions to path-planning for UAVs are becoming increasingly necessary as autonomous UAVs continue to enter commercial and government spaces. Path-planning is inherently challenging, as UAVs needs to account for dynamically changing flying conditions such as weather, obstacle or no-fly zones, degraded vehicle health and off-nominal battery power consumption. Machine learning methods such as Markov Decision Process (MDPs) have the potential to revolutionize how vehicles navigate in such uncertain environments. Previous papers have demonstrated the use of MDPs to optimize UAV path-planning for energy consumption under time-varying wind distribution. In this study, UAV trajectories from a pre-determined waypoint to target cell, will be computed on a 7X7 grid environment by optimizing parameters for mission assurance and safety limits in addition to the energy consumption, and operation time. The UAV navigates the grid by taking actions to move in either of the eight cardinal and intercardinal directions, under constant thrust profile. The next state of the UAV is calculated by considering its action, transition probability, obstacle cells and the wind speed magnitude and direction. Both constant and stochastic wind will be considered in this paper, the parameters being extracted from real wind measurements in proximity to an experimental UAV flight. One of the studies to be demonstrated in this paper is that as the unmanned airspace gets more complex with multiple vehicles and environmental uncertainties, trade-offs between energy consumption, operation time, risk tolerance, and mission assurance needs to be made. Further, MDPs are capable of fast computation of UAV trajectories under varying wind, hence making them suitable for in-flight path planners.

decision-making↗
DOE OSTI2022

Random field optimization

Herein we present a new modeling paradigm for optimization that we call random field optimization. Random fields are a powerful modeling abstraction that aims to capture the behavior of random variables that live on infinite-dimensional spaces (e.g., space and time) such as stochastic processes (e.g., time series, Gaussian processes, and Markov processes), random matrices, and random spatial fields. This paradigm involves sophisticated mathematical objects (e.g., stochastic differential equations and space-time kernel functions) and has been widely used in neuroscience, geoscience, physics, civil engineering, and computer graphics. Despite of this, however, random fields have seen limited use in optimization; specifically, existing optimization paradigms that involve uncertainty (e.g., stochastic programming and robust optimization) mostly focus on the use of finite random variables. This trend is rapidly changing with the advent of statistical optimization (e.g., Bayesian optimization) and multi-scale optimization (e.g., integration of molecular sciences and process engineering). Our work extends a recently-proposed abstraction for infinite-dimensional optimization problems by capturing more general uncertainty representations. Moreover, we discuss solution paradigms for this new class of problems based on finite transformations and sampling, and identify open questions and challenges.

97 MATHEMATICS AND COMPUTING↗