Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “Markov models”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 289 records · Page 16
All sourcesNASA NTRSDOE OSTI
DOE OSTI2026

Performing Numerical Analysis of Cybersecurity Options Using Dynamic Risk Analysis Tool EMRALD

Cyberattacks can have many different attack paths, durations, and goals. There are also many different mitigation options involving hardware, software, and/or humans. Considering a cyber threat should involve defense-in-depth methods and a quantitative or numerical evaluation of overall effectiveness against dynamic, time-dependent attacks to make cost and risk-informed decisions. Typical cyberattack modeling methods only provide a qualitative evaluation. The main areas of cybersecurity are confidentiality, integrity, and availability. For companies with cyber-physical systems such as advanced nuclear reactors, cyber-related safety is a requirement set by North American Electric Reliability and the U.S. Nuclear Regulatory Commission. They are also concerned about availability or reliability as a business case. As cyber threats are evolving to a business-for-hire structure, more attacks may focus on disrupting business success and reliability, causing financial and economic stability risk. Companies want to know business reliability and recovery from those threats, and that requires modeling physical behavior of the targets. Dynamic-state-based and Markov-based modeling provides a method for better cyber scenario modeling with different tools having issues such as state-base explosion. Dynamic modeling enables time and conditional features not found in other numerical evaluation methods. EMRALD (Event Modeling Risk Assessment using Lined Diagrams) is a dynamic risk analysis modeling and simulation tool and has features that reduce modeling issues. It has been used to model different time-dependent events including plant behavior and operator procedures. As a general modeling tool, EMRALD can also be used to model cyberattack scenarios with varying mitigation options and quantify effectiveness, producing numerical data for risk-informed decisions. This paper uses EMRALD to demonstrate that dynamic numerical risk analysis can be used for cyber threat modeling to provide insights for design decision-making and optimize defense strategies. Keywords: cyber modeling; cyber-physical systems; numerical cyber modeling

97 - MATHEMATICS AND COMPUTING↗
DOE OSTI2021

Attention Enabled Multi-Agent DRL for Decentralized Volt-VAR Control of Active Distribution System Using PV Inverters and SVCs

This paper proposes attention enabled multi-agent deep reinforcement learning (MADRL) framework for active distribution network decentralized Volt-VAR control. Using the unsupervised clustering, the whole distribution system can be decomposed into several sub-networks according to the voltage and reactive power sensitivity relationships. Then, the distributed control problem of each sub-network is modeled as Markov games and solved by the improved MADRL algorithm, where each sub-network is modeled as an adaptive agent. An attention mechanism is developed to help each agent focus on specific information that is mostly related to the reward. All agents are centrally trained offline to learn the optimal coordinated Volt-VAR control strategy and executed in a decentralized manner to make online decisions with only local information. Compared with other distributed control approaches, the proposed method can effectively deal with uncertainties, achieve fast decision makings, and significantly reduce the communication requirements. Comparison results with model-based and other data-driven methods on IEEE 33-bus and 123-bus systems demonstrate the benefits of the proposed approach.

distribution network↗
DOE OSTI2020

Probabilistic Power Consumption Modeling for Commercial Buildings Using Logistic Regression Markov Chain

The total energy consumed by buildings takes up to 40% of U.S. energy use, in which a large portion is contributed by commercial buildings. Building performance optimization is desirable but requires accurate building models with uncertainties taken into account. This paper proposes a novel probabilistic modeling method using Logistic Regression Markov Chain (LRMC). The LRMC model enhances the performance of traditional Markov Chain (MC) models by adopting time-variant transition matrices calibrated using logistic regression with exogenous inputs. Compared with existing building models, the proposed model produces accurate multi-step modeling results with full probability distribution. The proposed probabilistic building model is tested using actual commercial building measurements and modeling performance is evaluated with two probabilisitc metrics. The results show that the LRMC model has higher accuracy than traditional MC model and Logistic Regression (LR) model in that it yields lower error scores under both evaluation metrics.

Building modeling↗
DOE OSTI2023

Quantum logic gate synthesis as a Markov decision process

Reinforcement learning has witnessed recent applications to a variety of tasks in quantum programming. The underlying assumption is that those tasks could be modeled as Markov decision processes (MDPs). Here, we investigate the feasibility of this assumption by exploring its consequences for single-qubit quantum state preparation and gate compilation. By forming discrete MDPs, we solve for the optimal policy exactly through policy iteration. We find optimal paths that correspond to the shortest possible sequence of gates to prepare a state or compile a gate, up to some target accuracy. Our method works in both the absence and presence of noise and compares favorably to other quantum compilation methods, such as the Ross–Selinger algorithm. This work provides theoretical insight into why reinforcement learning may be successfully used to find optimally short gate sequences in quantum programming.

71 CLASSICAL AND QUANTUM MECHANICS, GENERAL PHYSIC↗
NASA NTRS1977

In-flight gyro drift rate calibration on the Viking Orbiters

The drift rates of the attitude control gyros onboard the Viking Orbiters were calibrated several times during flight. The calibration was performed by engaging the gyro control of attitude for a six-hour period, recording the Viking Orbiter coordinates of the sun and the roll reference star as a function of time, and processing these data by a computer program. The computer program calculated drift rates for each data increment, and then smoothed the calculated rate versus the time function for all the data on the basis of a Gauss-Markov gyro-drift model. The effects of data noise were minimized by using a relatively long time constant in the Gauss-Markov model and by empirically determining a set of corrections for celestial inertial scale factor error differences. In this way the rms error of the data fit was lessened.

Breckenridge, W. G.↗
NASA NTRS1994

Advantages of estimating rate corrections during dynamic propagation of spacecraft rates: Applications to real-time attitude determination of SAMPEX

This paper describes real-time attitude determination results for the Solar, Anomalous, and Magnetospheric Particle Explorer (SAMPEX), a gyroless spacecraft, using a Kalman filter/Euler equation approach denoted the real-time sequential filter (RTSF). The RTSF is an extended Kalman filter whose state vector includes the attitude quaternion and corrections to the rates, which are modeled as Markov processes with small time constants. The rate corrections impart a significant robustness to the RTSF against errors in modeling the environmental and control torques, as well as errors in the initial attitude and rates, while maintaining a small state vector. SAMPLEX flight data from various mission phases are used to demonstrate the robustness of the RTSF against a priori attitude and rate errors of up to 90 deg and 0.5 deg/sec, respectively, as well as a sensitivity of 0.0003 deg/sec in estimating rate corrections in torque computations. In contrast, it is shown that the RTSF attitude estimates without the rate corrections can degrade rapidly. RTSF advantages over single-frame attitude determination algorithms are also demonstrated through (1) substantial improvements in attitude solutions during sun-magnetic field coalignment and (2) magnetic-field-only attitude and rate estimation during the spacecraft's sun-acquisition mode. A robust magnetometer-only attitude-and-rate determination method is also developed to provide for the contingency when both sun data as well as a priori knowledge of the spacecraft state are unavailable. This method includes a deterministic algorithm used to initialize the RTSF with coarse estimates of the spacecraft attitude and rates. The combined algorithm has been found effective, yielding accuracies of 1.5 deg in attitude and 0.01 deg/sec in the rates and convergence times as little as 400 sec.

Challa, M. S.↗
DOE OSTI2021

Approximating Nash Equilibrium in Day-ahead Electricity Market Bidding with Multi-agent Deep Reinforcement Learning

In this paper, a day-ahead electricity market bidding problem with multiple strategic generation company (GEN-CO) bidders is studied. The problem is formulated as a Markov game model, where GENCO bidders interact with each other todevelop their optimal day-ahead bidding strategies. Considering unobservable information in the problem, a model-free and data-driven approach, known as multi-agent deep deterministic policy gradient (MADDPG), is applied for approximating the Nash equilibrium (NE) in the above Markov game. The MADDPG algorithm has the advantage of generalization due to the automatic feature extraction ability of the deep neural networks. The algorithm is tested on an IEEE 30-bus system with three competitive GENCO bidders in both an uncongested caseand a congested case. Comparisons with a truthful bidding strategy and state-of-the-art deep reinforcement learning methods including deep Q network and deep deterministic policy gradient (DDPG) demonstrate that the applied MADDPG algorithm can find a superior bidding strategy for all the market participants with increased profit gains. In addition, the comparison with a conventional model-based method shows that the MADDPG algorithm has higher computational efficiency, which is feasible for real-world applications.

24 POWER TRANSMISSION AND DISTRIBUTION↗
NASA NTRS1985

The conservativeness of reliability estimates based on instantaneous coverage

Reliability modeling must take into account two different types of phenomena, including the fault-occurrence behavior and the fault/error-handling behavior of a system. The effectiveness of the fault/error-handling behavior can be captured by instantaneous coverage probabilities. This paper has the objective to show that the assumption of instantaneous coverage leads to conservative predictions of system reliability for systems characterized by relatively long interevent times for fault occurrences and relatively short interevent times for fault/error-handling actions. The importance of this result is related to the fact that it can now be shown that model predictions based on instantaneous coverage are lower bounds on the true system reliability. Attention is given to a semi-Markov reliability model, instantaneous coverage approximations, the proof of conservative prediction, and the computation of coverage probabilities.

Mcgough, J.↗
DOE OSTI2021

Unveiling the Dynamics of KRAS4b on Lipid Model Membranes

Abstract Small GTPase proteins are ubiquitous and responsible for regulating several processes related to cell growth and differentiation. Mutations that stabilize their active state can lead to uncontrolled cell proliferation and cancer. Although these proteins are well characterized at the cellular scale, the molecular mechanisms governing their functions are still poorly understood. In addition, there is limited information about the regulatory function of the cell membrane which supports their activity. Thus, we have studied the dynamics and conformations of the farnesylated KRAS4b in various membrane model systems, ranging from binary fluid mixtures to heterogeneous raft mimics. Our approach combines long time-scale coarse-grained (CG) simulations and Markov state models to dissect the membrane-supported dynamics of KRAS4b. Our simulations reveal that protein dynamics is mainly modulated by the presence of anionic lipids and to some extent by the nucleotide state (activation) of the protein. In addition, our results suggest that both the farnesyl and the polybasic hypervariable region (HVR) are responsible for its preferential partitioning within the liquid-disordered (Ld) domains in membranes, potentially enhancing the formation of membrane-driven signaling platforms. Graphic Abstract

59 BASIC BIOLOGICAL SCIENCES↗
DOE OSTI2022

Reinforcement learning for adaptive maintenance policy optimization under imperfect knowledge of the system degradation model and partial observability of system states

Maintenance policy optimization usually is faced with challenges that arise from an imperfect knowledge of system degradation models and from the partial observability of system degradation states. Here, this paper proposes a reinforcement learning method to address these two challenges for a class of maintenance problems with Markov degradation processes. The reinforcement learning approach consists of a learning component and a planning component. Using sequentially collected observations, at each step of decision-making the learning component improves the knowledge of system degradation in terms of the probability distributions of the transition rates based on sequential Bayesian inference. Using the updated transition rates, at each step of decision-making the maintenance policy optimization problem is then formulated as a partially observable Markov decision problem, and the planning component computes the optimal maintenance policy that maximizes the expected cumulative reward. The proposed method is illustrated using a numerical example with repair and inspection maintenance actions. The result shows that as more observations are collected, the learning component progressively learns the true system degradation process, and the planning component adjusts the optimal maintenance policy accordingly as well, which leads to increased reward.

42 ENGINEERING↗
NASA NTRS1987

On the next generation of reliability analysis tools

The current generation of reliability analysis tools concentrates on improving the efficiency of the description and solution of the fault-handling processes and providing a solution algorithm for the full system model. The tools have improved user efficiency in these areas to the extent that the problem of constructing the fault-occurrence model is now the major analysis bottleneck. For the next generation of reliability tools, it is proposed that techniques be developed to improve the efficiency of the fault-occurrence model generation and input. Further, the goal is to provide an environment permitting a user to provide a top-down design description of the system from which a Markov reliability model is automatically constructed. Thus, the user is relieved of the tedious and error-prone process of model construction, permitting an efficient exploration of the design space, and an independent validation of the system's operation is obtained. An additional benefit of automating the model construction process is the opportunity to reduce the specialized knowledge required. Hence, the user need only be an expert in the system he is analyzing; the expertise in reliability analysis techniques is supplied.

Babcock, Philip S., IV↗
NASA NTRS1972

A Bayesian model for visual space perception

A model for visual space perception is proposed that contains desirable features in the theories of Gibson and Brunswik. This model is a Bayesian processor of proximal stimuli which contains three important elements: an internal model of the Markov process describing the knowledge of the distal world, the a priori distribution of the state of the Markov process, and an internal model relating state to proximal stimuli. The universality of the model is discussed and it is compared with signal detection theory models. Experimental results of Kinchla are used as a special case.

Curry, R. E.↗
NASA NTRS1993

Probabilistic Anomaly Detection in Dynamic Systems

This paper describes probabilistic methods for novelty detection when using pattern recognition methods for fault monitoring of dynamic systems. The problem of novelty detection is particularly acute when prior knowledge and data only allow one to construct an incomplete prior model of the system. Hence, some allowance must be made in model design so that a classifier will be robust to data generated by classes not included in the training phase.

Propabilistic Markov Model↗
DOE OSTI2021

A parallel evolutionary multiple-try metropolis Markov chain Monte Carlo algorithm for sampling spatial partitions

We develop an Evolutionary Markov Chain Monte Carlo (EMCMC) algorithm for sampling spatial partitions that lie within a large, complex, and constrained spatial state space. Our algorithm combines the advantages of evolutionary algorithms (EAs) as optimization heuristics for state space traversal and the theoretical convergence properties of Markov Chain Monte Carlo algorithms for sampling from unknown distributions. Local optimality information that is identified via a directed search by our optimization heuristic is used to adaptively update a Markov chain in a promising direction within the framework of a Multiple-Try Metropolis Markov Chain model that incorporates a generalized Metropolis-Hastings ratio. We further expand the reach of our EMCMC algorithm by harnessing the computational power afforded by massively parallel computing architecture through the integration of a parallel EA framework that guides Markov chains running in parallel.

97 MATHEMATICS AND COMPUTING↗
NASA NTRS1988

Integrated avionics reliability

The integrated avionics reliability task is an effort to build credible reliability and/or performability models for multisensor integrated navigation and flight control. The research was initiated by the reliability analysis of a multisensor navigation system consisting of the Global Positioning System (GPS), the Long Range Navigation system (Loran C), and an inertial measurement unit (IMU). Markov reliability models were developed based on system failure rates and mission time.

Alikiotis, Dimitri↗
DOE OSTI2024

Coarse-Graining Conformational Dynamics with Multidimensional Generalized Langevin Equation: How, When, and Why

A data-driven ab initio generalized Langevin equation (AIGLE) approach is developed to learn and simulate high-dimensional, heterogeneous, coarse-grained (CG) conformational dynamics. Constrained by the fluctuation–dissipation theorem, the approach can build CG models in dynamical consistency (DC) with all-atom molecular dynamics. Here, we also propose practical criteria for AIGLE to enforce long-term DC. Case studies of a toy polymer, with 20 CG sites, and the alanine dipeptide, with two dihedral angles, elucidate why one should adopt AIGLE or its Markovian limit for modeling CG conformational dynamics in practice.

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗
NASA NTRS2002

Measuring the Resilience of Advanced Life Support Systems

Despite the central importance of crew safety in designing and operating a life support system, the metric commonly used to evaluate alternative Advanced Life Support (ALS) technologies does not currently provide explicit techniques for measuring safety. The resilience of a system, or the system s ability to meet performance requirements and recover from component-level faults, is fundamentally a dynamic property. This paper motivates the use of computer models as a tool to understand and improve system resilience throughout the design process. Extensive simulation of a hybrid computational model of a water revitalization subsystem (WRS) with probabilistic, component-level faults provides data about off-nominal behavior of the system. The data can then be used to test alternative measures of resilience as predictors of the system s ability to recover from component-level faults. A novel approach to measuring system resilience using a Markov chain model of performance data is also developed. Results emphasize that resilience depends on the complex interaction of faults, controls, and system dynamics, rather than on simple fault probabilities.

Bell, Ann Maria↗