Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “MARKOV PROCESS”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 127 records · Page 7
All sourcesNASA NTRSDOE OSTI
NASA NTRS2000

Back to the Future: Consistency-Based Trajectory Tracking

Given a model of a physical process and a sequence of commands and observations received over time, the task of an autonomous controller is to determine the likely states of the process and the actions required to move the process to a desired configuration. We introduce a representation and algorithms for incrementally generating approximate belief states for a restricted but relevant class of partially observable Markov decision processes with very large state spaces. The algorithm presented incrementally generates, rather than revises, an approximate belief state at any point by abstracting and summarizing segments of the likely trajectories of the process. This enables applications to efficiently maintain a partial belief state when it remains consistent with observations and revisit past assumptions about the process' evolution when the belief state is ruled out. The system presented has been implemented and results on examples from the domain of spacecraft control are presented.

Kurien, James↗
DOE OSTI2025

Reinforcement Learning-Based Oscillation Dampening: Scaling Up Single-Agent Reinforcement Learning Algorithms to a 100-Autonomous-Vehicle Highway Field Operational Test

In this article, we explore the technical details of the reinforcement learning (RL) algorithms that were deployed in the largest field test of automated vehicles designed to smooth traffic flow in history as of 2023, uncovering the challenges and breakthroughs that come with developing RL controllers for automated vehicles. We delve into the fundamental concepts behind RL algorithms and their application in the context of self-driving cars, discussing the developmental process from simulation to deployment in detail, from designing simulators to reward function shaping. We present the results in both simulation and deployment, discussing the flow-smoothing benefits of the RL controller. From understanding the basics of Markov decision processes to exploring advanced techniques such as deep RL, our article offers a comprehensive overview and deep dive of the theoretical foundations and practical implementations driving this rapidly evolving field. We also showcase real-world case studies and alternative research projects that highlight the impact of RL controllers in revolutionizing autonomous driving. From tackling complex urban environments to dealing with unpredictable traffic scenarios, these intelligent controllers are pushing the boundaries of what automated vehicles can achieve. Furthermore, we examine the safety considerations and hardware-focused technical details surrounding deployment of RL controllers into automated vehicles. As these algorithms learn and evolve through interactions with the environment, ensuring their behavior aligns with safety standards becomes crucial. Here, we explore the methodologies and frameworks being developed to address these challenges, emphasizing the importance of building reliable control systems for automated vehicles.

32 ENERGY CONSERVATION, CONSUMPTION, AND UTILIZATI↗
NASA NTRS1986

An optimal repartitioning decision policy

A central problem to parallel processing is the determination of an effective partitioning of workload to processors. The effectiveness of any given partition is dependent on the stochastic nature of the workload. The problem of determining when and if the stochastic behavior of the workload has changed enough to warrant the calculation of a new partition is treated. The problem is modeled as a Markov decision process, and an optimal decision policy is derived. Quantification of this policy is usually intractable. A heuristic policy which performs nearly optimally is investigated empirically. The results suggest that the detection of change is the predominant issue in this problem.

Nicol, D. M.↗
NASA NTRS1996

Probabilistic Independence Networks for Hidden Markov Probability Models

In this paper we explore hidden Markov models(HMMs) and related structures within the general framework of probabilistic independence networks (PINs). The paper contains a self-contained review of the basic principles of PINs. It is shown that the well-known forward-backward (F-B) and Viterbi algorithms for HMMs are special cases of more general enference algorithms for arbitrary PINs.

random variables pattern recognition signal proces↗
NASA NTRS2003

Modeling, Analyzing, and Mitigating Dissonance Between Alerting Systems

Alerting systems are becoming pervasive in process operations, which may result in the potential for dissonance or conflict in information from different alerting systems that suggests different threat levels and/or actions to resolve hazards. Little is currently available to help in predicting or solving the dissonance problem. This thesis presents a methodology to model and analyze dissonance between alerting systems, providing both a theoretical foundation for understanding dissonance and a practical basis from which specific problems can be addressed. A state-space representation of multiple alerting system operation is generalized that can be tailored across a variety of applications. Based on the representation, two major causes of dissonance are identified: logic differences and sensor error. Additionally, several possible types of dissonance are identified. A mathematical analysis method is developed to identify the conditions for dissonance originating from logic differences. A probabilistic analysis methodology is developed to estimate the probability of dissonance originating from sensor error, and to compare the relative contribution to dissonance of sensor error against the contribution from logic differences. A hybrid model, which describes the dynamic behavior of the process with multiple alerting systems, is developed to identify dangerous dissonance space, from which the process can lead to disaster. Methodologies to avoid or mitigate dissonance are outlined. Two examples are used to demonstrate the application of the methodology. First, a conceptual In-Trail Spacing example is presented. The methodology is applied to identify the conditions for possible dissonance, to identify relative contribution of logic difference and sensor error, and to identify dangerous dissonance space. Several proposed mitigation methods are demonstrated in this example. In the second example, the methodology is applied to address the dissonance problem between two air traffic alert and avoidance systems: the existing Traffic Alert and Collision Avoidance System (TCAS) vs. the proposed Airborne Conflict Management system (ACM). Conditions on ACM resolution maneuvers are identified to avoid dynamic dissonance between TCAS and ACM. Also included in this report is an Appendix written by Lee Winder about recent and continuing work on alerting systems design. The application of Markov Decision Process (MDP) theory to complex alerting problems is discussed and illustrated with an abstract example system.

Song, Lixia↗
DOE OSTI2025

Control-Affine Schrödinger Bridge and Generalized Bohm Potential

From a stochastic control perspective, the Schrödinger bridge is a density-valued continuous curve parameterized by time that connects a given pair of initial and terminal probability densities via minimum effort controlled Brownian motion. The control-affine Schrödinger bridge extends this idea to a generic control-affine Itô diffusion, possibly with an additive state cost. Here, in this letter, we recast the necessary conditions of optimality for the control-affine Schrödinger bridge problem as a two point boundary value problem for a quantum mechanical Schrödinger PDE with complex potential. This complex-valued potential is a generalization of the real-valued Bohm potential in quantum mechanics. Our derived potential is akin to the optical potential in nuclear physics where the real part of the potential encodes elastic scattering (transmission of wave function), and the imaginary part encodes inelastic scattering (absorption of wave function). The key takeaway is that the process noise that drives the evolution of probability densities induces an absorbing medium in the evolution of wave function. These results make new connections between control theory and non-equilibrium statistical mechanics through the lens of quantum mechanics.

Markov processes↗
DOE OSTI2025

Coupling to rotational manifolds to improve gas-phase pump–probe spectroscopic models

The physical picture of gas-phase optical transitions is normally presented as an isolated two-level system balanced by upward and downward processes. Isolated models assume a phenomenological treatment of collisional dephasing but do not strictly account for collisional population exchange with the rotational baths. While this assumption is valid under low-intensity conditions, where excitation is rate-limiting, isolated models can deviate from Beer’s Law at sufficient pressures and monochromatic intensities when both collisional broadening and power broadening are comparable to (or greater than) lifetime broadening, which are not uncommon conditions for cavity enhanced spectroscopies in the mid-IR spectral range. Although this problem has been addressed by rate-equation models for linear absorption measurements, a general treatment for multi-level quantum mechanical models suitable for non-linear absorption measurements (two-photon/two-color/pump–probe) is lacking. Isolated models require physical parameter inputs that disagree with expected values by at least an order of magnitude. These non-physical models undermine the ability to predict non-linear signal strengths under untested conditions and thereby limit the potential to optimize the sensitivity of non-linear spectroscopies and to expand their analytical applications (e.g., new analytes and/or buffer gases, changes in cavity free-spectral-range, changes in intracavity powers or wavelengths, and accurate investigation of physical phenomena). In this study, we derive bath-coupled models for gaseous pump–probe spectroscopy by application of the quantum Lindblad equation and detailed balance. Bath-coupled models are shown to fit data consistently across variations in intensity and agree with all physically expected values.

Cavity ring-down spectroscopy↗
DOE OSTI2025

Workforce planning: a review of methodologies

Workforce planning deals with determining the number of employees and associated skills necessary to meet the future operational needs of an organization. A workforce system consists of six elements: recruitment, attrition, promotion, training, retention, and scheduling. Historically, several workforce modeling and analysis methodologies have been developed to capture these elements. This paper reviews the results of workforce and manpower models published within peer-reviewed literature between 1959 and 2021 to provide an in-depth analysis of current models. The focus of this review is on analytical, simulation, and empirical models found in literature that were collected based on a citation requirement and keyword search criteria. Results demonstrate the trends in workforce modeling research and discuss the common uses of each model type and the advantages/disadvantages related to each model. Based on the common attributes of workforce systems, the discussion focuses on the most frequently used model type for each element and the best use for each model. Lastly, recommendations are made for the development of workforce models that allow the most comprehensive view of the workforce systems of the future.

42 ENGINEERING↗
DOE OSTI2024

Federated Deep Reinforcement Learning for Decentralized VVO of BTM DERs

The future of grid control requires a hybrid approach combining centralized and decentralized methods to fully utilize the potential of smart edge devices with artificial intelligence (AI) capabilities. This paper aims to develop and evaluate a federated deep reinforcement learning (FDRL) framework for decentralized adaptive volt-var optimization (VVO) of behind-the-meter (BTM) distributed energy resources (DERs). First, this paper models a single deep reinforcement learning (DRL) agent using the Markov Decision Process (MDP) framework for decentralized adaptive VVO of BTM DERs. Two DRL algorithms, soft actor-critic (SAC) and twin-delayed deep deterministic policy gradient (TD3), are compared for their effectiveness in optimizing VVO. Results show that TD3 outperforms SAC, achieving a 71.3% improvement in mean reward. Finally, the DRL agent is deployed within the FDRL framework, using the Flower platform, to enhance learning, provide adaptive control, and ensure data privacy for BTM DERs.

Ravi, Abhijith↗
DOE OSTI2026

Safe Deep Reinforcement Learning for Robust Frequency and Voltage-Constrained Networked Microgrid Restoration

Here, this paper proposes a safe soft actor-critic reinforcement learning (RL) algorithm–based controller for networked microgrid restoration. It formulates the post black-start start as a finite-horizon constrained Markov decision process. The RL agent co-optimizes real and reactive power set-points for both grid-forming and grid-following inverters under explicit voltage and frequency constraints, while enforcing proper power sharing via the Mean Active Power Sharing Index (MPSI) and Mean Reactive Power Sharing Index (MQSI). Numerical results obtained on the IEEE 123-bus distribution system show that the proposed method achieves a mean voltage build-up time of 0.01 s without breaching the 5% sharing-violation budget under various load scenarios, considering MPSI and MQSI indices. These findings demonstrate that the proposed method yields fast and safe black-start schedules without resorting to heuristic penalties.

Selim, Alaa [Dartmouth College, Hanover, NH (Unite↗
NASA NTRS1974

Wave propagation in a random medium - A complete set of the moment equations with different wavenumbers

The propagation of waves in a random medium is studied in the 'quasi-optics' and the 'Markov random process' approximations. Under these assumptions, a Fokker-Planck equation satisfied by the characteristic functional of the random wave field is derived. A complete set of moment equations with different transverse coordinates and different wave numbers is then obtained from the Fokker-Planck equation of the characteristic functional. The application of those results to the pulse smearing of the pulsar signal and the frequency correlation function of the wave intensity in interstellar scintillation is briefly discussed.

Lee, L. C.↗
NASA NTRS1980

An application of extremal field guidance in statistical low thrust guidance analysis

The subject of this paper is the application of extremal field guidance to the covariance analysis of low thrust retargeting guidance. The general procedure for low thrust navigation analysis is briefly reviewed. The basic equations of trajectory optimization and extremal field guidance are presented with emphasis on the low thrust problem. The adopted formulation includes thruster on-off times as control variables and accounts for the effects of model parameter variations. A method of solution of the guidance equations is outlined; and the application of the solution in a covariance analysis, including a procedure for the treatment of model parameter variations as either biases or first order Gauss-Markov stochastic processes, is discussed.

Jacobson, R. A.↗