Engineering PapersSearch

SEARCH · Engineering Papers

Results for “MARKOV PROCESS”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 163 records · Page 9

Probabilistic Path Planning of Montgolfier Balloons in Strong, Uncertain Wind Fields

Lighter-than-air vehicles such as hot-air balloons have been proposed for exploring Saturn s moon Titan, as well as other bodies with significant atmospheres. For these vehicles to navigate effectively, it is critical to incorporate the effects of surrounding wind fields, especially as these winds will likely be strong relative to the control authority of the vehicle. Predictive models of these wind fields are available, and previous research has considered problems of planning paths subject to these predicted forces. However, such previous work has considered the wind fields as known a priori, whereas in practical applications, the actual wind vector field is not known exactly and may deviate significantly from the wind velocities estimated by the model. A probabilistic 3D path-planning algorithm was developed for balloons to use uncertain wind models to generate time-efficient paths. The nominal goal of the algorithm is to determine what altitude and what horizontal actuation, if any is available on the vehicle, to use to reach a particular goal location in the least expected time, utilizing advantageous winds. The solution also enables one to quickly evaluate the expected time-to-goal from any other location and to avoid regions of large uncertainty. This method is designed for balloons in wind fields but may be generalized for any buoyant vehicle operating in a vector field. To prepare the planning problem, the uncertainty in the wind field is modeled. Then, the problem of reaching a particular goal location is formulated as a Markov decision process (MDP) using a discretized space approach. Solving the MDP provides a policy of what actuation option (how much buoyancy change and, if applicable, horizontal actuation) should be selected at any given location to minimize the expected time-to-goal. The results provide expected time-to-goal values from any given location on the globe in addition to the action policy. This stochastic approach can also provide insights not accessible by deterministic methods; for example, one can evaluate variability and risk associated with different scenarios, rather than only viewing the expected outcome.

Wolf, Michael

Exact and Approximate Probabilistic Symbolic Execution

Probabilistic software analysis seeks to quantify the likelihood of reaching a target event under uncertain environments. Recent approaches compute probabilities of execution paths using symbolic execution, but do not support nondeterminism. Nondeterminism arises naturally when no suitable probabilistic model can capture a program behavior, e.g., for multithreading or distributed systems. In this work, we propose a technique, based on symbolic execution, to synthesize schedulers that resolve nondeterminism to maximize the probability of reaching a target event. To scale to large systems, we also introduce approximate algorithms to search for good schedulers, speeding up established random sampling and reinforcement learning results through the quantification of path probabilities based on symbolic execution. We implemented the techniques in Symbolic PathFinder and evaluated them on nondeterministic Java programs. We show that our algorithms significantly improve upon a state-of- the-art statistical model checking algorithm, originally developed for Markov Decision Processes.

Symbolic Execution

Probabilistic Motion Planning of Balloons in Strong, Uncertain Wind Fields

This paper introduces a new algorithm for probabilistic motion planning in arbitrary, uncertain vector fields, with emphasis on high-level planning for Montgolfiere balloons in the atmosphere of Titan. The goal of the algorithm is to determine what altitude--and what horizontal actuation, if any is available on the vehicle--to use to reach a goal location in the fastest expected time. The winds can vary greatly at different altitudes and are strong relative to any feasible horizontal actuation, so the incorporation of the winds is critical for guidance plans. This paper focuses on how to integrate the uncertainty of the wind field into the wind model and how to reach a goal location through the uncertain wind field, using a Markov decision process (MDP). The resulting probabilistic solutions enable more robust guidance plans and more thorough analysis of potential paths than existing methods.

Wolf, Michael T.

Dynamic Routing of Aircraft in the Presence of Adverse Weather Using a POMDP Framework

Each year weather-related airline delays result in hundreds of millions of dollars in additional fuel burn, maintenance, and lost revenue, not to mention passenger inconvenience. The current approaches for aircraft route planning in the presence of adverse weather still mainly rely on deterministic methods. In contrast, this work aims to deal with the problem using a Partially Observable Markov Decision Processes (POMDPs) framework, which allows for reasoning over uncertainty (including uncertainty in weather evolution over time) and results in solutions that are more robust to disruptions. The POMDP-based decision support system is demonstrated on several scenarios involving convective weather cells and is benchmarked against a deterministic planning system with functionality similar to those currently in use or under development.

Decision making

Adaptive Stress Testing: Using Reinforcement Learning to Find Failures in Safety-Critical Systems

Emerging applications in artificial intelligence, such as driverless cars and autonomous aircraft promise to be more efficient, cheaper to operate, and always available. However, ensuring the safety of these systems remains a major challenge to their certification and adoption. These autonomous systems are expected to routinely make safety-critical decisions where failures can have serious consequences including loss of life and property. Testing and validation techniques aim to identify and diagnose potential failures before the system is deployed. However, finding failure scenarios in autonomous systems can be very challenging due to high-dimensional and continuous state spaces, interaction with large environments over many time steps, and the rarity of failures. This talk presents Adaptive Stress Testing (AST), a simulation-based testing framework for finding the most likely path to a failure event of a safety-critical system. The key idea of AST is that stress testing can be formulated as a Partially Observable Markov Decision Process (POMDP), which enables reinforcement learning techniques to be used for finding failure events. Reinforcement learning algorithms can efficiently explore the search space and have been shown to scale to very large systems. We present applications of AST to find failures in various safety-critical systems including the aircraft collision avoidance systems, autonomous cars, and small unmanned aerial vehicles.

autonomous vehicles

Adaptive Stress Testing: Finding Likely Failure Events with Reinforcement Learning

Finding the most likely path to a set of failure states is important to the analysis of safety-critical systems that operate over a sequence of time steps, such as aircraft collision avoidance systems and autonomous cars. In many applications such as autonomous driving, failures cannot be completely eliminated due to the complex stochastic environment in which the system operates.As a result, safety validation is not only concerned about whether a failure can occur, but also discovering which failures are most likely to occur. This article presents adaptive stress testing (AST), a framework for finding the most likely path to a failure event in simulation. We consider a general black box setting for partially observable and continuous-valued systems operating in an environment with stochastic disturbances. We formulate the problem as a Markov decision process and use reinforcement learning to optimize it. The approach is simulation-based and does not require internal knowledge of the system, making it suitable for black-box testing of large systems. We present different formulations depending on whether the state is fully observable or partially observable. In the latter case, we present a modified Monte Carlo tree search algorithm that only requires access to the pseudorandom number generator of the simulator to overcome partial observability. We also present an extension of the framework, called differential adaptive stress testing (DAST), that can find failures that occur in one system but not in another. This type of differential analysis is useful in applications such as regression testing, where we are concerned with finding areas of relative weakness compared to a baseline. We demonstrate the effectiveness of the approach on an aircraft collision avoidance application, where a prototype aircraft collision avoidance system is stress tested to find the most likely scenarios of near mid-air collision.

Verification and Validation

Dynamic Spectrum Allocation in Urban Air Transportation System via Deep Reinforcement Learning

The emerging concepts of Urban Air Mobility (UAM) and Advanced Air Mobility (AAM) open a new paradigm for urban air transportation. A big challenge is that these new aerial vehicles will quickly saturate the already crowded aviation spectrum, which is an essential resource to ensure reliable communications for safe operations. In this paper, we consider an air transportation system where multiple aerial vehicles are operated to transport passengers or cargo from different sources to destinations along their pre-defined paths. During the flight, the minimum communication Quality of Service (QoS) requirement must be achieved to ensure flight safety. Our objective is to minimize the average mission completion time by jointly optimizing the velocity selection and spectrum allocation for all aerial vehicles. We formulate the optimization problem as a multi-stage Markov Decision Process (MDP) where the optimization variables are coupled together. A multi-agent Deep Reinforcement Learning (DRL) based solution is proposed where Value Decomposition Networks (VDN) algorithm is utilized to take discrete actions. Additionally, we propose a heuristic greedy algorithm as a baseline solution. Simulation results show that our learning based solution outperforms the heuristic greedy algorithm and another Orthogonal Multiple Access (OMA) solution in minimizing the mission completion time.

Ruixuan Han

Dynamic Spectrum Allocation in Urban Air Transportation System via Deep Reinforcement Learning

The emerging concepts of Urban Air Mobility (UAM) and Advanced Air Mobility (AAM) open a new paradigm for urban air transportation. A big challenge is that these new aerial vehicles will quickly saturate the already crowded aviation spectrum, which is an essential resource to ensure reliable communications for safe operations. In this paper, we consider an air transportation system where multiple aerial vehicles are operated to transport passengers or cargo from different sources to destinations along their pre-defined paths. During the flight, the minimum communication Quality of Service (QoS) requirement must be achieved to ensure flight safety. Our objective is to minimize the average mission completion time by jointly optimizing the velocity selection and spectrum allocation for all aerial vehicles. We formulate the optimization problem as a multi-stage Markov Decision Process (MDP) where the optimization variables are coupled together. A multi-agent Deep Reinforcement Learning (DRL) based solution is proposed where Value Decomposition Networks (VDN) algorithm is utilized to take discrete actions. Additionally, we propose a heuristic greedy algorithm as a baseline solution. Simulation results show that our learning based solution outperforms the heuristic greedy algorithm and another Orthogonal Multiple Access (OMA) solution in minimizing the mission completion time.

Ruixuan Han

Scheduling For Urban Air Mobility Using Safe Learning

This work considers the scheduling problem for Urban Air Mobility (UAM) vehicles travelling between origin-destination pairs with both hard and soft trip deadlines. Each route is described by a discrete probability distribution over trip completion times (or delay) and over interarrival times of requests (or demand) for the route along with a fixed hard or soft deadline. Soft deadlines carry a cost that is incurred when the deadline is missed. An online, safe scheduler is developed that ensures that hard deadlines are never missed and that average cost of missing soft deadlines is minimized. The system is modelled as a Markov Decision Process (MDP) and safe model based learning is used to find the probabilistic distributions over route delays and demand. Monte Carlo Tree Search (MCTS) Earliest Deadline First (EDF) is used to safely explore the learned models in an online fashion and develop a near-optimal non-preemptive scheduling policy. These results are compared with Value Iteration (VI) and MCTS (Random) scheduling solutions.

Urban Air Mobility

Scheduling for Urban Air Mobility using Safe Learning

This work considers the scheduling problem for Urban Air Mobility (UAM) vehicles travelling between origin-destination pairs with both hard and soft trip deadlines. Each route is described by a discrete probability distribution over trip completion times (or delay) and over interarrival times of requests (or demand) for the route along with a fixed hard or soft deadline. Soft deadlines carry a cost that is incurred when the deadline is missed. An online, safe scheduler is developed that ensures that hard deadlines are never missed and that average cost of missing soft deadlines is minimized. The system is modelled as a Markov Decision Process (MDP) and safe model based learning is used to find the probabilistic distributions over route delays and demand. Monte Carlo Tree Search (MCTS) Earliest Deadline First (EDF) is used to safely explore the learned models in an online fashion and develop a near-optimal non-preemptive scheduling policy. These results are compared with Value Iteration (VI) and MCTS (Random) scheduling solutions.

Urban Air Mobility

SatNet: A Benchmark for Satellite Scheduling Optimization

Satellites provide essential services such as networking and weather tracking, and the number of near-earth and deep space satellites are expected to grow rapidly in the coming years. Communications with terrestrial ground stations is one of the critical functionalities of any space mission. Satellite scheduling is a problem that has been scientifically investigated since the 1970s. A central aspect of this problem is the need to consider resource contention and satellite visibility constraints as they require line of sight. Due to the combinatorial nature of the problem, prior solutions such as linear programs and evolutionary algorithms require extensive compute capabilities to output a feasible schedule for each scenario. Machine learning based scheduling can provide an alternative solution by training a model with historical data and generating a schedule quickly with model inference. We present SatNet, a benchmark for satellite scheduling optimization based on historical data from the NASA Deep Space Network. We propose formulation of the satellite scheduling problem as a Markov Decision Process and use reinforcement learning (RL) policies to generate schedules. The nature of constraints imposed by SatNet differ from other combinatorial optimization problems such as vehicle routing studied in prior literature. Our initial results indicate that RL is an alternative optimization approach that can generate candidate solutions of comparable quality to existing state-of-the-practice results. However, we also find that RL policies overfit to the training dataset and do not generalize well to new data, thereby necessitating continued research on reusable and generalizable agents.

Wilson, Brian

Battery State-of-Health Aware Path Planning for a Mars Rover

A rover mission consists of visiting waypoints to gather scientific samples based on set requirements. However, rovers face operational uncertainties during the mission, affecting the performance of its electrical and mechanical components and overall mission success. Hence, it is critical to have a decision-making framework that is aware of the health state of the components when planning the path of the vehicle. In particular, battery degradation, and consequently the battery State of Health (SOH), can affect the optimality of decisions made by the autonomous system in the long term. This paper presents a decision-making system that incorporates information on the energy drawn from the battery (based on the vehicle’s velocity), terrain conditions, and model-based prognostic modules to assess the impact on the battery’s state of charge (SoC). The decision-making system was formulated as a Markov Decision Process (MDP) to reach the goal destination by sending commands in a determined amount of time while maintaining the battery SoC within the policy stated. The MDP problem was programmed using the open-source framework POMDPs.jl, which has a variety of online and offline solvers. To solve the MDP problem online, we used Monte Carlo Tree Search (MCTS). Results from simulations demonstrate the effect that battery degradation and charging plans have on decision-making.

Prognostics

Autonomous Ocean World Exploration: Advancement of a Virtual Testbed

The search for life (extinct or extant) and potentially habitable bodies in our solar system and beyond is one of the 12 priority science questions outlined in the National Acadamies’ 2022 decadal survey [5]. Extraterrestrial destinations containing liquid water present an opportunity to search for life as we know it, and in recent years an increasing number of such locations have been discovered within our solar system. Several Jovian moons—Europa, Ganymede, and Callisto [10]—and the Saturnian moons Enceladus [8] and Titan [9] are known or suspected to harbor massive subsurface oceans. Of these "ocean worlds", Europa is the focus of at least one planned NASA orbiter mission, Europa Clipper [4], and an early lander mission concept, the Europa Lander [2, 3]. Whereas most robotic missions to the Moon and Mars (e.g. orbiters, rovers, landers) to date have had ground controllers on Earth tightly involved in mission operations, missions to more distant worlds will require a high degree of onboard autonomy due to long communication lags and blackouts, harsh environments (radiation, cold), and more limited battery and hardware life. The past decade has seen great advances in both AI technologies and computing scalability and performance that offer promising solutions for spacecraft autonomy and motivate the software system and research programs described in this paper. The Ocean Worlds Autonomy Testbed for Exploration, Research, and Simulation (OceanWATERS) [1], which has been in development at the NASA Ames Research Center since 2018, is a virtual environment for testing lander autonomy solutions. It is built on the Robot Operating System (ROS), runs on consumer-grade Linux workstations, and was released as open source in 2020. OceanWATERS provides a physical and visual simulation of a prototypical lander in a Europa-like environment (Figure 1). The lander was modeled after requirements and specifications made in JPL’s Europa Lander Study of 2016 [3]. Simulated lander systems include stereo cameras and spotlights mounted on an antenna mast that pans and tilts, a 6 degrees of freedom (DoF) robotic arm with a force-torque sensor and two interchangeable end effectors, and a battery pack power system. The environment consists of multiple terrain models including a highly detailed model sourced from the FROST dataset [11], simulation of surrounding planetary bodies based on an ephemeris model, and lighting from the sun with associated surface illumination, reflectance, and shadows. Operations supported by OceanWATERS include panoramic and directed imaging of the environment and lander workspace, Cartesian and joint-level arm commanding, grinding of the terrain surface (e.g. digging a trench), and scooping of ground material (Figure 2) which can be discarded or collected as science samples in a receptacle that can be emptied (science operations themselves are not simulated). These operations are realized as ROS Actions and are complimented by a wide selection of telemetry that is continually produced by each lander subsystem. The power system model is driven by the open-source Generic Software Architecture for Prognostics (GSAP) [11] that predicts the battery’s remaining useful life and other characteristics. As a testbed for high-level autonomy, OceanWATERS provides an execution framework based on PLEXIL [12], an open-source plan specification language and execution engine developed largely at Ames. NASA's initial development of OceanWATERS, as well the Ocean Worlds Lander Autonomy Testbed (OWLAT) [6], a complimentary physical testbed developed at JPL, was the first step in a plan for realizing candidate onboard autonomy solutions for such planetary landers. In 2020 NASA solicited applications for its Autonomous Robotics Research for Ocean Worlds (ARROW) program, and in 2021 the similar Concepts for Ocean worlds Life Detection Technology (COLDTech) program. Collectively six research teams, based in universities and companies across the United States, were awarded grants to develop and demonstrate autonomy solutions on OceanWATERS and OWLAT. These 1–2-year projects have now finished or are nearing completion, and a wide variety of autonomy challenges in ocean world surface missions were addressed. Prototyped and demonstrated solutions have included autonomous discovery, response and adaptation to system faults and unexpected environmental events, world model synthesis through perception, plan synthesis using learned models, methods to optimize sample target selection and prioritize science data transmission, extension of PLEXIL for stochastic decision-making, and an integration of a model of JPL’s mission-ready COLDArm [7]. Technologies used in these projects include many forms of machine learning, causal reasoning, automated planning, Markov decision processes, formal methods, and other advanced techniques. A more detailed summary of the ARROW and COLDTech projects is given herein. OceanWATERS has had significant enhancements since its open-source release in 2020. Many of its new features were driven or shaped by feedback from the ARROW and COLDTech teams and requirements of their projects. In support of enabling autonomous adaptation to spacecraft faults (a specific capability solicited by both programs), a fault injection and detection framework was developed that supports a wide and growing range of fault types such as locked joints, image loss, and battery failures. The power system model was completed and integrated into the simulator, starting as a single-cell battery model and later upgraded to a multi-cell model with associated faults such as cell disconnection. Arm/terrain interaction was improved by adding a force-torque sensor and associated faults, and an analytic dig force model based on the Balovnev bucket force equations. Environment fidelity was increased by modeling terrain deformation resulting from digging and scooping; visual improvements were made in textures, lighting, and shadows. To facilitate interoperation with OWLAT, a unified command and telemetry interface between the testbeds was developed at the ROS level, along with a PLEXIL interface. The number of lander operations was greatly expanded (e.g. with Cartesian-based arm and antenna movement), and a framework was designed for users to build their own lander actions. A GUI for PLEXIL plan selection was created (Figure 3), and an expansive set of plans were added, such as those that illustrate patterns for fault handling. This paper provides a self-contained high-level description of OceanWATERS, focusing on more detailed coverage of the aforementioned enhancements. It provides a high-level summary of the projects undertaken by participants in the ARROW and COLDTech programs and how these efforts have helped shape OceanWATERS. Finally, potential future work and directions for the testbed are listed, as likely informed by the recent planetary science decadal survey [5].

K Michael Dalal

Tradeoffs When Considering Deep Reinforcement Learning for Contingency Management in Advanced Air Mobility

Air transportation is undergoing a rapid evolution globally with the introduction of Advanced Air Mobility (AAM) and with it comes novel challenges and opportunities for transforming aviation. As AAM operations introduce increasing heterogeneity in vehicle capabilities and density, increased levels of automation are likely necessary to achieve operational safety and efficiency goals. This paper focuses on one example where increased automation has been suggested. Autonomous operations will need contingency management systems that can monitor evolving risk across a span of interrelated (or interdependent) hazards and, if necessary, execute appropriate control interventions via supervised or automated decision making. Accommodating this complex environment may require automated functions (autonomy) that apply artificial intelligence (AI) techniques that can adapt and respond to a quickly changing environment. This paper explores the use of Deep Reinforcement Learning (DRL) which has shown promising performance in complex and high-dimensional environments where the objective can be constructed as a sequential decision-making problem. An extension of a prior formulation of the contingency management problem as a Markov Decision Process (MDP) is presented and uses a DRL framework to train agents that mitigate hazards present in the simulation environment. A comparison of these learning-based agents and classical techniques is presented in terms of their performance, verification difficulties, and development process.

machine learningautonomous systems; flight simulat

Energy-Optimized Path Planning for Uas in Varying Winds Via Reinforcement Learning

In this paper we propose a reinforcement learning (RL) algorithm for path planning of Unmanned Aviation Vehicles (UAVs) under varying wind conditions. Solutions to UAV path planning problems are becoming increasingly necessary as autonomous UAVs continue to enter commercial and government spaces. Path-planning is inherently challenging, as UAVs need to account for dynamically changing flying conditions such as weather, obstacle or no-fly zones, degraded vehicle health, and off-nominal battery power consumption. Machine learning methods such as reinforcement learning (RL) have the potential to revolutionize how vehicles navigate in such uncertain environments. In this study, we compute UAV trajectories from a pre-determined starting position to a target cell within a 7X7 grid environment by optimizing parameters for mission assurance and safety limits in addition to the energy consumption and operation time. The UAV navigates the grid by taking actions to move in any of the eight cardinal and inter-cardinal directions, under constant thrust profile. The resultant UAV state is sampled from a probability distribution which accounts for the UAV’s action, local wind velocity, and the presence of obstacles or boundaries. As the unmanned airspace gets more complex due to multiple vehicles and environmental uncertainties, trade-offs between energy consumption, operation time, risk tolerance, and mission assurance need to be made. Our Markov Decision Process (MDP) environment model can capture any combination of these in the optimization objective, making it novel compared to other work in the field.

trajectory planning

Tradeoffs When Considering Deep Reinforcement Learning for Contingency Management in Advanced Air Mobility

Air transportation is undergoing a rapid evolution globally with the introduction of Advanced Air Mobility (AAM) and with it comes novel challenges and opportunities for transforming aviation. As AAM operations introduce increasing heterogeneity in vehicle capabilities and density, increased levels of automation are likely necessary to achieve operational safety and efficiency goals. This paper focuses on one example where increased automation has been suggested. Autonomous operations will need contingency management systems that can monitor evolving risk across a span of interrelated (or interdependent) hazards and, if necessary, execute appropriate control interventions via supervised or automated decision making. Accommodating this complex environment may require automated functions (autonomy) that apply artificial intelligence (AI) techniques that can adapt and respond to a quickly changing environment. This paper explores the use of Deep Reinforcement Learning (DRL) which has shown promising performance in complex and high-dimensional environments where the objective can be constructed as a sequential decision-making problem. An extension of a prior formulation of the contingency management problem as a Markov Decision Process (MDP) is presented and uses a DRL framework to train agents that mitigate hazards present in the simulation environment. A comparison of these learning-based agents and classical techniques is presented in terms of their performance, verification difficulties, and development process.

machine learning

Separation Assurance in Urban Air Mobility Systems Using Shared Scheduling Protocols

Ensuring safe separation between aircraft is a critical challenge in air traffic management, particularly in urban air mobility (UAM) environments where high traffic density and low altitudes require precise control. In these environments, conflicts often arise at the intersections of flight corridors, posing significant risks. We propose a tactical separation approach leveraging shared scheduling protocols, originally designed for Ethernet networks and operating systems, to coordinate access to these intersections. Using a decentralized Markov decision process framework, the proposed approach enables aircraft to autonomously adjust their speed and timing as they navigate these critical areas, maintaining safe separation without a central controller. We evaluate the effectiveness of this approach in simulated UAM scenarios, demonstrating its ability to reduce separation violations to zero while acknowledging trade-offs in flight times as traffic density increases. Additionally, we explore the impact of non-compliant aircraft, showing that while shared scheduling protocols can no longer guarantee safe separation, they still provide significant improvements over systems without scheduling protocols.

Separation Assurance

Separation Assurance in Urban Air Mobility Systems Using Shared Scheduling Protocols

Ensuring safe separation between aircraft is a critical challenge in air traffic management, particularly in urban air mobility (UAM) environments where high traffic density and low altitudes require precise control. In these environments, conflicts often arise at the intersections of flight corridors, posing significant risks. We propose a tactical separation approach leveraging shared scheduling protocols, originally designed for Ethernet networks and operating systems, to coordinate access to these intersections. Using a decentralized Markov decision process framework, the proposed approach enables aircraft to autonomously adjust their speed and timing as they navigate these critical areas, maintaining safe separation without a central controller. We evaluate the effectiveness of this approach in simulated UAM scenarios, demonstrating its ability to reduce separation violations to zero while acknowledging trade-offs in flight times as traffic density increases. Additionally, we explore the impact of non-compliant aircraft, showing that while shared scheduling protocols can no longer guarantee safe separation, they still provide significant improvements over systems without scheduling protocols.

system safety