Dissertation
Reinforcement learning-based motion planning in partially observable environments for complex tasks
University of Iowa
Doctor of Philosophy (PhD), University of Iowa
Spring 2024
DOI: 10.25820/etd.007512
Abstract
Partially Observable Markov Decision Processes (POMDPs) have been used as mathematical models for sequential decision-making under uncertain and incomplete information. Since the state space is partially observable in a POMDP, the agent has to make a decision based on the integrated information over the past experiences of actions and observations. This dissertation aims to solve probabilistic motion planning problems in which the agent is assigned a complex task under a partially observable environment. We employ Linear Temporal Logic (LTL) to formulate the complex task and then convert it to a Limit-Deterministic Generalized Büchi Automaton (LDGBA). We formulate the problem as finding an optimal policy on the product of POMDP and LDGBA based on model-checking techniques. This dissertation also studies both model-based and model-free Reinforcement Learning (RL) algorithms as the approaches to solve such optimal policy.
Following the literature review in Chapter 1 and the introduction of RL in Chapter 2, Chapter 3 adopts two model-based RL approaches to approximate the value function of the belief state space in order to obtain the optimal policy on the corresponding product POMDP domain. Both approaches are model-based because the transition information of the underlying Markov Decision Process (MDP) is needed for the belief state update.
Inspired by the limitation of the model-based approaches employed in Chapter 3, a model-free approach is proposed in Chapter 4 by implementing Recurrent Neural Networks (RNNs) into Q-network architectures to directly process the agent's observation histories and task recognition without using the model knowledge. Two scenarios are considered, depending on whether the agent fully recognizes the LTL-induced automaton. The combination of observation histories with the automaton states is used as the input features to Q-networks as the agent previously acquired the automaton knowledge. Otherwise, the state labels with observations are utilized. Both settings give the agent a good sense of the problem goal for its decision-making.
The proposed model-free approach was then implemented to address the AI ethical decision-making in motion planning problems for complex tasks in partially observable environments. In Chapter 5, the developed approach is leveraged to model the ethical norms into `hard' and `soft' ethical constraints, where the formal logic language LTL is utilized to express the `hard' constraints, and a reward redesign method is applied to synthesize the `soft' ethical constraints. The innovated RNN-based Deep Q-learning approach is employed to solve such ethical decision-making problems. The simulation results showed its potential applicability to other scenarios and challenges in the field of ethical decision-making.
Chapter 6 provides a comprehensive summary of the proposed approaches for AI decision-making, with a particular focus on motion planning. Additionally, it provides a detailed comparative analysis, exploring the performance and limitations of these approaches relative to established models in the academic literature.
Details
- Title: Subtitle
- Reinforcement learning-based motion planning in partially observable environments for complex tasks
- Creators
- Junchao Li
- Contributors
- Shaoping Xiao (Advisor)Venanzio Cichella (Committee Member)Deema Totah (Committee Member)Rachel Vitali (Committee Member)
- Resource Type
- Dissertation
- Degree Awarded
- Doctor of Philosophy (PhD), University of Iowa
- Degree in
- Mechanical Engineering
- Date degree season
- Spring 2024
- Publisher
- University of Iowa
- DOI
- 10.25820/etd.007512
- Number of pages
- xvi, 165 pages
- Copyright
- Copyright 2024 Junchao Li
- Language
- English
- Date submitted
- 03/15/2024
- Description illustrations
- Illustrations, tables, graphs, charts
- Description bibliographic
- Includes bibliographical references (pages 155-165).
- Public Abstract (ETD)
Robotic motion planning is about finding a path for a robot to move from one place to another without encountering obstacles. Reinforcement learning is a commonly used approach in this field. However, despite extensive research, it has seldom addressed motion planning problems in environments with incomplete knowledge and uncertainties caused by the malfunction of actuators and/or sensors, which are commonly observed in real-world applications.
This thesis focuses on developing algorithms that enable robots to navigate optimally in environments with incomplete knowledge while also accomplishing complex tasks specified by humans, such as surveillance missions. To address these challenges, a high-level logical language is employed for task expression, which is then synthesized into our proposed framework. Two reinforcement learning approaches are developed to map the robot’s observations to its movement decisions. One approach is model-based reinforcement learning, which assumes that the robot knows the probabilities of its movements and the likelihood of the observations it can perceive. The other approach is model-free reinforcement learning, in which the robot does not know the movement and observation probabilities.
Furthermore, the model-free approach is utilized to address ethical considerations in motion planning. Human-defined ethical standards are transformed into specific constraints, and integrated into the reinforcement learning framework. This approach aligns robot decisions with established ethical standards and human values. The associated simulations also demonstrate its versatility and applicability to various AI decision-making challenges.
- Academic Unit
- Mechanical Engineering
- Record Identifier
- 9984647152002771
Metrics
1 File views/ downloads
6 Record Views