Reinforcement learning with soft temporal logic constraints using limit-deterministic generalized Büchi automaton

Mingyu Cai; Zhangli Zhou; Lin Li; Shaoping Xiao; Zhen Kan

doi:10.1016/j.jai.2024.12.005

Back

Reinforcement learning with soft temporal logic constraints using limit-deterministic generalized Büchi automaton

Journal article

Open access

Peer reviewed

Reinforcement learning with soft temporal logic constraints using limit-deterministic generalized Büchi automaton

Mingyu Cai, Zhangli Zhou, Lin Li, Shaoping Xiao and Zhen Kan

Journal of Automation and Intelligence, Vol.4(1), pp.39-51

03/2025

DOI: 10.1016/j.jai.2024.12.005

Files and links (1)

url

https://doi.org/10.1016/j.jai.2024.12.005View

Published (Version of record) Open Access

Abstract

This paper investigates control synthesis for motion planning under conditions of uncertainty, specifically in robot motion and environmental properties, which are modeled using a probabilistic labeled Markov decision process (PL-MDP). To address this, a model-free reinforcement learning (RL) approach is designed to produce a finite-memory control policy that meets complex tasks specified by linear temporal logic (LTL) formulas. Recognizing the presence of uncertainties and potentially conflicting objectives, this study centers on addressing infeasible LTL specifications. A relaxed LTL constraint enables the agent to adapt its motion plan, allowing for partial satisfaction by accounting for necessary task violations. Additionally, a new automaton structure is introduced to increase the density of accepting rewards, facilitating deterministic policy outcomes. The proposed RL framework is rigorously analyzed and prioritizes two key objectives: (1) satisfying the acceptance condition of the relaxed product MDP, and (2) minimizing long-term violation costs. Simulation and experimental results are presented to demonstrate the framework’s effectiveness and robustness.

Formal methods in robotics and automation

Linear temporal logic

Motion planning

Optimal control

Details

Title: Subtitle: Reinforcement learning with soft temporal logic constraints using limit-deterministic generalized Büchi automaton
Creators: Mingyu Cai - University of California, Riverside
Zhangli Zhou - University of Science and Technology of China
Lin Li - University of Science and Technology of China
Shaoping Xiao - The University of Iowa, 52246, IA, USA
Zhen Kan - University of Science and Technology of China, 230026 Hefei, Anhui, China
Resource Type: Journal article
Publication Details: Journal of Automation and Intelligence, Vol.4(1), pp.39-51
DOI: 10.1016/j.jai.2024.12.005
ISSN: 2949-8554
eISSN: 2949-8554
Publisher: Elsevier B.V
Language: English
Electronic publication date: 12/31/2024
Date published: 03/2025
Academic Unit: Iowa Technology Institute; Mechanical Engineering
Record Identifier: 9984771630502771

Metrics

2 Record Views