Preprint
Performance Bounds for Policy-Based Average Reward Reinforcement Learning Algorithms
ArXiv.org
Cornell University
02/02/2023
DOI: 10.48550/arxiv.2302.01450
Abstract
Many policy-based reinforcement learning (RL) algorithms can be viewed as instantiations of approximate policy iteration (PI), i.e., where policy improvement and policy evaluation are both performed approximately. In applications where the average reward objective is the meaningful performance metric, discounted reward formulations are often used with the discount factor being close to 1, which is equivalent to making the expected horizon very large. However, the corresponding theoretical bounds for error performance scale with the square of the horizon. Thus, even after dividing the total reward by the length of the horizon, the corresponding performance bounds for average reward problems go to infinity. Therefore, an open problem has been to obtain meaningful performance bounds for approximate PI and RL algorithms for the average-reward setting. In this paper, we solve this open problem by obtaining the first finite-time error bounds for average-reward MDPs, and show that the asymptotic error goes to zero in the limit as policy evaluation and policy improvement errors go to zero.
Details
- Title: Subtitle
- Performance Bounds for Policy-Based Average Reward Reinforcement Learning Algorithms
- Creators
- Yashaswini MurthyMehrdad MoharramiR Srikant
- Resource Type
- Preprint
- Publication Details
- ArXiv.org
- DOI
- 10.48550/arxiv.2302.01450
- ISSN
- 2331-8422
- Publisher
- Cornell University
- Number of pages
- 24
- Language
- English
- Date posted
- 02/02/2023
- Academic Unit
- Computer Science
- Record Identifier
- 9984446965802771
Metrics
64 Record Views