Preprint
Offline Constrained RLHF with Multiple Preference Oracles
ArXiv.org
Cornell University
03/31/2026
DOI: 10.48550/arxiv.2604.00200
Abstract
We study offline constrained reinforcement learning from human feedback with multiple preference oracles. Motivated by applications that trade off performance with safety or fairness, we aim to maximize target population utility subject to a minimum protected group welfare constraint. From pairwise comparisons collected under a reference policy, we estimate oracle-specific rewards via maximum likelihood and analyze how statistical uncertainty propagates through the dual program. We cast the constrained objective as a KL-regularized Lagrangian whose primal optimizer is a Gibbs policy, reducing learning to a convex dual problem. We propose a dual-only algorithm that ensures high-probability constraint satisfaction and provide the first finite-sample performance guarantees for offline constrained preference learning. Finally, we extend our theoretical analysis to accommodate multiple constraints and general f-divergence regularization.
Details
- Title: Subtitle
- Offline Constrained RLHF with Multiple Preference Oracles
- Creators
- Brenden LathamMehrdad Moharrami
- Resource Type
- Preprint
- Publication Details
- ArXiv.org
- DOI
- 10.48550/arxiv.2604.00200
- ISSN
- 2331-8422
- Publisher
- Cornell University; Ithaca, New York
- Language
- English
- Date posted
- 03/31/2026
- Academic Unit
- Computer Science
- Record Identifier
- 9985149705202771
Metrics
1 Record Views