Sharp Sample Complexity of KL-Regularized Reinforcement Learning
Andrew Liu
Abstract
KL-regularized reinforcement learning, the objective behind maximum-entropy RL and RL from human feedback, is variational inference in the graphical model of "control as inference": the regularized optimal policy is a posterior over trajectories, and the suboptimality of a policy is β times a KL divergence between path measures. However, it is not known how many samples it takes to learn this posterior, even with a generative model. Earlier 1/ε guarantees scale with H5 in the effective horizon H, and no lower bound for MDPs depends on the temperature β. In this work, we show that for a discounted MDP with K state-action pairs, the plug-in estimator finds a policy that is ε-optimal from every initial state with O(KH3/(ε\ε,β\) + KH) samples, and that no algorithm, even one that chooses where to sample, can do with fewer when ε \H, H2/β\. For ε< β this is a 1/ε rate with the same horizon dependence as unregularized RL. We also find the exact asymptotic constant of the plug-in estimator, a Fisher-weighted variance of the estimated soft Q-function. The estimator is locally asymptotically minimax among methods that sample every pair equally often, and we give the allocation of samples that is optimal among adaptive designs. Simulations on small MDPs match the rates and the constants.
Create a lesson
Related papers
Bio-PM-Guided Synthetic IMU Generation
Deepali Singh, Deepesh Parpani, Aviral Gupta
Active Feature Acquisition by Distilling an Oracle Teacher
Yang Meng, Zixin Ding, Robert L. Grossman et al.
The Window Lever: Longer Observation Windows Outweigh the Observed Effects of Data, Capacity, and Architecture in Customer Behavior Forecasting
Tara Sowrirajan, Yanqiao Wang
Hybrid Retrieval for Government Scheme Search in English and Marathi
Aishwarya Gunjal, Kanchan Khemnar, Pawan Poojary et al.
Grid Cells as Spectral Successor Features: A Reusable Basis for Continual Reinforcement Learning
Élodie Côté-Gauthier, Doina Precup, Isabeau Prémont-Schwarz
A new algorithmic approach for finding flat minima: critically damped Polyak stepsizes
Francesco Caporali, Jason M. Altschuler, Boris Hanin