Sharp Gaussian Approximation for Q-Learning
Xingtu Liu
Abstract
Gaussian approximation provides a foundation for online statistical inference in reinforcement learning. We establish sharp non-asymptotic Gaussian approximation rates in Wasserstein distance for asynchronous Q-learning with Polyak-Ruppert averaging. We prove an upper bound of O(n-1/4), where n is the number of iterations, together with a matching lower bound up to logarithmic factors. Q-learning is more challenging to analyze than linear stochastic approximation due to the nonlinearity of the Bellman operator. Asynchronous updates introduce further challenges due to Markovian noise. Our analysis separates the errors from Bellman nonlinearity, Markovian noise, and the finite-sample covariance discrepancy. In particular, we show that Markovian noise induces an unavoidable bias even when the Bellman operator is linear. The trade-off between this bias and the covariance discrepancy determines the convergence rate. We further extend our analysis to the synchronous setting and establish a sharp rate of O(n-1/2).
Create a lesson
Related papers
Bayesian optimization on amortized search space
Tristan Fauvel, Federico Pavone, Titouan Guillou et al.
Exact Bayesian layers via characteristic functions
William H. W. Thompson, Guillaume St-Onge, Laurent Hébert-Dufresne et al.
Flow Matching from Non-Gaussian Sources: A Theory of Efficient Sampling
Yuchen Zhou, Lili Zheng, Yuxin Chen et al.
Conformal Selection for Hierarchical Streaming Data
Zhenyuan Sun, Chengyao Yu, Ruixing Ming et al.
A Statistical Framework for Training Data Auditing of Large Language Models
Shengjie Niu, Yeheng Ge, Jian Huang
Hellinger Decision Trees for Density Ratio Estimation: Finite-Sample Guarantees
Ren-Rui Liu, Zheng-Chu Guo