Bounds on Spectral Gaps for Non-Reversible Markov Chains with Applications to Temporal Difference Learning

Abstract

This work is motivated by the analysis of temporal difference algorithms, where stability can be guaranteed by bounding the eigenvalues of an associated matrix derived from a, typically non-reversible, Markov kernel. We generalize the existing sufficient conditions for stability and show that the associated eigenvalues can be bounded in terms of the Dirichlet spectral gap of the Markov kernel. We derive a collection of methods for showing that non-reversible Markov chains have positive spectral gaps. We show that if a Markov chain has positive absolute spectral gap, then it has a positive Dirichlet spectral gap. In the case of discrete-time linear Gaussian systems, we give explicit bounds for both Dirichlet and absolute spectral gaps. Additionally, we present an example of a Markov chain which is $V$-uniformly geometrically ergodic but has zero Dirichlet spectral gap.