Ergodic Theory for Controlled Markov Chains with Stationary Inputs

Abstract

Consider a stochastic process $\{X(t)\}$ on a finite state space $ {\sf X}=\{1,\dots, d\}$. It is conditionally Markov, given a real-valued `input process' $\{\zeta(t)\}$. This is assumed to be small, which is modeled through the scaling, \[ \zeta_t = \varepsilon \zeta^1_t, \qquad 0\le \varepsilon \le 1\,, \] where $\{\zeta^1(t)\}$ is a bounded stationary process. The following conclusions are obtained, subject to smoothness assumptions on the controlled transition matrix and a mixing condition on $\{\zeta(t)\}$: (i) A stationary version of the process is constructed, that is coupled with a stationary version of the Markov chain $\{X^\bullet$(t)\}obtained with $\{\zeta(t)\}\equiv 0$. The triple $(\{X(t)\}, \{X^\bullet(t)\},\{\zeta(t)\})$ is a jointly stationary process satisfying \[ {\sf P}\{X(t) \neq X^\bullet(t)\} = O(\varepsilon) \] Moreover, a second-order Taylor-series approximation is obtained: \[ {\sf P}\{X(t) =i \} ={\sf P}\{X^\bullet(t) =i \} + \varepsilon^2 \varrho(i) + o(\varepsilon^2),\quad 1\le i\le d, \] with an explicit formula for the vector $\varrho\in\mathbb{R}^d$. (ii) For any $m\ge 1$ and any function $f\colon \{1,\dots,d\}\times \mathbb{R}\to\mathbb{R}^m$, the stationary stochastic process $Y(t) = f(X(t),\zeta(t))$ has a power spectral density $\text{S}_f$ that admits a second order Taylor series expansion: A function $\text{S}^{(2)}_f\colon [-\pi,\pi] \to \mathbb{C}^{ m\times m}$ is constructed such that \[ \text{S}_f(\theta) = \text{S}^\bullet_f(\theta) + \varepsilon^2 \text{S}_f^{(2)}(\theta) + o(\varepsilon^2),\quad \theta\in [-\pi,\pi] . \] An explicit formula for the function $\text{S}_f^{(2)}$ is obtained, based in part on the bounds in (i). The results are illustrated using a version of the timing channel of Anantharam and Verdu.