Continuous-time q-learning for Markov regime switching system under Tsallis entropy

Abstract

This paper studies the continuous-time q-learning (the continuous time counterpart of Q-learing) for Markov switching system under Tsallis entropy regularization. We address the difficulty in traditional RL algorithms where the Tsallis entropy regularization leads to an optimal policy distribution not necessarily a Gibbs measure, which often complicates algorithm design. Furthermore, to address the limited universality of current continuous time regime-switching RL algorithms (often restricted to the EMV framework), this study focuses on continuous-time q-learning for Markov regime-switching systems based on Tsallis entropy, aiming for a more universally applicable continuous-time RL method. We establish the martingale characterization of the q-function under Tsallis entropy for continuous-time Markov regime-switching systems. Based on this, we design two q-learning algorithms, distinguished by whether the Lagrange multiplier can be explicitly derived. We apply these algorithms to the continuous-time exploratory Mean-Variance (EMV) portfolio optimization problem in a regime-switching market. Numerical experiments demonstrate the satisfactory performance of our q-learning algorithms.