Global phase diagram, possible chiral spin liquid and topological superconductivity in the triangular Kitaev-Heisenberg model

Abstract

The possible ground states of the undoped and doped Kitaev-Heisenberg model on a triangular lattice are studied. For the undoped system, a combination of the numerical exact diagonalization calculation and the four-sublattice transformation analysis suggests one possible exotic phase and four magnetically ordered phases including a collinear stripe pattern and a noncollinear spiral pattern in the global phase diagram. The exotic phase near the antiferromagnetic (AF) Kitaev point is further investigated by using the Schwinger-fermion mean-field method, and we obtain an energetically favorable $Z_2$ chiral spin liquid with a Chern number $\pm2$ as a promising candidate. At finite doping, we find that the AF Heisenberg coupling supports an $s$-wave or a $d_{x^2-y^2}+id_{xy}$-wave superconductivity (SC), while the AF and the ferromagnetic Kitaev interactions favor a $d_{x^2-y^2}+id_{xy}$-wave SC and a time-reversal invariant topological $p$-wave SC, respectively. Possible experimental realizations and related candidate materials are also discussed.