Quantum spin liquid ground states of the Heisenberg-Kitaev model on the triangular lattice

Abstract

We study quantum disordered ground states of the two dimensional Heisenberg-Kitaev model on the triangular lattice using a Schwinger boson approach. Our aim is to identify and characterize potential gapped quantum spin liquid phases that are stabilized by anisotropic Kitaev interactions. For antiferromagnetic Heisenberg- and Kitaev couplings and sufficiently small spin $S$ we find three different symmetric $Z_2$ spin liquid phases, separated by two continuous quantum phase transitions. Interestingly, the gap of elementary excitations remains finite throughout the transitions. The first spin liquid phase corresponds to the well known zero-flux state in the Heisenberg limit, which is stable with respect to small Kitaev couplings and develops $120^\circ$ order in the semi-classical limit at large $S$. In the opposite Kitaev limit we find a different spin liquid ground-state, which is a quantum disordered version of a magnetically ordered state with antiferromagnetic chains, in accordance with results in the classical limit. Finally, at intermediate couplings we find a spin liquid state with unconventional spin correlations. Upon spinon condensation this state develops Bragg peaks at incommensurate momenta in close analogy to the magnetically ordered Z2 vortex crystal phase, which has been analyzed in recent theoretical works.