Global phase diagram and quantum spin liquids in spin-1/2 triangular antiferromagnet

Abstract

We study the spin-$1/2$ Heisenberg model on the triangular lattice with the nearest-neighbor $J_1 > 0$, the next-nearest-neighobr $J_2 > 0$ Heisenberg interactions, and the additional scalar chiral interaction $J_{\chi}(\vec{S}_i \times \vec{S}_j) \cdot \vec{S}_k$ for the three spins in all the triangles using large-scale density matrix renormalization group calculation on cylinder geometry. With increasing $J_2$ ($J_2/J_1 \leq 0.3$) and $J_{\chi}$ ($J_{\chi}/J_1 \leq 1.0$) interactions, we establish a quantum phase diagram with the magnetically ordered $120^{\circ}$ phase, stripe phase, and non-coplanar tetrahedral phase. In between these magnetic order phases, we find a chiral spin liquid (CSL) phase, which is identified as a $\nu = 1/2$ bosonic fractional quantum Hall state with possible spontaneous rotational symmetry breaking. By switching on the chiral interaction, we find that the previously identified spin liquid in the $J_1 - J_2$ triangular model ($0.08 \lesssim J_2/J_1 \lesssim 0.15$) shows a phase transition to the CSL phase at very small $J_{\chi}$. We also compute spin triplet gap in both spin liquid phases, and our finite-size results suggest large gap in the odd topological sector but small or vanishing gap in the even sector. We discuss the implications of our results to the nature of the spin liquid phases.