Finite-$U$ induced competing interactions, frustration, and quantum phase transition in a triangular-lattice antiferromagnet

Abstract

The $120^0$ ordered antiferromagnetic state of the Hubbard model on a triangular lattice presents an interesting case of $U$-controlled competing interactions and frustration. The spin stiffness is found to vanish at $U^* _{\rm stiff} \approx 6$ and the spin-wave energy $\omega_M$ at ${\bf q}_M=(2\pi/3,0)$ etc. is found to vanish at $U_M ^* \approx 6.8$ due to competing spin couplings generated at finite $U$. The loss of magnetic order due to the magnetic instability at ${\bf q}_M$ yields a first-order quantum phase transition in the insulating state at $U=U_{\rm M}^*$. Implications of the quantum spin disordered insulator to the spin-liquid state and Mott transition in the organic systems $\rm \kappa -(BEDT-TTF)_2 X$ are discussed. Effects of hole and electron doping on magnetic ordering and spin stiffness are also examined.