Novel spin-liquid states in the frustrated Heisenberg antiferromagnet on the honeycomb lattice

Abstract

Recent experiment on a honeycomb-lattice Heisenberg antiferromagnet (AF) Bi$_3$Mn$_4$O$_{12}$(NO$_3$) revealed a novel spin-liquid-like behavior down to low temperature, which was ascribed to the frustration effect due to the competition between the AF nearest- and next-nearest-neighbor interactions $J_1$ and $J_2$. Motivated by the experiment, we study the ordering of the $J_1$ -$J_2$ frustrated classical Heisenberg AF on a honeycomb lattice both by a low-temperature expansion and a Monte Carlo simulation. The model has been known to possess a massive degeneracy of the ground state, which, however, might be lifted due to thermal fluctuations leading to a unique ordered state, the effect known as 'order-by-disorder'. We find that the model exhibits an intriguing ordering behavior, particularly near the AF phase boundary. The energy scale of the order-by-disorder is suppressed there down to extremely low temperatures, giving rise to exotic spin-liquid states like a "ring-liquid" or a "pancake-liquid" state accompanied by the characteristic spin structure factor and the field-induced antiferromagnetism. We argue that the recent experimental data are explicable if the system is in such exotic spin-liquid states.