Intermediate phases in $\alpha$-RuCl$_3$ under in-plane magnetic field via interlayer spin interactions

Abstract

$\alpha$-RuCl$_3$ has attracted significant attention as a prime candidate for the spin-1/2 Kitaev spin liquid in two-dimensional honeycomb lattices. Although its ground state is magnetically ordered, the order is suppressed under a moderate in-plane magnetic field. The intermediate regime of the field has exotic behaviors, some of which are claimed to originate from a Kitaev spin liquid. In resolving debates surrounding these behaviors, interlayer interactions in $\alpha$-RuCl$_3$ have been largely overlooked due to their perceived weakness in van der Waals materials. However, near the transition, they may become significant as the field energy approaches the interlayer coupling scale. Here we investigate the effects of interlayer couplings in $\alpha$-RuCl$_3$ with $R\bar{3}$ and $C2/m$ structures. We first examine their effects on the transition temperature ($T_N$) using classical Monte Carlo simulations. We found that the interlayer couplings have minimal effects on $T_N$, and the different $T_N$ between the two structures are mainly due to the anisotropy in intralayer interactions. Focusing on the $R{\bar 3}$ structure, we show that the nearest neighbor interlayer interaction is the XXZ type due to the symmetry, and the next nearest neighbor interaction of the Kitaev type is crucial for the transition between two zigzag orders under an in-plane field. Furthermore, an intermediate phase with a large unit cell emerges due to the interlayer interactions. Our findings provide insights into the exotic behaviors and sample dependence reported in $\alpha$-RuCl$_3$.