Magnetization Plateaus in the Spin-Orbit Coupled Bilayer Triangular Lattice Antiferromagnet Rb2Co2(SeO3)3

Abstract

Geometric frustration among competing spin exchanges can give rise to novel quantum phases by enhancing fluctuations that drive magnetic systems beyond the classical regime. We investigate the frustrated array of strongly correlated spin dimers in the bilayer triangular lattice antiferromagnet \rcs{} under applied magnetic fields. A cascade of magnetization plateaus appears at \(M/M_s = 1/3, 1/2, 2/3,\) and \(5/6\), together with a weak anomalous feature near \(M/M_s = 1/6\), in fields up to 60 T. Concurrent changes in magneto-dielectric response follows the plateau boundaries. The finite slope of each plateau and the absence of a zero-field gap in our ultralow-temperature ac susceptibility down to 20 mK indicate broken \(U(1)\) spin-rotation symmetry. A minimal bilayer-dimer model treated with bond operator representation reproduces the low-field sequence only when \(U(1)\) symmetry is explicitly lifted by spin-orbit-driven, bond-dependent anisotropy. Near saturation, a projected triangular pseudospin model accounts for the high-field plateaus with modest further-neighbor interactions. These results demonstrate that anisotropic exchange arising from spin-orbit-coupled moments is essential for stabilizing the full plateau hierarchy in \rcs{}, a mechanism overlooked in previous interpretations of Co-based triangular bilayers.