Emergence of vortex state in the $S=1$ Kitaev-Heisenberg model with single-ion anisotropy

Abstract

The search for Kitaev spin liquid states has recently broadened to include a number of honeycomb materials with integer spin moments. The qualitative difference with their spin-1/2 counterparts is the presence of single-ion anisotropy (SIA). This motivates our investigation of the effects of SIA on the ground state of the spin-$1$ Kitaev-Heisenberg (KH) model using the density-matrix renormalization group which allows construction of detailed phase diagrams around the Kitaev points. We demonstrate that positive out-of-plane SIA induces an in-plane vortex state without the need for off-diagonal interactions. Conversely, negative SIA facilitates the emergence of a ferromagnetic state in presence of antiferromagnetic Heisenberg interactions, while a N\`eel state can emerge for ferromagnetic Heisenberg coupling. These findings, pertinent even for weak SIA, not only enhance our theoretical understanding of the spin-1 KH model but also propose experimental prospects for observing these novel magnetic states in material realizations.