Topological and magnetic phase transitions in the bilayer Kitaev-Ising model

Abstract

We investigate the phase diagram of a bilayer Kitaev honeycomb model with Ising interlayer interactions, deriving effective models via perturbation theory and performing Majorana mean-field theory calculations. We show that a diverse array of magnetic and topological phase transitions occur, depending on the direction of the interlayer Ising interaction and the relative sign of Kitaev interactions. When two layers have the same sign of the Kitaev interaction, a first-order transition from a Kitaev spin liquid to a magnetically ordered state takes place. The magnetic order points along the Ising axis and it is (anti)ferromagnetic for (anti)ferromagnetic Kitaev interactions. However, when two layers have opposite sign of the Kitaev interaction, we observe a notable weakening of magnetic ordering tendencies and the Kitaev spin liquid survives up to a remarkably larger interlayer exchange. Our mean-field analysis suggests the emergence of an intermediate gapped $\mathbb{Z}_2$ spin liquid state, which eventually becomes unstable upon vison condensation. The confined phase is described by a highly frustrated $120^\circ$ compass model. We furthermore use perturbation theory to study the model with the Ising axis pointing along $\hat{z}$-axis or lying in the $xy$-plane. In both cases, our analysis reveals the formation of 1D Ising chains, which remain decoupled in perturbation theory, resulting in a subextensive ground-state degeneracy. Our results highlight the interplay between topological order and magnetic ordering tendencies in bilayer quantum spin liquids.