$\mathcal{Z}_2$ classification for a novel antiferromagnetic topological insulating phase in three-dimensional topological Kondo insulator

Abstract

Antiferromagnetic topological insulator (AFTI) is a topological matter that breaks time-reversal symmetry. Since its proposal, explorations of AFTI in strong-correlated systems are still lacking. In this paper, we show for the first time that a novel AFTI phase can be realized in three-dimensional topological Kondo insulator (TKI). In a wide parameter region, the ground states of TKI undergo a second-order transition to antiferromagnetic insulating phases which conserve a combined symmetry of time reversal and a lattice translation, allowing us to derive a $\mathcal{Z}_2$-classification formula for these states. By calculating the $\mathcal{Z}_2$ index, the antiferromagnetic insulating states are classified into (AFTI) or non-topological antiferromagnetic insulator (nAFI) in different parameter regions. On the antiferromagnetic surfaces in AFTI, we find topologically protected gapless Dirac cones inside the bulk gap, leading to metallic Fermi rings exhibiting helical spin texture with weak spin-momentum locking. Depending on model parameters, the magnetic transitions take place either between AFTI and strong topological insulator, or between nAFI and weak topological insulator. By varying some model parameters, we find a topological transition between AFTI and nAFI, driving by closing of bulk gap. Our work may account for the pressure-induced magnetism in TKI compound SmB$_6$, and helps to explore richer AFTI phases in heavy-fermion systems as well as in other strong-correlated systems.