Dynamical Response of the Kitaev Spin Liquid under Third-Nearest-Neighbor Heisenberg Interaction

Abstract

Motivated by growing evidence for the significance of the third-nearest-neighbor Heisenberg ($J_3$) interaction in candidate Kitaev materials, we investigate the dynamical properties of the Kitaev spin liquid (KSL) under a $J_3$ perturbation, focusing on its spin dynamical structure factor (DSF) and Raman scattering. Within a self-consistent parton mean-field plus random-phase approximation framework, we find that $J_3$ induces coherent, paramagnon-like collective modes that coexist with a high-energy Majorana continuum in the spin DSF. The softening of these modes with increasing $|J_3|$ signals a quantum phase transition to magnetic order. Remarkably, magnetic ordering sets in at a common critical $J_3$ for both ferromagnetic ($K<0$) and antiferromagnetic ($K>0$) Kitaev models, with the resulting ordered states forming exact dual pairs under a four-sublattice duality transformation that maps $(K,J_3) \rightarrow (-K,J_3)$. An external magnetic field further softens the preexisting paramagnon modes, thereby enhancing magnetic order. Perturbative Raman calculations show that while the Kitaev-like Raman vertex probes only itinerant matter Majorana fermions, the response from the $J_3$-like vertex features both matter Majoranas and visons. Four-vison excitations produce a sharp peak accompanied by a two-fermion continuum, whereas two-vison excitations yield a continuum closely resembling the single-matter-fermion density of states. These results provide a unified perspective on the dynamical signatures of $J_3$-perturbed KSL and are helpful for interpreting experimental spectra in candidate Kitaev materials with sizable $J_3$ interactions.