Anisotropy governed competition of magnetic phases in the honeycomb quantum magnet Na$_3$Ni$_2$SbO$_6$ studied by dilatometry and high-frequency ESR

Abstract

Thermodynamic properties as well as low-energy magnon excitations of $S=1$ honeycomb-layered Na$_3$Ni$_2$SbO$_6$ have been investigated by high-resolution dilatometry, static magnetisation, and high-frequency electron spin resonance studies in magnetic fields up to 16 T. At $T_{\rm N}$ = 16.5 K, there is a tricritical point separating two distinct antiferromagnetic phases AF1 and AF2 from the paramagnetic regime. In addition, our data imply short-range antiferromagnetic correlations at least up to $\sim 5\cdot T_{\rm N}$. Well below $T_{\rm N}$, the magnetic field $B_{\rm C1}\approx$ 9.5 T is needed to stabilize AF2 against AF1. The thermal expansion and magnetostriction anomalies at $T_{\rm N}$ and $B_{\rm C1}$ imply significant magnetoelastic coupling, both of which associated with a sign change of $\partial L/\partial B$. The transition at $B_{\rm C1}$ is associated with softening of the antiferromagnetic resonance modes observed in the electron spin resonance spectra. The anisotropy gap $\Delta = 360$ GHz implies considerable uniaxial anisotropy. We conclude the crucial role of axial anisotropy favoring the AF1 spin structure over the AF2 one. While the magnetostriction data disprove a simple spin-flop scenario at $B_{\rm C1}$, the nature of a second transition at $B_{\rm C2}\approx$ 13 T remains unclear. Both the sign of the magnetostriction and Gr\"uneisen analysis suggest the short-range correlations at high temperatures to be of AF2-type.