Three-dimensional nematic spin liquid in the stacked triangular lattice 6H-B structure

Abstract

Recently, a number of experiments indicate the possible presence of spin liquid phases in quantum magnets with spin-1/2 and spin-1 moments sitting on triangular-lattice-based structures in Ba3CuSb2O9 and Ba3NiSb2O9 respectively. In relation to these experiments, several theoretical proposals have been made for spin liquid phases and spin-liquid-like behaviours on the stacked triangular lattice. While the crystal structures of these materials are currently under debate, it is nonetheless interesting to understand possible spin liquid phases on such frustrated lattices. In this work, we apply Schwinger boson mean-field theory and projective symmetry group (PSG) analysis to investigate spin liquid phases on the fully three-dimensional 6H-B structure, in contrast to previous works that considered two-dimensional systems. We find that a nematic Z2 spin liquid phase, where the lattice-rotational symmetry is spontaneously broken, is the most promising spin liquid phase that is consistent with spiral magnetic ordering in the classical limit. We discuss the implications of our results to future theoretical and experimental works.