Signatures of Z$_3$ Vestigial Potts-nematic order in van der Waals antiferromagnets

Abstract

Layered van der Waals magnets have attracted much recent attention as a promising and versatile platform for exploring intrinsic two-dimensional magnetism. Within this broader class, the transition metal phosphorous trichalcogenides $M$P$X_3$ stand out as particularly interesting, as they provide a realization of honeycomb lattice magnetism and are known to display a variety of magnetic ordering phenomena as well as superconductivity under pressure. One example, found in a number of different materials, is commensurate single-$Q$ zigzag antiferromagnetic order, which spontaneously breaks the spatial threefold $(C_3)$ rotation symmetry of the honeycomb lattice. The breaking of multiple distinct symmetries in the magnetic phase suggests the possibility of a sequence of distinct transitions as a function of temperature, and a resulting intermediate $\mathbb{Z}_3$-nematic phase which exists as a paramagnetic vestige of zigzag magnetic order -- a scenario known as vestigial ordering. Here, we report the observation of key signatures of vestigial Potts-nematic order in rhombohedral FePSe$_3$. By performing linear dichroism imaging measurements -- an ideal probe of rotational symmetry breaking -- we find that the $C_3$ symmetry is already broken above the N\'eel temperature. We show that these observations are explained by a general Ginzburg-Landau model of vestigial nematic order driven by magnetic fluctuations and coupled to residual strain. An analysis of the domain structure as temperature is lowered and a comparison with zigzag-ordered monoclinic FePS$_3$ reveals a broader applicability of the Ginzburg-Landau model in the presence of external strain, and firmly establishes the $M$P$X_3$ magnets as a new experimental venue for studying the interplay between Potts-nematicity, magnetism and superconductivity.