Magnetic Signatures of a Putative Fractional Topological Insulator in Twisted MoTe2

Abstract

The interplay among electronic correlation, topology, and time-reversal-symmetry (TRS) often leads to exotic quantum states of matter, as highlighted by the discoveries of fractional Chern insulators (FCIs) in twisted bilayer MoTe2 (tMoTe2). Among the FCIs in tMoTe2, the most robust is at a hole filling factor of v=-2/3 per moir\'e unit cell. Here, employing pump-probe circular dichroism (CD) measurement on tMoTe2 at twist angles (3.9 and 3.7 degrees), we show that a correlated state at v =-4/3 exhibits an unusual Ising antiferromagnet behavior. The v =-4/3 state with no net magnetization undergoes first order phase transitions at extremely low magnetic fields of ~ 2-6 mT to partially valley polarized (PVP) states. This behavior is notably absent for all other correlated states in tMoTe2 and also disappears for v =-4/3 at higher or lower twist angles (4.0 or 3.3 degree). The observed magnetic signature is consistent with a theoretically proposed fractional topological insulator (FTI), consisting of two copies of v =-2/3 FCIs with opposite chirality in the two K valleys. The experimental results are supported by interacting continuum model calculations that reveal the extreme closeness in energy ( < 1 meV) between the putative FTI and PVP states. Our findings present a candidate FTI with TRS and call for advanced transport and imaging measurements to establish the quantized helical edge modes.