Moir\'e Fractional Chern Insulators I: First-principles calculations and Continuum Models of Twisted Bilayer MoTe$_2$

Abstract

Recent experiments observed fractional Chern insulators (FCI) in twisted bilayer MoTe$_2$ at zero magnetic field, yet even the single-particle model of this material is controversial, leading to unreliable predictions of the experimental phase diagram as discussed in [Yu et al., 2023]. In this light, we revisit the single-particle model of twisted bilayer MoTe$_2$. Utilizing large-scale density functional theory, we calculate the band structure of twisted AA-stacked bilayer MoTe$_2$ at various twist angles relevant to experiment. We find that a band inversion occurs near $4.41^\circ$ between the second and third bands. Our ab initio band structure is in qualitative agreement with [Wang et al., 2023], but shows important differences in the remote bands and in the $\Gamma$ valley. We incorporate two higher harmonic terms into the continuum model to capture the highest 3 valence bands per valley. We confirm that the two highest valence bands per valley have opposite Chern numbers with $|C|=1$ for small angles, and also use our model to predict a variety of Chern states in the remote bands accessible by displacement field. Finally, we perform DFT calculations and build models for the AB stacking configuration. Our work serves as a foundation for accurate determination of the correlated phases in twisted bilayer MoTe$_2$.