Mott transition and magnetism of the triangular-lattice Hubbard model with next-nearest-neighbor hopping

Abstract

The variational cluster approximation is used to study the isotropic triangular-lattice Hubbard model at half filling, taking into account the nearest-neighbor ($t_1$) and next-nearest-neighbor ($t_2$) hopping parameters for magnetic frustrations. We determine the ground-state phase diagram of the model. In the strong correlation regime, the 120$^\circ$ N\'eel and stripe ordered phases appear, and a nonmagnetic insulating phase emerges in between. In the intermediate correlation regime, the nonmagnetic insulating phase expands to a wider parameter region, which goes into a paramagnetic metallic phase in the weak correlation regime. The critical phase boundary of the Mott metal-insulator transition is discussed in terms of the van Hove singularity evident in the calculated density of states and single-particle spectral function.