Epsilon dichotomy for linear models: the Archimedean case

Abstract

Let $G=\mathrm{GL}_{2n}(\mathbb{R})$ or $G=\mathrm{GL}_n(\mathbb{H})$ and $H=\mathrm{GL}_n(\mathbb{C})$ regarded as a subgroup of $G$. Here, $\mathbb{H}$ is the quaternion division algebra over $\mathbb{R}$. For a character $\chi$ on $\mathbb{C}^\times$, we say that an irreducible smooth admissible moderate growth representation $\pi$ of $G$ is $\chi_H$-distinguished if $\mathrm{Hom}_H(\pi, \chi\circ\det_H)\neq0$. We compute the root number of a $\chi_H$-distinguished representation $\pi$ twisted by the representation induced from $\chi$. This proves an Archimedean analogue of the conjecture by Prasad and Takloo-Bighash (J. Reine Angew. Math., 2011). The proof is based on the analysis of the contribution of $H$-orbits in a flag manifold of $G$ to the Schwartz homology of principal series representations. A large part of the argument is developed for general real reductive groups of inner type. In particular, we prove that the Schwartz homology $H_\ast(H, \pi\otimes\chi)$ is finite-dimensional and hence it is Hausdorff for a reductive symmetric pair $(G, H)$ and a finite-dimensional representation $\chi$ of $H$.