A summation formula for the Rankin-Selberg monoid and a nonabelian trace formula

Abstract

Let $F$ be a number field and let $\mathbb{A}_F$ be its ring of adeles. Let $B$ be a quaternion algebra over $F$ and let $\nu:B \to F$ be the reduced norm. Consider the reductive monoid $M$ over $F$ whose points in an $F$-algebra $R$ are given by \begin{align*} M(R):=\{(\gamma_1,\gamma_2) \in (B \otimes_F R)^{2}:\nu (\gamma_1)=\nu(\gamma_2)\}. \end{align*} Motivated by an influential conjecture of Braverman and Kazhdan we prove a summation formula analogous to the Poisson summation formula for certain spaces of functions on the monoid. As an application, we define new zeta integrals for the Rankin-Selberg $L$-function and prove their basic properties. We also use the formula to prove a nonabelian twisted trace formula, that is, a trace formula whose spectral side is given in terms of automorphic representations of the unit group of $M$ that are isomorphic (up to a twist by a character) to their conjugates under a simple nonabelian Galois group.