Towards the trace formula for convex-cocompact groups

Abstract

We derive a formula for the regularized trace of operators with compact spectrum which act on the space of square integrable functions on the quotient of a semisimple Liegroup of real rank one by a convex-cocompact subgroup. The sum of normalized orbital integrals associated to the hyperbolic conjugacy classes of this subgroup is an invariant distribution on the group, and we state precise conjecture about its Fouriertransform. We apply our trace formula to resolvent kernels. This yields functional equations of Selberg zeta functions and information about their singularities. The paper leaves open several interesting questions concerning the proof of the conjecture mentioned above on the one hand (this has to do with estimating the growth of the Eisenstein series as the spectral parameter turns to infinity), and the integrality of the multiplicities and of certain residues on the other hand.