The transfer operator for the Hecke triangle groups

Abstract

In this paper we extend the transfer operator approach to Selberg's zeta function for cofinite Fuchsian groups to the Hecke triangle groups G_q, q=3,4,..., which are non-arithmetic for q \not= 3,4,6. For this we make use of a Poincare map for the geodesic flow on the corresponding Hecke surfaces which has been constructed in arXiv:0801.3951 and which is closely related to the natural extension of the generating map for the so called Hurwitz-Nakada continued fractions. We derive simple functional equations for the eigenfunctions of the transfer operator which for eigenvalues rho =1 are expected to be closely related to the period functions of Lewis and Zagier for these Hecke triangle groups.