Geometry of symmetric group-based models

Abstract

Phylogenetic models have polynomial parametrization maps. For symmetric group-based models, Matsen studied the polynomial inequalities that characterize the joint probabilities in the image of these parametrizations. We employ this description for maximum likelihood estimation via numerical algebraic geometry. In particular, we explore an example where the maximum likelihood estimate does not exist, which would be difficult to discover without using algebraic methods. We also study the embedding problem for symmetric group-based models, i.e. we identify which mutation matrices are matrix exponentials of rate matrices that are invariant under a group action.