Power-law scaling of the effective population size in a branching particle system for moderate mutation-selection

Abstract

We consider a one-dimensional dyadic branching Brownian motion on $\mathbb{R}$ with positive drift $\beta \in (0,1)$, branching rate $1/2$, reflected at $0$ and killed at a boundary $L > 0$. The killing boundary $L$ is chosen so that the total population size remains approximately constant, proportional to $N \in \mathbb{N}$. This branching process models a population accumulating deleterious mutations. In the large-$N$ limit, we prove that when the typical width of the particle cloud is of order $c \log(N)$, with $c \in (0,1)$, the demographic fluctuations follow a Yaglom law on a polynomial time scale. Moreover, the limiting genealogy of the system involves only binary mergers, concentrated near the reflecting boundary. Our model is a version of the branching Brownian motion with absorption introduced by Berestycki, Berestycki, and Schweinsberg to study the effect of beneficial mutations on genealogies. In sharp contrast with their model, whose genealogy is given by a Bolthausen--Sznitman coalescent, we show that our system falls into the universality class of Kingman's coalescent.