Oscillatory evolutionarily stable state and limit cycle in replicator dynamics

Abstract

The idea of evolutionarily stable state (ESS) of a population is a cornerstone of evolutionary game theory; moreover, it coincides with the game-theoretic concept of Nash equilibrium. Such a state corresponds to a strategy adopted by the population such that a rare mutant strategy cannot invade the population. In parallel, the dynamical formulation of evolutionary game theory -- particularly through replicator dynamics embodying the tenet of survival of the fittest -- provides a framework for modelling frequency-dependent selection over time. While it is well known that an ESS corresponds to stable fixed point in replicator dynamics, the evolutionary game-theoretic characterization of limit cycles is unknown. Here we fill this lacuna by defining oscillatory ESS (OESS) which we prove to be a stable limit cycle. We also show when an OESS is unique and if there are multiple OESSes, then what their locations are in the phase space.