Search for the double Dalitz decays $\eta/\eta' \to e^+e^-\mu^+\mu^-$ and $\eta' \to \mu^+\mu^-\mu^+\mu^-$

Abstract

Using a data sample of $(10087 \pm 44) \times {10^{6}}$ $J/{\psi}$ events collected with the BESIII detector, we search for the decays $\eta/\eta'\to e^+e^-\mu^+\mu^-$ and $\eta' \to \mu^+\mu^-\mu^+\mu^-$ via the radiative decays $J/{\psi}\to\gamma\eta$/$\gamma\eta'$. No excess of events over expected background is observed for any of the decays of interest. At 90% confidence level, we report the first upper limits on the branching fractions of $\eta' \to e^{+}e^{-}\mu^{+}\mu^{-}$ and $\eta' \to \mu^{+}\mu^{-}\mu^{+}\mu^{-}$ to be $ 1.75 \times {10^{-6}}$ and $5.28 \times {10^{-7}}$, respectively. In addition, we set an upper limit on the branching fraction of $\eta \to e^{+}e^{-}\mu^{+}\mu^{-}$ to be $6.88 \times {10^{-6}}$, which improves the previous result by about two orders of magnitude.