A novel analysis of contamination in Lyman-break galaxy samples at $\boldsymbol{z\sim6-8}$: spatial correlation with intermediate-redshift galaxies at $\boldsymbol{z\sim1.3-2}$

Abstract

Potential contamination from low/intermediate-redshift galaxies, such as objects with a prominent Balmer break, affects the photometric selection of high-redshift galaxies through identification of a Lyman break. Traditionally, contamination is estimated from spectroscopic follow-up and/or simulations. Here, we introduce a novel approach to estimating contamination for Lyman-break galaxy (LBG) samples based on measuring spatial correlation with the parent population of lower redshift interlopers. We propose two conceptual approaches applicable to different survey strategies: a single large contiguous field and a survey consisting of multiple independent lines of sight. For a large single field, we compute the cross-correlation function between galaxies at redshift $z \sim 6$ and intermediate-redshift galaxies at $z \sim 1.3$. We apply the method to the CANDELS GOODS-S and XDF surveys and compare the measurement with simulated mock observations, finding that the contamination level in both cases is not measurable and lies below $5.5\%$ (at $90\%$ confidence). For random-pointing multiple field surveys, we measure instead the number count correlation between high-redshift galaxies and interlopers, as a two-point correlation analysis is not generally feasible. We show an application to the LBG samples at redshift $z \sim 8$ and the possible interloper population at $z \sim 2$ in the Brightest of Reionizing Galaxies (BoRG) survey. By comparing the Pearson correlation coefficient with the result from Monte Carlo simulations, we estimate a contamination fraction of $62^{+13}_{-39}\%$, consistent with previous estimates in the literature. These results validate the proposed approach and demonstrate its utility as an independent check of contamination in photometrically selected samples of high-redshift galaxies.