Efficient searches of small signals across two-channel noisy data: a challenge in Big Data observations

Abstract

Searching for novel small signals in noisy data is preferably pursued by correlation in two or more independently operating channels. Signals of potential interest exist in tails beyond $κσ$, where $κ$ denotes a multiple of the standard deviation $σ$ of the data. Since moving data is a major cost factor in Big Data analysis and heterogeneous computing more generally, efficiency may be optimized by restricting the correlations computation to the tails of two-channel data exceeding $κσ$. Already, a moderate value $κ\gtrsim2$ realizes a data-reduction by at least an order of magnitude. Here, we study this approach using a novel {\it Excess Probability Ratio} (EPR), correlating Boolean data resulting from tails beyond a cut-off $κσ$. We compare and rank EPR performance against conventional direct cross-correlation (DCC) and Pearson coefficient (PC), applicable to the original data with no cut-off. This benchmark is performed over different combinations of background noise Gaussian, Poisson and Uniform and signals Gaussian, Poisson, Uniform, Chirps and Sine waves. Results show performance of EPR to be comparable to that of PC, providing a new approach for significant improvements in efficiency with essentially no loss of sensitivity, relevant to the present era of Big Data observatories.