Efficient Thresholded Correlation using Truncated Singular Value Decomposition

Abstract

Efficiently computing a subset of a correlation matrix consisting of values above a specified threshold is important to many practical applications. Real-world problems in genomics, machine learning, finance other applications can produce correlation matrices too large to explicitly form and tractably compute. Often, only values corresponding to highly-correlated vectors are of interest, and those values typically make up a small fraction of the overall correlation matrix. We present a method based on the singular value decomposition (SVD) and its relationship to the data covariance structure that can efficiently compute thresholded subsets of very large correlation matrices.