Partial Correlation Screening for Estimating Large Precision Matrices, with Applications to Classification

Abstract

We propose Partial Correlation Screening (PCS) as a new row-by-row approach to estimating a large precision matrix $\Omega$. To estimate the $i$-th row of $\Omega$, $1 \leq i \leq p$, PCS uses a Screen step and a Clean step. In the Screen step, PCS recruits a (small) subset of indices using a stage-wise algorithm, where in each stage, the algorithm updates the set of recruited indices by adding the index $j$ that has the largest (in magnitude) empirical partial correlation with $i$. In the Clean step, PCS re-investigates all recruited indices and use them to reconstruct the $i$-th row of $\Omega$. PCS is computationally efficient and modest in memory use: to estimate a row of $\Omega$, it only needs a few rows (determined sequentially) of the empirical covariance matrix. This enables PCS to execute the estimation of a large precision matrix (e.g., $p=10K$) in a few minutes, and open doors to estimating much larger precision matrices. We use PCS for classification. Higher Criticism Thresholding (HCT) is a recent classifier that enjoys optimality, but to exploit its full potential in practice, one needs a good estimate of the precision matrix $\Omega$. Combining HCT with any approach to estimating $\Omega$ gives a new classifier: examples include HCT-PCS and HCT-glasso. We have applied HCT-PCS to two large microarray data sets ($p = 8K$ and $10K$) for classification, where it not only significantly outperforms HCT-glasso, but also is competitive to the Support Vector Machine (SVM) and Random Forest (RF). The results suggest that PCS gives more useful estimates of $\Omega$ than the glasso. We set up a general theoretical framework and show that in a broad context, PCS fully recovers the support of $\Omega$ and HCT-PCS yields optimal classification behavior. Our proofs shed interesting light on the behavior of stage-wise procedures.