On the explanatory power of principal components

Abstract

We show that if we have an orthogonal base ($u_1,\ldots,u_p$) in a $p$-dimensional vector space, and select $p+1$ vectors $v_1,\ldots, v_p$ and $w$ such that the vectors traverse the origin, then the probability of $w$ being to closer to all the vectors in the base than to $v_1,\ldots, v_p$ is at least 1/2 and converges as $p$ increases to infinity to a normal distribution on the interval [-1,1]; i.e., $\Phi(1)-\Phi(-1)\approx0.6826$. This result has relevant consequences for Principal Components Analysis in the context of regression and other learning settings, if we take the orthogonal base as the direction of the principal components.