Selection and Estimation Optimality in High Dimensions with the TWIN Penalty

Abstract

We introduce a novel class of variable selection penalties called TWIN, which provides sensible data-adaptive penalization. Under a linear sparsity regime and random Gaussian designs we show that penalties in the TWIN class have a high probability of selecting the correct model and furthermore result in minimax optimal estimators. The general shape of penalty functions in the TWIN class is the key ingredient to its desirable properties and results in improved theoretical and empirical performance over existing penalties. In this work we introduce two examples of TWIN penalties that admit simple and efficient coordinate descent algorithms, making TWIN practical in large data settings. We demonstrate in challenging and realistic simulation settings with high correlations between active and inactive variables that TWIN has high power in variable selection while controlling the number of false discoveries, outperforming standard penalties.