Super Ensemble Learning Using the Highly-Adaptive-Lasso

Abstract

We introduce the Meta Highly-Adaptive-Lasso Minimum Loss Estimator (M-HAL-MLE), a novel ensemble approach for estimating functional parameters of realistically modeled data distribution from independent and identically distributed observations. Given $J$ initial estimators, candidate ensembles are generated by finite-sectional-variation cadlag functions. Using $V$-fold cross-validation, the M-HAL-MLE selects the optimal cadlag ensemble minimizing the cross-validated empirical risk, with the sectional variation bound as a tuning parameter. The final estimator, M-HAL super-learner, is obtained by averaging ensemble compositions across folds. In contrast, the oracle ensemble and oracle estimator are defined by minimizing the population excess risk relative to the true function. We establish following theoretical properties: 1) the M-HAL super-learner converges to the oracle estimator at rate $n^{-2/3}$ in excess risk, up to log-n factors; 2) by appropriate undersmoothing, target features of the M-HAL super-learner are asymptotically linear for corresponding target features of the oracle estimator; 3) the excess risk between the oracle estimator and true function, along with the difference between their target features, is generally second-order. Simulations validate the theoretical results, demonstrating effectiveness in high-dimensional settings. We further illustrate the method in a real-data application involving mediation analysis of functional MRI from human pain studies.