Empirical risk minimization is optimal for the convex aggregation problem

Abstract

Let $F$ be a finite model of cardinality $M$ and denote by $\operatorname {conv}(F)$ its convex hull. The problem of convex aggregation is to construct a procedure having a risk as close as possible to the minimal risk over $\operatorname {conv}(F)$. Consider the bounded regression model with respect to the squared risk denoted by $R(\cdot)$. If ${\widehat{f}}_n^{\mathit{ERM-C}}$ denotes the empirical risk minimization procedure over $\operatorname {conv}(F)$, then we prove that for any $x>0$, with probability greater than $1-4\exp(-x)$, \[R({\widehat{f}}_n^{\mathit{ERM-C}})\leq\min_{f\in \operatorname {conv}(F)}R(f)+c_0\max \biggl(\psi_n^{(C)}(M),\frac{x}{n}\biggr),\] where $c_0>0$ is an absolute constant and $\psi_n^{(C)}(M)$ is the optimal rate of convex aggregation defined in (In Computational Learning Theory and Kernel Machines (COLT-2003) (2003) 303-313 Springer) by $\psi_n^{(C)}(M)=M/n$ when $M\leq \sqrt{n}$ and $\psi_n^{(C)}(M)=\sqrt{\log (\mathrm{e}M/\sqrt{n})/n}$ when $M>\sqrt{n}$.