Splines and Operators: Part II - Resolution of Inverse Problems

Abstract

We present a unified treatment of the data-fitting problems with quadratic (Hilbertian) regularization that arise in approximation theory, machine learning, engineering, and statistics. The generic problem is an optimization in the continuum: given $M$ linear measurements of an unknown function, one minimizes a convex data-fidelity term that is penalized by $\|\mathrm{L} f\|^2_{L_2}$, where $\mathrm{L} $ is a differential-like (spline-admissible) operator. The natural domain of the minimization is the native space $\mathcal{H}_\mathrm{L}$ of $\mathrm{L} $ - the largest Hilbert space over which the regularizer remains meaningful - for which we provide a concise and directly usable characterization. Central to our formulation is a measurement-adapted (conditioned) basis that decouples the null-space and spline components of the solution. This basis serves in three places. First, it tightens the formulation of our representer theorem, which confines the unique solution to an $M$-dimensional space without any orthogonality constraint on the coefficients. Second, it streamlines the implementation, as the exact discretization of the problem becomes an unconstrained convex program with closed-form solutions in the least-squares case, which covers ridge regression, Tikhonov regularization, and smoothing splines. Third, it makes the bridge with the minimum-mean-square-error estimation of generalized Gaussian processes, which is the topic of a companion paper.