Efficiency requires innovation

Abstract

In estimation a parameter $\theta\in{\mathbb R}$ from a sample $(x_1,\ldots,x_n)$ from a population $P_{\theta}$ a simple way of incorporating a new observation $x_{n+1}$ into an estimator $\tilde\theta_{n} = \tilde\theta_{n}(x_1,\ldots,x_n)$ is transforming $\tilde\theta_n$ to what we call the {\it jackknife extension} $\tilde\theta_{n+1}^{(e)} = \tilde\theta_{n+1}^{(e)}(x_1,\ldots,x_n,x_{n+1})$, \[\tilde\theta_{n+1}^{(e)} = \{\tilde\theta_n (x_1 ,\ldots,x_n)+ \tilde\theta_n (x_{n+1},x_2 ,\ldots,x_n) + \ldots + \tilde\theta_n (x_1 ,\ldots,x_{n-1},x_{n+1})\}/(n+1).\] Though $\tilde\theta_{n+1}^{(e)}$ lacks an innovation the statistician could expect from a larger data set, it is still better than $\tilde\theta_n$, \[{\rm var}(\tilde\theta_{n+1}^{(e)})\leq\frac{n}{n+1} {\rm var}(\tilde\theta_n).\] However, an estimator obtained by jackknife extension for all $n$ is asymptotically efficient only for samples from exponential families. For a general $P_{\theta}$, asymptotically efficient estimators require innovation when a new observation is added to the data. Some examples illustrate the concept.