Estimation and prediction in sparse and unbalanced tables

Abstract

We consider the problem where we have a multi-way table of means, indexed by several factors, where each factor can have a large number of levels. The entry in each cell is the mean of some response, averaged over the observations falling into that cell. Some cells may be very sparsely populated, and in extreme cases, not populated at all. We might still like to estimate an expected response in such cells. We propose here a novel hierarchical ANOVA (HANOVA) representation for such data. Sparse cells will lean more on the lower-order interaction model for the data. These in turn could have components that are poorly represented in the data, in which case they rely on yet lower-order models. Our approach leads to a simple hierarchical algorithm, requiring repeated calculations of sub-table means of modified counts. The algorithm has shown superiority over the unshrinked methods in both simulations and real data sets.