An Empirical Comparison of Methods for Quantifying the Similarity of Numeric Datasets

Abstract

Methods for quantifying the similarity of datasets are relevant in applications where two or more datasets, or their underlying distributions, need to be compared, ranging from two- and k-sample testing to applications in machine learning and synthetic data generation. Many methods for quantifying the similarity of datasets are available from the literature, but due to the lack of neutral comparison studies, it is unclear which method to choose when. Here, 36 methods applicable to continuous data are compared across various scenarios, including two or more datasets drawn from different distributions. Several deviations between datasets are considered, including shift and scale alternatives or differences in higher moments. An overall method ranking is established based on the methods' abilities to differentiate between datasets from different distributions, combined with computational aspects. Based on this, concrete decision rules for finding the best method based on characteristics of the datasets are determined. Moreover, combinations of four to six methods are proposed in the two-sample case such that in 90% to 95% of the considered scenarios, at least one of these methods is almost as good as the best method. In the multi-sample case, a combination of two to three methods is proposed analogously.