Heterogeneous item populations across individuals: Consequences for the factor model, item inter-correlations, and scale validity

Abstract

The paper is devoted to the consequences of blind random selection of items from different item populations that might be based on completely uncorrelated factors for item inter-correlations and corresponding factor loadings. Based on the model of essentially parallel measurements, we explore the consequences of presenting items from different populations across individuals and items from identical populations within each individual for the factor model and item inter-correlations in the total population of individuals. Moreover, we explore the consequences of presenting items from different as well as identical item populations across and within individuals. We show that correlations can be substantial in the total population of individuals even when -- in subpopulations of individuals -- items are drawn from populations with uncorrelated factors. In order to address this challenge for the validity of a scale, we propose a method that helps to detect whether item inter-correlations result from different item populations in different subpopulations of individuals and evaluate the method by means of a simulation study. Based on the analytical results and on results from a simulation study, we provide recommendations for the detection of subpopulations of individuals responding to items from different item populations.