Machine Learning Models Trained on a Longitudinal Survey Dataset Poorly Predict Hundreds of Life Outcomes

Abstract

Understanding the predictability of life outcomes is important for public policy and science, but existing studies focus on a limited number of outcomes and therefore struggle to describe the broad patterns of predictability. We developed a high-throughput approach that uses a single dataset to study the predictability of many future outcomes using past data. We deployed this approach with hundreds of outcomes and thousands of machine learning models per outcome in the Future of Families and Child Wellbeing Study. Three findings emerged. First, most outcomes were not very predictable, but some were much more predictable than others. Second, even for the most predictable outcomes, our complex prediction procedure typically exhibited little skill relative to a simple procedure. Third, we found modest patterns in how predictability varied across outcomes. We conclude by discussing scope conditions and how future research might address them.