Researchers have long theorized about the processes through which
childhood experiences shape life outcomes. However, statistical models
in the social science often have poor predictive performance. Despite
this track record, policy makers are increasingly considering using
complex predictive models for high-stakes decisions in settings such as
criminal justice and child protective services.
In this talk, we present results from the Fragile Families Challenge,
a scientific mass collaboration designed to assess the limits of
predictability of life outcomes and improve our understanding of these
limits. Using data from the Fragile Fragile Families and Child
Wellbeing Study, a high-quality, birth cohort study that has followed
about 5,000 mainly disadvantaged families for the past 15 years, 457
researchers built predictive models of six life outcomes, such as a
child’s grades in school or whether the family would be evicted from
their home. Research participants in the Challenge could use any
theoretical, statistical, or machine learning approach they wished and
could draw on the more than 12,000 features that had been measured about
the child, parents, and family since the birth of the child. All
predictions were evaluated on held-out data. Our empirical results have
implications for social science theory, data, and methods and for
algorithmic decision-making in high-stakes social settings.