Measuring the predictability of life outcomes with a scientific mass collaboration
Open accessAbstract
How predictable are life trajectories? We investigated this question with a scientific mass collaboration using the common task method; 160 teams built predictive models for six life outcomes using data from the Fragile Families and Child Wellbeing Study, a high-quality birth cohort study. Despite using a rich dataset and applying machine-learning methods optimized for prediction, the best predictions were not very accurate and were only slightly better than those from a simple benchmark model. Within each outcome, prediction error was strongly associated with the family being predicted and weakly associated with the technique used to generate the prediction. Overall, these results suggest practical limits to the predictability of life outcomes in some settings and illustrate the value of mass collaborations in the social sciences.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Complex data analysis has become feasible to the point that our predictive models frequently outperform theoretical ones.
“First, complex data analysis has become feasible to the point that our predictive models frequently outperform theoretical ones (Salganik 2020).”
From Computational Social Science of Social Cohesion and Polarization (2026) · cites this paper as Salganik 2020
The Results describe teams using varied data-processing and statistical-learning techniques, supporting the feasibility of complex predictive analysis. The Discussion does not compare those models with theoretical models or establish that they frequently outperform them; instead, it says a simple benchmark outperformed many submissions and was only slightly worse than the best. (LLM, conf=0.83)
Experimental — accuracy not yet measuredRecent research for predicting social outcomes has shown that even with complex models and large datasets, there are strong limits to predictive performance.
“Recent research for predicting social outcomes has shown that even with complex models and large datasets, there are strong limits to predictive performance.49,83”
From Leakage and the reproducibility crisis in machine-learning-based science (2023) · cites this paper as [49]
The finding that complex ML models are not substantively better at civil war prediction is consistent with similar findings in other tasks involving predicting social outcomes, such as children's life outcomes and recidivism.
“This is consistent with similar sobering findings in other tasks involving predicting social outcomes, such as children’s life outcomes49 and recidivism.83”
From Leakage and the reproducibility crisis in machine-learning-based science (2023) · cites this paper as [49]
In the Fragile Families Challenge, state-of-the-art ML models predicting individual life outcomes (including psychological and socio-economic status) showed only marginal improvement over simple benchmark models.
“A salient illustration of this phenomenon is the “Fragile Families Challenge,” wherein diverse state-of-the-art ML models were employed to predict individual life outcomes, including psychological and socio-economic status [66]. The resultant performance exhibited only marginal improvement over simple benchmark models [66].”
From Integrating machine learning and artificial intelligence in life-course epidemiology: pathways to innovative public health solutions (2024) · cites this paper as [66]
ML models have limited prediction value when applied to complex social phenomena, in contrast to their success in physical and biological contexts.
“This limited prediction value highlights the potential limitations of ML models when applied to complex social phenomena, contrasting with their success in physical and biological contexts [66, 67].”
From Integrating machine learning and artificial intelligence in life-course epidemiology: pathways to innovative public health solutions (2024) · cites this paper as [66]
The more socially complex a problem is, the less capable machine-learning systems are of accurately or reliably classifying or predicting.
“Crucially, the more socially complex a problem is, the less capable machine-learning systems are of accurately or reliably classifying or predicting.66”
From Algorithmic injustice: A relational ethics approach (2021) · cites this paper as [66]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
The Elements of Statistical Learning
Hastie, T., Tibshirani, R., & Friedman, J. (2009) · Springer Series in Statistics · doi:10.1007/978-0-387-84858-7 · Cited by 25,970 · see citing works
Rudin, C. (2019) · Nature Machine Intelligence · doi:10.1038/s42256-019-0048-x · Cited by 10,561 · see citing works
Estimating the reproducibility of psychological science
Open Science Collaboration (2015) · Science · doi:10.1126/science.aac4716 · Cited by 8,515 · see citing works
The American Occupational Structure.
Blalock, H. M., Blau, P. M., Duncan, O. D., & Tyrée, A. (1968) · American Sociological Review · doi:10.2307/2092399 · Cited by 4,412 · see citing works
Barocas, S., & Selbst, A. D. (2014) · doi:10.2139/ssrn.2477899 · Cited by 3,048 · see citing works
Where is the land of Opportunity? The Geography of Intergenerational Mobility in the United States *
Chetty, R., Hendren, N., Kline, P., & Saez, E. (2014) · The Quarterly Journal of Economics · doi:10.1093/qje/qju022 · Cited by 2,922 · see citing works
“Clinical versus Actuarial Judgment.”
Dawes, R. M., Faust, D., & Meehl, P. E. (1989) · Science · doi:10.1126/science.2648573 · Cited by 2,298 · see citing works
Experimental Study of Inequality and Unpredictability in an Artificial Cultural Market
Salganik, M. J., Dodds, P. S., & Watts, D. J. (2006) · Science · doi:10.1126/science.1121066 · Cited by 2,219 · see citing works
Self-citationThe Efficient Market Hypothesis and Its Critics
Malkiel, B. G. (2003) · Journal of Economic Perspectives · doi:10.1257/089533003321164958 · Cited by 2,161 · see citing works
Proceedings of The 32nd International Conference on Machine Learning
Germain, M., Gregor, K., Murray, I., & Larochelle, H. (2015) · Cited by 2,020 · see citing works
Machine Learning: An Applied Econometric Approach
Mullainathan, S., & Spiess, J. (2017) · Journal of Economic Perspectives · doi:10.1257/jep.31.2.87 · Cited by 1,989 · see citing works
Fragile Families: sample and design
Reichman, N. E., Teitler, J. O., Garfinkel, I., & McLanahan, S. (2001) · Children and Youth Services Review · doi:10.1016/s0190-7409(01)00141-4 · Cited by 1,616 · see citing works
Self-citationHuman Decisions and Machine Predictions
Kleinberg, J., Lakkaraju, H., Leskovec, J., Ludwig, J., & Mullainathan, S. (2018) · The Quarterly Journal of Economics · doi:10.1093/qje/qjx032 · Cited by 1,040 · see citing works
The American Occupational Structure.
Marsh, R. M., Blau, P. M., & Duncan, O. D. (1968) · Social Forces · doi:10.2307/2575413 · Cited by 823 · see citing works
Classifier Technology and the Illusion of Progress
Hand, D. J. (2006) · Statistical Science · doi:10.1214/088342306000000060 · Cited by 723 · see citing works
Donoho, D. (2017) · Journal of Computational and Graphical Statistics · doi:10.1080/10618600.2017.1384734 · Cited by 695 · see citing works
Kleinberg, J., Ludwig, J., Mullainathan, S., & Obermeyer, Z. (2015) · American Economic Review · doi:10.1257/aer.p20151023 · Cited by 584 · see citing works
Beyond prediction: using big data for policy problems
Athey, S. (2017) · Science · doi:10.1126/science.aal4321 · Cited by 575 · see citing works
Human Decisions and Machine Predictions
Kleinberg, J., Lakkaraju, H., Leskovec, J., Ludwig, J., & Mullainathan, S. (2017) · doi:10.3386/w23180 · Cited by 545 · see citing works
Bit by Bit: Social Research in the Digital Age
King, T. (2018) · Journal of the Royal Statistical Society Series A (Statistics in Society) · doi:10.1111/rssa.12375 · Cited by 445 · see citing works
Prediction and explanation in social systems
Hofman, J. M., Sharma, A., & Watts, D. J. (2017) · Science · doi:10.1126/science.aal3856 · Cited by 429 · see citing works
Oliver, M. L., & Shapiro, T. M. (2013) · Routledge eBooks · doi:10.4324/9780203707425 · Cited by 405 · see citing works
Bit by Bit: Social Research in the Digital Age
Elgesem, D. (2020) · European Journal of Communication · doi:10.1177/0267323120922090 · Cited by 302 · see citing works
Machine Learning for Sociology
Molina, M., & Garip, F. (2019) · Annual Review of Sociology · doi:10.1146/annurev-soc-073117-041106 · Cited by 295 · see citing works
Advances in weather prediction
Alley, R. B., Emanuel, K. A., & Zhang, F. (2019) · Science · doi:10.1126/science.aav7274 · Cited by 242 · see citing works
(2018) · Cited by 230 · see citing works
Common Sense and Sociological Explanations
Watts, D. J. (2014) · American Journal of Sociology · doi:10.1086/678271 · Cited by 176 · see citing works
Self-citationPredicting armed conflict: Time to adjust our expectations?
Cederman, L.-E., & Weidmann, N. B. (2017) · Science · doi:10.1126/science.aal4483 · Cited by 156 · see citing works
Exploring Limits to Prediction in Complex Social Systems
Martin, T., Hofman, J. M., Sharma, A. K., Anderson, A., & Watts, D. J. (2016) · International World Wide Web Conference (WWW) · doi:10.1145/2872427.2883001 · Cited by 142 · see citing works
Self-citationThe Ladder: A Reliable Leaderboard for Machine Learning Competitions
Blum, A., & Hardt, M. (2015) · arXiv · doi:10.48550/arxiv.1502.04585 · Cited by 92 · see citing works
Self-citationStatistical significance of the Netflix challengePreprint
Feuerverger, A., He, Y. H., & Khatri, S. (2016) · Cited by 75 · see citing works
Bringing probability judgments into policy debates via forecasting tournaments
Tetlock, P. E., Mellers, B. A., & Scoblic, J. P. (2017) · Science · doi:10.1126/science.aal3147 · Cited by 67 · see citing works
Dynamic Modeling and Statistical Analysis of Event Times
Peña, E. A. (2006) · Statistical Science · doi:10.1214/088342306000000349 · Cited by 47 · see citing works
Augarten, S. (1985) · Cited by 34 · see citing works
Privacy, Ethics, and Data Access: A Case Study of the Fragile Families Challenge
Lundberg, I., Narayanan, A., Levy, K. E. C., & Salganik, M. J. (2019) · Socius Sociological Research for a Dynamic World · doi:10.1177/2378023118813023 · Cited by 29 · see citing works
Self-citationRisi, J., Sharma, A., Shah, R., Connelly, M., & Watts, D. J. (2019) · Nature Human Behaviour · doi:10.1038/s41562-019-0620-8 · Cited by 19 · see citing works
Self-citation
Flagged first, then by citations
Cited by
Sorted by citations
Leakage and the reproducibility crisis in machine-learning-based science
Kapoor, S., & Narayanan, A. (2023) · Patterns · doi:10.1016/j.patter.2023.100804 · Cited by 1,387 · see citing works
“Recent research for predicting social outcomes has shown that even with complex models and large datasets, there are strong limits to predictive performance.(49,83) However, results such as the better-than-human performance of ML models in perception tasks such as image classification(88,89) give the impression of ML…”
BackgroundMethodAlgorithmic injustice: A relational ethics approach
Birhane, A. (2021) · Patterns · doi:10.1016/j.patter.2021.100205 · Cited by 707 · see citing works
Early life stress and development: potential mechanisms for adverse outcomes
Smith, K. E., & Pollak, S. D. (2020) · Journal of Neurodevelopmental Disorders · doi:10.1186/s11689-020-09337-y · Cited by 500 · see citing works
Integrating explanation and prediction in computational social science
Hofman, J. M., Watts, D. J., Athey, S., Garip, F., Griffiths, T. L., Kleinberg, J., Margetts, H., Mullainathan, S., Salganik, M. J., Vazire, S., et al. (2021) · Nature · doi:10.1038/s41586-021-03659-0 · Cited by 371 · see citing works
Self-citationCollaborating During Coronavirus: The Impact of COVID-19 on the Nature of Work
DeFilippis, E., Impink, S. M., Singell, M., Polzer, J., & Sadun, R. (2020) · doi:10.3386/w27612 · Cited by 273 · see citing works
Chen, S., Yu, J., Chamouni, S., Wang, Y., & Li, Y. (2024) · BMC Medicine · doi:10.1186/s12916-024-03566-x · Cited by 73 · see citing works
“The resultant performance exhibited only marginal improvement over simple benchmark models [66].”
BackgroundMethodTuomi, I. (2022) · European Journal of Education · doi:10.1111/ejed.12531 · Cited by 66 · see citing works
Theory In, Theory Out: The Uses of Social Theory in Machine Learning for Social Science
Radford, J., & Joseph, K. (2020) · Frontiers in Big Data · doi:10.3389/fdata.2020.00018 · Cited by 58 · see citing works
Liu, D. M., & Salganik, M. J. (2019) · Socius Sociological Research for a Dynamic World · doi:10.1177/2378023119849803 · Cited by 47 · see citing works
Self-citationThe AI-mediated intimacy economy: a paradigm shift in digital interactions
Bozdağ, A. A. (2025) · AI & SOCIETY · doi:10.1007/s00146-024-02132-6 · Cited by 42 · see citing works
The language of opinion change on social media under the lens of communicative action
Monti, C., Aiello, L. M., De Francisci Morales, G., & Bonchi, F. (2022) · Scientific Reports · doi:10.1038/s41598-022-21720-4 · Cited by 32 · see citing works
Zilcha-Mano, S., & Webb, C. A. (2021) · Frontiers in Psychiatry · doi:10.3389/fpsyt.2020.609585 · Cited by 25 · see citing works
Coupland, H., Scheidwasser, N., Katsiferis, A., Davies, M., Flaxman, S., Hulvej Rod, N., Mishra, S., Bhatt, S., & Unwin, H. J. T. (2025) · BMC Public Health · doi:10.1186/s12889-025-22705-4 · Cited by 26 · see citing works
Predicting and explaining with machine learning models: Social science as a touchstone
Buchholz, O., & Grote, T. (2023) · Studies in History and Philosophy of Science · doi:10.1016/j.shpsa.2023.10.004 · Cited by 15 · see citing works
Computational Social Science of Social Cohesion and Polarization
Keijzer, M. A., Lorenz, J., & Bojanowski, M. (2026) · Computational Social Sciences · doi:10.1007/978-3-032-01373-6_1
- Authors
- Salganik, M. J., Lundberg, I., Kindel, A. T., Ahearn, C. E., Al-Ghoneim, K., Almaatouq, A., Altschul, D. M., Brand, J. E., Carnegie, N. B., Compton, R. J., Datta, D., Davidson, T., Filippova, A., Gilroy, C., Goode, B. J., Jahani, E., Kashyap, R., Kirchner, A., McKay, S., Morgan, A. C., Pentland, A., Polimis, K., Raes, L., Rigobon, D. E., Roberts, C. V., Stanescu, D. M., Suhara, Y., Usmani, A., Wang, E. H., Adem, M., Alhajri, A., AlShebli, B., Amin, R., Amos, R. B., Argyle, L. P., Baer-Bositis, L., Büchi, M., Chung, B.-R., Eggert, W., Faletto, G., Fan, Z., Freese, J., Gadgil, T., Gagné, J., Gao, Y., Halpern-Manners, A., Hashim, S. P., Hausen, S., He, G., Higuera, K., Hogan, B., Horwitz, I. M., Hummel, L. M., Jain, N., Jin, K., Jurgens, D., Kaminski, P., Karapetyan, A., Kim, E. H., Leizman, B., Liu, N., Möser, M., Mack, A. E., Mahajan, M., Mandell, N., Marahrens, H., Mercado-Garcia, D., Mocz, V., Mueller-Gastell, K., Musse, A., Niu, Q., Nowak, W., Omidvar, H., Or, A., Ouyang, K., Pinto, K. M., Porter, E., Porter, K. E., Qian, C., Rauf, T., Sargsyan, A., Schaffner, T., Schnabel, L., Schonfeld, B., Sender, B., Tang, J. D., Tsurkov, E., van Loon, A., Varol, O., Wang, X., Wang, Z., Wang, J., Wang, F., Weissman, S., Whitaker, K., Wolters, M. K., Woon, W. L., Wu, J., Wu, C., Yang, K., Yin, J., Zhao, B., Zhu, C., Brooks-Gunn, J., Engelhardt, B. E., Hardt, M., Knox, D., Levy, K., Narayanan, A., Stewart, B. M., Watts, D. J., & McLanahan, S.
- Affiliations
- Princeton University (US)
- University of California, Los Angeles (US)
- Hawaz, Riyadh 12363, Saudi Arabia;
- MIT Media Lab (US)
- University of Edinburgh (GB)
- Montana State University (US)
- University of California, Santa Cruz (US)
- Virginia Tech (US)
- Cornell University (US)
- GitHub, San Francisco, CA 94107;
- University of Washington (US)
- Massachusetts Institute of Technology (US)
- University of Oxford (GB)
- RTI International (US)
- University of Lincoln (GB)
- University of Colorado Boulder (US)
- Tilburg University (NL)
- Harvard University (US)
- Indiana University Bloomington (US)
- New York University Abu Dhabi (AE)
- Brigham Young University (US)
- Stanford University (US)
- University of Zurich (CH)
- University of Southern California (US)
- Columbia University (US)
- Department of Sociology, Stanford University, Stanford, CA 94305;
- New York University (US)
- The Ohio State University (US)
- University of Michigan (US)
- Khalifa University of Science and Technology (AE)
- Dataiku, New York, NY 10010;
- California State University, Dominguez Hills (US)
- George Washington University (US)
- Manpower Demonstration Research Corporation (US)
- Northeastern University (US)
- Syracuse University (US)
- University of Cambridge (GB)
- Expedia Group (United States) (US)
- Department of Human Development, Teachers College, Columbia University, New York, NY 10027;
- University of California, Berkeley (US)
- University of Pennsylvania (US)
- Corresponding author
- Salganik, Matthew J.
- McLanahan, Sara
- ORCID
- Salganik, Matthew J.
- Lundberg, Ian
- Kindel, Alexander T.
- Ahearn, Caitlin E.
- Almaatouq, Abdullah
- Altschul, Drew M.
- Brand, Jennie E.
- Carnegie, Nicole Bohme
- Datta, Debanjan
- Davidson, Thomas
- Gilroy, Connor
- Jahani, Eaman
- Kashyap, Ridhi
- Kirchner, Antje
- McKay, Stephen
- Morgan, Allison C.
- Pentland, Alex
- Polimis, Kivan
- Raes, Louis
- Rigobon, Daniel E.
- Roberts, Claudia V.
- Suhara, Yoshihiko
- Usmani, Adaner
- Wang, Erik H.
- Adem, Muna
- Alhajri, Abdulla
- AlShebli, Bedoor
- Argyle, Lisa P.
- Büchi, Moritz
- Fan, Zhilin
- Gadgil, Tejomay
- Gagné, Josh
- Gao, Yue
- Halpern-Manners, Andrew
- He, Guanhua
- Higuera, Kimberly
- Hogan, Bernie
- Horwitz, Ilana M.
- Hummel, Lisa M.
- Jin, Kun
- Jurgens, David
- Kaminski, Patrick
- Karapetyan, Areg
- Liu, Naijia
- Mandell, Noah
- Marahrens, Helge
- Mercado-Garcia, Diana
- Mocz, Viola
- Mueller-Gastell, Katariina
- Niu, Qiankun
- Omidvar, Hamidreza
- Ouyang, Karen
- Pinto, Katy M.
- Porter, Ethan
- Qian, Crystal
- Rauf, Tamkinat
- Sargsyan, Anahit
- Schnabel, Landon
- Schonfeld, Bryan
- Varol, Onur
- Wang, Xiafei
- Wang, Zhi
- Wang, Julia
- Whitaker, Kirstie
- Wolters, Maria K.
- Woon, Wei Lee
- Yang, Kengran
- Yin, Jingwen
- Zhao, Bingyu
- Engelhardt, Barbara E.
- Hardt, Moritz
- Knox, Dean
- Levy, Karen
- Narayanan, Arvind
- Stewart, Brandon M.
- Watts, Duncan J.
- McLanahan, Sara
- Venue
- Proceedings of the National Academy of Sciences
- Publisher
- National Academy of Sciences
- Published
- 2020-03-30
- Volume
- 117
- Issue
- 15
- Pages
- 8398-8403
- ISSN
- 0027-8424 (ISSN-L)
- 1091-6490
- PMC
- PMC7165437
- OpenAlex
- W3013182397
- Cited by
- 322
- References
- 36
- Citations by year
- 2026: 32 · 2025: 38 · 2024: 47 · 2023: 72 · 2022: 54 · 2021: 46 · 2020: 30 · 2019: 3
- FWCI
- 76.12
- Type
- journal article
- Language
- en
- Update notices
- correction (2021-12-02)
- Open access
- hybrid
- doi.org
- Licence
- Locations
- Proceedings of the National Academy of Sciences (publishedVersion, cc-by-nc-nd)
- PubMed Central (publishedVersion, cc-by-nc-nd)
- Research portal (Tilburg University) (publishedVersion, cc-by-nc-nd)
- PubMed (publishedVersion)
- Lincoln Repository (University of Lincoln) (acceptedVersion)
- Zurich Open Repository and Archive (University of Zurich) (acceptedVersion, cc-by-nc-nd)
- Edinburgh Research Explorer (University of Edinburgh) (submittedVersion, cc-by-nc-nd)
- Montana State University ScholarWorks (Montana State University) (submittedVersion, cc-by-nc-nd)
- Data Archiving and Networked Services (DANS) (submittedVersion, cc-by-nc-nd)
- RePEc: Research Papers in Economics (submittedVersion)
- DSpace@MIT (Massachusetts Institute of Technology) (submittedVersion, cc-by-nc-sa)
- DSpace@MIT (Massachusetts Institute of Technology) (submittedVersion, cc-by-nc-nd)
- eScholarship (California Digital Library) (submittedVersion)
- Figshare (submittedVersion, cc-by-nc-nd)
- Apollo (University of Cambridge) (submittedVersion, cc-by-nc-nd)
- Zurich Open Repository and Archive (University of Zurich) (submittedVersion, cc-by-nc-nd)
- Apollo
- Zurich Open Repository and Archive (University of Zurich)
- Funders
- Russell Sage Foundation (NA)
- National Science Foundation (1761810, 1750729, NA, 1750555, 1122374, IIS-1704444)
- HHS | NIH | Eunice Kennedy Shriver National Institute of Child Health and Human Development (P2CHD047879, P2CHD041022, T32HD007545, P2CHD042828)
- HHS | National Institutes of Health (R01HL133218)
- MRC Mental Health Data Pathfinder Award (MC_PC_17209)
- DOD | Defense Advanced Research Projects Agency (D17AC00003 (NGS2))
- ED | Institute of Education Sciences (R305B140009)
- Chicano Studies Research Center, UCLA (NA)
- RCUK | Engineering and Physical Sciences Research Council (EP/N510129/1)
- Topics
- Social Sciences › Demography › Family Dynamics and Relationships
- Physical Sciences › Computer Science › Computer Science Applications › Online Learning and Analytics
- Social Sciences › Decision Sciences › Information Systems and Management › Interdisciplinary Research and Collaboration
- Keywords
- life trajectories, prediction accuracy, mass collaboration, Fragile Families and Child Wellbeing Study, birth cohort, prediction error, machine learning prediction, social forecasting
- MeSH terms
- Machine Learning, Adolescent, Child, Child, Preschool, Family, Female, Humans, Infant, Male, Predictive Value of Tests, Social Sciences, Cohort Studies, Life
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
