Privacy, Ethics, and Data Access: A Case Study of the Fragile Families Challenge
Open accessThe DOI matches its Crossref record; 19 of the 19 references it cites were found in a registry or an index.
Abstract
Stewards of social data face a fundamental tension. On one hand, they want to make their data accessible to as many researchers as possible to facilitate new discoveries. At the same time, they want to restrict access to their data as much as possible to protect the people represented in the data. In this article, we provide a case study addressing this common tension in an uncommon setting: the Fragile Families Challenge, a scientific mass collaboration designed to yield insights that could improve the lives of disadvantaged children in the United States. We describe our process of threat modeling, threat mitigation, and third-party guidance. We also describe the ethical principles that formed the basis of our process. We are open about out process and the trade-offs we made in the hope that others can improve on what we have done.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Communication Theory of Secrecy Systems
Shannon, C. E. (1949) · Bell System Technical Journal · doi:10.1002/j.1538-7305.1949.tb00928.x · Cited by 9,440
Differential Privacy: A Survey of Results
Dwork, C. (2008) · Lecture Notes in Computer Science · doi:10.1007/978-3-540-79228-4_1 · Cited by 3,542
Barocas, S., & Selbst, A. D. (2014) · doi:10.2139/ssrn.2477899 · Cited by 3,048
Robust De-anonymization of Large Sparse Datasets
Narayanan, A., & Shmatikov, V. (2008) · 2008 IEEE Symposium on Security and Privacy (sp 2008) · doi:10.1109/sp.2008.33 · Cited by 2,372
Self-citationPrivacy in Context: Technology, Policy, and the Integrity of Social Life
Nissenbaum, H. (2009) · doi:10.1515/9780804772891 · Cited by 2,129
Machine Learning: An Applied Econometric Approach
Mullainathan, S., & Spiess, J. (2017) · Journal of Economic Perspectives · doi:10.1257/jep.31.2.87 · Cited by 1,989
Human Decisions and Machine Predictions
Kleinberg, J., Lakkaraju, H., Leskovec, J., Ludwig, J., & Mullainathan, S. (2018) · The Quarterly Journal of Economics · doi:10.1093/qje/qjx032 · Cited by 1,040
Fairness in Criminal Justice Risk Assessments: The State of the Art
Berk, R., Heidari, H., Jabbari, S., Kearns, M., & Roth, A. (2021) · Sociological Methods & Research · doi:10.1177/0049124118782533 · Cited by 988
“But the data is already public”: on the ethics of research in Facebook
Zimmer, M. (2010) · Ethics and Information Technology · doi:10.1007/s10676-010-9227-5 · Cited by 900
Donoho, D. (2017) · Journal of Computational and Graphical Statistics · doi:10.1080/10618600.2017.1384734 · Cited by 695
Cited by
Measuring the predictability of life outcomes with a scientific mass collaboration
Salganik, M. J., Lundberg, I., Kindel, A. T., Ahearn, C. E., Al-Ghoneim, K., Almaatouq, A., Altschul, D. M., Brand, J. E., Carnegie, N. B., Compton, R. J., et al. (2020) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1915006117 · Cited by 322
“For privacy reasons, the background data excluded genetic and geographic information (11).”
BackgroundSelf-citationComputational Social Science and Sociology
Edelmann, A., Wolff, T., Montagne, D., & Bail, C. A. (2020) · Annual Review of Sociology · doi:10.1146/annurev-soc-121919-054621 · Cited by 305
“He invited teams of researchers to develop models based on a subsample of the new wave of the dataset—commonly referred to as a training dataset—to predict outcomes of interest on the full dataset that was released later (Salganik et al. 2020, Lundberg et al. 2018).”
BackgroundTheory In, Theory Out: The Uses of Social Theory in Machine Learning for Social Science
Radford, J., & Joseph, K. (2020) · Frontiers in Big Data · doi:10.3389/fdata.2020.00018 · Cited by 58
“On the contrary, Matt Salganik and colleagues [77] use the statistical sampling and survey measurement theories baked into the Fragile Families Wellbeing Study to create the Fragile Families Challenge - a common data set computational social scientists can use to develop models predicting critical…”
BackgroundLiu, D. M., & Salganik, M. J. (2019) · Socius Sociological Research for a Dynamic World · doi:10.1177/2378023119849803 · Cited by 47
Self-citationMitigating Dataset Harms Requires Stewardship: Lessons from 1000 Papers
Peng, K., Mathur, A., & Narayanan, A. (2021) · arXiv · doi:10.48550/arxiv.2108.02922 · Cited by 31
Self-citation
Metadata
- Authors
- Lundberg, I., Narayanan, A., Levy, K., & Salganik, M. J.
- Venue
- Socius: Sociological Research for a Dynamic World
- Publisher
- SAGE Publications
- Published
- 2019-01
- Volume
- 5
- ISSN
- 2378-0231 (ISSN-L)
- OpenAlex
- W2891882871
- Cited by
- 22
- References
- 19
- FWCI
- 1.27
- Type
- journal article
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Privacy, Ethics, and Data Access: A Case Study of the Fragile Families Challenge. https://app.cochat.ai/papers/10.1177/2378023118813023
- The paper
- Lundberg, I., Narayanan, A., Levy, K., & Salganik, M. J. (2019). Privacy, Ethics, and Data Access: A Case Study of the Fragile Families Challenge. Socius: Sociological Research for a Dynamic World. https://doi.org/10.1177/2378023118813023
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.