Successes and Struggles with Computational Reproducibility: Lessons from the Fragile Families Challenge
Open accessPublished version of a preprint
Abstract
Reproducibility is fundamental to science, and an important component of reproducibility is computational reproducibility: the ability of a researcher to recreate the results in a published paper using the original author's raw data and code. Although most people agree that computational reproducibility is important, it is still difficult to achieve in practice. In this paper, we describe our approach to enabling computational reproducibility for the 12 papers in this special issue of Socius about the Fragile Families Challenge. Our approach draws on two tools commonly used by professional software engineers but not widely used by academic researchers: software containers (e.g., Docker) and cloud computing (e.g., Amazon Web Services). These tools enabled us to standardize the computing environment around each submission, which will ease computational reproducibility both today and in the future. Drawing on our successes and struggles, we conclude with recommendations to authors and journals.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Rudin, C. (2019) · Nature Machine Intelligence · doi:10.1038/s42256-019-0048-x · Cited by 10,561 · see citing works
Wolpert, D. H. (1992) · Neural Networks · doi:10.1016/s0893-6080(05)80023-1 · Cited by 7,951 · see citing works
(2012) · American Economic Review · doi:10.1257/aer.102.7.i · Cited by 7,275 · see citing works
Refactoring: Improving the Design of Existing Code
Fowler, M. (2002) · Lecture notes in computer science · doi:10.1007/3-540-45672-4_31 · Cited by 5,615 · see citing works
Making the most of statistical analyses: improving interpretation and presentation
King, G., Tomz, M., & Wittenberg, J. (2000) · American Journal of Political Science · doi:10.2307/2669316 · Cited by 3,232 · see citing works
Singularity: Scientific containers for mobility of compute
Kurtzer, G. M., Sochat, V., & Bauer, M. W. (2017) · PLOS ONE · doi:10.1371/journal.pone.0177459 · Cited by 2,740 · see citing works
Promoting an open research culture
Nosek, B. A., Alter, G., Banks, G. C., Borsboom, D., Bowman, S. D., Breckler, S. J., Buck, S., Chambers, C. D., Chin, G., Christensen, G., et al. (2015) · Science · doi:10.1126/science.aab2374 · Cited by 2,708 · see citing works
Proceedings of The 32nd International Conference on Machine Learning
Germain, M., Gregor, K., Murray, I., & Larochelle, H. (2015) · Cited by 2,020 · see citing works
Breiman, L. (1996) · Machine Learning · doi:10.1023/a:1018046112532 · Cited by 1,966 · see citing works
Knuth, D. (1984) · Computer/law journal · doi:10.1093/comjnl/27.2.97 · Cited by 1,863 · see citing works
Reproducible research in computational science
Peng, R. D. (2011) · Science · doi:10.1126/science.1213847 · Cited by 1,449 · see citing works
Numerical recipes: The art of scientific computing
DAVIDROPER, L. (1987) · Bulletin of Mathematical Biology · doi:10.1016/s0092-8240(87)90019-x · Cited by 1,428 · see citing works
What does research reproducibility mean?
Goodman, S. N., Fanelli, D., & Ioannidis, J. P. A. (2016) · Science Translational Medicine · doi:10.1126/scitranslmed.aaf5027 · Cited by 1,293 · see citing works
PS : political science & politics
(n.d.) · RWTH Publications (RWTH Aachen) · Cited by 1,230 · see citing works
An introduction to Docker for reproducible research
Boettiger, C. (2015) · ACM SIGOPS Operating Systems Review · doi:10.1145/2723872.2723882 · Cited by 1,059 · see citing works
Ten Simple Rules for Reproducible Computational Research
Sandve, G. K., Nekrutenko, A., Taylor, J., & Hovig, E. (2013) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1003285 · Cited by 880 · see citing works
Children and Youth Services Review
Zhan, M., & Sherraden, M. W. (2011) · Cited by 746 · see citing works
Kidwell, M. C., Lazarević, L. B., Baranski, E., Hardwicke, T. E., Piechowski, S., Falkenberg, L.-S., Kennett, C., Slowik, A., Sonnleitner, C., Hess-Holden, C., et al. (2016) · PLOS Biology · doi:10.1371/journal.pbio.1002456 · Cited by 700 · see citing works
Donoho, D. (2017) · Journal of Computational and Graphical Statistics · doi:10.1080/10618600.2017.1384734 · Cited by 695 · see citing works
Artificial intelligence faces reproducibility crisis
Hutson, M. (2018) · Science · doi:10.1126/science.359.6377.725 · Cited by 573 · see citing works
WaveLab and Reproducible Research
Buckheit, J. B., & Donoho, D. L. (1995) · Lecture notes in statistics · doi:10.1007/978-1-4612-2544-7_5 · Cited by 551 · see citing works
Repeatability of published microarray gene expression analyses
Ioannidis, J. P. A., Allison, D. B., Ball, C. A., Coulibaly, I., Cui, X., Culhane, A. C., Falchi, M., Furlanello, C., Game, L., Jurman, G., et al. (2009) · Nature Genetics · doi:10.1038/ng.295 · Cited by 551 · see citing works
The poor availability of psychological research data for reanalysis.
Wicherts, J. M., Borsboom, D., Kats, J., & Molenaar, D. (2006) · American Psychologist · doi:10.1037/0003-066x.61.7.726 · Cited by 543 · see citing works
The Availability of Research Data Declines Rapidly with Article Age
Vines, T. H., Albert, A. Y. K., Andrew, R. L., Débarre, F., Bock, D. G., Franklin, M. T., Gilbert, K. J., Moore, J.-S., Renaut, S., & Rennison, D. J. (2014) · Current Biology · doi:10.1016/j.cub.2013.11.014 · Cited by 482 · see citing works
King, G. (1995) · PS Political Science & Politics · doi:10.1017/s1049096500057607 · Cited by 455 · see citing works
Bit by Bit: Social Research in the Digital Age
King, T. (2018) · Journal of the Royal Statistical Society Series A (Statistics in Society) · doi:10.1111/rssa.12375 · Cited by 445 · see citing works
Reproducibility vs. Replicability: A Brief History of a Confused Terminology
Pleßer, H. E. (2018) · Frontiers in Neuroinformatics · doi:10.3389/fninf.2017.00076 · Cited by 426 · see citing works
International Studies Perspectives
(2016) · International Studies Perspectives · doi:10.1111/(issn)1528-3585 · Cited by 425 · see citing works
Wicherts, J. M., Bakker, M., & Molenaar, D. (2011) · PLoS ONE · doi:10.1371/journal.pone.0026828 · Cited by 424 · see citing works
Enhancing reproducibility for computational methods
Stodden, V., Mcnutt, M. K., Bailey, D. H., Deelman, E., Gil, Y., Hanson, B., Heroux, M. A., Ioannidis, J. P. A., & Taufer, M. (2016) · Science · doi:10.1126/science.aah6168 · Cited by 414 · see citing works
Empirical Study of Data Sharing by Authors Publishing in PLoS Journals
Savage, C. J., & Vickers, A. J. (2009) · PLoS ONE · doi:10.1371/journal.pone.0007078 · Cited by 411 · see citing works
An empirical analysis of journal policy effectiveness for computational reproducibility
Stodden, V., Seiler, J., & Ma, Z. (2018) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1708290115 · Cited by 369 · see citing works
Statistical Analyses and Reproducible Research
Gentleman, R. C., & Lang, D. T. (2007) · Journal of Computational and Graphical Statistics · doi:10.1198/106186007x178663 · Cited by 364 · see citing works
Public Availability of Published Research Data in High-Impact Journals
Alsheikh-Ali, A. A., Qureshi, W., Al-Mallah, M. H., & Ioannidis, J. P. A. (2011) · PLoS ONE · doi:10.1371/journal.pone.0024357 · Cited by 363 · see citing works
Hardwicke, T. E., Mathur, M. B., MacDonald, K., Nilsonne, G., Banks, G. C., Kidwell, M. C., Hofelich Mohr, A., Clayton, E., Yoon, E. J., Henry Tessler, M., et al. (2018) · Royal Society Open Science · doi:10.1098/rsos.180448 · Cited by 336 · see citing works
Jeremy Freese, A., & Peterson, D. (2017) · Annual Review of Sociology · doi:10.1146/annurev-soc-060116-053450 · Cited by 323 · see citing works
Measuring the predictability of life outcomes with a scientific mass collaboration
Salganik, M. J., Lundberg, I., Kindel, A. T., Ahearn, C. E., Al-Ghoneim, K., Almaatouq, A., Altschul, D. M., Brand, J. E., Carnegie, N. B., Compton, R. J., et al. (2020) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1915006117 · Cited by 322 · see citing works
Self-citationPublish your computer code: it is good enough
Barnes, N. (2010) · Nature · doi:10.1038/467753a · Cited by 317 · see citing works
Reproducible Epidemiologic Research
Peng, R. D., Dominici, F., & Zeger, S. L. (2006) · American Journal of Epidemiology · doi:10.1093/aje/kwj093 · Cited by 292 · see citing works
Gilbert, K. J., Andrew, R. L., Bock, D. G., Franklin, M. T., Kane, N. C., Moore, J., Moyers, B. T., Renaut, S., Rennison, D. J., Veen, T., et al. (2012) · Molecular Ecology · doi:10.1111/j.1365-294x.2012.05754.x · Cited by 290 · see citing works
Reproducible research in signal processing
Vandewalle, P., Kovacevic, J., & Vetterli, M. (2009) · IEEE Signal Processing Magazine · doi:10.1109/msp.2009.932122 · Cited by 276 · see citing works
Eich, E. (2013) · Psychological Science · doi:10.1177/0956797613512465 · Cited by 258 · see citing works
Repeatability in computer systems research
Collberg, C., & Proebsting, T. A. (2016) · Communications of the ACM · doi:10.1145/2812803 · Cited by 252 · see citing works
Marwick, B. (2016) · Journal of Archaeological Method and Theory · doi:10.1007/s10816-015-9272-9 · Cited by 246 · see citing works
THE MEANING OF FAILED REPLICATIONS: A REVIEW AND PROPOSAL
Clemens, M. A. (2015) · Journal of Economic Surveys · doi:10.1111/joes.12139 · Cited by 246 · see citing works
Reproducible research and Biostatistics
Peng, R. D. (2009) · Biostatistics · doi:10.1093/biostatistics/kxp014 · Cited by 244 · see citing works
Most computational hydrology is not reproducible, so is it really science?
Hutton, C., Wagener, T., Freer, J. E., Han, D., Duffy, C., & Arheimer, B. (2016) · Water Resources Research · doi:10.1002/2016wr019285 · Cited by 233 · see citing works
Practical Tips for Ethical Data Sharing
Meyer, M. N. (2018) · Advances in Methods and Practices in Psychological Science · doi:10.1177/2515245917747656 · Cited by 232 · see citing works
Naudet, F., Sakarovitch, C., Janiaud, P., Cristea, I., Fanelli, D., Moher, D., & Ioannidis, J. P. A. (2018) · BMJ · doi:10.1136/bmj.k400 · Cited by 227 · see citing works
Electronic documents give reproducible research a new meaning
Claerbout, J. F., & Karrenbach, M. (1992) · Second International Meeting for Applied Geoscience & Energy · doi:10.1190/1.1822162 · Cited by 224 · see citing works
Reproducible research can still be wrong: Adopting a prevention approach
Leek, J. T., & Peng, R. D. (2015) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1421412111 · Cited by 193 · see citing works
Replication Standards for Quantitative Social Science
Jeremy Freese, A. (2007) · Sociological Methods & Research · doi:10.1177/0049124107306659 · Cited by 192 · see citing works
Journal of Computational Neuroscience
(2013) · Journal of Computational Neuroscience · doi:10.1007/10827.1573-6873 · Cited by 192 · see citing works
Gil, Y., David, C. H., Demir, İ., Essawy, B. T., Fulweiler, R. W., Goodall, J. L., Karlstrom, L., Lee, H., Mills, H. J., Oh, J., et al. (2016) · Earth and Space Science · doi:10.1002/2015ea000136 · Cited by 182 · see citing works
An invitation to reproducible computational research
Donoho, D. L. (2010) · Biostatistics · doi:10.1093/biostatistics/kxq028 · Cited by 167 · see citing works
Tools and techniques for computational reproducibility
Piccolo, S., & Frampton, M. (2016) · GigaScience · doi:10.1186/s13742-016-0135-4 · Cited by 167 · see citing works
Openness in Political Science: Data Access and Research Transparency
Lupia, A., & Elman, C. (2013) · PS Political Science & Politics · doi:10.1017/s1049096513001716 · Cited by 161 · see citing works
Practical Computational Reproducibility in the Life Sciences
Grüning, B. A., Chilton, J. M., Köster, J., et al. (2017) · Cell Systems · doi:10.1016/j.cels.2018.03.014 · Cited by 158 · see citing works
Please Stop Explaining Black Box Models for High Stakes Decisions.Preprint
Rudin, C. D. (2018) · arXiv (Cornell University) · Cited by 138 · see citing works
Terminologies for Reproducible ResearchPreprint
Barba, L. A. (2018) · arXiv (Cornell University) · doi:10.4230/lipics.giscience.2021.ii.2 · Cited by 136 · see citing works
Computing environments for reproducibility: Capturing the “Whole Tale”
Brinckman, A., Chard, K., Gaffney, N. I., Hategan, M., Jones, M. B., Kowalik, K., Kulasekaran, S., Ludäscher, B., Mecum, B., Nabrzyski, J., et al. (2018) · Future Generation Computer Systems · doi:10.1016/j.future.2017.12.029 · Cited by 134 · see citing works
ReproZip: Computational Reproducibility With Ease
Chirigati, F., Rampin, R., Shasha, D., & Freire, J. (2016) · Proceedings of the 2016 International Conference on Management of Data · doi:10.1145/2882903.2899401 · Cited by 128 · see citing works
What incentives increase data sharing in health and medical research? A systematic review
Rowhani-Farid, A., Allen, M. J., & Barnett, A. G. (2017) · Research Integrity and Peer Review · doi:10.1186/s41073-017-0028-9 · Cited by 127 · see citing works
In silico research in the era of cloud computing
Dudley, J. T., & Butte, A. J. (2010) · Nature Biotechnology · doi:10.1038/nbt1110-1181 · Cited by 125 · see citing works
Sustainable computational science: the ReScience initiative
Rougier, N. P., Hinsen, K., Alexandre, F., Arildsen, T., Barba, L. A., Benureau, F., Brown, C. T., de Buyl, P., Çağlayan, O., Davison, A. P., et al. (2017) · PeerJ Computer Science · doi:10.7717/peerj-cs.142 · Cited by 123 · see citing works
Journal of Open Research Software
(2022) · Journal of Open Research Software · doi:10.5334/jors · Cited by 121 · see citing works
Miłkowski, M., Hensel, W. M., & Hohol, M. (2018) · Journal of Computational Neuroscience · doi:10.1007/s10827-018-0702-z · Cited by 110 · see citing works
Packaging Data Analytical Work Reproducibly Using R (and Friends)
Marwick, B., Boettiger, C., & Mullen, L. A. (2017) · The American Statistician · doi:10.1080/00031305.2017.1375986 · Cited by 109 · see citing works
Toward standard practices for sharing computer code and programs in neuroscience
Eglen, S. J., Marwick, B., Halchenko, Y. O., Hanke, M., Sufi, S., Gleeson, P., Silver, R. A., Davison, A. P., Lanyon, L., Abrams, M. B., et al. (2017) · Nature Neuroscience · doi:10.1038/nn.4550 · Cited by 108 · see citing works
Easterbrook, S. (2014) · Nature Geoscience · doi:10.1038/ngeo2283 · Cited by 98 · see citing works
On reproducible econometric research
Koenker, R. W., & Zeileis, A. (2009) · Journal of Applied Econometrics · doi:10.1002/jae.1083 · Cited by 95 · see citing works
Re-run, Repeat, Reproduce, Reuse, Replicate: Transforming Code into Scientific Contributions
Benureau, F., & Rougier, N. P. (2018) · Frontiers in Neuroinformatics · doi:10.3389/fninf.2017.00069 · Cited by 94 · see citing works
A statistical definition for reproducibility and replicabilityPreprint
Patil, P., Peng, R. D., & Leek, J. T. (2016) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/066803 · Cited by 90 · see citing works
Mangul, S., Mosqueiro, T., Abdill, R. J., Duong, D., Mitchell, K., Sarwal, V., Hill, B., Brito, J., Littman, R. J., Statz, B., et al. (2019) · PLOS Biology · doi:10.1371/journal.pbio.3000333 · Cited by 90 · see citing works
The role of data/code archives in the future of economic research
Anderson, R. G., Greene, W. H., McCullough, B. D., & Vinod, H. D. (2008) · Journal of Economic Methodology · doi:10.1080/13501780801915574 · Cited by 87 · see citing works
Has open data arrived at the <i>British Medical Journal (BMJ)</i> ? An observational study
Rowhani-Farid, A., & Barnett, A. G. (2016) · BMJ Open · doi:10.1136/bmjopen-2016-011784 · Cited by 84 · see citing works
Virtual Appliances, Cloud Computing, and Reproducible Research
Howe, B. (2012) · Computing in Science & Engineering · doi:10.1109/mcse.2012.62 · Cited by 83 · see citing works
How to make replication the norm
Gertler, P., Galiani, S., & Romero, M. (2018) · Nature · doi:10.1038/d41586-018-02108-9 · Cited by 81 · see citing works
Hardwicke, T. E., & Ioannidis, J. P. A. (2018) · PLOS ONE · doi:10.1371/journal.pone.0201856 · Cited by 78 · see citing works
Reproducing Statistical Results
Stodden, V. (2015) · Annual Review of Statistics and Its Application · doi:10.1146/annurev-statistics-010814-020127 · Cited by 77 · see citing works
When open data is a Trojan Horse: The weaponization of transparency in science and governance
Levy, K. E. C., & Johns, D. M. (2016) · Big Data & Society · doi:10.1177/2053951715621568 · Cited by 76 · see citing works
Konkol, M., Kray, C., & Pfeiffer, M. (2018) · International Journal of Geographical Information Systems · doi:10.1080/13658816.2018.1508687 · Cited by 74 · see citing works
An Introduction to Rocker: Docker Containers for R
Boettiger, C., & Eddelbuettel, D. (2017) · The R Journal · doi:10.32614/rj-2017-065 · Cited by 67 · see citing works
Learning from the Past: Approaches for Reproducibility in Computational Neuroscience
Crook, S. M., Davison, A. P., & Pleßer, H. E. (2013) · Springer eBooks · doi:10.1007/978-1-4614-1424-7_4 · Cited by 61 · see citing works
Advancing Science with VGI: Reproducibility and Replicability of Recent Studies using VGI
Ostermann, F., & Granell, C. (2016) · Transactions in GIS · doi:10.1111/tgis.12195 · Cited by 59 · see citing works
Hung, L., Kristiyanto, D., Lee, S. B., & Yeung, K. Y. (2016) · PLoS ONE · doi:10.1371/journal.pone.0152686 · Cited by 54 · see citing works
Early Career Researchers Embrace Data Sharing
Campbell, H. A., Campbell, M. A., & Udyawer, V. (2018) · Trends in Ecology & Evolution · doi:10.1016/j.tree.2018.11.010 · Cited by 49 · see citing works
Fehr, J., Heiland, J., Himpe, C., & Saak, J. (2016) · AIMS Mathematics · doi:10.3934/math.2016.3.261 · Cited by 46 · see citing works
Boisvert, R. F. (2016) · Communications of the ACM · doi:10.1145/2994031 · Cited by 45 · see citing works
AlgoRun: a Docker-based packaging system for platform-agnostic implemented algorithms
Hosny, A., Vera‐Licona, P., Laubenbacher, R., & Favre, T. (2016) · Bioinformatics · doi:10.1093/bioinformatics/btw120 · Cited by 42 · see citing works
Reproducibility and Comparability of Computational Models for Astrocyte Calcium Excitability
Manninen, T., Havela, R., & Linne, M. (2017) · Frontiers in Neuroinformatics · doi:10.3389/fninf.2017.00011 · Cited by 41 · see citing works
Data Access, Transparency, and Replication: New Insights from the Political Behavior Literature
Stockemer, D., Koehler, S., & Lentz, T. (2018) · PS: Political Science and Politics · doi:10.1017/s1049096518000926 · Cited by 35 · see citing works
Johnson, E. J., & Goldstein, D. (2003) · Science · doi:10.1126/science.1091721 · Cited by 35 · see citing works
Introduction to the Special Collection on the Fragile Families Challenge
Salganik, M., Lundberg, I., Kindel, A. T., & McLanahan, S. (2019) · Socius Sociological Research for a Dynamic World · doi:10.1177/2378023119871580 · Cited by 33 · see citing works
Self-citationWhy scientists must share their research code
Baker, M. (2016) · Nature · doi:10.1038/nature.2016.20504 · Cited by 32 · see citing works
Reproducible and Replicable Computational Fluid Dynamics: It’s Harder Than You Think
Mesnard, O., & Barba, L. A. (2017) · Computing in Science & Engineering · doi:10.1109/mcse.2017.3151254 · Cited by 32 · see citing works
Privacy, Ethics, and Data Access: A Case Study of the Fragile Families Challenge
Lundberg, I., Narayanan, A., Levy, K. E. C., & Salganik, M. J. (2019) · Socius Sociological Research for a Dynamic World · doi:10.1177/2378023118813023 · Cited by 29 · see citing works
Self-citationHeroux, M. A. (2015) · ACM Transactions on Mathematical Software · doi:10.1145/2743015 · Cited by 27 · see citing works
Institutionalizing Transparency
Jeremy Freese, A., & King, M. M. (2018) · Socius Sociological Research for a Dynamic World · doi:10.1177/2378023117739216 · Cited by 27 · see citing works
The Limits of Reproducibility in Numerical Simulation
Diethelm, K. (2011) · Computing in Science & Engineering · doi:10.1109/mcse.2011.21 · Cited by 26 · see citing works
A Pragmatic Approach for Reproducible Research With Sensitive Data
Shepherd, B. E., Peratikos, M. B., Rebeiro, P. F., Duda, S. N., & McGowan, C. C. (2017) · American Journal of Epidemiology · doi:10.1093/aje/kwx066 · Cited by 26 · see citing works
Push button replication: Is impact evaluation evidence for international development verifiable?
Wood, B. D. K., Müller, R., & Brown, A. N. (2018) · PLOS ONE · doi:10.1371/journal.pone.0209416 · Cited by 24 · see citing works
Proceedings of the 17th Python in Science Conference
(2018) · Proceedings of the Python in Science Conferences · doi:10.25080/majora-4af1f417-018 · Cited by 24 · see citing works
Schnell, S. D. (2018) · Bulletin of Mathematical Biology · doi:10.1007/s11538-018-0500-9 · Cited by 23 · see citing works
Distributed Reproducible Research Using Cached Computations
Peng, R. D., & Eckel, S. P. (2008) · Computing in Science & Engineering · doi:10.1109/mcse.2009.6 · Cited by 22 · see citing works
packrat: A Dependency Management System for Projects and their R Package Dependencies
Atkins, A., Allen, T., Ushey, K., et al. (2014) · doi:10.32614/cran.package.packrat · Cited by 22 · see citing works
archivist: An R Package for Managing, Recording and Restoring Data Analysis Results
Biecek, P., & Kosiński, M. (2017) · Journal of Statistical Software · doi:10.18637/jss.v082.i11 · Cited by 21 · see citing works
Computational Reproducibility via Containers in Social PsychologyPreprint
Clyburne-Sherin, A., Fei, X., & Green, S. A. (2018) · doi:10.31234/osf.io/mf82t · Cited by 19 · see citing works
In’nami, Y., & Koizumi, R. (2010) · International Journal of Testing · doi:10.1080/15305058.2010.482219 · Cited by 15 · see citing works
Assessing the reproducibility of discriminant function analyses
Andrew, R. L., Albert, A., Renaut, S., Rennison, D. J., Bock, D. G., & Vines, T. (2015) · PeerJ · doi:10.7717/peerj.1137 · Cited by 11 · see citing works
Filippova, A., Gilroy, C., Kashyap, R., Kirchner, A., Morgan, A. C., Polimis, K., Usmani, A., & Wang, T. (2019) · Socius Sociological Research for a Dynamic World · doi:10.1177/2378023118820157 · Cited by 11 · see citing works
Reproducing Epidemiologic Research and Ensuring Transparency
Coughlin, S. S. (2017) · American Journal of Epidemiology · doi:10.1093/aje/kwx065 · Cited by 10 · see citing works
Good Enough Practices in Scientific Computing
Wilson, G., Bryan, J., Cranston, K., Kitzes, J., Nederbragt, L., & Teal, T. K. (2017) · PLOS Computational Biology · doi:10.1371/journal.pcbi.1005510 · Cited by 8 · see citing works
The poor availability of syntaxes of structural equation modeling
Wicherts, J. M., & Crompvoets, E. A. V. (2017) · Accountability in Research · doi:10.1080/08989621.2017.1396214 · Cited by 6 · see citing works
Freckleton, R. P. (2018) · Methods in Ecology and Evolution · doi:10.1111/2041-210x.12947 · Cited by 5 · see citing works
Predicting Layoff among Fragile Families
Ahearn, C. E., & Brand, J. E. (2019) · Socius Sociological Research for a Dynamic World · doi:10.1177/2378023118809757 · Cited by 5 · see citing works
Coding Style and Good Computing Practices
Nagler, J. (1995) · PS Political Science & Politics · doi:10.1017/s1049096500057747 · Cited by 5 · see citing works
Keister, L. A., & Moody, J. W. (2015) · Socius Sociological Research for a Dynamic World · doi:10.1177/2378023115596525 · Cited by 4 · see citing works
Heuberger, S. (2018) · Political Analysis · doi:10.1017/pan.2018.43 · Cited by 2 · see citing works
Flagged first, then by citations
Research works printed but not linked: 11
- Bowers Jake, and Voors Maarten. 2016. “ How to Improve Your Relationship with Your Future Self.” Revista de Ciencia Poltica (Santiago) 36(3): 829– 48.
- Chang Andrew C., and Li Phillip. 2018. “ Is Economics Research Replicable? Sixty Published Papers from Thirteen Journals Say Often Not.” Critical Finance Review 7.
- Eubank Nicholas. 2016. “ Lessons from a Decade of Replications at the Quarterly Journal of Political Science.” PS: Political Science & Politics 49(2): 273– 76.
- Heroux Michael A. 2015. “ Editorial: ACM TOMS Replicated Computational Results Initiative.” ACM Transactions on Mathematical Software 41(3): 13:1– 13:5.
- Liu DM 2018. Computational Reproducibility and the Fragile Families Challenge: Lessons Learned and Suggestions for the Future. B.S.E. Thesis, Department of Computer Science, Princeton University, Princeton, NJ.
- McCullough BD, McGeary Kerry Anne, and Harrison Teresa D.. 2006. “ Lessons from the JMCB Archive.” Journal of Money, Credit and Banking 38(4): 1093– 1107.
- Merkel Dirk. 2014. “ Docker: Lightweight Linux Containers for Consistent Development and Deployment.” Linux Journal 2014(239).
- Pasquier Thomas, Lau Matthew K., Han Xueyuan, Fong Elizabeth, Lerner Barbara S., Boose Emery, Crosas Mercè, Ellison Aaron M., and Seltzer Margo. 2018. “ Sharing and Preserving Computational Analyses for Posterity with encapsulator.” arXiv.
- Rampin Remi, Chirigati Fernando, Steeves Vicky, and Freire Juliana. 2018. “ ReproServer: Making Reproducibility Easier and Less Intensive.” arXiv.
- Shah Nihar B., Tabibian Behzad, Muandet Krikamol, Guyon Isabelle, and Luxburg von Ulrike. 2018. “ Design and Analysis of the Nips 2016 Review Process.” Journal of Machine Learning Research 19(49): 1– 34.
- Tatman Rachael, VanderPlas Jake, and Dane Sohier. 2018. “ A Practical Taxonomy of Reproducibility for Machine Learning Research.” Presented at the Reproducibility in Machine Learning Workshop at ICML 2018, Stockholm, Sweden.
Cited by
Sorted by citations
Leakage and the reproducibility crisis in machine-learning-based science
Kapoor, S., & Narayanan, A. (2023) · Patterns · doi:10.1016/j.patter.2023.100804 · Cited by 1,387 · see citing works
Computational Reproducibility via Containers in Psychology
Clyburne-Sherin, A., Fei, X., & Green, S. A. (2019) · Meta-Psychology · doi:10.15626/mp.2018.892 · Cited by 69 · see citing works
Siegel, Z. S., Kapoor, S., Nadgir, N., Stroebl, B., & Narayanan, A. (2024) · arXiv · doi:10.48550/arxiv.2409.11363 · Cited by 4 · see citing works
- Authors
- Liu, D. M., & Salganik, M. J.
- Affiliations
- Princeton University (US)
- Corresponding author
- Liu, David M.
- ORCID
- Salganik, Matthew J.
- Venue
- Socius Sociological Research for a Dynamic World
- Publisher
- SAGE Publications
- Published
- 2019-01
- Volume
- 5
- ISSN
- 2378-0231 (ISSN-L)
- PMC
- PMC10260256
- OpenAlex
- W4244185376
- Cited by
- 47
- References
- 119
- Citations by year
- 2026: 6 · 2025: 8 · 2024: 10 · 2023: 5 · 2022: 5 · 2021: 4 · 2020: 6 · 2019: 2 · 2018: 1
- FWCI
- 5.03
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- Socius Sociological Research for a Dynamic World (publishedVersion, cc-by-nc)
- PubMed (publishedVersion)
- PubMed Central (acceptedVersion, cc-by-nc)
- SocArXiv (OSF Preprints) (submittedVersion)
- SocArXiv (OSF Preprints) (submittedVersion, public-domain)
- Funders
- Russell Sage Foundation
- eunice kennedy shriver national institute of child health and human development (P2-CHD047879, R01-HD36916, R01-HD39135, R01-HD40421)
- Robert Wood Johnson Foundation
- Topics
- Social Sciences › Decision Sciences › Information Systems and Management › Scientific Computing and Data Management
- Physical Sciences › Computer Science › Information Systems › Research Data Management Practices
- Physical Sciences › Computer Science › Information Systems › Cloud Computing and Resource Management
- Keywords
- computational reproducibility, software containers, Docker, cloud computing, Amazon Web Services, research software engineering
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
