The Brazilian Reproducibility Initiative
Open accessThe DOI matches its Crossref record; 24 of the 24 references it cites were found in a registry or an index.
Abstract
Most efforts to estimate the reproducibility of published findings have focused on specific areas of research, even though science is usually assessed and funded on a regional or national basis. Here we describe a project to assess the reproducibility of findings in biomedical science published by researchers based in Brazil. The Brazilian Reproducibility Initiative is a systematic, multicenter effort to repeat between 60 and 100 experiments: the project will focus on a set of common methods, repeating each experiment in three different laboratories from a countrywide network. The results, due in 2021, will allow us to estimate the level of reproducibility of biomedical science in Brazil, and to investigate what aspects of the published literature might help to predict whether a finding is reproducible.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Estimating the reproducibility of psychological science
Open Science Collaboration (2015) · Science · doi:10.1126/science.aac4716 · Cited by 8,515
1,500 scientists lift the lid on reproducibility
Baker, M. (2016) · Nature · doi:10.1038/533452a · Cited by 4,209
A manifesto for reproducible science
Munafò, M. R., Nosek, B. A., Bishop, D. V. M., Button, K. S., Chambers, C. D., Percie du Sert, N., Simonsohn, U., Wagenmakers, E.-J., Ware, J. J., & Ioannidis, J. P. A. (2017) · Nature Human Behaviour · doi:10.1038/s41562-016-0021 · Cited by 3,397
Raise standards for preclinical cancer research
Begley, C. G., & Ellis, L. M. (2012) · Nature · doi:10.1038/483531a · Cited by 2,979 · cited 1×
“…reproducibility of published results in certain areas of biomedical research were initially raised by theoretical models (Ioannidis, 2005a), systematic reviews of the existing literature (Ioannidis, 2005b) and alarm calls by the pharmaceutical industry (Begley and Ellis, 2012; Prinz et al., 2011).”
BackgroundBelieve it or not: how much can we rely on published data on potential drug targets?
Prinz, F., Schlange, T., & Asadullah, K. (2011) · Nature Reviews Drug Discovery · doi:10.1038/nrd3439-c1 · Cited by 1,888 · cited 1×
“Many analysts criticize the country’s research as largely composed of "salami science", growing in absolute numbers but lacking in impact, originality and influence (Righetti, 2013).”
BackgroundGenetics of Mouse Behavior: Interactions with Laboratory Environment
Crabbe, J. C., Wahlsten, D., & Dudek, B. C. (1999) · Science · doi:10.1126/science.284.5420.1670 · Cited by 1,666
Camerer, C. F., Dreber, A., Holzmeister, F., Ho, T.-H., Huber, J., Johannesson, M., Kirchler, M., Nave, G., Nosek, B. A., Pfeiffer, T., et al. (2018) · Nature Human Behaviour · doi:10.1038/s41562-018-0399-z · Cited by 1,521
Contradicted and Initially Stronger Effects in Highly Cited Clinical Research
Ioannidis, J. P. A. (2005) · JAMA · doi:10.1001/jama.294.2.218 · Cited by 1,465
Evaluating replicability of laboratory experiments in economics
Camerer, C. F., Dreber, A., Forsell, E., Ho, T.-H., Huber, J., Johannesson, M., Kirchler, M., Almenberg, J., Altmejd, A., Chan, T., et al. (2016) · Science · doi:10.1126/science.aaf0918 · Cited by 1,314
What does research reproducibility mean?
Goodman, S. N., Fanelli, D., & Ioannidis, J. P. A. (2016) · Science Translational Medicine · doi:10.1126/scitranslmed.aaf5027 · Cited by 1,293
Policy: NIH plans to enhance reproducibility
Collins, F. S., & Tabak, L. A. (2014) · Nature · doi:10.1038/505612a · Cited by 1,247 · cited 1×
“While funding agencies have expressed concerns about reproducibility (Collins and Tabak, 2014), efforts to replicate published findings in specific areas of research have mostly been conducted by bottom-up collaborations and supported by private funders.”
BackgroundInvestigating Variation in Replicability: A “Many Labs” Replication Project
Klein, R. A., Ratliff, K. A., Vianello, M., Adams, R. B., Bahník, Š., Bernstein, M. J., Bocian, K., Brandt, M. J., Brooks, B., Brumbaugh, C. C., et al. (2014) · Social Psychology · doi:10.1027/1864-9335/a000178 · Cited by 1,208
Kilkenny, C., Parsons, N., Kadyszewski, E., Festing, M. F. W., Cuthill, I. C., Fry, D., Hutton, J., & Altman, D. G. (2009) · PLoS ONE · doi:10.1371/journal.pone.0007824 · Cited by 860
How to Make More Published Research True
Ioannidis, J. P. A. (2014) · PLoS Medicine · doi:10.1371/journal.pmed.1001747 · Cited by 760
Small Telescopes: Detectability and the Evaluation of Replication Results
Simonsohn, U. (2015) · Psychological Science · doi:10.1177/0956797614567341 · Cited by 742 · cited 1×
“…effect (e.g. two results finding a statistically significant difference in the same direction) might not be accompanied by reproducibility of the effect size; conversely, studies with effect sizes that are similar to each other might have different outcomes in significance tests (Simonsohn, 2015).”
Comment on “Estimating the reproducibility of psychological science.”
Gilbert, D. T., King, G., Pettigrew, S., & Wilson, T. D. (2016) · Science · doi:10.1126/science.aad7243 · Cited by 474
Many Labs 3: Evaluating participant pool quality across the academic semester via replication
Ebersole, C. R., Atherton, O. E., Belanger, A. L., Skulborstad, H. M., Allen, J. M., Banks, J. B., Baranski, E., Bernstein, M. J., Bonfiglio, D. B. V., Boucher, L., et al. (2016) · Journal of Experimental Social Psychology · doi:10.1016/j.jesp.2015.10.012 · Cited by 468
Wicherts, J. M., Bakker, M., & Molenaar, D. (2011) · PLoS ONE · doi:10.1371/journal.pone.0026828 · Cited by 424
An empirical analysis of journal policy effectiveness for computational reproducibility
Stodden, V., Seiler, J., & Ma, Z. (2018) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1708290115 · Cited by 369
Using prediction markets to estimate the reproducibility of scientific research
Dreber, A., Pfeiffer, T., Almenberg, J., Isaksson, S., Wilson, B., Chen, Y., Nosek, B. A., & Johannesson, M. (2015) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1516179112 · Cited by 331
Reproducibility of preclinical animal research improves with heterogeneity of study samples
Voelkl, B., Vogt, L., Sena, E. S., & Würbel, H. (2018) · PLOS Biology · doi:10.1371/journal.pbio.2003693 · Cited by 262
Patil, P., Peng, R. D., & Leek, J. T. (2016) · Perspectives on Psychological Science · doi:10.1177/1745691616646366 · Cited by 243
on behalf of the IICARus Collaboration, Hair, K., Macleod, M. R., & Sena, E. S. (2019) · Research Integrity and Peer Review · doi:10.1186/s41073-019-0069-3 · Cited by 174
Hardwicke, T. E., & Ioannidis, J. P. A. (2018) · PLOS ONE · doi:10.1371/journal.pone.0201856 · Cited by 78
Research works printed but not linked: 22
- Angelo, 2016, Brazil's scientists battle to escape 20-year funding freeze, Nature, 539, 10.1038/nature.2016.21014 · doi:10.1038/nature.2016.21014
- Baker, 2017, Cancer reproducibility project releases first results, Nature, 541, 269, 10.1038/541269a · DOI · Paper page
- Barata, 2016, Dez coisas que você deveria saber sobre o Qualis, Revista Brasileira De Pós-Graduação, 13, 13, 10.21713/2358-2332.2016.v13.947 · DOI
- Ciscati R. 2018. Projeto vai replicar experimentos de cientistas brasileiros para checar sua eficiência . O Globo. https://oglobo.globo.com/sociedade/ciencia/projeto-vai-replicar-experimentos-de-cientistas-brasileiros-para-checar-sua-eficiencia-22615152 [Accessed January 25, 2019].
- Cova, 2018, Estimating the reproducibility of experimental philosophy, Review of Philosophy and Psychology, 28, 1, 10.1007/s13164-018-0400-9 · DOI · Paper page
- Coyne, 2016, Replication initiatives will not salvage the trustworthiness of psychology, BMC Psychology, 4, 28, 10.1186/s40359-016-0134-3 · DOI
- Errington, 2014, An open investigation of the reproducibility of cancer biology research, eLife, 3, 10.7554/eLife.04333 · DOI · Paper page
- Floresti F. 2017. A ciência brasileira vai quebrar?. Revista Galileu. https://revistagalileu.globo.com/Revista/noticia/2017/09/ciencia-brasileira-vai-quebrar.html [Accessed January 25, 2019].
- Harris, 2017, Rigor Mortis
- Hines, 2014, Sorting out the FACS: a devil in the details, Cell Reports, 6, 779, 10.1016/j.celrep.2014.02.021 · DOI · Paper page
- Hostins, 2006, Os planos nacionais de Pós-graduação (PNPG) e suas repercussões na pós-graduação brasileira, Perspectiva, 24, 133
- Ioannidis, 2005, Why most published research findings are false, PLOS Medicine, 2, 10.1371/journal.pmed.0020124 · DOI · Paper page
- Jamieson, 2018, Crisis or self-correction: Rethinking media narratives about the well-being of science, PNAS, 115, 2620, 10.1073/pnas.1708276114 · DOI
- Kaiser, 2018, Plan to replicate 50 high-impact cancer papers shrinks to just 18, Science, 10.1126/science.aau9619 · DOI
- Klein, 2018, Many Labs 2: Investigating variation in replicability across sample and setting, PsyArXiv, 10.31234/osf.io/9654g · DOI · Paper page
- Massonnet, 2010, Probing the reproducibility of leaf growth and molecular phenotypes: a comparison of three Arabidopsis accessions cultivated in ten laboratories, Plant Physiology, 152, 2142, 10.1104/pp.109.148338 · DOI · Paper page
- Nature Medicine, 2016, Take the long view, Nature Medicine, 22, 10.1038/nm.4033 · doi:10.1038/nm.4033
- Neves K, Amaral OB. 2018. Abrindo a caixa-preta. Ciência Hoje. http://cienciahoje.org.br/artigo/abrindo-a-caixa-preta [Accessed January 25, 2019].
- Pesquisa FAPESP. 2018. Uma rede para reproduzir experimentos. Revista Pesquisa FAPESP. http://revistapesquisa.fapesp.br/2018/05/17/uma-rede-para-reproduzir-experimentos [Accessed January 25, 2019].
- Pinto, 1999, Fator de impacto de revistas científicas: qual o significado deste parâmetro?, Química Nova, 22, 448, 10.1590/S0100-40421999000300026 · DOI
- Righetti S. 2013. Brasil cresce em produção científica, mas índice de qualidade cai. Folha De S. Paulo. https://www1.folha.uol.com.br/ciencia/2013/04/1266521-brasil-cresce-em-producao-cientifica-mas-indice-de-qualidade-cai.shtml [Accessed January 25, 2019].
- Silberzahn, 2018, Many analysts, one data set: Making transparent how variations in analytic choices affect results, Advances in Methods and Practices in Psychological Science, 1, 337, 10.1177/2515245917747646 · DOI · Paper page
Cited by
Investigating the replicability of preclinical cancer biology
Errington, T. M., Mathur, M., Soderberg, C. K., Denis, A., Perfito, N., Iorns, E., & Nosek, B. A. (2021) · eLife · doi:10.7554/elife.71601 · Cited by 462
Loss, C. M., Melleu, F. F., Domingues, K., Lino-de-Oliveira, C., & Viola, G. G. (2021) · Frontiers in Behavioral Neuroscience · doi:10.3389/fnbeh.2021.763428 · Cited by 44
Addressing selective reporting of experiments through predefined exclusion criteria
Neves, K., & Amaral, O. B. (2020) · eLife · doi:10.7554/elife.56626 · Cited by 15
Self-citationEstimating the replicability of Brazilian biomedical science
Initiative, T. B. R., Amaral, O. B., Carneiro, C. F. D., Neves, K., Sampaio, A. P. W., Gomes, B. V., Abreu, M. B. D., Tan, P. B., Mota, G. P. S., Goulart, R. N., et al. (2025) · bioRxiv · doi:10.1101/2025.04.02.645026 · Cited by 13
“With this in mind, we set up the Brazilian Reproducibility Initiative, a multicenter assessment of the replicability of experiments published by Brazilian institutions using common laboratory methods (Amaral et al., 2019; Neves et al., 2020).”
Self-citation
Metadata
- Authors
- Amaral, O. B., Neves, K., Wasilewska-Sampaio, A. P., & Carneiro, C. F.
- Venue
- eLife
- Publisher
- eLife Sciences Publications, Ltd
- Published
- 2019-02-05
- Volume
- 8
- ISSN
- 2050-084X (ISSN-L)
- OpenAlex
- W2901258965
- Cited by
- 49
- References
- 24
- FWCI
- 7.37
- Type
- journal article
- Language
- en
- Open access
- gold
- Licence
- creativecommons.org/licenses/by/4.0/ (vor)
- creativecommons.org/licenses/by/4.0/ (am)
- creativecommons.org/licenses/by/4.0/ (tdm)
- Funders
- Instituto Serrapilheira
- Conselho Nacional de Desenvolvimento Científico e Tecnológico
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.