Addressing selective reporting of experiments through predefined exclusion criteria
Open accessAbstract
The pressure for every research article to tell a clear story often leads researchers in the life sciences to exclude experiments that 'did not work' when they write up their results. However, this practice can lead to reporting bias if the decisions about which experiments to exclude are taken after data have been collected and analyzed. Here we discuss how to balance clarity and thoroughness when reporting the results of research, and suggest that predefining the criteria for excluding experiments might help researchers to achieve this balance.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Our attempts to establish predefined inclusion/exclusion criteria to assess the validity of individual samples or experiments were less successful than we had expected.
“Our attempts to establish predefined inclusion/exclusion criteria to assess the validity of individual samples or experiments (Neves & Amaral, 2020) were also less successful than we had expected.”
From Estimating the replicability of Brazilian biomedical science (2025) · cites this paper as Neves & Amaral, 2020
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · PLoS Medicine · doi:10.1371/journal.pmed.0020124 · Cited by 10,505 · see citing works
The file drawer problem and tolerance for null results.
Rosenthal, R. (1979) · Psychological Bulletin · doi:10.1037/0033-2909.86.3.638 · Cited by 9,068 · cited 1× · see citing works
“A considerable body of literature exists on the omission of whole studies, a phenomenon best known as the ’file drawer effect’ (Rosenthal, 1979).”
BackgroundSimmons, J. P., Nelson, L. D., & Simonsohn, U. (2011) · Psychological Science · doi:10.1177/0956797611417632 · Cited by 6,567 · see citing works
Promoting an open research culture
Nosek, B. A., Alter, G., Banks, G. C., Borsboom, D., Bowman, S. D., Breckler, S. J., Buck, S., Chambers, C. D., Chin, G., Christensen, G., et al. (2015) · Science · doi:10.1126/science.aab2374 · Cited by 2,708 · see citing works
The preregistration revolution
Nosek, B. A., Ebersole, C. R., DeHaven, A. C., & Mellor, D. T. (2018) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1708274114 · Cited by 1,998 · see citing works
Increasing Transparency Through a Multiverse Analysis
Steegen, S., Tuerlinckx, F., Gelman, A., & Vanpaemel, W. (2016) · Perspectives on Psychological Science · doi:10.1177/1745691616658637 · Cited by 1,388 · see citing works
Negative results are disappearing from most disciplines and countries
Fanelli, D. (2012) · Scientometrics · doi:10.1007/s11192-011-0494-7 · Cited by 1,248 · see citing works
Dwan, K., Gamble, C., Williamson, P. R., Kirkham, J. J., & the Reporting Bias Group (2013) · PLoS ONE · doi:10.1371/journal.pone.0066844 · Cited by 1,127 · see citing works
Wicherts, J. M., Veldkamp, C. L. S., Augusteijn, H. E. M., Bakker, M., van Aert, R. C. M., & van Assen, M. A. L. M. (2016) · Frontiers in Psychology · doi:10.3389/fpsyg.2016.01832 · Cited by 754 · see citing works
A proposal for validation of antibodies
Uhlen, M., Bandrowski, A., Carr, S., Edwards, A., Ellenberg, J., Lundberg, E., Rimm, D. L., Rodriguez, H., Hiltke, T., Snyder, M., et al. (2016) · Nature Methods · doi:10.1038/nmeth.3995 · Cited by 626 · see citing works
Ferguson, C. J., & Heene, M. (2012) · Perspectives on Psychological Science · doi:10.1177/1745691612459059 · Cited by 603 · see citing works
Addressing the theory crisis in psychology
Oberauer, K., & Lewandowsky, S. (2019) · Psychonomic Bulletin & Review · doi:10.3758/s13423-019-01645-2 · Cited by 462 · see citing works
The ironic effect of significant results on the credibility of multiple-study articles.
Schimmack, U. (2012) · Psychological Methods · doi:10.1037/a0029487 · Cited by 455 · see citing works
Detecting and avoiding likely false‐positive findings – a practical guide
Forstmeier, W., Wagenmakers, E., & Parker, T. H. (2017) · Biological Reviews · doi:10.1111/brv.12315 · Cited by 436 · see citing works
On the Plurality of (Methodological) Worlds: Estimating the Analytic Flexibility of fMRI Experiments
Carp, J. (2012) · Frontiers in Neuroscience · doi:10.3389/fnins.2012.00149 · Cited by 381 · see citing works
The introduction, methods, results, and discussion (IMRAD) structure: a fifty-year survey.
Sollaci, L. B., & Pereira, M. G. (2004) · PubMed · Cited by 365 · see citing works
Wallach, J. D., Boyack, K. W., & Ioannidis, J. P. A. (2018) · PLOS Biology · doi:10.1371/journal.pbio.2006930 · Cited by 269 · see citing works
Publication bias and the canonization of false facts
Nissen, S. B., Magidson, T., Gross, K., & Bergstrom, C. T. (2016) · eLife · doi:10.7554/elife.21451 · Cited by 266 · see citing works
Evidence of experimental bias in the life sciences: why we need blind data recording
Holman, L., Head, M. L., Lanfear, R., & Jennions, M. D. (2015) · PLOS Biology · doi:10.1371/journal.pbio.1002190 · Cited by 263 · see citing works
Statistical methods for dealing with publication bias in meta‐analysis
Jin, Z., Zhou, X., & He, J. (2014) · Statistics in Medicine · doi:10.1002/sim.6342 · Cited by 252 · see citing works
(1974) · Cited by 251 · see citing works
Distinguishing between exploratory and confirmatory preclinical research will improve translation
Kimmelman, J., Mogil, J. S., & Dirnagl, U. (2014) · PLoS Biology · doi:10.1371/journal.pbio.1001863 · Cited by 236 · see citing works
Harking, Sharking, and Tharking
Hollenbeck, J. R., & Wright, P. M. (2016) · Journal of Management · doi:10.1177/0149206316679487 · Cited by 223 · see citing works
Blind analysis: Hide results to seek the truth
MacCoun, R., & Perlmutter, S. (2015) · Nature · doi:10.1038/526187a · Cited by 218 · see citing works
Match criteria for human cell line authentication: Where do we draw the line?
Capes‐Davis, A., Reid, Y. A., Kline, M. C., Storts, D. R., Strauss, E., Dirks, W. G., Drexler, H. G., MacLeod, R. A. F., Sykes, G., Kohara, A., et al. (2013) · International Journal of Cancer · doi:10.1002/ijc.27931 · Cited by 189 · see citing works
Outcome selection bias in meta-analysis
Williamson, P. R., Gamble, C., Altman, D. G., & Hutton, J. L. (2005) · Statistical Methods in Medical Research · doi:10.1191/0962280205sm415oa · Cited by 188 · see citing works
Lakens, D., & Etz, A. J. (2017) · Social Psychological and Personality Science · doi:10.1177/1948550617693058 · Cited by 145 · see citing works
Marcon, E., Jain, H., Bhattacharya, A., Guo, H., Phanse, S., Pu, S., Byram, G., Collins, B. C., Dowdell, E., Fenner, M., et al. (2015) · Nature Methods · doi:10.1038/nmeth.3472 · Cited by 127 · see citing works
Holman, C., Piper, S. K., Grittner, U., Diamantaras, A. A., Kimmelman, J., Siegerink, B., & Dirnagl, U. (2016) · PLOS Biology · doi:10.1371/journal.pbio.1002331 · Cited by 114 · see citing works
Science mapping analysis characterizes 235 biases in biomedical research
Chavalarias, D., & Ioannidis, J. P. A. (2010) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2009.12.011 · Cited by 97 · see citing works
MetaQC: objective quality control and inclusion/exclusion criteria for genomic meta-analysis
Kang, D., Sibille, E. L., Kaminski, N., & Tseng, G. C. (2011) · Nucleic Acids Research · doi:10.1093/nar/gkr1071 · Cited by 96 · see citing works
Phillips, C. V. (2004) · BMC Medical Research Methodology · doi:10.1186/1471-2288-4-20 · Cited by 94 · see citing works
No publication without confirmation
Mogil, J. S., & Macleod, M. R. (2017) · Nature · doi:10.1038/542409a · Cited by 74 · see citing works
Powell-Smith, A., & Goldacre, B. M. (2016) · F1000Research · doi:10.12688/f1000research.10010.1 · Cited by 59 · see citing works
The Brazilian Reproducibility Initiative
Amaral, O. B., Neves, K., Wasilewska‐Sampaio, A. P., & Carneiro, C. F. D. (2019) · eLife · doi:10.7554/elife.41602 · Cited by 56 · see citing works
Self-citationSelf-Control, Ego Depletion, and Social Psychology’s Replication CrisisPreprint
Baumeister, R. F. (2019) · doi:10.31234/osf.io/uf3cn · Cited by 35 · see citing works
Sanes, J. R. (2019) · eLife · doi:10.7554/elife.50527 · Cited by 28 · see citing works
Variability in the analysis of a single neuroimaging dataset by many teamsPreprint
Botvinik‐Nezer, R., Holzmeister, F., Camerer, C. F., Dreber, A., Huber, J., Johannesson, M., Kirchler, M., Iwanir, R., Mumford, J. A., Adcock, R. A., et al. (2019) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/843193 · Cited by 21 · see citing works
What If We Were Texas Sharpshooters? Predictor Reporting Bias in Regression Analysis
Biemann, T. (2013) · Organizational Research Methods · doi:10.1177/1094428113485135 · Cited by 9 · see citing works
Flagged first, then by citations
1 of the 41 references couldn’t be matched to a known paper.
Cited by
Sorted by citations
Addressing the theory crisis in psychology
Oberauer, K., & Lewandowsky, S. (2019) · Psychonomic Bulletin & Review · doi:10.3758/s13423-019-01645-2 · Cited by 462 · see citing works
Improving preclinical studies through replications
Drude, N., Martinez‐Gamboa, L., Danziger, M., Dirnagl, U., & Toelch, U. (2021) · eLife · doi:10.7554/elife.62101 · Cited by 54 · see citing works
Loss, C. M., Melleu, F. F., Domingues, K., Lino-de-Oliveira, C., & Viola, G. G. (2021) · Frontiers in Behavioral Neuroscience · doi:10.3389/fnbeh.2021.763428 · Cited by 44 · see citing works
The Benefits of Candidly Reporting Consumer Research
Janiszewski, C. A., & van Osselaer, S. M. J. (2021) · Journal of Consumer Psychology · doi:10.1002/jcpy.1263 · Cited by 25 · see citing works
Experiments from unfinished Registered Reports in the Reproducibility Project: Cancer Biology
Errington, T. M., Denis, A., Allison, A. B., Araiza, R., Aza-Blanc, P., Bower, L. R., Campos, J., Chu, H., Denson, S., Donham, C., et al. (2021) · eLife · doi:10.7554/elife.73430 · Cited by 21 · see citing works
Gardener, A. D., Hick, E. J., Jacklin, C., Tan, G., Cashin, A. G., Lee, H., Nunan, D., Toomey, E. C., & Richards, G. C. (2022) · JRSM Open · doi:10.1177/20542704221132139 · Cited by 19 · see citing works
Reporting In Vitro Experiments Responsibly – the RIVER RecommendationsPreprint
The RIVER working group (2023) · doi:10.31222/osf.io/x6aut · Cited by 16 · see citing works
Estimating the replicability of Brazilian biomedical science
Initiative, T. B. R., Amaral, O. B., Carneiro, C. F. D., Neves, K., Sampaio, A. P. W., Gomes, B. V., Abreu, M. B. D., Tan, P. B., Mota, G. P. S., Goulart, R. N., et al. (2025) · bioRxiv · doi:10.1101/2025.04.02.645026 · Cited by 13 · see citing works
“Our attempts to establish predefined inclusion/exclusion criteria to assess the validity of individual samples or experiments (Neves & Amaral, 2020) were also less successful than we had expected.”
Self-citationNeves, K., Carneiro, C. F. D., Wasilewska‐Sampaio, A. P., de Abreu, M. B., Valério‐Gomes, B., Tan, P. B., & Amaral, O. B. (2020) · Memórias do Instituto Oswaldo Cruz · doi:10.1590/0074-02760200328 · Cited by 13 · see citing works
Self-citationBardon, C., Guillemette, A., Rioux, M.-E., & Rivard, M. (2023) · Disability and Rehabilitation · doi:10.1080/09638288.2022.2104943 · Cited by 8 · see citing works
Neves, K., Carneiro, C. F. D., Wasilewska‐Sampaio, A. P., Abreu, M., Valério‐Gomes, B., Tan, P. B., & Amaral, O. B. (2020) · doi:10.31222/osf.io/njs6k · Cited by 1 · see citing works
Self-citationRigor in Experimental Design in Nonregulated Preclinical Research in Neuroscience Drug Discovery
Steele, S., Bespalov, A., & Dequeker, E. M. C. (2025) · Wiley eBooks · doi:10.1002/9783527844678.ch6
A MCDM-Based Framework for Quantifying Reproducibility Readiness in Machine Learning Research
Leščinskaitė, P., Paulavičius, R., & Filatovas, E. (2026) · Mathematics · doi:10.3390/math14091536
Estimating the replicability of Brazilian biomedical sciencePreprint
Initiative, T. B. R. (2026) · eLife · doi:10.7554/elife.111001
Estimating the replicability of Brazilian biomedical sciencePreprint
Initiative, T. B. R. (2026) · eLife · doi:10.7554/elife.111001.1
- Authors
- Neves, K., & Amaral, O. B.
- Affiliations
- Universidade Federal do Rio de Janeiro (BR)
- Corresponding author
- Amaral, Olavo B
- Venue
- eLife
- Publisher
- eLife Sciences Publications, Ltd
- Published
- 2020-05-22
- Volume
- 9
- ISSN
- 2050-084X (ISSN-L)
- PMC
- PMC7244322
- OpenAlex
- W3026292303
- Cited by
- 15
- References
- 41
- Citations by year
- 2026: 3 · 2025: 2 · 2023: 1 · 2022: 2 · 2021: 4 · 2020: 2 · 2019: 1
- FWCI
- 1.37
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- eLife (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Funders
- Serrapilheira Institute (Brazilian Reproducibility Initiative)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Decision Sciences › Information Systems and Management › Academic Publishing and Open Access
- Social Sciences › Arts and Humanities › History and Philosophy of Science › Academic Writing and Publishing
- Keywords
- selective reporting, reporting bias, exclusion criteria, pre-analysis plan, research transparency
- MeSH terms
- Humans, Publications, Research, Science, Bias
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
