Blind analysis: Hide results to seek the truth
The DOI matches its Crossref record; 9 of the 9 references it cites were found in a registry or an index.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Eliminating false positives entirely is likely to be impossible [86].”
From The natural selection of bad science (2016) · cites this paper as [86]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Considering the opposite: A corrective strategy for social judgment.
Lord, C. G., Lepper, M. R., & Preston, E. (1984) · Journal of Personality and Social Psychology · doi:10.1037//0022-3514.47.6.1231 · Cited by 740
Promoting Transparency in Social Science Research
Miguel, E., Camerer, C., Casey, K., Cohen, J., Esterling, K. M., Gerber, A., Glennerster, R., Green, D. P., Humphreys, M., Imbens, G., et al. (2014) · Science · doi:10.1126/science.1245317 · Cited by 540
BIASES IN THE INTERPRETATION AND USE OF RESEARCH RESULTS
MacCoun, R. J. (1998) · Annual Review of Psychology · doi:10.1146/annurev.psych.49.1.259 · Cited by 417
Self-citationLikelihood of Null Effects of Large NHLBI Clinical Trials Has Increased over Time
Kaplan, R. M., & Irvin, V. L. (2015) · PLOS ONE · doi:10.1371/journal.pone.0132382 · Cited by 356
Simonton, D. K. (2011) · Review of General Psychology · doi:10.1037/a0022912 · Cited by 199
BLIND ANALYSIS IN NUCLEAR AND PARTICLE PHYSICS
Klein, J. R., & Roodman, A. (2005) · Annual Review of Nuclear and Particle Science · doi:10.1146/annurev.nucl.55.090704.151521 · Cited by 156
The blind leading the blind: Use and misuse of blinding in randomized controlled trials
Miller, L. E., & Stewart, M. E. (2010) · Contemporary Clinical Trials · doi:10.1016/j.cct.2010.11.004 · Cited by 117
Conley, A., Goldhaber, G., Wang, L., Aldering, G., Amanullah, R., Commins, E. D., Fadeyev, V., Folatelli, G., Garavini, G., Gibbons, R., et al. (2006) · The Astrophysical Journal · doi:10.1086/503533 · Cited by 75
Self-citationMasked Monitoring in Clinical Trials — Blind Stupidity?
Meinert, C. L. (1998) · New England Journal of Medicine · doi:10.1056/nejm199805073381911 · Cited by 58
Research works printed but not linked: 1
- Lord, C. G., Lepper, M. R. & Preston, E. J. Pers. Soc. Psychol. 47, 1231-1243 (1984). · doi:10.1037/0022-3514.47.6.1231 · Paper page
Cited by
A manifesto for reproducible science
Munafò, M. R., Nosek, B. A., Bishop, D. V. M., Button, K. S., Chambers, C. D., Percie du Sert, N., Simonsohn, U., Wagenmakers, E.-J., Ware, J. J., & Ioannidis, J. P. A. (2017) · Nature Human Behaviour · doi:10.1038/s41562-016-0021 · Cited by 3,397
“In some physical sciences this approach has been extended to include deliberate perturbations in or masking of data to allow data preparation (for example, identification of outliers) to proceed without the analyst being able to see the corresponding results 30 .”
Planck 2018 results: I. Overview and the cosmological legacy of Planck
Planck Collaboration, Aghanim, N., Akrami, Y., Arroja, F., Ashdown, M., Aumont, J., Baccigalupi, C., Ballardini, M., Banday, A. J., Barreiro, R. B., et al. (2020) · Astronomy & Astrophysics · doi:10.1051/0004-6361/201833880 · Cited by 1,635
The preregistration revolution
Nosek, B. A., Ebersole, C. R., DeHaven, A. C., & Mellor, D. T. (2018) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1708274114 · Cited by 1,998
“A more robust option is to blind the dataset by scrambling some of the observations so that distributional forms are still retained, but there is no way to know the actual outcomes until the dataset is unblinded (47, 48).”
BackgroundIncreasing Transparency Through a Multiverse Analysis
Steegen, S., Tuerlinckx, F., Gelman, A., & Vanpaemel, W. (2016) · Perspectives on Psychological Science · doi:10.1177/1745691616658637 · Cited by 1,372
“Preregistration (e.g., Chambers, 2013; Wagenmakers et al., 2012) or blind analysis (e.g., MacCoun & Perlmutter, 2015) are not useful strategies for deflating the multiverse.”
MethodThe natural selection of bad science
Smaldino, P. E., & McElreath, R. (2016) · Royal Society Open Science · doi:10.1098/rsos.160384 · Cited by 860
“…for career success in science (Nosek & Lakens, 2014; Nosek et al., 2012; Ioannidis, 2014a; Vankov et al., 2014; Fischer, Ritchie & Hanspach, 2012; Brembs, Button & Munafò, 2013; Begley, Buchan & Dirnagl, 2015; MacCoun & Perlmutter, 2015; Gigerenzer & Marewski, 2015; Sills, 2016; Sarewitz, 2016).”
BackgroundBest practices in data analysis and sharing in neuroimaging using MRI
Nichols, T. E., Das, S., Eickhoff, S. B., Evans, A. C., Glatard, T., Hanke, M., Kriegeskorte, N., Milham, M. P., Poldrack, R. A., Poline, J.-B., et al. (2017) · Nature Neuroscience · doi:10.1038/nn.4500 · Cited by 714
Union through UNITY: Cosmology with 2000 SNe Using a Unified Bayesian Framework
Rubin, D., Aldering, G., Betoule, M., Fruchter, A., Huang, X., Kim, A. G., Lidman, C., Linder, E., Perlmutter, S., Ruiz-Lapuente, P., et al. (2025) · The Astrophysical Journal · doi:10.3847/1538-4357/adc0a5 · Cited by 246
Self-citationThe past, present and future of Registered Reports
Chambers, C. D., & Tzavella, L. (2021) · Nature Human Behaviour · doi:10.1038/s41562-021-01193-7 · Cited by 499
Transparency, Reproducibility, and the Credibility of Economics Research
Christensen, G., & Miguel, E. (2018) · Journal of Economic Literature · doi:10.1257/jel.20171350 · Cited by 446
Swedberg, R. (2020) · The Production of Knowledge · doi:10.1017/9781108762519.002 · Cited by 293
Detecting and avoiding likely false‐positive findings – a practical guide
Forstmeier, W., Wagenmakers, E., & Parker, T. H. (2017) · Biological Reviews · doi:10.1111/brv.12315 · Cited by 436
Planck 2018 results. I. Overview and the cosmological legacy of Planck
Collaboration, P., Akrami, Y., Arroja, F. M. A., Ashdown, M., Aumont, J., Baccigalupi, C., Ballardini, M., Banday, A. J., Barreiro, R. B., Bartolo, N., et al. (2020) · arXiv (Cornell University) · doi:10.17863/cam.33674 · Cited by 280
How Much Does Your Data Exploration Overfit? Controlling Bias via Information Usage
Russo, D., & Zou, J. (2020) · IEEE Transactions on Information Theory · doi:10.1109/tit.2019.2945779 · Cited by 119
“The raw, uncorrupted, dataset is only used in computing the final reported values [32].”
MethodRecommendations for Increasing the Transparency of Analysis of Preexisting Data Sets
Weston, S. J., Ritchie, S. J., Rohrer, J. M., & Przybylski, A. K. (2019) · Advances in Methods and Practices in Psychological Science · doi:10.1177/2515245919848684 · Cited by 203
The Preregistration Revolution
Nosek, B. A., Ebersole, C. R., DeHaven, A. C., & Mellor, D. T. (2017) · doi:10.31219/osf.io/2dxu5 · Cited by 168
More Than Privacy: Applying Differential Privacy in Key Areas of Artificial Intelligence
Zhu, T., Ye, D., Wang, W., Zhou, W., & Yu, P. S. (2022) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2020.3014246 · Cited by 219
“[34] believed: when deciding which results to report, the analyst interacts with a dataset that has been obfuscated through adding noise to observations, removing some data points, or switching data labels.”
BackgroundReducing bias, increasing transparency and calibrating confidence with preregistration
Hardwicke, T. E., & Wagenmakers, E.-J. (2023) · Nature Human Behaviour · doi:10.1038/s41562-022-01497-2 · Cited by 179
Protecting against researcher bias in secondary data analysis: challenges and potential solutions
Baldwin, J. R., Pingault, J.-B., Schoeler, T., Sallis, H. M., & Munafò, M. R. (2022) · European Journal of Epidemiology · doi:10.1007/s10654-021-00839-0 · Cited by 178
“Various types of blinding strategies exist [54], but one method that is appropriate for epidemiological data is “data scrambling” [55].”
MethodThe Production of Knowledge: Enhancing Progress in Social Science
(2020) · Cambridge University Press eBooks · doi:10.1017/9781108762519 · Cited by 241
Schweinsberg, M., Feldman, M., Staub, N., van den Akker, O. R., van Aert, R. C. M., van Assen, M. A. L. M., Liu, Y., Althoff, T., Heer, J., Kale, A. M., et al. (2021) · Organizational Behavior and Human Decision Processes · doi:10.1016/j.obhdp.2021.02.003 · Cited by 171
Jarantow, S. W., Pisors, E. D., & Chiu, M. L. (2023) · Current Protocols · doi:10.1002/cpz1.801 · Cited by 162
“For some time, there has been an apparent crisis in the scientific community regarding the low reproducibility of experimental data (Begley & Ellis, 2012; Flier, 2017; Ioannidis, 2005; MacCoun & Perlmutter, 2015; Nature, 2016; Nuzzo, 2015).”
BackgroundA blinded determination of H0 from low-redshift Type Ia supernovae, calibrated by Cepheid variables
Zhang, B. R., Childress, M. J., Davis, T. M., Karpenka, N. V., Lidman, C., Schmidt, B. P., & Smith, M. (2017) · Monthly Notices of the Royal Astronomical Society · doi:10.1093/mnras/stx1600 · Cited by 130
“Croft & Dailey (2011) find evidence of confirmation bias in recent measurements of cosmological parameters and recommend blinding; similarly Maccoun & Perlmutter (2015) argue for its necessity.”
BackgroundBrain and psychological determinants of placebo pill response in chronic pain patients
Vachon‐Presseau, É., Berger, S., Abdullah, T. B., Huang, L., Cecchi, G., Griffith, J. W., Schnitzer, T. J., & Apkarian, A. V. (2018) · Nature Communications · doi:10.1038/s41467-018-05859-1 · Cited by 129
Falsifiability is not optional.
LeBel, E. P., Berger, D., Campbell, L., & Loving, T. J. (2017) · Journal of Personality and Social Psychology · doi:10.1037/pspi0000106 · Cited by 105
Designing a rigorous microscopy experiment: Validating methods and avoiding bias
Jost, A. P.-T., & Waters, J. C. (2019) · The Journal of Cell Biology · doi:10.1083/jcb.201812109 · Cited by 116
Rath, A., Salamon, V., Peixoto, S., Hivert, V., Laville, M., Segrestin, B., Neugebauer, E. A. M., Eikermann, M., Bertelè, V., Garattini, S. K., et al. (2017) · Trials · doi:10.1186/s13063-017-2287-7 · Cited by 111
Predicting Violent Behavior: What Can Neuroscience Add?
Poldrack, R. A., Monahan, J. T., Imrey, P. B., Reyna, V. F., Raichle, M. E., Faigman, D. L., & Buckholtz, J. W. (2017) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2017.11.003 · Cited by 107
“Yet another provocative alternative is the use of “blind analysis” [70], in which the researcher analyzing the data is blinded to some aspect of the data (e.”
MethodSpecific barriers to the conduct of randomised clinical trials on medical devices
Neugebauer, E. A. M., Rath, A., Antoine, S., Eikermann, M., Seidel, D., Koenen, C., Jacobs, E. M. G., Pieper, D., Laville, M., Pitel, S., et al. (2017) · Trials · doi:10.1186/s13063-017-2168-0 · Cited by 103
Biases in research: risk factors for non-replicability in psychotherapy and pharmacotherapy research
Leichsenring, F., Abbass, A., Hilsenroth, M. J., Leweke, F., Luyten, P., Keefe, J. R., Midgley, N., Rabung, S., Salzer, S., & Steinert, C. (2017) · Psychological Medicine · doi:10.1017/s003329171600324x · Cited by 78
Ito, S., Feldheim, D. A., & Litke, A. M. (2017) · The Journal of Neuroscience · doi:10.1523/jneurosci.3689-16.2017 · Cited by 92
Big Data Analytics for Scanning Transmission Electron Microscopy Ptychography
Jesse, S., Chi, M., Belianinov, A., Beekman, C., Kalinin, S. V., Borisevich, A. Y., & Lupini, A. R. (2016) · Scientific Reports · doi:10.1038/srep26348 · Cited by 88
“Alternatively, to put it bluntly, prior expectations might both limit the potential information that can be extracted 24 and create false positives for expected results 25–27 .”
A survey on how preregistration affects the research workflow: better science but more work
Sarafoglou, A., Kovács, M., Bakos, B. E., Wagenmakers, E., & Aczél, B. (2022) · Royal Society Open Science · doi:10.1098/rsos.211997 · Cited by 95
Hoffmann, S., Schönbrodt, F. D., Elsas, R., Wilson, R., Strasser, U., & Boulesteix, A. (2021) · Royal Society Open Science · doi:10.1098/rsos.201925 · Cited by 85
“Strictly confirmatory analyses can be realized either through the pre-registration of analysis plans [92, 93] and registered reports [94], where the analysis strategy is specified in detail before observing the data; or through blind analyses, where researchers select an analysis strategy while being blinded to the ou…”
BackgroundMannhardt, I., Eder, A., Dumotier, B., Prondzynski, M., Krämer, E., Traebert, M., Söhren, K.-D., Flenner, F., Stathopoulou, K., Lemoine, M. D., et al. (2017) · Toxicological Sciences · doi:10.1093/toxsci/kfx081 · Cited by 68
“Given the well-known differences between blinded and nonblinded analysis (MacCoun and Perlmutter, 2015), the validity of these test systems is probably best evaluated by blinded interlaboratory experiments.”
BackgroundApplying Differential Privacy Mechanism in Artificial Intelligence
Zhu, T., & Yu, P. S. (2019) · IEEE International Conference on Distributed Computing Systems (ICDCS) · doi:10.1109/icdcs.2019.00159 · Cited by 66
“[22] supported the blind data analysis: when deciding which results to report, the analyst interacts with a dataset that has been obfuscated through adding noise to observations, removing some data points, or switching data labels.”
BackgroundTransparency, Reproducibility, and the Credibility of Economics Research
Christensen, G., & Miguel, E. (2017) · doi:10.31222/osf.io/9a3rw · Cited by 50
Olazabal, M., Neumann, M. B., Foudi, S., & Chiabai, A. (2018) · Systems Research and Behavioral Science · doi:10.1002/sres.2519 · Cited by 56
Schoenfeld, B. J., Vigotsky, A. D., Grgic, J., Haun, C., Contreras, B., Delcastillo, K., Francis, A., Cote, G., & Alto, A. (2020) · Physiological Reports · doi:10.14814/phy2.14427 · Cited by 56
“First, the principal investigator, who obtained the measurements of the primary outcome, was blinded to group allocation; second, the statistician performed blinded analyses in the form of cell scrambling (MacCoun & Perlmutter, 2015).”
MethodWhy are we not evaluating multiple competing hypotheses in ecology and evolution?
Betini, G. S., Avgar, T., & Fryxell, J. M. (2017) · Royal Society Open Science · doi:10.1098/rsos.160756 · Cited by 59
“More recently, blind analysis has also become popular, at least in some fields [10,38,39].”
BackgroundFeature—Is a Replicability Crisis on the Horizon for Environmental and Resource Economics?
Ferraro, P. J., & Shukla, P. (2020) · Review of Environmental Economics and Policy · doi:10.1093/reep/reaa011 · Cited by 57
Flexible yet fair: blinding analyses in experimental psychology
Dutilh, G., Sarafoglou, A., & Wagenmakers, E. (2019) · Synthese · doi:10.1007/s11229-019-02456-7 · Cited by 55
“To address the lack of flexibility inherent to preregistration, we follow MacCoun and Perlmutter (2015) and argue for the broader adoption of a technique known as analysis blinding .”
Shedding light on X17: community report
Alves, D. S. M., Barducci, D., Cavoto, G., Darmé, L. J.-M., Rose, L. D., Doria, L., Feng, J. L., Frankenthal, A., Gasparian, A., Goudzovski, E., et al. (2023) · The European Physical Journal C · doi:10.1140/epjc/s10052-023-11271-x · Cited by 71
Waltemath, D., & Wolkenhauer, O. (2016) · IEEE Transactions on Biomedical Engineering · doi:10.1109/tbme.2016.2555481 · Cited by 49
“Contrarily, the desire to support one’s own theory, to be the first person to report a finding, or even the pressure to publish a result may distort – consciously or unconsciously – what we draw from research results [16, 17].”
BackgroundQuantifying research waste in ecology
Purgar, M., Klanjšček, T., & Čulina, A. (2022) · Nature Ecology & Evolution · doi:10.1038/s41559-022-01820-0 · Cited by 54
The Creativity-Verification Cycle in Psychological Science: New Methods to Combat Old Idols
Wagenmakers, E.-J., Dutilh, G., & Sarafoglou, A. (2018) · Perspectives on Psychological Science · doi:10.1177/1745691618771357 · Cited by 46
“A method that could help here is blinding (e.g., MacCoun & Perlmutter, 2015), possibly in combination with preregistration (Dutilh et al., 2017); in a blinded analysis, the analyst is initially given only part of the data—enough to create a sensible statistical model and account for unexpected data…”
Law and Psychology Grows Up, Goes Online, and Replicates
Irvine, K., Hoffman, D. A., & Wilkinson‐Ryan, T. (2017) · Journal of Empirical Legal Studies · doi:10.1111/jels.12180 · Cited by 53
Assessing Theoretical Conclusions With Blinded Inference to Investigate a Potential Inference Crisis
Starns, J. J., Cataldo, A. M., Rotello, C. M., Annis, J. S., Aschenbrenner, A. J., Bröder, A., Cox, G. E., Criss, A. H., Curl, R. A., Dobbins, I. G., et al. (2019) · Advances in Methods and Practices in Psychological Science · doi:10.1177/2515245919869583 · Cited by 54
“Blinded-analysis techniques, increasingly common in physics (MacCoun & Perlmutter, 2015), hide from the data analyst either the true experimental condition from which each observation is drawn (e.g., scrambled conditions) or the true value of each observation (i.e., addition of removable random…”
Interventions for Organizational Climate and Culture in Academia: A Scoping Review
Viđak, M., Barać, L., Tokalić, R., Buljan, I., & Marušić, A. (2021) · Science and Engineering Ethics · doi:10.1007/s11948-021-00298-6 · Cited by 48
Fucci, D., Scanniello, G., Romano, S., Shepperd, M., Sigweni, B., Uyaguari, F., Turhan, B., Juristo, N., & Oivo, M. (2016) · Proceedings of the ... ACM-IEEE International Symposium on Empirical Software Engineering and Measurement/International Symposium on Empirical Software Engineering and Measurement · doi:10.1145/2961111.2962592 · Cited by 52
“In order to reduce possible researcher bias [26], we used a two-step protocol for blind analysis: 1) blind data extraction (UPM); 2) blind statistical analysis (BRUNEL).”
MethodStrong Inference in Mathematical Modeling: A Method for Robust Science in the Twenty-First Century
Ganusov, V. V. (2016) · Frontiers in Microbiology · doi:10.3389/fmicb.2016.01131 · Cited by 49
“…of science including use of strong inference (Nuzzo, 2015), improved trainings (Moher and Altman, 2015), performing blind analyses of the data (MacCoun and Perlmutter, 2015), the need for independent analyses of the same data/models by different teams prior to publication (Silberzahn and…”
Who Believes in the Storybook Image of the Scientist?
Veldkamp, C. L. S., Hartgerink, C. H. J., van Assen, M. A. L. M., & Wicherts, J. M. (2016) · Accountability in Research · doi:10.1080/08989621.2016.1268922 · Cited by 47
“…Motyl 2012; Wagenmakers et al. 2012), collaboration on statistical analysis (Veldkamp et al. 2014; Wicherts 2011), blind data analysis (MacCoun and Perlmutter 2015), reforms in incentive structures (Chambers 2015; Nosek, Spies, and Motyl 2012), training in research integrity (Steneck 2013), and…”
BackgroundAlthor, G., & Witt, B. (2019) · Journal of Environmental Studies and Sciences · doi:10.1007/s13412-019-00582-9 · Cited by 48
“As the reason for such a high statistic is unclear, we highly recommend that future EDJ studies take precautions to prevent potential biases where possible, and utilize methods such as counterfactual thinking and randomized trials, quasi-experimental designs, blind analysis and the use of control variables (MacCoun an…”
BackgroundMethodBerger, S., & Baria, A. T. (2022) · Frontiers in Pain Research · doi:10.3389/fpain.2022.896276 · Cited by 47
“Additionally, labs can engage in data practices which combat individual and systemic biases in healthcare and medical pain research, including but not limited to: using simulated or randomized data sets to blind researchers during analyses (378, 379) conducting fairness audits or applying de-biasing techniques, and ai…”
BackgroundQuantitative Electroencephalography Characteristics for Parkinson’s Disease: A Systematic Review
Shirahige, L., Berenguer-Rocha, M., Mendonça, S., Rocha, S., Rodrigues, M. C. A., & Monte-Silva, K. K. (2020) · Journal of Parkinson s Disease · doi:10.3233/jpd-191840 · Cited by 48
“Blinding analysis is considered the only way to trust results [41] and a precise and accurate conclusion with lesser probability to leading a type II error can only be made by an appropriate sample size [42].”
BackgroundBalachandran, A. T., Vigotsky, A. D., Quiles, N. N., Mokkink, L. B., Belio, M., & Glenn, J. M. (2020) · Experimental Gerontology · doi:10.1016/j.exger.2020.111202 · Cited by 48
A manifesto for reproducible science.
Munafo, M. R., Nosek, B. A., Bishop, D., Button, K. S., Chambers, C., du Sert, N. P., Simonsohn, U., Wagenmakers, E., Ware, J. J., & Ioannidis, J. P. A. (2024) · American Psychological Association eBooks · doi:10.1037/0000409-041 · Cited by 42
Efficacy of attention bias modification training for depressed adults: a randomized clinical trial
Hsu, K. J., Shumake, J., Longoria, K. D., Risom, S., Labrada, J., Smits, J. A. J., Schnyer, D. M., & Beevers, C. G. (2021) · Psychological Medicine · doi:10.1017/s0033291721000702 · Cited by 44
“Importantly, we used a data blind analysis approach (MacCoun & Perlmutter, 2015): training condition assignments were permuted when testing model assumptions so that observed results would not influence decision making.”
MethodOn researcher bias in Software Engineering experiments
Romano, S., Fucci, D., Scanniello, G., Baldassarre, M. T., Turhan, B., & Juristo, N. (2021) · Journal of Systems and Software · doi:10.1016/j.jss.2021.111068 · Cited by 42
“Although extensively used in other research fields like medicine and physics [6, 7], SE researchers have used these techniques only in few occasions [8, 9].”
BackgroundDutilh, G., Vandekerckhove, J., Ly, A., Matzke, D., Pedroni, A. F., Frey, R., Rieskamp, J., & Wagenmakers, E. (2017) · Attention Perception & Psychophysics · doi:10.3758/s13414-017-1304-y · Cited by 40
“A novel element to our preregistration proposal is the inclusion of a blinding procedure, where an analyst (in this case, author JV) is sent the data with the key variable shuffled (MacCoun & Perlmutter, 2015).”
MethodThe Discovery of a Gravitationally Lensed Supernova Ia at Redshift 2.22
Rubin, D., Hayden, B., Huang, X., Aldering, G., Amanullah, R., Barbary, K., Boone, K., Brodwin, M., Deustua, S. E., Dixon, S., et al. (2018) · The Astrophysical Journal · doi:10.3847/1538-4357/aad565 · Cited by 37
“In previous work with high-redshift SNe, (e.g., Suzuki et al. 2012; Rubin et al. 2013; Nordin et al. 2014), we were able to make use of “blinded” analyses, in which the results are hidden until the analysis is finalized (e.g., Conley et al. 2006; Maccoun & Perlmutter 2015).”
MethodSelf-citationCashin, A. G., Lee, H., Bagg, M. K., O’Hagan, E. T., Traeger, A. C., Kamper, S. J., Folly, T., Jones, M. D., Booth, J., & McAuley, J. H. (2020) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2020.06.025 · Cited by 35
“blinding of the data analyst is an additional practice which can be more easily implemented to minimise the risk of bias [48,49].”
BackgroundTrautwein, F., Schweitzer, Y., Dor‐Ziderman, Y., Nave, O., Ataria, Y., Fulder, S. J., & Berkovich‐Ohana, A. (2024) · Journal of Neuroscience · doi:10.1523/jneurosci.1182-23.2024 · Cited by 39
Teaching Good Research Practices: Protocol of a Research Master Course
Sarafoglou, A., Hoogeveen, S., Matzke, D., & Wagenmakers, E. (2019) · Psychology Learning & Teaching · doi:10.1177/1475725719858807 · Cited by 38
Blind Analysis as a Correction for Confirmatory Bias in Physics and in Psychology
MacCoun, R. J., & Perlmutter, S. (2017) · Wiley eBooks · doi:10.1002/9781119095910.ch15 · Cited by 38
Self-citationTransparency, Reproducibility, and the Credibility of Economics Research
Christensen, G., & Miguel, E. A. (2016) · National Bureau of Economic Research · doi:10.3386/w22989 · Cited by 48
Bestiary of Questionable Research Practices in Psychology
Nagy, T., Hergert, J., Elsherif, M. M., Wallrich, L., Schmidt, K., Waltzer, T., Payne, J. W., Gjoneska, B., Seetahul, Y., Wang, Y. A., et al. (2025) · Advances in Methods and Practices in Psychological Science · doi:10.1177/25152459251348431 · Cited by 37
“Blind data analysis is another emerging trend that can help prevent several QRPs (MacCoun & Perlmutter, 2015).”
van den Ende, C. H. M., Minten, M. J. M., Leseman-Hoogenboom, M. M., van den Hoogen, F. H. J., den Broeder, A. A., Mahler, E. A. M., & Poortmans, P. M. (2019) · The Lancet Rheumatology · doi:10.1016/s2665-9913(19)30096-7 · Cited by 35
The Impact of Educative News Articles about Suicide Prevention: A Randomized Controlled Trial
Till, B., Tran, U. S., & Niederkrotenthaler, T. (2020) · Health Communication · doi:10.1080/10410236.2020.1813953 · Cited by 32
“It is commonly recommended for randomized controlled trials to ensure the blinding of researchers until data analysis is completed in order to avoid bias (Karanicolas et al., 2010; MacCoun & Perlmutter, 2015).”
BackgroundKarp, N. A., Pearl, E. J., Stringer, E. J., Barkus, C., Ulrichsen, J. C., & du Sert, N. P. (2022) · PLoS Biology · doi:10.1371/journal.pbio.3001873 · Cited by 36
“Adding cell bias to equalise the means Hide experimental group differences by adding the same hidden number to all observations within an experimental group, which leaves the distribution intact but obscures the differences [42].”
BackgroundMethodInfluentialPre-registration and Results-Free Review in Observational and Qualitative Research
Jacobs, A. M. (2020) · Cambridge University Press eBooks · doi:10.1017/9781108762519.009 · Cited by 34
Field-recorded data on the diet of six species of European Hydromantes cave salamanders
Lunghi, E., Cianferoni, F., Ceccolini, F., Mulargia, M., Cogoni, R., Barzaghi, B., Cornago, L., Avitabile, D., Veith, M., Manenti, R., et al. (2018) · Scientific Data · doi:10.1038/sdata.2018.83 · Cited by 30
Spectral cues are necessary to encode azimuthal auditory space in the mouse superior colliculus
Ito, S., Si, Y., Feldheim, D. A., & Litke, A. M. (2020) · Nature Communications · doi:10.1038/s41467-020-14897-7 · Cited by 30
Photographic database of the European cave salamanders, genus Hydromantes
Lunghi, E., Giachello, S., Zhao, Y., Corti, C., Ficetola, G. F., & Manenti, R. (2020) · Scientific Data · doi:10.1038/s41597-020-0513-8 · Cited by 29
Nonlinear visuoauditory integration in the mouse superior colliculus
Ito, S., Si, Y., Litke, A. M., & Feldheim, D. A. (2021) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1009181 · Cited by 28
“We used blind analysis, an effective method for reducing false-positive reporting of results[40,41].”
MethodA Guide and Toolbox to Replicability and Open Science in Entomology
Wittman, J. T., & Aukema, B. H. (2020) · Journal of Insect Science · doi:10.1093/jisesa/ieaa036 · Cited by 29
“Data that are made publically available with appropriate metadata can be used by other researchers in meta-analyses or reanalyzed to ensure the robustness of results (Michener 2006, 2015; Piwowar et al.”
MethodManenti, R., Melotto, A., Guillaume, O., Ficetola, G. F., & Lunghi, E. (2020) · Behavioral Ecology and Sociobiology · doi:10.1007/s00265-020-02909-x · Cited by 27
“A blind video analysis was performed by two operators: i.e. operators were not aware of the typology of trial they were going to analyse; this procedure allows us to limit biased interpretation (MacCoun and Perlmutter 2015).”
MethodBelieving is seeing – the deceptive influence of bias in quantitative microscopy
Lee, R., Eisenman, L. R., Khuon, S., Aaron, J., & Chew, T. (2024) · Journal of Cell Science · doi:10.1242/jcs.261567 · Cited by 26
“Careful testing of the hypothesis should be ensured by a combination of masked image assessment (Holman et al., 2015; Jost and Waters, 2019; MacCoun and Perlmutter, 2015), automatic analysis techniques (discussed in a later section), and, whenever suitable, complementary approaches, such as…”
BackgroundMethodImproving Statistical Analysis in Team Science: The Case of a Bayesian Multiverse of Many Labs 4
Hoogeveen, S., Berkhout, S. W., Gronau, Q. F., Wagenmakers, E., & Haaf, J. M. (2023) · Advances in Methods and Practices in Psychological Science · doi:10.1177/25152459231182318 · Cited by 25
“Another option to ensure uncontaminated data analysis would be to use “blinded analysis” (MacCoun & Perlmutter, 2015, 2018), in which analysts perform their analysis on an altered version of the data (e.g., shuffling the dependent variable, adding noise to the data, or switching labels of…”
BackgroundOnce an optimist, always an optimist? Studying cognitive judgment bias in mice
Bračić, M., Bohn, L., Siewert, V., von Kortzfleisch, V. T., Schielzeth, H., Kaiser, S., Sachser, N., & Richter, S. H. (2022) · Behavioral Ecology · doi:10.1093/beheco/arac040 · Cited by 23
Kirkwood, B. R., Sikander, S., Roy, R., Soremekun, S., Bhopal, S., Avan, B. I., Lingam, R., Gram, L., Amenga‐Etego, S., Khan, B., et al. (2023) · Frontiers in Nutrition · doi:10.3389/fnut.2023.1155763 · Cited by 22
“This was to agree the exact analyses to be used in reporting the trial findings and to ensure that all data procedures, methodological and statistical decisions were completed and agreed with the DSMB and TSC members before the true results were revealed, to minimise biases (37).”
MethodConsumer-centric electricity markets: Six design principles
Michaelis, A., Hanny, L., Körner, M.-F., Strüker, J., & Weibelzahl, M. (2023) · Renewable and Sustainable Energy Reviews · doi:10.1016/j.rser.2023.113817 · Cited by 27
“We use these expert interviews as a blind evaluation to mitigate bias [52].”
MethodSound Inference in Complicated Research: A Multi-Strategy ApproachPreprint
Srivastava, S. (2018) · PsyArXiv (OSF Preprints) · doi:10.31234/osf.io/bwr48 · Cited by 26
Objectivity for the research worker
van Dongen, N. N. N., & Sikorski, M. (2021) · European Journal for Philosophy of Science · doi:10.1007/s13194-021-00400-6 · Cited by 25
Rubin, D., Hoyt, T. J., Aldering, G., & Perlmutter, S. (2026) · The Astrophysical Journal · doi:10.48550/arxiv.2601.19854 · Cited by 29
Self-citationSipion, M., Ferreira, F. M., Scholler, J., Brana, C., Gora, M. J., Kouvas, G., Barthet, G., & Sobolewski, A. (2023) · Scientific Reports · doi:10.1038/s41598-023-47039-2 · Cited by 22
“However, despite overwhelmingly positive outcomes reported in literature, PBM has also faced controversy in general 21–23 and in the neurodegenerative field in particular 24 , while many studies in this field can be seen as methodologically challenged as not being fully blinded 25–27 .”
BackgroundResearch protocols: A forest of hypotheses
Rosén, J. (2016) · Nature · doi:10.1038/nj7615-239a · Cited by 20
González–Fraile, E., Domínguez-Panchón, A. I., Berzosa-Grande, P., Costas-González, A. B., Jiménez, I. G., Rufino-Ventura, D., Aparicio, J. I. L., & Martín-Carrasco, M. (2019) · Research in Developmental Disabilities · doi:10.1016/j.ridd.2019.103458 · Cited by 25
Héroux, M. E., Butler, A. A., Cashin, A. G., McCaughey, E. J., Affleck, A. J., Green, M. A., Cartwright, A., Jones, M. D., Kiely, K. M., van Schooten, K. S., et al. (2022) · BMJ Open · doi:10.1136/bmjopen-2022-060976 · Cited by 21
Flink, M., Lindblad, M., Frykholm, O., Kneck, Å., Nilsén, P., Årestedt, K., & Ekstedt, M. (2017) · BMJ Open · doi:10.1136/bmjopen-2016-014178 · Cited by 20
Improving statistical reporting in psychology
Schubert, A., Snijder-Steinhilber, M., Kang, H., & Quintana, D. S. (2025) · Communications Psychology · doi:10.1038/s44271-025-00356-w · Cited by 24
Weeks, K. L., Henstridge, D. C., Salim, A., Shaw, J. E., Marwick, T. H., & McMullen, J. R. (2019) · American Journal of Physiology-Heart and Circulatory Physiology · doi:10.1152/ajpheart.00327.2019 · Cited by 18
“Leaders in the field have highlighted that there is a natural tendency for scientists to “see” results and report data in a manner that confirms their original hypothesis and that failure to control for bias and rationalizing behaviors such as P-hacking can lead to results and conclusions that are less likely to be re…”
Narrative simulation of social experiences in naturalistic context – A neurocinematic approach
Tikka, P., Kaipainen, M., & Salmi, J. (2023) · Neuropsychologia · doi:10.1016/j.neuropsychologia.2023.108654 · Cited by 23
Perceiving personality through the grapevine: A network approach to reputations.
Costello, C., & Srivastava, S. (2020) · Journal of Personality and Social Psychology · doi:10.1037/pspp0000362 · Cited by 18
Replicability in Empirical Legal Research
Chin, J. M., & Zeiler, K. (2021) · Annual Review of Law and Social Science · doi:10.1146/annurev-lawsocsci-121620-085055 · Cited by 21
Ávila, C. G., May, E. S., Bott, F. S., Tiemann, L., Hohn, V. D., Heitmann, H., Zebhauser, P. T., Groß, J., & Ploner, M. (2025) · eLife · doi:10.7554/elife.101727 · Cited by 19
Czaczkes, T. J., Castorena, M., Schürch, R., & Heınze, J. (2016) · Physiological Entomology · doi:10.1111/phen.12174 · Cited by 16
“To ensure an unbiased analysis, the initial data analysis was carried out ‘blind’ (MacCoun & Perlmutter, 2015).”
MethodZika virus infection and risk of Guillain-Barré syndrome: A meta-analysis
Bautista, L. E. (2019) · Journal of the Neurological Sciences · doi:10.1016/j.jns.2019.06.019 · Cited by 16
Comparing Analysis Blinding With Preregistration in the Many-Analysts Religion Project
Sarafoglou, A., Hoogeveen, S., & Wagenmakers, E. (2023) · Advances in Methods and Practices in Psychological Science · doi:10.1177/25152459221128319 · Cited by 18
“An alternative to preregistration is analysis blinding (Dutilh et al., 2019; MacCoun, 2020; MacCoun & Perlmutter, 2015, 2018).”
The post hoc measurement as a safe and reliable method to age and size plethodontid salamanders
Lunghi, E., Giachello, S., Manenti, R., Zhao, Y., Corti, C., Ficetola, G. F., & Bradley, J. G. (2020) · Ecology and Evolution · doi:10.1002/ece3.6748 · Cited by 16
“…this area shows lateral “folds,” of which the third from the hind limb roughly corresponds to the tail base (Figure 2b), and thus the posterior end of the cloaca used in SVL. Photographs were provided without any additional information to ensure unbiased measurements (MacCoun & Perlmutter, 2015).”
MethodAddressing selective reporting of experiments through predefined exclusion criteria
Neves, K., & Amaral, O. B. (2020) · eLife · doi:10.7554/elife.56626 · Cited by 15
Metadata
- Authors
- MacCoun, R., & Perlmutter, S.
- Affiliations
- Stanford University (US)
- University of California, Berkeley (US)
- Corresponding author
- MacCoun, Robert
- Perlmutter, Saul
- Venue
- Nature
- Published
- 2015-10-07
- Volume
- 526
- Issue
- 7572
- Pages
- 187-189
- ISSN
- 0028-0836 (ISSN-L)
- 1476-4687
- OpenAlex
- W2192523408
- Cited by
- 218
- References
- 9
- Citations by year
- 2026: 14 · 2025: 15 · 2024: 19 · 2023: 22 · 2022: 12 · 2021: 17 · 2020: 43 · 2019: 19 · 2018: 14 · 2017: 23 · 2016: 20
- FWCI
- 17.33
- Type
- journal article
- Language
- en
- Open access
- closed
- Licence
- www.springernature.com/gp/researchers/text-and-data-mining (tdm)
- www.springernature.com/gp/researchers/text-and-data-mining (vor)
- Locations
- Nature (publishedVersion)
- PubMed (publishedVersion)
- RePEc: Research Papers in Economics (submittedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Library and Information Sciences › Academic Publishing and Open Access
- Physical Sciences › Mathematics › Statistics and Probability › Advanced Causal Inference Techniques
- Keywords
- blind analysis, research bias, publication bias, information hiding, scientific integrity, particle physics, reproducibility
- MeSH terms
- Biological Science Disciplines, Double-Blind Method, Humans, Models, Theoretical, Motivation, Physics, Prejudice, Psychology, Research Design, Research Personnel, Social Sciences, Reproducibility of Results, Bias
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.