Type S error rates for classical and Bayesian single and multiple comparison procedures
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Bayesian analysis allows researchers to conduct interim analyses without being penalized because decisions at analysis points are based on posterior probabilities of treatment effects.
“Bayesian analysis allows researchers to conduct interim analyses without being penalized because decisions at analysis points are based on the posterior probabilities of treatment effects [47].”
From Outcomes of an App-Based Intervention to Target Naming Among Individuals With Poststroke Aphasia: Virtual Randomized Controlled Trial (2025) · cites this paper as [47]
Bayesian inference was chosen for accurate multiple comparisons.
“Bayesian inference was chosen for accurate multiple comparisons [47].”
From Outcomes of an App-Based Intervention to Target Naming Among Individuals With Poststroke Aphasia: Virtual Randomized Controlled Trial (2025) · cites this paper as [47]
Cites
Gelman, A., Carlin, J. B., Stern, H. S., Dunson, D. B., Vehtari, A., & Rubin, D. B. (2013) · doi:10.1201/b16018 · Cited by 27,276 · see citing works
Self-citationExperimental Design: Procedures for the Behavioral Sciences
Kirk, R. (2013) · doi:10.4135/9781483384733 · Cited by 8,539 · see citing works
Journal of The American Statistical Association
Fienberg, S. E. (2004) · Encyclopedia of Statistical Sciences · doi:10.1002/0471667196.ess1317 · Cited by 6,289 · see citing works
Scheffé, H. (1960) · Soil Science · doi:10.1097/00010694-196006000-00016 · Cited by 5,625 · see citing works
Bayes and Empirical Bayes Methods for Data Analysis
Carlin, B. P., & Louis, T. A. (2001) · Technometrics · doi:10.1198/tech.2001.s608 · Cited by 2,110 · see citing works
That BLUP is a Good Thing: The Estimation of Random Effects
Robinson, G. K. (1991) · Statistical Science · doi:10.1214/ss/1177011926 · Cited by 1,694 · see citing works
Bayesianly Justifiable and Relevant Frequency Calculations for the Applied Statistician
Rubin, D. (1984) · The Annals of Statistics · doi:10.1214/aos/1176346785 · Cited by 1,503 · see citing works
Parametric empirical Bayes inference: theory and applications
Morris, C. N. (1983) · Journal of the American Statistical Association · doi:10.1080/01621459.1983.10477920 · Cited by 1,448 · see citing works
Klockars, A. J., & Sax, G. (1986) · Sage eBooks · doi:10.4135/9781412985185 · Cited by 1,015 · see citing works
Testing a Point Null Hypothesis: The Irreconcilability of P Values and Evidence
Berger, J. O., & Sellke, T. (1987) · Journal of the American Statistical Association · doi:10.1080/01621459.1987.10478397 · Cited by 971 · see citing works
Hierarchical Generalized Linear Models
Lee, Y., & Nelder, J. A. (1996) · Journal of the Royal Statistical Society Series B (Statistical Methodology) · doi:10.1111/j.2517-6161.1996.tb02105.x · Cited by 958 · see citing works
Posterior Predictive $p$-Values
Meng, X. (1994) · The Annals of Statistics · doi:10.1214/aos/1176325622 · Cited by 792 · see citing works
Berger, J. O., & Delampady, M. (1987) · Statistical Science · doi:10.1214/ss/1177013238 · Cited by 664 · see citing works
What If There Were No Significance Tests?
(2016) · Routledge eBooks · doi:10.4324/9781315629049 · Cited by 543 · see citing works
Bayes and Empirical Bayes Methods for Data Analysis
Rukhin, A. L., Carlin, B. P., & Louis, T. A. (1997) · Technometrics · doi:10.2307/1271144 · Cited by 454 · see citing works
Reconciling Bayesian and Frequentist Evidence in the One-Sided Testing Problem
Casella, G., & Berger, R. L. (1987) · Journal of the American Statistical Association · doi:10.1080/01621459.1987.10478396 · Cited by 417 · see citing works
Asymptotic Distribution of P Values in Composite Null Models
Robins, J. M., van der Vaart, A., & Ventura, V. (2000) · Journal of the American Statistical Association · doi:10.1080/01621459.2000.10474310 · Cited by 176 · see citing works
Tukey, J. W. (1960) · Technometrics · doi:10.1080/00401706.1960.10489909 · Cited by 169 · see citing works
Asymptotic Distribution of P Values in Composite Null Models
Robins, J. M. W., van der Vaart, A., & Ventura, V. (2000) · Journal of the American Statistical Association · doi:10.2307/2669750 · Cited by 111 · see citing works
Testing 'Small', not Null, Hypotheses: Classical and Bayesian Approaches
Harlow, L. L., Mulaik, S. A., & Steiger, J. H. (2013) · doi:10.4324/9781315629049-26 · Cited by 27 · see citing works
Reforming Significance Testing via Three-Valued Logic
Harris, R. J. (2016) · doi:10.4324/9781315629049-19 · Cited by 19 · see citing works
An Introduction to Bayesian Inference and Its Applications
Pruzek, R. M. (2016) · doi:10.4324/9781315629049-25 · Cited by 9 · see citing works
Flagged first, then by citations
1 of the 23 references couldn’t be matched to a known paper.
Cited by
Sorted by citations
Multimodel inference in ecology and evolution: challenges and solutions
Grueber, C. E., Nakagawa, S., Laws, R., & Jamieson, I. G. (2011) · Journal of Evolutionary Biology · doi:10.1111/j.1420-9101.2010.02210.x · Cited by 2,277 · see citing works
“It should be borne in mind, however, that assigning the incorrect sign to a weak parameter estimate is a possibility in any regression (Gelman & Tuerlinckx, 2000), and further research as to the effects of model averaging on this type of error would be useful.”
BackgroundBeyond Power Calculations: Assessing Type S (Sign) and Type M (Magnitude) Errors
Gelman, A., & Carlin, J. (2014) · Perspectives on Psychological Science · doi:10.1177/1745691614551642 · Cited by 1,343 · see citing works
“We frame our calculations not in terms of Type 1 and Type 2 errors but rather Type S (sign) and Type M (magnitude) errors, which relate to the probability that claims with confidence have the wrong sign or are far in magnitude from underlying effect sizes (Gelman & Tuerlinckx, 2000).”
False discovery rates: a new deal
Stephens, M. (2016) · Biostatistics · doi:10.1093/biostatistics/kxw041 · Cited by 1,053 · see citing works
“Along the same lines, Gelman and co-workers (Gelman and Tuerlinckx, 2000; Gelman and others, 2012) suggest focusing on “type S errors,” meaning errors in sign, rather than the more traditional type I errors.”
BackgroundWhy We (Usually) Don't Have to Worry About Multiple Comparisons
Gelman, A., Hill, J., & Yajima, M. (2012) · Journal of Research on Educational Effectiveness · doi:10.1080/19345747.2011.618213 · Cited by 1,274 · see citing works
“These are both examples of what is referred to as “Type S” (sign) errors (Gelman and Tuerlinckx, 2000).”
BackgroundInfluentialBayes factor design analysis: Planning for compelling evidence
Schönbrodt, F. D., & Wagenmakers, E. (2017) · Psychonomic Bulletin & Review · doi:10.3758/s13423-017-1230-y · Cited by 732 · see citing works
Linear and generalized linear mixed models
Bolker, B. M. (2015) · Oxford University Press eBooks · doi:10.1093/acprof:oso/9780199672547.003.0014 · Cited by 358 · see citing works
Humphreys, M., Sierra, R., & van der Windt, P. C. (2013) · Political Analysis · doi:10.1093/pan/mps021 · Cited by 293 · see citing works
“2 But as highlighted above the problem extends beyond this type of testing and could obtain for example in approaches that focus on “Type S” errors, seeking to make statements of the form “θ1 > θ2, with confidence” (see for example Gelman and Tuerlinckx (2000)). This kind of concern has led some to worry that “most cu…”
BackgroundMethodInfluentialWhen Null Hypothesis Significance Testing Is Unsuitable for Research: A Reassessment
Szucs, D., & Ioannidis, J. P. A. (2017) · Frontiers in Human Neuroscience · doi:10.3389/fnhum.2017.00390 · Cited by 298 · see citing works
“, Wellek, 2010) or researchers may check for the sign of expected effects (Gelman and Tuerlinckx, 2000).”
BackgroundMethodInfluentialProcessing Polarity: How the Ungrammatical Intrudes on the Grammatical
Vasishth, S., Brüssow, S., Lewis, R. L., & Drenhaus, H. (2008) · Cognitive Science · doi:10.1080/03640210802066865 · Cited by 268 · see citing works
How Much Can We Generalize From Impact Evaluations?
Vivalt, E. (2020) · Journal of the European Economic Association · doi:10.1093/jeea/jvaa019 · Cited by 244 · see citing works
(2015) · doi:10.1093/acprof:oso/9780199672547.001.0001 · Cited by 190 · see citing works
A knockoff filter for high-dimensional selective inference
Barber, R. F., & Candès, E. (2016) · Annals of Statistics · doi:10.1214/18-aos1755 · Cited by 212 · see citing works
“[12] Andrew Gelman and Francis Tuerlinckx.”
BackgroundHanks, E. M., Schliep, E. M., Hooten, M. B., & Hoeting, J. A. (2015) · Environmetrics · doi:10.1002/env.2331 · Cited by 166 · see citing works
“A Bayesian analogue to the Type-1 error is the Type-S error of Gelman and Tuerlinckx (2000).”
MethodDetecting Differentially Expressed Genes in Microarrays Using Bayesian Model Selection
Ishwaran, H., & Rao, J. S. (2003) · Journal of the American Statistical Association · doi:10.1198/016214503000224 · Cited by 165 · see citing works
“However, whenever O ¿ 20 1 , the Bayesian test (9) will always be more conservative (see Gelman and Tuerlinckx 2000 for further discussion).”
BackgroundGrace, J. B., Scheiner, S. M., & Schoolmaster, D. R. (2015) · Oxford University Press eBooks · doi:10.1093/acprof:oso/9780199672547.003.0009 · Cited by 156 · see citing works
Is the statistic value all we should care about in neuroimaging?
Chen, G., Taylor, P. A., & Cox, R. W. (2016) · NeuroImage · doi:10.1016/j.neuroimage.2016.09.066 · Cited by 139 · see citing works
“…of failure or success, respectively, to detect the effect), but it would also be impossible to assess two other useful but less known errors (Gelman and Tuerlinckx, 2000): type M (tendency to over- or under-estimate the effect magnitude) and type S (likelihood of obtaining the incorrect…”
BackgroundGelman, A., & Weakliem, D. L. (2009) · American Scientist · doi:10.1511/2009.79.310 · Cited by 137 · see citing works
Bayesian Meta-Analysis with Weakly Informative Prior Distributions
Williams, D. R., Rast, P., & Bürkner, P. C. (2018) · doi:10.31234/osf.io/7tbrm · Cited by 124 · see citing works
“Further, there are other concepts such as type S (sign) and M (magnitude) errors (Gelman & Carlin, 2014; Gelman & Tuerlinckx, 2000), which are arguably more appropriate to describe errors in statistical inference than type I and II errors (Chen et al., 2017; Gelman, Hill, & Yajima, 2012).”
BackgroundSchell, T. L., Cefalu, M., Griffin, B. A., Smart, R., & Morral, A. R. (2020) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1921965117 · Cited by 118 · see citing works
“Using Bayesian inference generally avoids these problems in the same data when estimated with modestly informative priors (22).”
BackgroundThe Connection Between Varying Treatment Effects and the Crisis of Unreplicable Research
Gelman, A. (2015) · Journal of Management · doi:10.1177/0149206314525208 · Cited by 125 · see citing works
“First, uncertainty: when studying small effects, it is very possible for a large proportion of statistically significant findings to be in the wrong direction as well as be gross overestimates of the magnitude of the underlying effect (Gelman and Tuerlinckx, 2000).”
BackgroundPeyrot, W. J., & Price, A. L. (2021) · Nature Genetics · doi:10.1038/s41588-021-00787-1 · Cited by 106 · see citing works
Kanen, J. W., Ersche, K. D., Fineberg, N. A., Robbins, T. W., & Cardinal, R. N. (2019) · Psychopharmacology · doi:10.1007/s00213-019-05325-w · Cited by 101 · see citing works
“the top level), leading to an automatic multiple comparisons correction without the reduction in power seen with the classical approach (Gelman and Tuerlinckx 2000; Gelman et al. 2012; Kruschke 2011b).”
BackgroundExploring author gender in book rating and recommendation
Ekstrand, M. D., Tian, M., Kazi, M. I. R., Mehrpouyan, H., & Kluver, D. (2018) · ACM Conference on Recommender Systems (RecSys) · doi:10.1145/3240323.3240373 · Cited by 110 · see citing works
Brown, V. M., Zhu, L., Solway, A., Wang, J. M., McCurry, K. L., King-Casas, B., & Chiu, P. H. (2021) · JAMA Psychiatry · doi:10.1001/jamapsychiatry.2021.1844 · Cited by 112 · see citing works
Introducing an automated high content confocal imaging approach for Organs-on-Chips
Peel, S. E., Corrigan, A. M., Ehrhardt, B., Jang, K., Caetano-Pinto, P., Boeckeler, M., Rubins, J. E., Kodella, K. R., Petrópolis, D. B., Ronxhi, J., et al. (2019) · Lab on a Chip · doi:10.1039/c8lc00829a · Cited by 98 · see citing works
Losapio, G., Schöb, C., Staniczenko, P. P. A., Carrara, F., Palamara, G. M., Moraes, C. Μ. D., Mescher, M. C., Brooker, R. W., Butterfield, B. J., Callaway, R. M., et al. (2021) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2005759118 · Cited by 94 · see citing works
“We quantified the frequency of four network modules formed by the associations among three species: 1) intransitive competition (A→B, B→C, C→A) by using the triad census function in igraph R package (40); 2) facilitation-driven competition (A→B, A→C, B↔C) by searching for a species j that had a positive association wi…”
MethodInvestigating the Effect of the Multiple Comparisons Problem in Visual Analysis
Zgraggen, E., Zhao, Z., Zeleznik, R. C., & Kraska, T. (2018) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3173574.3174053 · Cited by 94 · see citing works
“Beyond Type I and Type II, other error types have been proposed to quantify the likelihood of mistaking the sign (Type S) or overestimating the magnitude (Type M) [22, 20].”
BackgroundECOLOGICAL APPLICATIONS OF MULTILEVEL ANALYSIS OF VARIANCE
Qian, S. S., & Shen, Z. (2007) · Ecology · doi:10.1890/06-2041.1 · Cited by 90 · see citing works
“Gelman and Tuerlinckx (2000) suggested59 that the hierarchical Bayesian approach for ANOVA includes the classical ANOVA60 as a special case.”
BackgroundMethodChoosing Your Neighbors: Networks of Diffusion in International Relations
Zhukov, Y., & Stewart, B. M. (2013) · International Studies Quarterly · doi:10.1111/isqu.12008 · Cited by 85 · see citing works
Yang, Y., Sánchez‐Tójar, A., O’Dea, R. E., Noble, D. W. A., Koricheva, J., Jennions, M. D., Parker, T. H., Lagisz, M., & Nakagawa, S. (2023) · BMC Biology · doi:10.1186/s12915-022-01485-y · Cited by 81 · see citing works
“Psychological and economic research also confirm that meta-analyses without bias adjustments overestimate the estimate of statistical power [27, 28].”
BackgroundSmith, R. S., Schwartenbeck, P., Stewart, J. L., Kuplicki, R., Ekhtiari, H., T Investigators, & Paulus, M. P. (2020) · Drug and Alcohol Dependence · doi:10.1016/j.drugalcdep.2020.108208 · Cited by 78 · see citing works
Handling Multiplicity in Neuroimaging Through Bayesian Lenses with Multilevel Modeling
Chen, G., Xiao, Y., Taylor, P. A., Rajendra, J. K., Riggins, T., Geng, F., Redcay, E., & Cox, R. W. (2019) · Neuroinformatics · doi:10.1007/s12021-018-9409-6 · Cited by 74 · see citing works
“Simulations (Gelman and Tuerlinckx 2000) indicate that, when making inference based on the 95% quantile interval of the posterior distribution for a single effect θj (j is fixed, e.”
MethodBlinding Us to the Obvious? The Effect of Statistical Training on the Evaluation of Evidence
McShane, B. B., & Gal, D. (2015) · Management Science · doi:10.1287/mnsc.2015.2212 · Cited by 77 · see citing works
“For instance, a large fraction of effects that attain statistical significance are either of the wrong sign or of a greatly biased magnitude when the underlying effect size is small, a problem known as the statistical significance filter (Gelman and Tuerlinckx 2000, Gelman and Weakliem 2009).”
BackgroundNakagawa, S. (2015) · Oxford University Press eBooks · doi:10.1093/acprof:oso/9780199672547.003.0005 · Cited by 75 · see citing works
Zampieri, F. G., Costa, E. L. V., Iwashyna, T. J., de Carvalho, C. R. R., Damiani, L. P., Taniguchi, L. U., Amato, M. B. P., & Cavalcanti, A. B. (2019) · British Journal of Anaesthesia · doi:10.1016/j.bja.2019.02.026 · Cited by 76 · see citing works
Investigating evolutionary models of leadership among recently settled Ethiopian hunter-gatherers
Garfield, Z. H., & Hagen, E. (2020) · Leadership Quarterly · doi:10.1016/j.leaqua.2019.03.005 · Cited by 62 · see citing works
“Additionally, these regularizing priors introduce greater conservatism on parameter estimates and have been demonstrated to reduce Type-S error rates relative to frequentist approaches or flat priors (Gelman & Tuerlinckx, 2000; Ghosh, Li, & Mitra, 2018).”
BackgroundComparing sign language and gesture: Insights from pointing
Fenlon, J., Cooperrider, K., Keane, J., Brentari, D., & Susan Goldin-meadow, A. (2019) · Glossa a journal of general linguistics · doi:10.5334/gjgl.499 · Cited by 72 · see citing works
False discovery rate control via debiased lasso
Javanmard, A., & Javadi, H. (2019) · Electronic Journal of Statistics · doi:10.1214/19-ejs1554 · Cited by 68 · see citing works
“[GT00] argue that type S error is a more relevant notion in many applications.”
BackgroundExploring author gender in book rating and recommendation
Ekstrand, M. D., & Kluver, D. (2021) · User Modeling and User-Adapted Interaction · doi:10.1007/s11257-020-09284-2 · Cited by 66 · see citing works
Should the Randomistas (Continue to) Rule?
Ravallion, M. D. (2020) · Oxford University Press eBooks · doi:10.1093/oso/9780198865360.003.0003 · Cited by 62 · see citing works
Mowinckel, A. M., Alnæs, D., Pedersen, M. L., Ziegler, S., Fredriksen, M., Kaufmann, T., Sonuga-Barke, E. J. S., Endestad, T., Westlye, L. T., & Biele, G. (2017) · NeuroImage Clinical · doi:10.1016/j.nicl.2017.03.008 · Cited by 62 · see citing works
“…models, the point estimates are shifted towards each other and towards the main effect(s), resulting in posterior densities that are appropriately conservative without sacrificing power (Gelman et al., 2012), which negates the need for multiple comparison corrections (Gelman and Tuerlinckx, 2000).”
Bayesian Data Analysis in Empirical Software Engineering Research
Furia, C. A., FELDT, R. H., & Torkar, R. (2019) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2019.2935974 · Cited by 57 · see citing works
“Instead, Gelman recommends [40] focusing on type S and type M errors.”
MethodHarrison, P. F., Pattison, A., Powell, D., & Beilharz, T. H. (2019) · Genome biology · doi:10.1186/s13059-019-1674-7 · Cited by 55 · see citing works
Statistical evidence and sample size determination for Bayesian hypothesis testing
Santis, F. D. (2004) · Journal of Statistical Planning and Inference · doi:10.1016/s0378-3758(03)00198-8 · Cited by 55 · see citing works
Our better nature: Does resource stress predict beyond-household sharing?
Ember, C. R., Skoggard, I. A., Ringen, E. J., & Farrer, M. (2018) · Evolution and Human Behavior · doi:10.1016/j.evolhumbehav.2018.03.001 · Cited by 52 · see citing works
Likelihood and model selection
Richards, S. A. (2015) · Oxford University Press eBooks · doi:10.1093/acprof:oso/9780199672547.003.0004 · Cited by 48 · see citing works
Looking at Blazar Light-curve Periodicities with Gaussian Processes
Covino, S., Landoni, M., Sandrinelli, A., & Treves, A. (2020) · The Astrophysical Journal · doi:10.3847/1538-4357/ab8bd4 · Cited by 48 · see citing works
“However, the computation of the Bayes factor already includes the multi-trial correction, since it is embedded in the integration on the allowed parameter space defined by the priors on the analysis (e.g., Gelman & Tuerlinckx 2000; Trotta 2007; Gelman et al. 2012).”
MethodCorrelated evolution of social organization and lifespan in mammals
Zhu, P., Liu, W., Zhang, X., Li, M., Liu, G., Yu, Y., Li, X., Li, X., Du, J., Wang, X., et al. (2023) · Nature Communications · doi:10.1038/s41467-023-35869-7 · Cited by 48 · see citing works
Quantitative Political Science Research Is Greatly Underpowered
Arel‐Bundock, V., Briggs, R. C., Doucouliagos, H., Aviña, M. M., & Stanley, T. D. (2024) · The Journal of Politics · doi:10.1086/734279 · Cited by 49 · see citing works
“When power is low and one happens to find a statistically significant estimate, that estimate is likely to be many-fold greater than the “true” underlying effect, and it may well have the wrong sign (Gelman and Carlin 2014; Gelman and Tuerlinckx 2000; Ioannidis, Stanley, and Doucouliagos 2017).”
Effect of lysergic acid diethylamide (LSD) on reinforcement learning in humans
Kanen, J. W., Luo, Q., Kandroodi, M. R., Cardinal, R. N., Robbins, T. W., Nutt, D. J., Carhart-Harris, R. L., & den Ouden, H. E. M. (2022) · Psychological Medicine · doi:10.1017/s0033291722002963 · Cited by 49 · see citing works
“This approach also removes distributional assumptions and provides an automatic multiple comparisons correction (Gelman and Tuerlinckx 2000; Gelman et al. 2012; Kruschke 2011).”
BackgroundFranco, A., Malhotra, N., & Simonovits, G. (2015) · Political Analysis · doi:10.1093/pan/mpv006 · Cited by 44 · see citing works
Benoit, C.-É., Solopchuk, O., Borragán, G., Carbonnelle, A., van Durme, S., & Zénon, A. (2018) · Neuropsychologia · doi:10.1016/j.neuropsychologia.2018.06.017 · Cited by 53 · see citing works
“Finally, it mitigates considerably the issue of multiple comparison (Gelman and Tuerlinckx, 2000).”
The evolution of daily food sharing: A Bayesian phylogenetic analysis
Ringen, E. J., Duda, P., & Jaeggi, A. V. (2019) · Evolution and Human Behavior · doi:10.1016/j.evolhumbehav.2019.04.003 · Cited by 47 · see citing works
“Using such priors greatly reduces both Type-S (inferring the wrong sign for an effect) and Type-M error (inferring that an effect is of greater magnitude than it is) (Gelman & Carlin, 2014; Gelman & Tuerlinckx, 2000).”
Randomized Control Trials in the Field of Development
Bédécarrats, F., Guérin, I., & Roubaud, F. (2020) · doi:10.1093/oso/9780198865360.001.0001 · Cited by 48 · see citing works
Breitborde, N. J. K., Woolverton, C. B., Dawson, S. C., Bismark, A. W., Bell, E. K., Bathgate, C. J., & Norman, K. R. (2015) · Early Intervention in Psychiatry · doi:10.1111/eip.12289 · Cited by 45 · see citing works
“21,22 The benefits of this approach compared with other strategies (e.g. Bonferroni corrections) are well documented elsewhere.”
ResultA Review of Cognitive Abilities in Dogs, 1911 Through 2016
Arden, R., Bensky, M. K., & Adams, M. J. (2016) · Current Directions in Psychological Science · doi:10.1177/0963721416667718 · Cited by 44 · see citing works
“…significance (e.g., α = .001), which promotes higher replicability in animal cognition studies, reduces the number of false positives reported, enables the accurate estimation of all effects both large and small, and allows negative findings to be more conclusive (Gelman & Tuerlinckx, 2000).”
BackgroundEvidence for third-party mediation but not punishment in Mentawai justice
Singh, M., & Garfield, Z. H. (2022) · Nature Human Behaviour · doi:10.1038/s41562-022-01341-7 · Cited by 47 · see citing works
Statistical Significance Testing in Information Retrieval
Urbano, J., Lima, H., & Hanjalić, A. (2019) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3331184.3331259 · Cited by 43 · see citing works
Response to CO 2 enrichment of understory vegetation in the shade of forests
Kim, D., Ram Oren, G., & Qian, S. S. (2015) · Global Change Biology · doi:10.1111/gcb.13126 · Cited by 42 · see citing works
“…the null was considered to be less rigorous as the probability of type II error is unknown (e.g. Steidl et al., 1997), we used the BHM approach (as discussed in Gelman & Tuerlinckx, 2000) for our inference, thereby avoiding the problems associated with the classical hypothesis testing method.”
MethodThe dark side of the force: Multiplicity issues in network meta‐analysis and how to address them
Efthimiou, O., & White, I. R. (2019) · Research Synthesis Methods · doi:10.1002/jrsm.1377 · Cited by 39 · see citing works
Methodological challenges and proposed solutions for evaluating opioid policy effectiveness
Schuler, M. S., Griffin, B. A., Cerdá, M., McGinty, E. E., & Stuart, E. A. (2020) · Health Services and Outcomes Research Methodology · doi:10.1007/s10742-020-00228-2 · Cited by 40 · see citing works
“Second, the use of Bayesian inference with modestly informative priors mitigates the limited power problems inherent in frequentist models applied to state-level policy evaluations (Gelman and Tuerlinckx 2000).”
MethodA New Look at P Values for Randomized Clinical Trials
van Zwet, E. W., Gelman, A., Greenland, S., Imbens, G. W., Schwab, S., & Goodman, S. N. (2023) · NEJM Evidence · doi:10.1056/evidoa2300003 · Cited by 39 · see citing works
Estimation and Prediction in the Presence of Spatial Confounding for Spatial Linear Models
Page, G. L., Liu, Y., He, Z., & Sun, D. (2017) · Scandinavian Journal of Statistics · doi:10.1111/sjos.12275 · Cited by 39 · see citing works
“See Gelman and Tuerlinckx (2000) for more details.) Paciorek (2010) also showed that effective range is an important factor in spatial confounding bias. We briefly note that Paciorek (2010) used the term ‘spatial scale’ in place of ‘spatial range’. This terminology may be a bit confusing (‘spatial scale’ can also refe…”
BackgroundInfluentialMixed-effects design analysis for experimental phonetics
Kirby, J., & Sonderegger, M. (2018) · Journal of Phonetics · doi:10.1016/j.wocn.2018.05.005 · Cited by 37 · see citing works
Time-Based Learning Curve for Robotic-Assisted Total Knee Arthroplasty: A Multicenter Study
Chen, Z., Bhowmik-Stoker, M., Palmer, M., Coppolecchia, A., Harder, B., Mont, M. A., & Marchand, R. C. (2022) · The Journal of Knee Surgery · doi:10.1055/s-0042-1744193 · Cited by 37 · see citing works
A note on Type S/M errors in hypothesis testing
Lu, J., Qiu, Y., & Deng, A. (2018) · British Journal of Mathematical and Statistical Psychology · doi:10.1111/bmsp.12132 · Cited by 35 · see citing works
“…a true effect size l is non-zero, that is, H0 : l ¼ 0 versus H1 : l 6¼ 0: The above two-sided setting is common inpsychological research, andwe followprevious discussions (e.g., Gelman & Carlin, 2014; Gelman & Tuerlinckx, 2000) and adopt it throughout this paper, for the purpose of illustration.”
BackgroundInfluentialHuygelier, H., Schraepen, B., Demeyere, N., & Gillebert, C. R. (2019) · Aging Neuropsychology and Cognition · doi:10.1080/13825585.2019.1680598 · Cited by 35 · see citing works
“Moreover, the Bayesian method is more conservative than Frequentist methods and allows to compare multiple contrasts without inflation of false positives (Gelman & Tuerlinckx, 2000; Kruschke, 2010).”
MethodLoot box spending is associated with problem gambling but not mental wellbeing
Etchells, P. J., Morgan, A. L., & Quintana, D. S. (2022) · Royal Society Open Science · doi:10.1098/rsos.220111 · Cited by 38 · see citing works
“However, this is not a concern when using a Bayesian approach as long as an appropriate prior distribution is used [32].”
BackgroundPawel, S., Kook, L., & Reeve, K. A. (2023) · Biometrical Journal · doi:10.1002/bimj.202200091 · Cited by 37 · see citing works
“…more likely to remain undetected (increased Type II errors), detected differences are more likely to be in the wrong direction (increased “Type S” errors, see Gelman & Tuerlinckx, 2000), and their magnitude is more likely to be overestimated (increased “Type M” errors, see van Zwet & Cator, 2021).”
BackgroundNakagawa, S., Lagisz, M., Yang, Y., & Drobniak, S. M. (2024) · PLoS Biology · doi:10.1371/journal.pbio.3002423 · Cited by 40 · see citing works
Determinants of coastal treeline and the role of abiotic and biotic interactions
Poulter, B., Qian, S. S., & Christensen, N. L. (2008) · Plant Ecology · doi:10.1007/s11258-008-9465-3 · Cited by 36 · see citing works
“The graphical presentation of the multilevel ANOVA results, which facilitates the new hypothesis testing and multiple comparison procedures of Gelman and Tuerlinckx (2000), are discussed in detail in Qian and Shen (2007, Appendix B, Ecological Archives E088-150-A2).”
MethodMarcondes, R. S., Nations, J. A., Seeholzer, G. F., & Brumfield, R. T. (2021) · The American Naturalist · doi:10.1086/713386 · Cited by 36 · see citing works
“…us to fit linear models with multiple predictor variables (Gelman 2006; McElreath 2016), use regularizing priors that minimize false discovery (Gelman and Tuerlinckx 2000; Gelman et al. 2012), and include group-level effects that incorporate the error from the statistical nonindependence of…”
BackgroundUncertainty in Online Experiments with Dependent Data: An Evaluation of Bootstrap MethodsPreprint
Bakshy, E., & Eckles, D. (2013) · arXiv (Cornell University) · doi:10.48550/arxiv.1304.7406 · Cited by 35 · see citing works
“In each of our three datasets, the iid bootstrap performed very poorly, such that using it (or other methods assuming iid observations) would result in reaching incorrect conclusions about the presence, sign, and magnitude of treatment effects [9].”
MethodBuckley, Y. M. (2015) · Oxford University Press eBooks · doi:10.1093/acprof:oso/9780199672547.003.0007 · Cited by 36 · see citing works
Uncertainty in online experiments with dependent data
Bakshy, E., & Eckles, D. (2013) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/2487575.2488218 · Cited by 34 · see citing works
“In each of our three datasets, the iid bootstrap performed very poorly, such that using it (or other methods assuming iid observations) would result in reaching incorrect conclusions about the presence, sign, and magnitude of treatment effects [9].”
MethodStatistical analysis of genetic interactions in Tn-Seq data
DeJesus, M. A., Nambi, S., Smith, C. M., Baker, R. E., Sassetti, C. M., & Ioerger, T. R. (2017) · Nucleic Acids Research · doi:10.1093/nar/gkx128 · Cited by 33 · see citing works
“The method may also detect spurious differences which are not biologically meaningful, because it only tests genes against a null hypothesis of no difference, a common criticism of classical hypothesis tests typical of frequentist methods (41).”
MethodType M Error Might Explain Weisburd’s Paradox
Gelman, A., Skarðhamar, T., & Aaltonen, M. (2017) · Journal of Quantitative Criminology · doi:10.1007/s10940-017-9374-5 · Cited by 33 · see citing works
“With these concerns in mind, Gelman and Tuerlinckx (2000) introduced the concept of type S (“sign”) and type M (“magnitude”) errors, both of which can occur when a researcher makes a claim with confidence (typically, a p-value of less than 0.05 or a confidence interval that excludes zero, but more…”
BackgroundInfluentialQuantifying sentence acceptability measures: Reliability, bias, and variability
Langsford, S., Hendrickson, A. T., Perfors, A., Kennedy, L., & Navarro, D. (2018) · Glossa a journal of general linguistics · doi:10.5334/gjgl.396 · Cited by 33 · see citing works
Friedl, W. M., III, & Keil, A. (2021) · Journal of Neuroscience · doi:10.1523/jneurosci.2889-20.2021 · Cited by 28 · see citing works
“Questions of when, whether, and how to adjust Bayesian statistical results for multiple comparisons are currently unsettled (Berry and Hochberg, 1999; Gelman and Tuerlinckx, 2000; Dienes, 2008, 2014; Gelman et al., 2012; Johnson, 2013).”
BackgroundNations, J. A., Heaney, L. R., Demos, T. C., Achmadi, A. S., Rowe, K. C., & Esselstyn, J. A. (2019) · Biological Journal of the Linnean Society · doi:10.1093/biolinnean/blz085 · Cited by 31 · see citing works
Richard‐Greenblatt, M., Ziegler, M. J., Bromberg, V., Huang, E., Abdallah, H. O., Tolomeo, P., Lautenbach, E., Glaser, L. J., & Kelly, B. J. (2021) · Open Forum Infectious Diseases · doi:10.1093/ofid/ofab235 · Cited by 31 · see citing works
“We anticipated that approximately 800 subjects would detect a 10% reduction in sensitivity of SARS-CoV-2 detection related to a β-actin Ct increase of 10, with CrI precision ensuring type S error <5% [31, 32].”
BackgroundEvaluating Methods to Estimate the Effect of State Laws on Firearm Deaths: A Simulation Study
Schell, T. L., Griffin, B. A., & Morral, A. R. (2018) · RAND Corporation eBooks · doi:10.7249/rr2685 · Cited by 31 · see citing works
How Pointing is Integrated into Language: Evidence From Speakers and Signers
Cooperrider, K., Fenlon, J., Keane, J., Brentari, D., & Susan Goldin-meadow, A. (2021) · Frontiers in Communication · doi:10.3389/fcomm.2021.567774 · Cited by 30 · see citing works
“To judge the direction and magnitude of effects, we use CIs, rather than p values (e.g., Gelman and Tuerlinckx, 2000; Cumming and Finch, 2005).”
MethodAltoè, G., Bertoldo, G., Callegher, C. Z., Toffalini, E., Calcagnì, A., Finos, L., & Pastore, M. (2020) · Frontiers in Psychology · doi:10.3389/fpsyg.2019.02893 · Cited by 31 · see citing works
Of beauty, sex, and power: Statistical challenges in estimating small eects
Gelman, A., & Weakliem, D. L. (2008) · Cited by 29 · see citing works
Alleviating Spatial Confounding for Areal Data Problems by Displacing the Geographical Centroids
Prates, M. O., Assunção, R., & Rodrigues, É. C. (2018) · Bayesian Analysis · doi:10.1214/18-ba1123 · Cited by 29 · see citing works
“The third, study the Type-S error rate (Gelman and Tuerlinckx, 2000) and coverage of the proposed methods from an areal data perspective.”
Computer Mouse Movements as an Indicator of Work Stress: Longitudinal Observational Field Study
Banholzer, N., Feuerriegel, S., Fleisch, E., Bauer, G. F., & Kowatsch, T. (2021) · Journal of Medical Internet Research · doi:10.2196/27121 · Cited by 29 · see citing works
“However, when choosing a Bayesian prior (eg, a normal prior), the results are different, and sign errors are less frequent with a Bayesian prior [25].”
Pritchett, L. H. (2020) · Oxford University Press eBooks · doi:10.1093/oso/9780198865360.003.0004 · Cited by 28 · see citing works
Smith, R. S., Taylor, S., Stewart, J. L., Guinjoan, S. M., Ironside, M., Kirlić, N., Ekhtiari, H., White, E. J., Zheng, H., Kuplicki, R., et al. (2022) · Computational Psychiatry · doi:10.5334/cpsy.85 · Cited by 27 · see citing works
Chen, G., Taylor, P. A., Stoddard, J., Cox, R. W., Bandettini, P. A., & Pessoa, L. (2022) · Aperture Neuro · doi:10.52294/apertureneuro.2022.2.zrji8542 · Cited by 28 · see citing works
Abramowicz, M., & Szafarz, A. (2019) · Oxford University Press eBooks · doi:10.1093/oso/9780198865360.003.0012 · Cited by 34 · see citing works
An integrative Bayesian approach to matrix‐based analysis in neuroimaging
Chen, G., Bürkner, P., Taylor, P. A., Li, Z., Yin, L., Glen, D., Kinnison, J., Cox, R. W., & Pessoa, L. (2019) · Human Brain Mapping · doi:10.1002/hbm.24686 · Cited by 26 · see citing works
“…which is often interested in modeling the data, not in making dichotomous decisions (Chen et al., 2019; Gelman & Carlin, 2014); and (b) inferences under BML most likely have the same directionality as the true effect because type S errors are well controlled under BML (Gelman & Tuerlinckx, 2000).”
BackgroundPublished estimates of group differences in multisensory integration are inflated
Magnotti, J., & Beauchamp, M. (2018) · bioRxiv · doi:10.1371/journal.pone.0202908 · Cited by 26 · see citing works
“We suggest that a more likely explanation than a variety of post hoc moderators is that small sample sizes inevitably result in high variability in population effect estimates, causing wild swings in their magnitude and sign across studies(29, 30).”
BackgroundMethodPan, V. S., McMunn, M. S., Karban, R., Goidell, J., Weber, M. G., & LoPresti, E. F. (2021) · Functional Ecology · doi:10.1111/1365-2435.13881 · Cited by 26 · see citing works
van Zwet, E. W., & Goodman, S. N. (2022) · Statistics in Medicine · doi:10.1002/sim.9406 · Cited by 25 · see citing works
Okamoto, J., Wang, L., Yin, X., Luca, F., Piqué-Regi, R., Helms, A. S., Im, H. K., Morrison, J., & Wen, X. (2023) · The American Journal of Human Genetics · doi:10.1016/j.ajhg.2022.12.002 · Cited by 26 · see citing works
RCTs in Development Economics, Their Critics and Their Evolution
Ogden, T. N. (2020) · Oxford University Press eBooks · doi:10.1093/oso/9780198865360.003.0006 · Cited by 26 · see citing works
What is our power to detect device effects in animal tracking studies?
Cleasby, I., Morrissey, B. J., Bolton, M., Owen, E., Wilson, L., Wischnewski, S., & Nakagawa, S. (2021) · Methods in Ecology and Evolution · doi:10.1111/2041-210x.13598 · Cited by 18 · see citing works
Quantifying “Promising Trials Bias” in Randomized Controlled Trials in Education
Sims, S., Anders, J., Inglis, M., & Lortie‐Forgues, H. (2022) · Journal of Research on Educational Effectiveness · doi:10.1080/19345747.2022.2090470 · Cited by 25 · see citing works
“In the first section of the paper, we begin by setting out a conceptual framework for understanding erroneous findings in RCTs, using an adapted version of the diagrams developed by Gelman and Tuerlinckx (2000).”
InfluentialGuidelines for Multiple Testing in Impact Evaluations of Educational InterventionsPreprint
Schochet, P. Z. (2008) · RePEc: Research Papers in Economics · Cited by 25 · see citing works
- Authors
- Gelman, A., & Tuerlinckx, F.
- Affiliations
- Columbia University (US)
- KU Leuven (BE)
- Venue
- Computational Statistics
- Published
- 2000-09
- Volume
- 15
- Issue
- 3
- Pages
- 373-390
- ISSN
- 0943-4062 (ISSN-L)
- 1613-9658
- OpenAlex
- W2083599439
- Cited by
- 380
- References
- 23
- Citations by year
- 2026: 17 · 2025: 25 · 2024: 36 · 2023: 37 · 2022: 36 · 2021: 33 · 2020: 44 · 2019: 33 · 2018: 18 · 2017: 18 · 2016: 12 · 2015: 29 · 2014: 6 · 2013: 8 · 2012: 4
- FWCI
- 0.83
- Type
- journal article
- Language
- en
- Open access
- closed
- Licence
- www.springer.com/tdm (tdm)
- Locations
- Computational Statistics (publishedVersion)
- Topics
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods and Bayesian Inference
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Clinical Trials
- Physical Sciences › Mathematics › Statistics and Probability › Probability and Statistical Research
- Keywords
- sign errors, confidence statements, multiple comparisons, confidence regions, error rate
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
