The Problems in Using Fixed-Effects Models of Meta-Analysis on Real-World Data
Open accessAbstract
There are 2 approaches to meta-analysis: One assumes that studies in a meta-analysis are sampled from populations with the same effect size (the fixed-effects case), the other assumes that studies are taken from populations that have varying effect sizes (the random-effects case). As such, 2 corresponding meta-analytic frameworks have been developed: fixed- and random-effects methods. Recent evidence suggests that the assumption of fixed population effect sizes is not tenable for virtually all real-world data (e.g., Hunter & Schmidt, 2000), and yet fixed-effects methods of meta-analysis are routinely applied to real-world data (see National Research Council, 1992). This article describes some of the problems in using fixed-effects models on random-effects data by presenting 2 Monte Carlo simulations. In keeping with statistical theory (e.g., Hunter & Schmidt, 2000) results show a radical inflation of the significance tests of the mean effect sizes (above and beyond theoretical predictions). These results are discussed in terms of the implications for previously published meta-analytic reviews and those yet to be done.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The study used random effects models, which produce larger standard errors and reduce the probability of Type I errors.
“Random effects models produce larger standard errors, therefore reducing the probability of Type I errors (for a review of risks resulting when not using random effects models when appropriate, see Field, 2003).”
From Gender differences in emotion expression in children: A meta-analytic review. (2013) · cites this paper as Field, 2003
Cites
THE BIG FIVE PERSONALITY DIMENSIONS AND JOB PERFORMANCE: A META‐ANALYSIS
BARRICK, M. R., & MOUNT, M. K. (1991) · Personnel Psychology · doi:10.1111/j.1744-6570.1991.tb00688.x · Cited by 8,689 · see citing works
Meta-Analytic Procedures for Social Research
Rosenthal, R. (1991) · doi:10.4135/9781412984997 · Cited by 4,635 · see citing works
Fixed-and random-effects models in meta-analysis.
Hedges, L. V., & Vevea, J. L. (1998) · Psychological Methods · doi:10.1037/1082-989x.3.4.486 · Cited by 2,977 · see citing works
Hunter, J. E., & Schmidt, F. L. (2000) · International Journal of Selection and Assessment · doi:10.1111/1468-2389.00156 · Cited by 833 · see citing works
What do data really mean? Research findings, meta-analysis, and cumulative knowledge in psychology.
Schmidt, F. L. (1992) · American Psychologist · doi:10.1037/0003-066x.47.10.1173 · Cited by 564 · see citing works
Hedges, L. V. (1992) · Journal of Educational Statistics · doi:10.3102/10769986017004279 · Cited by 186 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Emotion-regulation strategies across psychopathology: A meta-analytic review
Aldao, A., Nolen-Hoeksema, S., & Schweizer, S. (2010) · Clinical Psychology Review · doi:10.1016/j.cpr.2009.11.004 · Cited by 6,686 · see citing works
Assessing heterogeneity in meta-analysis: Q statistic or I² index?
Huedo-Medina, T. B., Sánchez-Meca, J., Marín-Martínez, F., & Botella, J. (2006) · Psychological Methods · doi:10.1037/1082-989x.11.2.193 · Cited by 4,256 · see citing works
Ellis, P. D. (2010) · doi:10.1017/cbo9780511761676 · Cited by 2,461 · see citing works
Ego depletion and the strength model of self-control: a meta-analysis
Hagger, M. S., Wood, C., Stiff, C., & Chatzisarantis, N. L. D. (2010) · Psychological Bulletin · doi:10.1037/a0019486 · Cited by 2,265 · see citing works
The effects of physical activity on sleep: A meta-analytic review
Kredlow, M. A., Capozzoli, M. C., Hearon, B. A., Calkins, A. W., & Otto, M. W. (2015) · Journal of Behavioral Medicine · doi:10.1007/s10865-015-9617-6 · Cited by 1,333 · see citing works
Gender differences in emotion expression in children: A meta-analytic review.
Chaplin, T. M., & Aldao, A. (2013) · Psychological Bulletin · doi:10.1037/a0030737 · Cited by 1,247 · see citing works
“Therefore, we conducted random effects models, as they assume that effect sizes differ from the population by sampling error plus random variability among the studies (Field, 2003; Hedges & Vevea, 1998; Hunter & Schmidt, 1990; Lipsey & Wilson, 2001; Rosenthal & DiMatteo, 2001).”
BackgroundMethodField, A. P., & Gillett, R. (2010) · British Journal of Mathematical and Statistical Psychology · doi:10.1348/000711010x502733 · Cited by 1,162 · see citing works
Self-citationDestination image and tourist loyalty: a meta-analysis
Zhang, H., Fu, X., Cai, L. A., & Lu, L. (2014) · Tourism Management · doi:10.1016/j.tourman.2013.06.006 · Cited by 1,098 · see citing works
Prati, G., & Pietrantoni, L. (2009) · Journal of Loss and Trauma · doi:10.1080/15325020902724271 · Cited by 978 · see citing works
A meta-analysis of risk factors for post-traumatic stress disorder in children and adolescents
Trickey, D., Siddaway, A. P., Meiser-Stedman, R., Serpell, L., & Field, A. P. (2012) · Clinical Psychology Review · doi:10.1016/j.cpr.2011.12.001 · Cited by 828 · see citing works
Self-citationHan, B., & Eskin, E. (2011) · The American Journal of Human Genetics · doi:10.1016/j.ajhg.2011.04.014 · Cited by 680 · see citing works
The effect of acute aerobic exercise on positive activated affect: A meta-analysis
Reed, J., & Ones, D. S. (2006) · Psychology of Sport and Exercise · doi:10.1016/j.psychsport.2005.11.003 · Cited by 622 · see citing works
Taylor, B., Irving, H. M., Kanteres, F., Room, R., Borges, G., Cherpitel, C., Greenfield, T., & Rehm, J. (2010) · Drug and Alcohol Dependence · doi:10.1016/j.drugalcdep.2010.02.011 · Cited by 509 · see citing works
Activity Achievement Emotions and Academic Performance: A Meta-analysis
Camacho-Morles, J., Slemp, G. R., Pekrun, R., Loderer, K., Hou, H., & Oades, L. G. (2021) · Educational Psychology Review · doi:10.1007/s10648-020-09585-3 · Cited by 475 · see citing works
Wang, M.-T., L. Degol, J., Amemiya, J., Parr, A., & Guo, J. (2020) · Developmental Review · doi:10.1016/j.dr.2020.100912 · Cited by 470 · see citing works
Psychological treatment of obsessive–compulsive disorder: A meta-analysis
Rosa-Alcázar, A. I., Sánchez-Meca, J., Gómez-Conesa, A., & Marín-Martínez, F. (2008) · Clinical Psychology Review · doi:10.1016/j.cpr.2008.07.001 · Cited by 444 · see citing works
Carolan, S., Harris, P. R., & Cavanagh, K. (2017) · Journal of Medical Internet Research · doi:10.2196/jmir.7583 · Cited by 385 · see citing works
“Since we expected considerable heterogeneity, a random effects model was performed [37].”
Leader autonomy support in the workplace: A meta-analytic review
Slemp, G. R., Kern, M. L., Patrick, K. J., & Ryan, R. M. (2018) · Motivation and Emotion · doi:10.1007/s11031-018-9698-y · Cited by 407 · see citing works
Emotion regulation and posttraumatic stress symptoms: a meta-analysis
Seligowski, A. V., Lee, D. J., Bardeen, J. R., & Orcutt, H. K. (2015) · Cognitive Behaviour Therapy · doi:10.1080/16506073.2014.980753 · Cited by 408 · see citing works
Trauma coping strategies and psychological distress: A meta‐analysis
Littleton, H., Horsley, S., John, S., & Nelson, D. V. (2007) · Journal of Traumatic Stress · doi:10.1002/jts.20276 · Cited by 380 · see citing works
Taylor, B., Irving, H. M., Baliunas, D., Roerecke, M., Patra, J., Mohapatra, S., & Rehm, J. (2009) · Addiction · doi:10.1111/j.1360-0443.2009.02694.x · Cited by 346 · see citing works
Johnsen, T. J., & Friborg, O. (2015) · Psychological Bulletin · doi:10.1037/bul0000015 · Cited by 331 · see citing works
Prati, G., & Pietrantoni, L. (2010) · Journal of Community Psychology · doi:10.1002/jcop.20371 · Cited by 331 · see citing works
Depressive realism: a meta-analytic review
Moore, M. T., & Fresco, D. M. (2012) · Clinical Psychology Review · doi:10.1016/j.cpr.2012.05.004 · Cited by 291 · see citing works
Review: Alleviating loneliness in young people – a meta‐analysis of interventions
Eccles, A. M., & Qualter, P. (2021) · Child and Adolescent Mental Health · doi:10.1111/camh.12389 · Cited by 285 · see citing works
Brewin, C. R., Kleiner, J. S., Vasterling, J. J., & Field, A. P. (2007) · Journal of Abnormal Psychology · doi:10.1037/0021-843x.116.3.448 · Cited by 290 · see citing works
Self-citationPsychological treatment of panic disorder with or without agoraphobia: A meta-analysis
Sánchez-Meca, J., Rosa-Alcázar, A. I., Marín-Martínez, F., & Gómez-Conesa, A. (2010) · Clinical Psychology Review · doi:10.1016/j.cpr.2009.08.011 · Cited by 291 · see citing works
Friborg, O., Martinsen, E. W., Martinussen, M., Kaiser, S., Øvergård, K. T., & Rosenvinge, J. H. (2014) · Journal of Affective Disorders · doi:10.1016/j.jad.2013.08.023 · Cited by 242 · see citing works
Emotion Regulation and Substance Use: A Meta-Analysis
Weiss, N. H., Kiefer, R., Goncharenko, S., Raudales, A. M., Forkus, S. R., Schick, M. R., & Contractor, A. A. (2022) · Drug and Alcohol Dependence · doi:10.1016/j.drugalcdep.2021.109131 · Cited by 235 · see citing works
Cognitive and perceptual variables in hypochondriasis and health anxiety: A systematic review
Marcus, D. K., Gurley, J. R., Marchi, M. M., & Bauer, C. (2007) · Clinical Psychology Review · doi:10.1016/j.cpr.2006.09.003 · Cited by 218 · see citing works
Schoth, D. E., Nunes, V. D., & Liossi, C. (2012) · Clinical Psychology Review · doi:10.1016/j.cpr.2011.09.004 · Cited by 217 · see citing works
The relationship between adult attachment style and post-traumatic stress symptoms: A meta-analysis
Woodhouse, S., Ayers, S., & Field, A. P. (2015) · Journal of Anxiety Disorders · doi:10.1016/j.janxdis.2015.07.002 · Cited by 210 · see citing works
Self-citationBoemo, T., Nieto, I., Vazquez, C., & Sanchez-Lopez, A. (2022) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2022.104747 · Cited by 191 · see citing works
“variability, which reduces the risk of type I errors (Field, 2003).”
BackgroundChandler, M. J., Parks, A. C., Marsiske, M., Rotblatt, L. J., & Smith, G. E. (2016) · Neuropsychology Review · doi:10.1007/s11065-016-9330-4 · Cited by 169 · see citing works
A meta-analysis of autonomous and controlled forms of teacher motivation
Slemp, G. R., Field, J. G., & Cho, A. S. H. (2020) · Journal of Vocational Behavior · doi:10.1016/j.jvb.2020.103459 · Cited by 179 · see citing works
Is the Dodo bird endangered in the 21st century? A meta-analysis of treatment comparison studies
Marcus, D. K., O'Connell, D., Norris, A. L., & Sawaqdeh, A. (2014) · Clinical Psychology Review · doi:10.1016/j.cpr.2014.08.001 · Cited by 148 · see citing works
A meta-analysis of cognitive therapy for worry in generalized anxiety disorder
Hanrahan, F., Field, A. P., Jones, F. W., & Davey, G. C. L. (2013) · Clinical Psychology Review · doi:10.1016/j.cpr.2012.10.008 · Cited by 135 · see citing works
“The reporting of such factors in future studies would benefit this field by allowing for more in-depth analysis of moderator variables, as recommended by Newman (2000). Fourth, the inclusion of AR in the ‘non-CT treatment control’ group may mask the size of the advantage of CT approaches over non-CT therapies other th…”
BackgroundSelf-citationPsychotherapies for hypochondriasis
Thomson, A., & Page, L. (2007) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd006520.pub2 · Cited by 141 · see citing works
Burlingame, G. M., Seebeck, J. D., Janis, R. A., Whitcomb, K. E., Barkowski, S., Rosendahl, J., & Strauss, B. (2016) · Psychotherapy · doi:10.1037/pst0000090 · Cited by 118 · see citing works
Taylor, B., & Rehm, J. (2012) · Alcoholism: Clinical and Experimental Research · doi:10.1111/j.1530-0277.2012.01785.x · Cited by 117 · see citing works
The effects of online education on academic success: A meta-analysis study
Ulum, H. (2022) · Education and Information Technologies · doi:10.1007/s10639-021-10740-8 · Cited by 96 · see citing works
MacDonald, L., Chapman, S., Syrett, M., Bowskill, R., & Horne, R. (2016) · Journal of Affective Disorders · doi:10.1016/j.jad.2016.01.002 · Cited by 80 · see citing works
BEAULIEU‐PRÉVOST, D., & ZADRA, A. (2007) · Journal of Sleep Research · doi:10.1111/j.1365-2869.2007.00572.x · Cited by 59 · see citing works
Spielmans, G. I., Benish, S. G., Marin, C., Bowman, W. M., Menster, M., & Wheeler, A. J. (2013) · Clinical Psychology Review · doi:10.1016/j.cpr.2013.01.008 · Cited by 53 · see citing works
Random or Fixed? An Empirical Examination of Meta-Analysis Model Choices
Slaney, K. L., Tafreshi, D., & Hohn, R. (2018) · Review of General Psychology · doi:10.1037/gpr0000140 · Cited by 13 · see citing works
- Authors
- Field, A. P.
- Affiliations
- University of Sussex (GB)
- Corresponding author
- Field, Andy P.
- ORCID
- Field, Andy P.
- Venue
- Understanding Statistics
- Publisher
- Informa UK Limited
- Published
- 2003-04-03
- Volume
- 2
- Issue
- 2
- Pages
- 105-124
- ISSN
- 1532-8031 (ISSN-L)
- 1534-844X
- OpenAlex
- W1979583539
- Cited by
- 237
- References
- 6
- Citations by year
- 2026: 3 · 2025: 5 · 2024: 9 · 2023: 11 · 2022: 13 · 2021: 13 · 2020: 11 · 2019: 9 · 2018: 11 · 2017: 9 · 2016: 13 · 2015: 13 · 2014: 7 · 2013: 10 · 2012: 20
- FWCI
- 6.39
- Type
- journal article
- Language
- en
- Open access
- green
- figshare.com
- Licence
- cc-by
- Locations
- Understanding Statistics (publishedVersion)
- Figshare (submittedVersion, cc-by)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Psychology Research and Bibliometrics
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods and Bayesian Inference
- Keywords
- fixed effects model, random effects models, meta-analysis, effect size heterogeneity, Monte Carlo simulation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
