Statistical and measurement pitfalls in the use of meta-regression in meta-analysis
The DOI matches its Crossref record; 4 of the 4 references it cites were found in a registry or an index.
Abstract
Purpose: Meta-regression is widely used and misused today in meta-analyses in psychology, organizational behavior, marketing, management, and other social sciences, as an approach to the identification and calibration of moderators, with most users being unaware of serious problems in its use. The purpose of this paper is to describe nine serious methodological problems that plague applications of meta-regression. Design/methodology/approach: This paper is methodological in nature and is based on well-established principles of measurement and statistics. These principles are used to illuminate the potential pitfalls in typical applications of meta-regression. Findings: The analysis in this paper demonstrates that many of the nine statistical and measurement pitfalls in the use of meta-regression are nearly universal in applications in the literature, leading to the conclusion that few meta-regressions in the literature today are trustworthy. A second conclusion is that in almost all cases, hierarchical subgrouping of studies is superior to meta-regression as a method of identifying and calibrating moderators. Finally, a third conclusion is that, contrary to popular belief among researchers, the process of accurately identifying and calibrating moderators, even with the best available methods, is complex, difficult, and data demanding. Practical implications: This paper provides useful guidance to meta-analytic researchers that will improve the practice of moderator identification and calibration in social science research literatures. Social implications: Today, many important decisions are made on the basis of the results of meta-analyses. These include decisions in medicine, pharmacology, applied psychology, management, marketing, social policy, and other social sciences. The guidance provided in this paper will improve the quality of such decisions by improving the accuracy and trustworthiness of meta-analytic results. Originality/value: This paper is original and valuable in that there is no similar listing and discussion of the pitfalls in the use of meta-regression in the literature, and there is currently a widespread lack of knowledge of these problems among meta-analytic researchers in all disciplines.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
HARKing: Hypothesizing after the results are known
Kerr, N. L. (1998) · Personality and Social Psychology Review · doi:10.1207/s15327957pspr0203_4 · Cited by 1,989
The power of statistical tests for moderators in meta-analysis
Hedges, L. V., & Pigott, T. D. (2004) · Psychological Methods · doi:10.1037/1082-989x.9.4.426 · Cited by 672
Accounting for heterogeneity via random-effects models and moderator analyses in meta-analysis.
Viechtbauer, W. (2007) · Zeitschrift für Psychologie / Journal of Psychology · doi:10.1027/0044-3409.215.2.104 · Cited by 193
Methods for second order meta-analysis and illustrative applications
Schmidt, F. L., & Oh, I.-S. (2013) · Organizational Behavior and Human Decision Processes · doi:10.1016/j.obhdp.2013.03.002 · Cited by 151
Self-citation
Cited by
Supply chain sustainability and performance of firms: A meta-analysis of the literature
Govindan, K., Rajeev, A., Padhi, S. S., & Pati, R. K. (2020) · Transportation Research Part E: Logistics and Transportation Review · doi:10.1016/j.tre.2020.101923 · Cited by 253
How to conduct a meta-analysis in eight steps: A practical guide
Hansen, C., Steinmetz, H., & Block, J. (2022) · Management Review Quarterly · doi:10.1007/s11301-021-00247-4 · Cited by 218
A meta‐analysis of the effects of electronic performance monitoring on work outcomes
Ravid, D. M., White, J. C., Tomczak, D. L., Miles, A. F., & Behrend, T. S. (2023) · Personnel Psychology · doi:10.1111/peps.12514 · Cited by 159
“We used hierarchical subgroup moderator analysis due to the relatively small subgroup K (Schmidt, 2017).”
MethodVetrovsky, T., Borowiec, A., Juřík, R., Wahlich, C., Śmigielski, W., Steffl, M., Tufano, J. J., Drygas, W., Stastny, P., Harris, T., et al. (2022) · British Journal of Sports Medicine · doi:10.1136/bjsports-2021-105198 · Cited by 68
Shan, W., Keller, J., & Joseph, D. (2019) · Journal of Organizational Behavior · doi:10.1002/job.2357 · Cited by 60
Metadata
- Authors
- Schmidt, F. L.
- Venue
- Career Development International
- Publisher
- Emerald
- Published
- 2017-09-11
- Volume
- 22
- Issue
- 5
- Pages
- 469-476
- ISSN
- 1362-0436 (ISSN-L)
- OpenAlex
- W2753768357
- Cited by
- 89
- References
- 4
- FWCI
- 4.57
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Statistical and measurement pitfalls in the use of meta-regression in meta-analysis. https://app.cochat.ai/papers/10.1108/cdi-08-2017-0136
- The paper
- Schmidt, F. L. (2017). Statistical and measurement pitfalls in the use of meta-regression in meta-analysis. Career Development International. https://doi.org/10.1108/cdi-08-2017-0136
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.