Difference-based meta-analytic procedures for between-participant and/or within-participant designs: A tutorial review for sports and exercise scientists
Open accessAbstract
The aim of this paper is to provide a contemporary summary of statistical and non-statistical meta-analytic procedures that have relevance to the type of experimental designs often used by sport scientists when examining differences/change in dependent measure(s) as a result of one or more independent manipulation(s). Using worked examples from studies on observational learning in the motor behaviour literature, we adopt a random effects model and give a detailed explanation of the statistical procedures for the three types of raw score difference-based analyses applicable to between-participant, within-participant, and mixed-participant designs. Major merits and concerns associated with these quantitative procedures are identified and agreed methods are reported for minimizing biased outcomes, such as those for dealing with multiple dependent measures from single studies, design variation across studies, different metrics (i.e., raw scores and difference scores), and variations in sample size. To complement the worked examples, we summarize the general considerations required when conducting and reporting a meta-analysis, including how to deal with publication bias, what information to present regarding the primary studies, and approaches for dealing with outliers. By bringing together these statistical and non-statistical meta-analytic procedures, we provide the tools required to clarify understanding of key concepts and principles.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Sixteen articles (17%) provided recommendations on the key item "protocol development", with publication dates between 1994 and 2015 (median year 2009).
“Sixteen articles (17%) provided recommendations on the key item “protocol development” (Table 3), [29–44] with publication dates between 1994 and 2015 (median year of publication 2009).”
From Methods to systematically review and meta-analyse observational studies: a systematic scoping review of recommendations (2018) · cites this paper as [36]
The articles recommended that the protocol should state the objectives, hypotheses to be tested and the rationale of the review, describe eligibility criteria, define the type of studies to be included, and give the reason when including observational studies in the review.
“The articles recommended that the protocol should state the objectives, hypotheses to be tested and the rationale of the review, [29] and that it should describe eligibility criteria [29, 33, 35, 36, 38, 39, 41], define the type of studies to be included [35, 37, 42, 44], and give the reason when including observational studies in the review [35, 37, 40].”
From Methods to systematically review and meta-analyse observational studies: a systematic scoping review of recommendations (2018) · cites this paper as [36]
Twenty articles (22%) gave recommendations on the key item "research question", with publication dates between 1997 and 2015 (median year 2009).
“Twenty articles (22%) gave recommendations on the key item “research question” [29, 30, 35, 36, 38–43, 45–54], with publication dates between 1997 and 2015 (median year of publication 2009).”
From Methods to systematically review and meta-analyse observational studies: a systematic scoping review of recommendations (2018) · cites this paper as [36]
Thirty-three articles (35%) made recommendations about the key item "search strategy", with publication dates between 1995 and 2016 (median year 2007).
“Thirty-three articles (35%) made recommendations about the key item “search strategy” [12, 29, 30, 33–36, 38, 39, 43, 46–51, 53–69], with publication dates between 1995 and 2016 (median year of publication 2007).”
From Methods to systematically review and meta-analyse observational studies: a systematic scoping review of recommendations (2018) · cites this paper as [36]
The majority of articles discussed aspects general to systematic reviews including advantages and limitations of different literature sources and databases, search tools, the importance of identifying unpublished studies, literature searching techniques including how to build a search string and reporting.
“The majority of articles discussed aspects general to systematic reviews including advantages and limitations of different literature sources and databases, search tools, the importance of identifying unpublished studies, literature searching techniques including how to build a search string and reporting [12, 29, 33–36, 46, 48, 50, 51, 53, 54, 57, 59–62, 67, 68].”
From Methods to systematically review and meta-analyse observational studies: a systematic scoping review of recommendations (2018) · cites this paper as [36]
Twenty (22%) articles reported on item "publication bias", with publication dates between 1994 and 2012 (median year 2001).
“Twenty (22%) articles reported on item “publication bias” [16, 29, 33, 34, 36, 46, 48–50, 55, 56, 61, 70, 78, 85, 95–99], with publication dates between 1994 and 2012 (median year of publication 2001).”
From Methods to systematically review and meta-analyse observational studies: a systematic scoping review of recommendations (2018) · cites this paper as [36]
Different articles advise to graphically examine the presence of publication bias using funnel plots, or to assess it with statistical tests.
“Since “journals are typically more likely to publish results that establish a difference than those that do not” [48], and publication bias is difficult to eliminate, different articles advise to graphically examine its presence using funnel plots, or to assess it with statistical tests [16, 36, 48, 55, 85, 97, 99].”
From Methods to systematically review and meta-analyse observational studies: a systematic scoping review of recommendations (2018) · cites this paper as [36]
Thirty-nine articles (42%) provided recommendations on heterogeneity and its assessment, with publication dates between 1994 and 2016 (median year 2006).
“Thirty-nine articles (42%) provided recommendations on heterogeneity and its assessment, [8, 12, 16, 29, 30, 33, 34, 36, 38, 39, 41–44, 47, 49–52, 54–56, 61, 66, 70, 72, 77, 82, 85, 87, 98, 100–107] with publication dates between 1994 and 2016 (median year of publication 2006).”
From Methods to systematically review and meta-analyse observational studies: a systematic scoping review of recommendations (2018) · cites this paper as [36]
Fifty-two articles (56%) provided recommendations on the statistical analysis in the context of meta-analysis of observational studies, with publication dates between 1994 and 2016 (median year 2006).
“Fifty-two articles (56%), provided recommendations on the statistical analysis in the context of meta-analysis of observational studies [8, 12, 16, 29–31, 33–36, 38, 41–44, 47, 48, 52, 54–56, 61, 66, 70, 72, 73, 77, 79–83, 85, 87, 96, 100, 101, 103–105, 107–118], with publication dates between 1994 and 2016 (median year of publication 2006).”
From Methods to systematically review and meta-analyse observational studies: a systematic scoping review of recommendations (2018) · cites this paper as [36]
Many considered heterogeneity important for the decision whether to pool the data.
“Many considered heterogeneity important for the decision whether to pool the data [34, 36, 47, 55, 56, 66, 101, 103].”
From Methods to systematically review and meta-analyse observational studies: a systematic scoping review of recommendations (2018) · cites this paper as [36]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Stroup, D. F. (2000) · JAMA · doi:10.1001/jama.283.15.2008 · Cited by 20,623 · see citing works
The file drawer problem and tolerance for null results.
Rosenthal, R. (1979) · Psychological Bulletin · doi:10.1037/0033-2909.86.3.638 · Cited by 9,068 · see citing works
Cohen, J. (1992) · Psychological Bulletin · doi:10.1037//0033-2909.112.1.155 · Cited by 8,771 · see citing works
Distribution Theory for Glass's Estimator of Effect Size and Related Estimators
Hedges, L. V. (1981) · Journal of Educational Statistics · doi:10.2307/1164588 · Cited by 5,695 · see citing works
Moher, D., Cook, D. J., Eastwood, S., Olkin, I., Rennie, D., & Stroup, D. F. (1999) · The Lancet · doi:10.1016/s0140-6736(99)04149-5 · Cited by 4,826 · see citing works
Meta-Analytic Procedures for Social Research
Rosenthal, R. (1991) · doi:10.4135/9781412984997 · Cited by 4,635 · see citing works
Publication and related bias in meta-analysis
Sterne, J. A. C., Gavaghan, D., & Egger, M. (2000) · Journal of Clinical Epidemiology · doi:10.1016/s0895-4356(00)00242-0 · Cited by 2,282 · see citing works
Light, R. J., & Pillemer, D. B. (1984) · Harvard University Press eBooks · doi:10.4159/9780674040243 · Cited by 1,422 · see citing works
Publication bias in meta-analysis its causes and consequences
Thornton, A. (2000) · Journal of Clinical Epidemiology · doi:10.1016/s0895-4356(99)00161-4 · Cited by 1,291 · cited 2× · see citing works
“Analytical correction methods have also been introduced, which are increasingly important where analysts elect to report effects from published sources only (Cooper & Hedges, 1994; Rosenthal, 1995; Sterne, Gavaghan, & Egger, 2000; Thornton & Lee, 2000).”
BackgroundThe Effects of Mental Practice on Motor Skill Learning and Performance: A Meta-analysis
Feltz, D. L., & Landers, D. M. (1983) · Journal of Sport Psychology · doi:10.1123/jsp.5.1.25 · Cited by 1,178 · see citing works
Writing meta-analytic reviews.
Rosenthal, R. (1995) · Psychological Bulletin · doi:10.1037/0033-2909.118.2.183 · Cited by 1,108 · see citing works
Estimation of effect size from a series of independent experiments.
Hedges, L. V. (1982) · Psychological Bulletin · doi:10.1037/0033-2909.92.2.490 · Cited by 1,065 · see citing works
Synthesizing standardized mean‐change measures
Becker, B. J. (1988) · British Journal of Mathematical and Statistical Psychology · doi:10.1111/j.2044-8317.1988.tb00901.x · Cited by 889 · see citing works
Snowling, N. J., & Hopkins, W. G. (2006) · Diabetes Care · doi:10.2337/dc06-1317 · Cited by 813 · see citing works
What do data really mean? Research findings, meta-analysis, and cumulative knowledge in psychology.
Schmidt, F. L. (1992) · American Psychologist · doi:10.1037/0003-066x.47.10.1173 · Cited by 564 · cited 1× · see citing works
“They state that l* reflects N times the ratio of t2 to vi, and also note that in an analyses of psychological reviews this ratio rarely exceeds 1.0 and is more typically around 0.33 (cf. Schmidt, 1992).”
BackgroundInfluentialVariability and Determinism in Motor Behavior
Riley, M. A., & Turvey, M. T. (2002) · Journal of Motor Behavior · doi:10.1080/00222890209601934 · Cited by 409 · see citing works
A random effects model for effect sizes.
Hedges, L. V. (1983) · Psychological Bulletin · doi:10.1037/0033-2909.93.2.388 · Cited by 399 · see citing works
Review of developments in meta-analytic method.
Bangert‐Drowns, R. L. (1986) · Psychological Bulletin · doi:10.1037/0033-2909.99.3.388 · Cited by 327 · see citing works
Meta-analytic procedures for combining studies with multiple effect sizes.
Rosenthal, R., & Rubin, D. B. (1986) · Psychological Bulletin · doi:10.1037/0033-2909.99.3.400 · Cited by 296 · see citing works
APPLICATIONS OF THE NON-CENTRAL t-DISTRIBUTION
Johnson, N. L., & Welch, B. L. (1940) · Biometrika · doi:10.1093/biomet/31.3-4.362 · Cited by 261 · see citing works
Distribution of the standardized mean change effect size for meta‐analysis on repeated measures
Morris, S. B. (2000) · British Journal of Mathematical and Statistical Psychology · doi:10.1348/000711000159150 · Cited by 206 · see citing works
Validity generalization results for computer programmers.
Schmidt, F. L., Gast-Rosenberg, I., & Hunter, J. E. (1980) · Journal of Applied Psychology · doi:10.1037/0021-9010.65.6.643 · Cited by 166 · see citing works
Schmidt's schema theory: the empirical base of the variability of practice hypothesis
van Rossum, J. H. A. (1990) · Human Movement Science · doi:10.1016/0167-9457(90)90010-b · Cited by 162 · see citing works
Horn, R. R., Williams, A. M., & Scott, M. (2002) · Journal of Sports Sciences · doi:10.1080/026404102317284808 · Cited by 108 · see citing works
Ashford, D., Bennett, S. J., & Davids, K. (2006) · Journal of Motor Behavior · doi:10.3200/jmbr.38.3.185-205 · Cited by 103 · see citing works
Self-citationThe Use of Meta-Analysis in Exercise and Sport: A Tutorial
Thomas, J. R., & French, K. E. (1986) · Research Quarterly for Exercise and Sport · doi:10.1080/02701367.1986.10605397 · Cited by 91 · see citing works
Effects of a Learning Model and Augmented Feedback on Tennis Skill Acquisition
HEBERT, E. P., & Landin, D. K. (1994) · Research Quarterly for Exercise and Sport · doi:10.1080/02701367.1994.10607626 · Cited by 83 · see citing works
Chapter 1 The Rise and Fall of Dominant Paradigms in Motor Behaviour Research
Abernethy, B., & Sparrow, W. A. (1992) · Advances in psychology · doi:10.1016/s0166-4115(08)61681-7 · Cited by 82 · see citing works
Teacher versus Peer Models Revisited: Effects on Motor Performance and Self-Efficacy
Lirgg, C., & Feltz, D. L. (1991) · Research Quarterly for Exercise and Sport · doi:10.1080/02701367.1991.10608713 · Cited by 70 · see citing works
Estimation of Effect Size from a Series of Experiments Involving Paired Comparisons
Gibbons, R. D., Hedeker, D., & Davis, J. M. (1993) · Journal of Educational Statistics · doi:10.2307/1165136 · Cited by 64 · see citing works
The Effect of the Availability of a Dynamic Model on the Acquisition of a Complex Cyclical Action
Whiting, H. T. A., Bijlard, M. J., & den Brinker, B. P. L. M. (1987) · The Quarterly Journal of Experimental Psychology Section A · doi:10.1080/02724988743000024 · Cited by 57 · see citing works
The Effects of Age and Number of Demonstrations on Modeling of Form and Performance
Feltz, D. L. (1982) · Research Quarterly for Exercise and Sport · doi:10.1080/02701367.1982.10605251 · Cited by 56 · see citing works
An unbiased correction for sampling error in validity generalization studies.
Hedges, L. V. (1989) · Journal of Applied Psychology · doi:10.1037/0021-9010.74.3.469 · Cited by 55 · see citing works
Observational Learning of Ballet Sequences: The Role of Kinematic Information
Gray, J. T., Neisser, U., Shapiro, B. A., & Kouns, S. (1991) · Ecological Psychology · doi:10.1207/s15326969eco0302_4 · Cited by 53 · see citing works
Hedges, L. V., Cooper, H. M., & Bushman, B. J. (1992) · Psychological Bulletin · doi:10.1037/0033-2909.111.1.188 · Cited by 46 · see citing works
Clustering estimates of effect magnitude from independent studies.
Hedges, L. V., & Olkin, I. (1983) · Psychological Bulletin · doi:10.1037/0033-2909.93.3.563 · Cited by 45 · see citing works
Videotape Replay and the Learning of Skills in Open and Closed Environments
Cooper, L. K., & Rothstein, A. L. (1981) · Research Quarterly for Exercise and Sport · doi:10.1080/02701367.1981.10607857 · Cited by 27 · see citing works
Looney, M. A., Feltz, C. J., & Vanvleet, C. N. (1994) · Research Quarterly for Exercise and Sport · doi:10.1080/02701367.1994.10607641 · Cited by 15 · see citing works
Twenty-five years of psychology in theJournal of Sports Sciences: A historical overview
Williams, A. M., Hardy, L., & Mutrie, N. (2008) · Journal of Sports Sciences · doi:10.1080/02640410701765631 · Cited by 14 · see citing works
Flagged first, then by citations
Research works printed but not linked: 19
- Glass, G. V., McGraw, B., & Smith, M. L. (1981). Meta-analysis in social research. Beverly Hills, CA: Sage.
- *Gould, D. (1978). The influence of motor task types on model effectiveness . Unpublished doctoral thesis.
- Gregoire, G., Derderian, F., & Le Lorier, J. (1995). Selecting the language of the publications included in a meta-analysis: is there a tower of babel bias? Journal of Clinical Epidemiology, 48 (1), 159-163.
- Grimm, L. G. (1993). Statistical applications for the behavioural sciences. New York: John Wiley and Sons, Inc.
- Harris, M. J., & Rosenthal, R. (1985). Mediation of interpersonal expectancy effects: 31 meta-analyses. Psychological Bulletin, 97 , 363-386.
- Hedges, L. V., & Olkin, I. (1985). Statistical Methods for Meta-Analysis. London: Academic 24 Press, inc. 25
- Hunter, J. E., & Schmidt, F. L. (1990). Methods of meta-analysis: correcting error and bias in research findings. London: Sage Publications Inc.
- *Laguna, P. L. (1996). The effects of model demonstration strategies on motor skill acquisition and performance. Journal of Human Movement Studies , 30, 55-79.
- Light, R. J., & Pillemer, D. B. (1984). Summing Up: The Science of Reviewing Research . Cambridge, MA: Harvard University Press.
- *Magill, R. A. and Schoenfelder-Zohdi. (1996). A visual model and knowledge of performance as sources of information for learning a rhythmical gymnastics skill. International Journal of Sports Psychology , 27, 7-22.
- Meijer, O.G. (2001). Making things happen: An introduction to the history of movement science. In M. L. Latash & V. M. Zatsiorsky (Eds.), Classics in Movement Science, (pp.1-57). Champaign, Ill.: Human Kinetics.
- Morris, S. B. (2000). Distribution of the standardised mean change effect size for metaanalysis on repeated measures. British Journal of Mathematics and Statistical Psychology, 53 , 17-29.
- Morris, S. B., & DeShon, R. P. (2002). Combining effect size estimates in meta-analysis with repeated measures and independent-groups designs. Psychological Methods, 7 (1), 105-125.
- *Roach, N. K. and Burwitz, L. (1986). Observational learning in motor skill acquisition: The effects of verbal directing cues. In VIII Commonwealth and International Conference on Sport Physical Education Dance Recreation and Health . pp.349-354. Glasgow: Spon.
- Rosenthal, R. (1976). Experimenter Effects in Behavioral Research. (Enlarged edition) . New York: Irvington.
- Rosenthal, R., & Rosnov, R. L. (1984). Essential of behavioral research: methods of data analysis. New York: McGraw-Hill.
- *Schoenfelder-Zohdi, B. G. (1992). Investigating the informational nature of a modeled visual demonstration . Unpublished doctoral thesis. Louisiana State University.
- Sterne, J. A. C., Gavaghan, D., & Egger, M. (2000). Publication and related bias in metanalysis: Power of statistical tests and prevalence in the literature. Journal of Clinical Epidemiology, 53 , 1119-1129.
- Summers, J.J. (2004). A historical perspective on skill acquisition. In A. M. Williams & N. Hodges (Eds.), Skill Acquisition in Sport (pp.1-26). London: Routledge, Taylor & Francis.
Cited by
Sorted by citations
Mueller, M., D’Addario, M., Egger, M., Cevallos, M., Dekkers, O., Mugglin, C., & Scott, P. (2018) · BMC Medical Research Methodology · doi:10.1186/s12874-018-0495-9 · Cited by 400 · see citing works
The effects of physical exercise on functional outcomes in the treatment of ADHD: a meta-analysis
Vysniauske, R., Verburgh, L., Oosterlaan, J., & Molendijk, M. L. (2020) · Journal of Attention Disorders · doi:10.1177/1087054715627489 · Cited by 129 · see citing works
- Authors
- Ashford, D., Davids, K., & Bennett, S. J.
- Affiliations
- Manchester Metropolitan University (GB)
- Queensland University of Technology (AU)
- Liverpool John Moores University (GB)
- Corresponding author
- Ashford, Derek
- Venue
- Journal of Sports Sciences
- Publisher
- Informa UK Limited
- Published
- 2009-02
- Volume
- 27
- Issue
- 3
- Pages
- 237-255
- ISSN
- 0264-0414 (ISSN-L)
- 1466-447X
- OpenAlex
- W1519673859
- Cited by
- 11
- References
- 39
- Citations by year
- 2024: 1 · 2019: 2 · 2018: 2 · 2017: 2 · 2016: 1 · 2015: 1 · 2013: 1
- FWCI
- 0.41
- Type
- journal article
- Language
- en
- Open access
- green
- eprints.qut.edu.au
- Locations
- Journal of Sports Sciences (publishedVersion)
- PubMed (publishedVersion)
- SHURA (Sheffield Hallam University Research Archive) (Sheffield Hallam University) (acceptedVersion)
- QUT ePrints (Queensland University of Technology) (submittedVersion)
- Faculty of Health; Institute of Health and Biomedical Innovation
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Psychology Research and Bibliometrics
- Social Sciences › Psychology › Developmental and Educational Psychology › Sport Psychology and Performance
- Keywords
- within-participant design, random effects models, observational learning, motor behavior, design variation, publication bias, outliers
- MeSH terms
- Humans, Meta-Analysis as Topic, Exercise, Biomedical Research, Athletic Performance
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
