Much ado about something: violent video game effects and a school of red herring: reply to Ferguson and Kilburn (2010)
The DOI matches its Crossref record; 56 of the 56 references it cites were found in a registry or an index.
Abstract
In this article we reply to C. J. Ferguson and J. Kilburn’s (2010) critique of our meta-analysis on violent video game effects (C. A. Anderson et al., 2010). We rely on well-established methodological and statistical theory and on empirical data to show that claims of bias and misinterpretation on our part are simply wrong. One should not systematically exclude unpublished studies from meta-analytic reviews. There is no evidence of publication or selection bias in our data. We did not purposely exclude certain studies; we included all studies that met our inclusion criteria. Although C. J. Ferguson and J. Kilburn believe that the effects we obtained are trivial in size, they are larger than many effects that are deemed sufficiently large to warrant action in medical and violence domains. The claim that we (and other media violence scholars) are attempting to create a false crisis is a red herring.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“For example, as of the 1980s, there has been a fierce debate among scholars about the effects of game violence on aggression (e.g.see the dispute in Psychological Bulletin about whether this effect is trivial or meaningful [51,52]).”
From Social media use and its impact on adolescent mental health: An umbrella review of the evidence. (2022) · cites this paper as [51]
Citations
Cites
Statistical power analysis for the behavioral sciences
(1990) · Computers, Environment and Urban Systems · doi:10.1016/0198-9715(90)90050-4 · Cited by 65,269
Statistical Power Analysis for the Behavioral Sciences (2nd ed.).
Lachenbruch, P. A., & Cohen, J. (1989) · Journal of the American Statistical Association · doi:10.2307/2290095 · Cited by 37,395
Borenstein, M., Hedges, L. V., Higgins, J. P. T., & Rothstein, H. R. (2021) · doi:10.1002/9781119558378 · Cited by 9,799
Self-citation(2000) · Cited by 9,198
Systematic reviews in the social sciences: a practical guide
(2006) · Choice Reviews Online · doi:10.5860/choice.43-5664 · Cited by 4,089
Funnel plots for detecting bias in meta-analysis
Sterne, J. A. C., & Egger, M. (2001) · Journal of Clinical Epidemiology · doi:10.1016/s0895-4356(01)00377-8 · Cited by 3,779 · cited 1×
“Sterne and Egger (2001) noted that it is possible that studies with smaller samples actually do have larger effects, perhaps because the smaller studies used different populations or designs than did the larger ones.”
ResultThe motivational pull of video games: a self-determination theory approach
Ryan, R. M., Rigby, C. S., & Przybylski, A. (2006) · Motivation and Emotion · doi:10.1007/s11031-006-9051-8 · Cited by 3,332
Systematic Reviews in Health Care : Meta-Analysis in Context
Egger, M., Smith, G. D., & Altman, D. G. (2000) · Cited by 2,401
Publication Bias in Meta‐Analysis: Prevention, Assessment and Adjustments
(2005) · doi:10.1002/0470870168 · Cited by 2,368
Anderson, C. A., Shibuya, A., Ihori, N., Swing, E. L., Bushman, B. J., Sakamoto, A., Rothstein, H. R., & Saleem, M. (2010) · Psychological Bulletin · doi:10.1037/a0018251 · Cited by 2,048 · cited 5×
“…of Michigan and Vrije Universiteit Hannah R. Rothstein Baruch College, City University of New York Craig A. Anderson Iowa State University In this article we reply to C. J. Ferguson and J. Kilburn’s (2010) critique of our meta-analysis on violent video game effects (C. A. Anderson et al., 2010).”
BackgroundMethodResultInfluentialSelf-citationResearch Synthesis and Meta-Analysis: A Step-by-Step Approach
Cooper, H. (2017) · doi:10.4135/9781071878644 · Cited by 1,716 · cited 1×
“Cooper (2009) specifically pointed out the limitations of relying only on peer-reviewed journal articles, stating that bias against null findings and confirmatory bias means that qualitycontrolled journal articles (and conference presentations) should not be used as the sole source of information…”
BackgroundOne Hundred Years of Social Psychology Quantitatively Described
Richard, F. D., Bond, C. F., & Stokes-Zoota, J. J. (2003) · Review of General Psychology · doi:10.1037/1089-2680.7.4.331 · Cited by 1,558
How Many Studies Do You Need?: A Primer on Statistical Power for Meta-Analysis
Valentine, J. C., Pigott, T. D., & Rothstein, H. R. (2010) · Journal of Educational and Behavioral Statistics · doi:10.3102/1076998609346961 · Cited by 1,294
Self-citationSystematic Reviews and Meta-Analysis
Littell, J. H., Corcoran, J., & Pillai, V. (2008) · doi:10.1093/acprof:oso/9780195326543.001.0001 · Cited by 1,293
Interpreting the magnitudes of correlation coefficients.
Hemphill, J. F. (2003) · American Psychologist · doi:10.1037/0003-066x.58.1.78 · Cited by 1,237 · cited 1×
“For this reason Hemphill (2003) recommended a reconceptualization of effect size, in which r .1 is small, r .2 is medium, and r .3 is large.”
MethodDesign Sensitivity: Statistical Power for Experimental Research.
Eliason, S. R., & Lipsey, M. W. (1991) · Contemporary Sociology · doi:10.2307/2072167 · Cited by 1,040
Taylor, S. P. (1967) · Journal of Personality · doi:10.1111/j.1467-6494.1967.tb01430.x · Cited by 872 · cited 1×
“Variations of the competitive reaction time task developed by Taylor (1967) have been used by aggression researchers for over 40 years.”
MethodAdjusting for publication bias in the presence of heterogeneity
Terrin, N., Schmid, C. H., Lau, J., & Olkin, I. (2003) · Statistics in Medicine · doi:10.1002/sim.1461 · Cited by 809
Design Sensitivity: Statistical Power for Experimental Research.
Lachenbruch, P. A., & Lipsey, M. W. (1990) · Journal of the American Statistical Association · doi:10.2307/2289636 · Cited by 782
Peters, J. L., Sutton, A. J., Jones, D. R., Abrams, K. R., & Rushton, L. (2007) · Statistics in Medicine · doi:10.1002/sim.2889 · Cited by 716
Media violence and the American public: Scientific facts versus media misinformation.
Bushman, B. J., & Anderson, C. A. (2001) · American Psychologist · doi:10.1037//0003-066x.56.6-7.477 · Cited by 571 · cited 1×
“The effects we obtained for violent video games are similar in size to the effects of risk factors for physical health, such as exposure to lead, asbestos, or secondhand smoke (Bushman & Anderson, 2001).”
BackgroundSelf-citationResearch synthesis and meta-analysis: A step-by-step approach (4th ed.).
Cooper, H. M. (2010) · Cited by 541
How are we doing in soft psychology?
Rosenthal, R. (1990) · American Psychologist · doi:10.1037//0003-066x.45.6.775 · Cited by 515 · cited 4×
“Rothstein, H. R. (2008a). Publication bias. In Wiley encyclopedia of clinical trials. Chichester, England: Wiley. Rothstein, H. R. (2008b). Publication bias as a threat to the validity of meta-analytic results.”
BackgroundMethodInfluentialFerguson, C. J. (2007) · Psychiatric Quarterly · doi:10.1007/s11126-007-9056-9 · Cited by 503 · cited 2×
“In their meta-analytic reviews (Ferguson, 2007a, 2007b; Ferguson & Kilburn, 2009), Ferguson and his colleagues claimed that the trim and fill technique produces a “corrected” coefficient; it does not.”
BackgroundDuval, S. (2005) · Publication Bias in Meta‐Analysis · doi:10.1002/0470870168.ch8 · Cited by 496 · cited 1×
“Both the originators of trim and fill technique (cf. Duval, 2005) and other meta-analysis experts who advocate its use have stated unequivocally that one should not view the adjusted estimate as a corrected or more accurate estimate of the effect, because it is based on imputed data points.”
BackgroundViolent video game effects on children and adolescents: theory, research, and public policy
(2007) · Choice Reviews Online · doi:10.5860/choice.44-6518 · Cited by 493
Reasons or excuses for avoiding meta-analysis in forest plots
Ioannidis, J. P. A., Patsopoulos, N. A., & Rothstein, H. R. (2008) · BMJ · doi:10.1136/bmj.a117 · Cited by 457
Self-citationThe effects of violent video games on aggression. A meta-analysis
Sherry, J. (2001) · Human Communication Research · doi:10.1093/hcr/27.3.409 · Cited by 444 · cited 1×
“Other meta-analyses cited by Ferguson and Kilburn as supposedly refuting the effect of violent video games on aggressive and violent behavior have found correlations in the same range (e.g., Sherry, 2001).”
BackgroundViolent Video Games: Specific Effects of Violent Content on Aggressive Thoughts and Behavior
Anderson, C. A., Carnagey, N. L., Flanagan, M., Benjamin, A. J., Eubanks, J., & Valentine, J. C. (2004) · Advances in Experimental Social Psychology · doi:10.1016/s0065-2601(04)36004-1 · Cited by 389 · cited 11×
“In their comment on our meta-analysis, Ferguson and Kilburn (2010) stated that “Anderson et al. fail to note that many scholars have been critical of the inclusion of unpublished studies in meta-analyses.” This is simply false, at least when considering the writings of meta-analytic scholars. Consider the following st…”
BackgroundResultInfluentialSelf-citationEvidence for publication bias in video game violence effects literature: A meta-analytic review
Ferguson, C. J. (2007) · Aggression and Violent Behavior · doi:10.1016/j.avb.2007.01.001 · Cited by 363 · cited 4×
“In their meta-analytic reviews (Ferguson, 2007a, 2007b; Ferguson & Kilburn, 2009), Ferguson and his colleagues claimed that the trim and fill technique produces a “corrected” coefficient; it does not.”
BackgroundInfluentialWiley Encyclopedia of Clinical Trials
(2007) · Wiley eBooks · doi:10.1002/9780471462422 · Cited by 341
The Public Health Risks of Media Violence: A Meta-Analytic Review
Ferguson, C. J., & Kilburn, J. (2009) · The Journal of Pediatrics · doi:10.1016/j.jpeds.2008.11.033 · Cited by 327 · cited 2×
“In their meta-analytic reviews (Ferguson, 2007a, 2007b; Ferguson & Kilburn, 2009), Ferguson and his colleagues claimed that the trim and fill technique produces a “corrected” coefficient; it does not.”
BackgroundFerguson, C. J., & Kilburn, J. (2010) · Psychological Bulletin · doi:10.1037/a0018566 · Cited by 311 · cited 10×
“Keywords: meta-analysis, violent video games, aggression We appreciate the opportunity to reply to the Ferguson and Kilburn (2010) critique of our meta-analysis on violent video game effects (Anderson et al., 2010).”
BackgroundInfluentialFerguson, C. J., San Miguel, C., & Hartley, R. D. (2009) · The Journal of Pediatrics · doi:10.1016/j.jpeds.2009.06.021 · Cited by 274
Przybylski, A. K., Weinstein, N., Ryan, R. M., & Rigby, C. S. (2009) · CyberPsychology & Behavior · doi:10.1089/cpb.2009.0083 · Cited by 262 · cited 7×
“, & Rothstein, H. R. (2009). Introduction to meta-analysis. New York, NY: Wiley. Bushman, B. J., & Anderson, C. A. (2001). Media violence and the American public: Scientific facts versus media misinformation. American Psychologist, 56, 477–489. Bushman, B. J., & Geen, R. G. (1990). The role of cognitive–emotional medi…”
BackgroundMethodResultInfluentialAffect of the Game Player: Short-Term Effects of Highly and Mildly Aggressive Video Games
Anderson, C. A., & Ford, C. M. (1986) · Personality and Social Psychology Bulletin · doi:10.1177/0146167286124002 · Cited by 226
Self-citationRothstein, H. R. (1990) · Journal of Applied Psychology · doi:10.1037//0021-9010.75.3.322 · Cited by 221 · cited 2×
“, Rothstein, H. R., & Whetzel, D. (2006). Publication bias:”
Self-citationPublication bias as a threat to the validity of meta-analytic results
Rothstein, H. R. (2008) · Journal of Experimental Criminology · doi:10.1007/s11292-007-9046-9 · Cited by 216 · cited 4×
“One of us (Rothstein) is an expert on publication bias in meta-analysis (McDaniel, Rothstein, & Whetzel, 2006; Rothstein, 2004, 2008a, 2008b; Rothstein & Hopewell, 2009; Rothstein, Sutton, & Borenstein, 2005).”
InfluentialSelf-citationUsing the normal quantile plot to explore meta-analytic data sets.
Wang, M. C., & Bushman, B. J. (1998) · Psychological Methods · doi:10.1037//1082-989x.3.1.46 · Cited by 216 · cited 1×
“…and methods (Bushman & Wang, 1995, 1996; Hedges, Cooper, & Bushman, 1992; Ioannidis, Patsopoulos, & Rothstein, 2008; Rothstein, 2008b; Rothstein & McDaniel, 1989; Schmidt et al., 1993; Valentine, Pigott, & Rothstein, in press; Wade, Turner, Rothstein, & Lavenberg, 2006; Wang & Bushman, 1998).”
Self-citationProblems and Limitations in Conducting Systematic Reviews
Egger, M., Dickersin, K., & Smith, G. D. (2001) · Systematic Reviews in Health Care · doi:10.1002/9780470693926.ch3 · Cited by 202
Ferguson, C. J., & Rueda, S. M. (2010) · European Psychologist · doi:10.1027/1016-9040/a000010 · Cited by 167
Bushman, B. J., & Geen, R. G. (1990) · Journal of Personality and Social Psychology · doi:10.1037/0022-3514.58.1.156 · Cited by 166 · cited 1×
“Two of us (Anderson and Bushman) have been conducting research on violent media (including violent video games) for at least 20 years (e.g., Anderson & Ford, 1986; Bushman & Geen, 1990).”
BackgroundSelf-citationPUBLICATION BIAS: A CASE STUDY OF FOUR TEST VENDORS
McDANIEL, M. A., ROTHSTEIN, H. R., & WHETZEL, D. L. (2006) · Personnel Psychology · doi:10.1111/j.1744-6570.2006.00059.x · Cited by 141
Self-citationEvidence for a General Construct of Aggression
Carlson, M., Marcus-Newhall, A., & Miller, N. (1989) · Personality and Social Psychology Bulletin · doi:10.1177/0146167289153008 · Cited by 133 · cited 2×
“Duval, 2005) and other meta-analysis experts who advocate its use have stated unequivocally that one should not view the adjusted estimate as a corrected or more accurate estimate of the effect, because it is based on imputed data points. Trim and fill is most appropriately considered a useful sensitivity analysis tha…”
ResultEscobar-Chaves, S. L., & Anderson, C. A. (2008) · The Future of Children · doi:10.1353/foc.0.0007 · Cited by 109 · cited 1×
“Second, there is evidence that certain types of youth violence rates are increasing (Escobar-Chaves & Anderson, 2008).”
BackgroundSelf-citationSchmidt, F. L., Law, K., Hunter, J. E., & Rothstein, H. R. (1993) · Journal of Applied Psychology · doi:10.1037//0021-9010.78.1.3 · Cited by 108 · cited 2×
“Two of us (Bushman and Rothstein) teach graduate-level courses on metaanalysis, have written meta-analysis books (Borenstein, Hedges, Higgins, & Rothstein, 2009; Wang & Bushman, 1999), have contributed chapters to reference books on meta-analysis (Bushman & Wang, 2009; Rothstein, 2003; Rothstein & Hopewell, 2009; Roth…”
Self-citationMedia violence and the American public revisited.
Anderson, C. A., & Bushman, B. J. (2002) · American Psychologist · doi:10.1037/0003-066x.57.6-7.448 · Cited by 93 · cited 1×
“Because violent crime is influenced by so many risk factors, simple studies of national crime rate changes are difficult to interpret (Anderson & Bushman, 2002).”
BackgroundSelf-citationStrube, M. (2004) · Academy of Management Review · doi:10.5465/amr.2004.12736129 · Cited by 74
M-Rated Video Games and Aggressive or Problem Behavior Among Young Adolescents
Olson, C. K., Kutner, L. A., Baer, L., Beresin, E. V., Warner, D. E., & Nicholi II, A. M. (2009) · Applied Developmental Science · doi:10.1080/10888690903288748 · Cited by 69 · cited 1×
“Several studies that are now available were not available at the time of the cutoff for the meta-analysis (i.e., Ferguson & Rueda, in press; Ferguson, San Miguel, & Hartley, in press; Olson et al., 2009; Przybylski, Weinstein, Ryan, & Rigby, 2009).”
BackgroundHedges, L. V., Cooper, H. M., & Bushman, B. J. (1992) · Psychological Bulletin · doi:10.1037//0033-2909.111.1.188 · Cited by 46
Self-citationWade, C. A., Turner, H. M., Rothstein, H. R., & Lavenberg, J. G. (2006) · Evidence & Policy · doi:10.1332/174426406775249705 · Cited by 40
Self-citationGuidelines for Conducting and Reporting Meta-Analyses
Rothstein, H. R., & McDaniel, M. A. (1989) · Psychological Reports · doi:10.2466/pr0.1989.65.3.759 · Cited by 29 · cited 1×
“…theory and methods (Bushman & Wang, 1995, 1996; Hedges, Cooper, & Bushman, 1992; Ioannidis, Patsopoulos, & Rothstein, 2008; Rothstein, 2008b; Rothstein & McDaniel, 1989; Schmidt et al., 1993; Valentine, Pigott, & Rothstein, in press; Wade, Turner, Rothstein, & Lavenberg, 2006; Wang &…”
Self-citationBushman, B. J., & Wang, M. C. (1995) · Psychological Bulletin · doi:10.1037/0033-2909.117.3.530 · Cited by 27
Self-citationVote counting methods in meta-analysis
Bushman, B. J., & Wang, M. C. (2009) · Data Archiving and Networked Services (DANS) · Cited by 18
Self-citationBushman, B. J., & Wang, M. C. (1996) · Psychological Methods · doi:10.1037//1082-989x.1.1.66 · Cited by 17 · cited 1×
“…& Hopewell, 2009; Rothstein, McDaniel, & Borenstein, 2001), and have written peer-reviewed articles that advance meta-analytic theory and methods (Bushman & Wang, 1995, 1996; Hedges, Cooper, & Bushman, 1992; Ioannidis, Patsopoulos, & Rothstein, 2008; Rothstein, 2008b; Rothstein & McDaniel, 1989;…”
Self-citationRothstein, H. R. (2003) · Cited by 14
Self-citation
Cited by
Social media use and its impact on adolescent mental health: An umbrella review of the evidence.
Valkenburg, P. M., Meier, A., & Beyens, I. (2022) · Current opinion in psychology · doi:10.1016/j.copsyc.2021.08.017 · Cited by 767
Greitemeyer, T., & Mügge, D. O. (2014) · Personality and Social Psychology Bulletin · doi:10.1177/0146167213520459 · Cited by 636
“Whereas Anderson, Bushman and colleagues (Anderson et al., 2010; Bushman et al., 2010) argue that violent video games do increase aggression, this claim has been disputed by Ferguson and his research group (e.g., Ferguson & Kilburn, 2010).”
BackgroundMedia use and ADHD-related behaviors in children and adolescents: A meta-analysis
Nikkelen, S. W. C., Valkenburg, P. M., Huizinga, M., & Bushman, B. J. (2014) · Developmental Psychology · doi:10.1037/a0037318 · Cited by 339
“…movement; Christakis, 2009; Christakis, Zimmerman, DiGiuseppe, & McCarty, 2004; Geist & Gibson, 2000; Halpern, 1975; Jensen et al., 1997; Landhuis, Poulton, Welch, & Hancox, 2007; Levine & Waite, 2000) or to the overall amount of time children spent consuming media (Christakis, 2009; Zimmerman &…”
Self-citationMetaanalysis of the relationship between violent video game play and physical aggression over time
Prescott, A. T., Sargent, J. D., & Hull, J. G. (2018) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1611617114 · Cited by 303
Krahé, B., Möller, I., Huesmann, L. R., Kirwil, L., Felber, J., & Berger, A. (2011) · Journal of Personality and Social Psychology · doi:10.1037/a0021711 · Cited by 236
“…of media violence to increase aggression is far from being over, as reflected in a recent issue of Psychological Bulletin (Anderson et al., 2010; Bushman et al., 2010; Ferguson & Kilburn, 2010; Huesmann, 2010) and in a 2008 special issue of the American Behavioral Scientist, in which Grimes,…”
Metadata
- Authors
- Bushman, B. J., Rothstein, H. R., & Anderson, C. A.
- Affiliations
- University of Michigan (US)
- Baruch College (US)
- Iowa State University (US)
- Venue
- Psychological Bulletin
- Published
- 2010-03
- Volume
- 136
- Issue
- 2
- Pages
- 182-187
- ISSN
- 0033-2909 (ISSN-L)
- 1939-1455
- OpenAlex
- W2100193398
- Semantic Scholar
- 0b854b5a8d89106845e7651d09d36fc7829c8509
- Cited by
- 104
- References
- 56
- Citations by year
- 2026: 3 · 2025: 2 · 2024: 1 · 2023: 3 · 2022: 1 · 2021: 7 · 2020: 2 · 2019: 2 · 2018: 3 · 2017: 5 · 2016: 8 · 2015: 8 · 2014: 26 · 2013: 13 · 2012: 10
- FWCI
- 14.43
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Psychological Bulletin (publishedVersion)
- VU Research Portal (publishedVersion)
- Digital Academic REpository of VU University Amsterdam (Vrije Universiteit Amsterdam) (submittedVersion)
- Data Archiving and Networked Services (DANS) (submittedVersion)
- Topics
- Social Sciences › Psychology › Social Psychology › Bullying, Victimization, and Aggression
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Sociology and Political Science › Digital Games and Media
- Keywords
- violent video games, media violence, publication bias, selection bias, unpublished research, effect size, meta-analysis methodology
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.