Computing Contrasts, Effect Sizes, and Counternulls on Other People's Published Data: General Procedures for Research Consumers
Open accessAbstract
We describe convenient statistical procedures that will enable research consumers (e.g,, professional psychologists, graduate students, and researchers themselves) to reach beyond the published conclusions and make an independent assessment of the reported results. Appropriately conceived contrasts accompanied by effect size estimates often allow researchers to address precise predictions that the authors of the published report may have ignored or abandoned prematurely. We describe the use of t, F, and Z to compute contrasts with different raw ingredients, and we review 3 effect size indices (Cohen's d, Hedges's g, and the Pearson r) and a way of displaying the magnitude of any effect size r. We also describe how to construct confidence limits for the obtained effect as well as its null-counternull interval.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Cites
Experimental Design: Procedures for the Behavioral Sciences
Kirk, R. (2013) · doi:10.4135/9781483384733 · Cited by 8,539 · see citing works
The earth is round (p < .05).
Cohen, J. (1994) · American Psychologist · doi:10.1037/0003-066x.49.12.997 · Cited by 4,003 · see citing works
The statistical power of abnormal-social psychological research: A review.
Cohen, J. (1962) · The Journal of Abnormal and Social Psychology · doi:10.1037/h0045186 · Cited by 1,824 · see citing works
Things I have learned (so far).
Cohen, J. (1990) · American Psychologist · doi:10.1037/0003-066x.45.12.1304 · Cited by 1,781 · see citing works
Jacobson, N. S., Follette, W. C., & Revenstorf, D. (1984) · Behavior Therapy · doi:10.1016/s0005-7894(84)80002-7 · Cited by 1,294 · cited 1× · see citing works
“ings (e.g., Jacobson et al., 1984; Kendall & Grove, 1988;”
Writing meta-analytic reviews.
Rosenthal, R. (1995) · Psychological Bulletin · doi:10.1037/0033-2909.118.2.183 · Cited by 1,108 · see citing works
Self-citationDo studies of statistical power have an effect on the power of studies?
Sedlmeier, P., & Gigerenzer, G. (1989) · Psychological Bulletin · doi:10.1037/0033-2909.105.2.309 · Cited by 748 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Ohayon, M. M., Carskadon, M. A., Guilleminault, C., & Vitiello, M. V. (2004) · Sleep · doi:10.1093/sleep/27.7.1255 · Cited by 3,349 · see citing works
Chiesa, A., & Serretti, A. (2009) · The Journal of Alternative and Complementary Medicine · doi:10.1089/acm.2008.0495 · Cited by 1,681 · see citing works
Jeynes, W. H. (2005) · Urban Education · doi:10.1177/0042085905274540 · Cited by 1,396 · see citing works
Jain, S., Shapiro, S. L., Swanick, S., Roesch, S. C., Mills, P. J., Bell, I., & Schwartz, G. E. R. (2007) · Annals of Behavioral Medicine · doi:10.1207/s15324796abm3301_2 · Cited by 1,300 · see citing works
Taras, V., Kirkman, B. L., & Steel, P. (2010) · Journal of Applied Psychology · doi:10.1037/a0018938 · Cited by 1,160 · see citing works
Nguyen, H.-H. D., & Ryan, A. M. (2008) · Journal of Applied Psychology · doi:10.1037/a0012702 · Cited by 845 · see citing works
EULAR evidence-based recommendations for the management of fibromyalgia syndrome
Carville, S. F., Arendt-Nielsen, S., Bliddal, H., Blotman, F., Branco, J. C., Buskila, D., Da Silva, J. A. P., Danneskiold-Samsøe, B., Dincer, F., Henriksson, C., et al. (2008) · Annals of the Rheumatic Diseases · doi:10.1136/ard.2007.071522 · Cited by 752 · see citing works
Augmentation of Exposure Therapy With D-Cycloserine for Social Anxiety Disorder
Hofmann, S. G., Meuret, A. E., Smits, J. A. J., Simon, N. M., Pollack, M. H., Eisenmenger, K., Shiekh, M., & Otto, M. W. (2006) · Archives of General Psychiatry · doi:10.1001/archpsyc.63.3.298 · Cited by 560 · see citing works
Contrasts and Correlations in Effect-Size Estimation
Rosnow, R. L., Rosenthal, R., & Rubin, D. B. (2000) · Psychological Science · doi:10.1111/1467-9280.00287 · Cited by 513 · see citing works
Self-citationShi, F., Liu, B., Zhou, Y., Yu, C., & Jiang, T. (2009) · Hippocampus · doi:10.1002/hipo.20573 · Cited by 470 · see citing works
Gadow, K. D., Devincent, C. J., Pomeroy, J., & Azizian, A. (2005) · Autism · doi:10.1177/1362361305056079 · Cited by 388 · see citing works
Mindfulness training affects attention—Or is it attentional effort?
Jensen, C. G., Vangkilde, S., Frokjaer, V., & Hasselbalch, S. G. (2012) · Journal of Experimental Psychology: General · doi:10.1037/a0024931 · Cited by 284 · see citing works
Dietary advanced glycation end products and their relevance for human health
Nowotny, K., Schröter, D., Schreiner, M., & Grune, T. (2018) · Ageing Research Reviews · doi:10.1016/j.arr.2018.06.005 · Cited by 255 · see citing works
Self‐compassion training for binge eating disorder: A pilot randomized controlled trial
Kelly, A. C., & Carter, J. C. (2015) · Psychology and Psychotherapy: Theory, Research and Practice · doi:10.1111/papt.12044 · Cited by 233 · see citing works
O'Connor, D. W., Ames, D., Gardner, B., & King, M. (2009) · International Psychogeriatrics · doi:10.1017/s1041610208007588 · Cited by 239 · see citing works
Antidepressant studies in Parkinson's disease: A review and meta‐analysis
Weintraub, D., Morales, K. H., Moberg, P. J., Bilker, W. B., Balderston, C., Duda, J. E., Katz, I. R., & Stern, M. B. (2005) · Movement Disorders · doi:10.1002/mds.20555 · Cited by 191 · see citing works
Guest, E., Costa, B., Williamson, H., Meyrick, J., Halliwell, E., & Harcourt, D. (2019) · Body Image · doi:10.1016/j.bodyim.2019.04.002 · Cited by 184 · see citing works
“Where possible, effect sizes were calculated using Cohen’s d by examining group differences divided by the pooled standard deviation (Rosnow & Rosenthal, 1996).”
MethodStrunk, D. R., Brotman, M. A., & DeRubeis, R. J. (2010) · Behaviour Research and Therapy · doi:10.1016/j.brat.2010.03.011 · Cited by 169 · see citing works
Almeida, L. G., Ricardo-Garcell, J., Prado, H., Barajas, L., Fernández-Bouzas, A., Ávila, D., & Martínez, R. B. (2010) · Journal of Psychiatric Research · doi:10.1016/j.jpsychires.2010.04.026 · Cited by 165 · see citing works
Predictors of sustained therapeutic change.
Blatt, S. J., Zuroff, D. C., Hawley, L. L., & Auerbach, J. S. (2010) · Psychotherapy Research · doi:10.1080/10503300903121080 · Cited by 151 · see citing works
Rutledge, T., & Loh, C. (2004) · Annals of Behavioral Medicine · doi:10.1207/s15324796abm2702_9 · Cited by 144 · see citing works
Differences between preschool children with ODD, ADHD, and ODD+ADHD symptoms
Gadow, K. D., & Nolan, E. E. (2002) · Journal of Child Psychology and Psychiatry · doi:10.1111/1469-7610.00012 · Cited by 132 · see citing works
O'Connor, D. W., Ames, D., Gardner, B., & King, M. (2009) · International Psychogeriatrics · doi:10.1017/s1041610208008223 · Cited by 126 · see citing works
Cocaine craving and use during daily life
Preston, K. L., Vahabzadeh, M., Schmittner, J., Lin, J.-L., Gorelick, D. A., & Epstein, D. H. (2009) · Psychopharmacology · doi:10.1007/s00213-009-1655-8 · Cited by 123 · see citing works
Pasieczny, N., & Connor, J. (2011) · Behaviour Research and Therapy · doi:10.1016/j.brat.2010.09.006 · Cited by 116 · see citing works
Kelly, A. C., Wisniewski, L., Martin‐Wagar, C., & Hoffman, E. (2017) · Clinical Psychology & Psychotherapy · doi:10.1002/cpp.2018 · Cited by 112 · see citing works
Working memory training and interpreting interactions in intelligence interventions
Redick, T. S. (2015) · Intelligence · doi:10.1016/j.intell.2015.01.014 · Cited by 107 · see citing works
Wang, C., Bomberg, E., Billington, C. J., Levine, A. S., & Kotz, C. M. (2010) · Brain Research · doi:10.1016/j.brainres.2010.04.013 · Cited by 104 · see citing works
Labelle, L. E., Campbell, T. S., & Carlson, L. E. (2010) · Mindfulness · doi:10.1007/s12671-010-0005-6 · Cited by 102 · see citing works
Internet-based exercise intervention systems: Are more interactive designs better?
Hurling, R., Fairley, B. W., & Dias, M. B. (2006) · Psychology & Health · doi:10.1080/14768320600603257 · Cited by 94 · see citing works
Finnes, A., Ghaderi, A., Dahl, J., Nager, A., & Enebrink, P. (2019) · Journal of Occupational Health Psychology · doi:10.1037/ocp0000097 · Cited by 67 · see citing works
Temporal patterns of physical activity and sedentary behavior in 10–14 year-old children on weekdays
De Baere, S., Lefevre, J., De Martelaer, K., Philippaerts, R., & Seghers, J. (2015) · BMC Public Health · doi:10.1186/s12889-015-2093-7 · Cited by 57 · see citing works
Field, C. A., Caetano, R., Harris, T. R., Frankowski, R., & Roudsari, B. (2010) · Addiction · doi:10.1111/j.1360-0443.2009.02737.x · Cited by 63 · see citing works
Self‐management intervention for head and neck cancer couples: Results of a randomized pilot trial
Badr, H., Herbert, K., Chhabria, K., Sandulache, V. C., Chiao, E. Y., & Wagner, T. (2019) · Cancer · doi:10.1002/cncr.31906 · Cited by 56 · see citing works
Field, C. A., & Caetano, R. (2010) · Drug and Alcohol Dependence · doi:10.1016/j.drugalcdep.2009.11.025 · Cited by 53 · see citing works
Schroevers, M. J., Tovote, K. A., Keers, J. C., Links, T. P., Sanderman, R., & Fleer, J. (2015) · Mindfulness · doi:10.1007/s12671-013-0235-5 · Cited by 50 · see citing works
Gadow, K. D., Nolan, E. E., Sverd, J., Sprafkin, J., & Schneider, J. (2008) · Journal of Child Neurology · doi:10.1177/0883073808315412 · Cited by 49 · see citing works
Is food addiction a predictor of treatment outcome among patients with eating disorder?
Romero, X., Agüera, Z., Granero, R., Sánchez, I., Riesco, N., Jiménez‐Murcia, S., Gisbert‐Rodriguez, M., Sánchez‐González, J., Casalé, G., Baenas, I., et al. (2019) · European Eating Disorders Review · doi:10.1002/erv.2705 · Cited by 47 · see citing works
Measurement of Acute Pelvic Pain Intensity in Gynecology: A Comparison of Five Methods
Fauconnier, A., Dallongeville, E., Huchon, C., Ville, Y., & Falissard, B. (2009) · Obstetrics & Gynecology · doi:10.1097/aog.0b013e318195bd6c · Cited by 41 · see citing works
Epstein, D. H., & Preston, K. L. (2012) · The American Journal on Addictions · doi:10.1111/j.1521-0391.2012.00230.x · Cited by 28 · see citing works
Benefits of Increased Home-Visitation Services: Exploring a Case Management Model
de la Rosa, I. A., Perry, J., & Johnson, V. (2009) · Family & Community Health · doi:10.1097/01.fch.0000342817.95390.7e · Cited by 28 · see citing works
Generative AI and Creativity: A Systematic Literature Review and Meta-Analysis
Holzner, N., Maier, S., & Feuerriegel, S. (2025) · arXiv · doi:10.48550/arxiv.2505.17241 · Cited by 17 · see citing works
“When effect sizes were not directly reported, we converted other statistical measures (e.g., 𝑡 -values, 𝐹 -values, means and standard deviations) to Cohen’s 𝑑 using widely accepted conversion formulas [3, 17, 48, 49], which are documented in our project repository.”
de la Rosa, I. A., Perry, J., Dalton, L. E., & Johnson, V. (2005) · Research on Social Work Practice · doi:10.1177/1049731505277004 · Cited by 16 · see citing works
Kelly, A. C., Zuroff, D. C., Leybman, M. J., Martin, E. A., & Koestner, R. (2008) · Journal of Applied Social Psychology · doi:10.1111/j.1559-1816.2008.00370.x · Cited by 14 · see citing works
Song, W., Shari, N. I., Song, J., Zhang, R., Mansor, N. S., Leong Bin Abdullah, M. F. I., & Zhang, Z. (2024) · PLoS ONE · doi:10.1371/journal.pone.0312669 · Cited by 9 · see citing works
Rangel, M. L., Milbury, K., Kayser, K., Ripley, R. T., Kvale, E., & Badr, H. (2024) · JTO Clinical and Research Reports · doi:10.1016/j.jtocrr.2024.100736 · Cited by 5 · see citing works
“38 Finally, a post hoc, observed power analysis was conducted.”
- Authors
- Rosnow, R. L., & Rosenthal, R.
- Affiliations
- Temple University (US)
- Venue
- Psychological Methods
- Published
- 1996-12
- Volume
- 1
- Issue
- 4
- Pages
- 331-340
- ISSN
- 1082-989X (ISSN-L)
- 1939-1463
- OpenAlex
- W2061647004
- Cited by
- 761
- References
- 7
- Citations by year
- 2026: 9 · 2025: 14 · 2024: 20 · 2023: 19 · 2022: 32 · 2021: 39 · 2020: 40 · 2019: 38 · 2018: 28 · 2017: 27 · 2016: 28 · 2015: 35 · 2014: 32 · 2013: 32 · 2012: 35
- FWCI
- 6.31
- Type
- journal article
- Language
- en
- Open access
- green
- nrs.harvard.edu
- Licence
- cc-by
- Locations
- Psychological Methods (publishedVersion)
- Digital Access to Scholarship at Harvard (DASH) (Harvard University) (submittedVersion, cc-by)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Psychology Research and Bibliometrics
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods and Bayesian Inference
- Keywords
- research consumers, effect size, Cohen's d, confidence intervals
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
