The relative benefits of meta-analysis conducted with individual participant data versus aggregated data
The DOI matches its Crossref record; 42 of the 43 references it cites were found in a registry or an index.
Abstract
The authors describe the relative benefits of conducting meta-analyses with (a) individual participant data (IPD) gathered from the constituent studies and (b) aggregated data (AD), or the group-level statistics (in particular, effect sizes) that appear in reports of a study's results. Given that both IPD and AD are equally available, meta-analysis of IPD is superior to meta-analysis of AD. IPD meta-analysis permits synthesists to perform subgroup analyses not conducted by the original collectors of the data, to check the data and analyses in the original studies, to add new information to the data sets, and to use different statistical methods. However, the cost of IPD meta-analysis and the lack of available IPD data sets suggest that the best strategy currently available is to use both approaches in a complementary fashion such that the first step in conducting an IPD meta-analysis would be to conduct an AD meta-analysis. Regardless of whether a meta-analysis is conducted with IPD or AD, synthesists must remain vigilant in how they interpret their results. They must avoid ecological fallacies, Simpson's paradox, and interpretation of synthesis-generated evidence as supporting causal inferences.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“While the majority of published meta-analyses compare group means and yield only a single estimate of an intervention effect (Alemayehu, 2011), meta-analyses of the rich individual-level data from single case experiments can provide detailed knowledge of individual patient, therapist, and context variables that influence treatment effectiveness (Cooper & Patall, 2009; Iwakabe & Gazzola, 2009; Th”
From Acceptance and Commitment Therapy modules: Differential impact on treatment processes and outcomes (2016) · cites this paper as Cooper & Patall, 2009
Citations
Cites
Ecological Correlations and the Behavior of Individuals
Robinson, W. S. (1950) · American Sociological Review · doi:10.2307/2087176 · Cited by 4,513
The Handbook of Research Synthesis.
Howel, D., Cooper, H., & Hedges, L. V. (1995) · Biometrics · doi:10.2307/2533295 · Cited by 3,642
Self-citationFixed-and random-effects models in meta-analysis.
Hedges, L. V., & Vevea, J. L. (1998) · Psychological Methods · doi:10.1037/1082-989x.3.4.486 · Cited by 2,977
Publication Bias in Meta‐Analysis: Prevention, Assessment and Adjustments
(2005) · doi:10.1002/0470870168 · Cited by 2,368
The interpretation of interaction in contingency tables
Simpson, E. H. (1951) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.2517-6161.1951.tb00088.x · Cited by 2,301 · cited 1×
“Related to the ecological fallacy is another phenomenon called “Simpson’s paradox” (Simpson, 1951).”
BackgroundMeta-analysis of psychotherapy outcome studies.
Smith, M. L., & Glass, G. V. (1977) · American Psychologist · doi:10.1037/0003-066x.32.9.752 · Cited by 2,154
Lipsey, M. W., & Wilson, D. B. (1993) · American Psychologist · doi:10.1037/0003-066x.48.12.1181 · Cited by 2,033 · cited 3×
“Mathew and Nordström (1999) then relaxed some of Olkin and Sampson’s assumptions and demonstrated that the estimate of the treatment main effect also was the same for IPD meta-analysis and AD meta-analysis even when observations were not independent within studies and the observations had different covariation matrice…”
MethodResultResearch Synthesis and Meta-Analysis: A Step-by-Step Approach
Cooper, H. (2017) · doi:10.4135/9781071878644 · Cited by 1,716
Self-citationHow Many Studies Do You Need?: A Primer on Statistical Power for Meta-Analysis
Valentine, J. C., Pigott, T. D., & Rothstein, H. R. (2010) · Journal of Educational and Behavioral Statistics · doi:10.3102/1076998609346961 · Cited by 1,294
Interpersonal expectancy effects: the first 345 studies
Rosenthal, R., & Rubin, D. B. (1978) · Behavioral and Brain Sciences · doi:10.1017/s0140525x00075506 · Cited by 1,158
Cited by
The New Statistics: Why and How
Cumming, G. (2014) · Psychological Science · doi:10.1177/0956797613504966 · Cited by 3,200
“As mentioned earlier, extensions of most of the discussions in this estimation section, and information about ESCI, are available elsewhere (Cumming, 2012, 2013). Fidler and I (Fidler & Cumming, 2013) also discussed many of the topics of this article. Kline (2013b) discussed many new-statistics issues and provided gui…”
InfluentialGu, J., Strauss, C., Bond, R., & Cavanagh, K. (2015) · Clinical Psychology Review · doi:10.1016/j.cpr.2015.01.006 · Cited by 1,712
Integrative data analysis: The simultaneous analysis of multiple data sets
Curran, P. J., & Hussong, A. M. (2009) · Psychological Methods · doi:10.1037/a0015914 · Cited by 867
Why Interventions to Influence Adolescent Behavior Often Fail but Could Succeed
Yeager, D. S., Dahl, R. E., & Dweck, C. S. (2018) · Perspectives on Psychological Science · doi:10.1177/1745691617722620 · Cited by 512
“Metaregressions compare different interventions given to children of different ages and therefore mask the possibility that the same intervention given to different age groups in the same study might show a different moderation pattern (for a commentary, see Cooper & Patall, 2009).”
BackgroundA meta-analytic review of the Motivated Strategies for Learning Questionnaire
Credé, M., & Phillips, L. A. (2011) · Learning and Individual Differences · doi:10.1016/j.lindif.2011.03.002 · Cited by 494
Yeager, D. S., Fong, C. J., Lee, H. Y., & Espelage, D. L. (2015) · Journal of Applied Developmental Psychology · doi:10.1016/j.appdev.2014.11.005 · Cited by 472
Single-Paper Meta-Analysis: Benefits for Study Summary, Theory Testing, and Replicability
McShane, B. B., & Böckenholt, U. (2017) · Journal of Consumer Research · doi:10.1093/jcr/ucw085 · Cited by 457
“…known as statistical sufficiency, to that underlying the “gold standard” metaanalytic approach—namely, an appropriately specified hierarchical (or multilevel) model fit to the individual-level observations (Cooper and Patall 2009; Haidich 2010; Simmonds et al. 2005; Stewart and Tierney 2002).”
BackgroundDigital interventions for the treatment of depression: a meta-analytic review
Moshe, I., Terhorst, Y., Philippi, P., Domhardt, M., Cuijpers, P., Cristea, I., Pulkki-Råback, L., Baumeister, H., & Sander, L. B. (2021) · Psychological Bulletin · doi:10.1037/bul0000334 · Cited by 390
Yeager, D. S., Johnson, R., Spitzer, B. J., Trzesniewski, K. H., Powers, J., & Dweck, C. S. (2014) · Journal of Personality and Social Psychology · doi:10.1037/a0036335 · Cited by 389
“To examine this further, we maximized power by stacking data from Studies 2 and 3 into a single data set (see Cooper & Patall, 2009, for a review of raw data meta-analysis; also see Schimmack, 2012).”
MethodMorris, T. F., Murrell, T. S., Beegle, D. B., Camberato, J. J., Ferguson, R. B., Grove, J., Ketterings, Q., Kyveryga, P. M., Laboski, C. A. M., McGrath, J. M., et al. (2018) · Agronomy Journal · doi:10.2134/agronj2017.02.0112 · Cited by 372
“Meta-analysis of individual plot data, which is called individual participant data in other scientific disciplines such as the medical and social sciences, provides a much richer and more robust analysis of research data compared with meta-analysis of aggregate data (Cooper and Patall, 2009).”
MethodKaryotaki, E., Ebert, D. D., Donkin, L., Riper, H., Twisk, J., Burger, S., Rozental, A., Lange, A., Williams, A. D., Zarski, A. C., et al. (2018) · Clinical Psychology Review · doi:10.1016/j.cpr.2018.06.007 · Cited by 303
Annual Research Review: Building a science of personalized intervention for youth mental health
Ng, M. Y., & Weisz, J. R. (2016) · Journal of Child Psychology and Psychiatry · doi:10.1111/jcpp.12470 · Cited by 291
“In addition, individual patient (or participant) data (IPD) meta-analysis (Cooper & Patall, 2009) may reveal treatment moderators that are undetectable in standard metaanalyses using aggregated data.”
Arfè, A., Scotti, L., Varas-Lorenzo, C., Nicotra, F., Zambon, A., Kollhorst, B., Schink, T., Garbe, E., Herings, R., Straatman, H., et al. (2016) · BMJ · doi:10.1136/bmj.i4857 · Cited by 290
Changes in suicide rates following media reports on celebrity suicide: a meta-analysis
Niederkrotenthaler, T., Fu, K.-W., Yip, P. S. F., Fong, D. Y. T., Stack, S., Cheng, Q., & Pirkis, J. (2012) · Journal of Epidemiology and Community Health · doi:10.1136/jech-2011-200707 · Cited by 281
“Like the primary studies included in this review, this meta-analysis could only identify temporal associations and could not establish causality.(41) Furthermore, our emphasis on celebrity suicides meant that only 10 studies were eligible for inclusion, which had implications for the statistical power of the analysis.”
ResultRethinking “Generation Me”: A Study of Cohort Effects from 1976-2006
Trzesniewski, K. H., & Donnellan, M. B. (2010) · Perspectives on Psychological Science · doi:10.1177/1745691609356789 · Cited by 275
“…correlation (e.g., average yearly score on a measure correlated with year of data collection) and the sign of the individual coefficients (e.g., individual scores correlated with year of data collection) were different: one was positive and one was negative (see also Cooper & Patall, 2009).”
BackgroundEbert, D. D., Cuijpers, P., Muñoz, R. F., & Baumeister, H. (2017) · Frontiers in Psychiatry · doi:10.3389/fpsyt.2017.00116 · Cited by 244
“By collecting and pooling the primary data of individual trials, multiple underpowered trials can contribute to a large enough pooled sample size with sufficient power to examine effects on the incidence of MHD as well as analyzing patient subgroups and other effect modifying variables (74).”
MethodIntegrative Data Analysis in Clinical Psychology Research
Hussong, A. M., Curran, P. J., & Bauer, D. J. (2013) · Annual Review of Clinical Psychology · doi:10.1146/annurev-clinpsy-050212-185522 · Cited by 244
Meijer, A., Conradi, H. J., Bos, E. H., Anselmino, M., Carney, R. M., Denollet, J., Doyle, F., Freedland, K. E., Grace, S. L., Hosseini, S. H., et al. (2013) · British Journal of Psychiatry · doi:10.1192/bjp.bp.112.111195 · Cited by 219
Fokkema, M., Smits, N., Zeileis, A., Hothorn, T., & Kelderman, H. (2018) · Behavior Research Methods · doi:10.3758/s13428-017-0971-x · Cited by 218
Tackman, A. M., Sbarra, D. A., Carey, A. L., Donnellan, M. B., Horn, A. B., Holtzman, N. S., Edwards, T. S., Pennebaker, J. W., & Mehl, M. R. (2019) · Journal of Personality and Social Psychology · doi:10.1037/pspp0000187 · Cited by 213
“Compared with meta-analyses with sample-level data, analyses with participantlevel data (a) have more power and flexibility to examine moderators (Cooper & Patall, 2009), such as how the depression–I-talk effect varies by first-person singular pronoun type, communication context, and gender, and…”
MethodThe within and cross domain effects of work-family enrichment: A meta-analysis
Zhang, Y., Xu, S., Jin, J., & Ford, M. T. (2018) · Journal of Vocational Behavior · doi:10.1016/j.jvb.2017.11.003 · Cited by 206
He, J., Cai, Z., & Fan, X. (2017) · International Journal of Eating Disorders · doi:10.1002/eat.22661 · Cited by 203
Toolbox of individual-level interventions against online misinformation
Kozyreva, A., Lorenz-Spreen, P., Herzog, S. M., Ecker, U. K. H., Lewandowsky, S., Hertwig, R., Ali, A., Bak-Coleman, J., Barzilai, S., Basol, M., et al. (2024) · Nature Human Behaviour · doi:10.1038/s41562-024-01881-0 · Cited by 187
“It can also be worthwhile to create databases for individual participant data, which allows for much richer re-analyses than standard meta-analyses based on study-level effect sizes 73 .”
Ross, R. M., McKay, R., Coltheart, M., & Langdon, R. (2015) · Schizophrenia Bulletin · doi:10.1093/schbul/sbu187 · Cited by 198
Statistical models for meta-analysis: A brief tutorial
Kelley, G. A. (2012) · World Journal of Methodology · doi:10.5662/wjm.v2.i4.27 · Cited by 196
Estimation of the predictive power of the model in mixed‐effects meta‐regression: A simulation study
López‐López, J. A., Marín‐Martínez, F., Sánchez‐Meca, J., Van den Noortgate, W., & Viechtbauer, W. (2014) · British Journal of Mathematical and Statistical Psychology · doi:10.1111/bmsp.12002 · Cited by 191
“Formore details on so-called individual participant datameta-analyses, see, for example, Cooper and Patall (2009). y ¼ Xbþ uþ e; ð1Þ whereb is a (p + 1) 9 1vector containing the regression coefficients b0; b1; . . .; bp ,u is a k 9 1 vector of independent between-studies errors {ui} with…”
BackgroundIndividual participant data meta-analyses compared with meta-analyses based on aggregate data
Tudur Smith, C., Marcucci, M., Nolan, S. J., Iorio, A., Sudell, M., Riley, R., Rovers, M. M., & Williamson, P. R. (2016) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.mr000007.pub3 · Cited by 183
Eldevik, S., Hastings, R. P., Hughes, J. C., Jahr, E., Eikeseth, S., & Cross, S. (2010) · American Journal on Intellectual and Developmental Disabilities · doi:10.1352/1944-7558-115.5.381 · Cited by 168
“An aggregated data meta-analysis of individual study effect sizes derived from individual participant data is a recommended first step in any analysis of evidence for an intervention using individual level data (Cooper & Patall, 2009).”
BackgroundMethodInfluentialMiu, A. S., & Yeager, D. S. (2015) · Clinical Psychological Science · doi:10.1177/2167702614548317 · Cited by 164
“Thus, more reliable effect size estimates were possible (for descriptions of raw-data meta-analysis, see Cooper & Patall, 2009; Schimmack, 2012).”
BackgroundA new life-span approach to conscientiousness and health: Combining the pieces of the causal puzzle.
Friedman, H. S., Kern, M. L., Hampson, S. E., & Duckworth, A. L. (2014) · Developmental Psychology · doi:10.1037/a0030373 · Cited by 164
Ebert, D. D., Donkin, L., Andersson, G., Andrews, G., Berger, T., Carlbring, P., Rozenthal, A., Choi, I., Laferton, J. A. C., Johansson, R., et al. (2016) · Psychological Medicine · doi:10.1017/s0033291716001562 · Cited by 154
“By combining the primary data of multiple trials using an IPDMA approach, it is possible to obtain a large sample size with sufficient power to examine effects in relevant 2680 D. D. Ebert et al. subgroups and identify moderators of outcome (Cooper & Patall, 2009).”
MethodCurran, P. J., McGinley, J. S., Bauer, D. J., Hussong, A. M., Burns, A., Chassin, L., Sher, K., & Zucker, R. (2014) · Multivariate Behavioral Research · doi:10.1080/00273171.2014.889594 · Cited by 153
“Despite the many potential advantages of this approach, there remain a number of distinct challenges to conducting IDA in practice (Cooper & Patall, 2009; Bauer & Hussong, 2009; Curran & Hussong, 2009).”
Di Bello, M., Carnevali, L., Petrocchi, N., Thayer, J. F., Gilbert, P., & Ottaviani, C. (2020) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2020.06.016 · Cited by 154
Dora, J., Piccirillo, M. L., Foster, K. T., Arbeau, K. J., Armeli, S., Auriacombe, M., Bartholow, B. D., Beltz, A. M., Blumenstock, S. M., Bold, K. W., et al. (2023) · Psychological Bulletin · doi:10.1037/bul0000387 · Cited by 202
Acceptance and Commitment Therapy modules: Differential impact on treatment processes and outcomes
Villatte, J. L., Vilardaga, R., Villatte, M., Plumb Vilardaga, J. C., Atkins, D. C., & Hayes, S. C. (2016) · Behaviour Research and Therapy · doi:10.1016/j.brat.2015.12.001 · Cited by 136
Outcome-Reporting Bias in Education Research
Pigott, T. D., Valentine, J. C., Polanin, J. R., Williams, R. T., & Canada, D. D. (2013) · Educational Researcher · doi:10.3102/0013189x13507104 · Cited by 127
“…it would likely lead to increased use of Individual Participant Data meta-analyses, where data from original studies when available are combined with studylevel data in a single systematic research review (Cooper & Patall, 2009; Pigott, Williams, & Polanin, 2012; Valentine & Thompson, 2012).”
Huh, D., Mun, E., Larimer, M. E., White, H. R., Ray, A. E., Rhew, I. C., Kim, S., Jiao, Y., & Atkins, D. C. (2015) · Alcoholism: Clinical and Experimental Research · doi:10.1111/acer.12714 · Cited by 131
“Of the many advantages of IPD meta-analysis over AD meta-analysis (see Cooper and Patall, 2009; Curran and Hussong, 2009; Simmonds et al., 2005), the opportunity to utilize more appropriate, flexible analytic techniques is perhaps most significant for the current article.”
BackgroundMethodPonikowski, P. P., Mentz, R. J., Hernandez, A. F., Butler, J., Khan, M. S., van Veldhuisen, D. J., Roubert, B., Blackman, N. J., Friede, T., Jankowska, E. A., et al. (2023) · European Heart Journal · doi:10.1093/eurheartj/ehad586 · Cited by 153
“17,18”
How Effective Are Intentional Vocabulary‐Learning Activities? A Meta‐Analysis
WEBB, S., YANAGISAWA, A., & UCHIHARA, T. (2020) · The Modern Language Journal · doi:10.1111/modl.12671 · Cited by 100
“More-over, it would also enhance the transparency of the research, and allow more precise and robust estimations of individual participant data (see, e.g., Cooper & Patall, 2009).”
Haug, T., Nordgreen, T., Öst, L. G., & Havik, O. E. (2012) · Clinical Psychology Review · doi:10.1016/j.cpr.2012.04.002 · Cited by 128
Behavioral Training as New Treatment for Adult Amblyopia: A Meta-Analysis and Systematic Review
Inna Tsirlin, A., Colpa, L., Goltz, H. C., & WONG, A. M. F. (2015) · Investigative Ophthalmology & Visual Science · doi:10.1167/iovs.15-16583 · Cited by 141
Polyakova, M., Schroeter, M. L., Elzinga, B. M., Holiga, S., Schoenknecht, P., de Kloet, E. R., & Molendijk, M. L. (2015) · PLOS ONE · doi:10.1371/journal.pone.0141564 · Cited by 109
“This omission may be due to a limited number of trials and patients and the use of group-level statistics [17].”
InfluentialSocial identity mapping online.
Bentley, S. V., Greenaway, K. H., Haslam, S. A., Cruwys, T., Steffens, N. K., Haslam, C., & Cull, B. (2020) · Journal of Personality and Social Psychology · doi:10.1037/pspa0000174 · Cited by 105
The Relation of Personality and Intelligence—What Can the Brunswik Symmetry Principle Tell Us?
Kretzschmar, A., Spengler, M., Schubert, A., Steinmayr, R., & Ziegler, M. D. (2018) · Journal of Intelligence · doi:10.3390/jintelligence6030030 · Cited by 115
“However, it should be noted that the present study is not a meta-analysis based on individual data in a strict sense as described by Cooper and Patall [67].”
BackgroundKelley, G. A., & Kelley, K. S. (2010) · Journal of Hypertension · doi:10.1097/hjh.0b013e3283357d16 · Cited by 105
Kelley, G. A., & Kelley, K. S. (2013) · Journal of Obesity · doi:10.1155/2013/783103 · Cited by 103
“However, the use of the IPD approach needs to be considered with respect to the ability to retrieve IPD from investigators as well as the increased costs associated with the conduct of such [48], although methods to address the former have recently been developed [49].”
Kelley, G. A., & Kelley, K. S. (2015) · PLoS ONE · doi:10.1371/journal.pone.0129181 · Cited by 109
“However, such a decision needs to be made while considering factors such as (1) the ability to retrieve IPD from original study investigators, (2) the increased costs associated with conducting an IPD meta-analysis, and (3) any actual benefit that may be attained by conducting an IPD versus aggregate data meta-analysi…”
BackgroundReins, J. A., Buntrock, C., Zimmermann, J., Grund, S., Harrer, M., Lehr, D., Baumeister, H., Weisel, K., Domhardt, M., Imamura, K., et al. (2021) · Psychotherapy and Psychosomatics · doi:10.1159/000507819 · Cited by 98
Kelley, G. A., Kelley, K. S., & Hootman, J. M. (2015) · Arthritis Research & Therapy · doi:10.1186/s13075-015-0533-5 · Cited by 92
“3 times greater cost has been suggested by others to be 8 times greater since the research team continued to work on the project after funding ended [77].”
BackgroundKern, M. L., Hampson, S. E., Goldberg, L. R., & Friedman, H. S. (2012) · Developmental Psychology · doi:10.1037/a0030874 · Cited by 97
“…data (IDA; or mega-analysis), in which, like in aggregate level (AD) meta-analysis, a comprehensive review of existing studies is performed, but the raw participant data from each study, rather than standardized effect sizes, are aggregated (Cooper & Patall, 2009; Stewart & Clarke, 1995).”
BackgroundSystematic Reviews With Meta-Analysis: Why, When, and How?
Crocetti, E. (2016) · Emerging Adulthood · doi:10.1177/2167696815617076 · Cited by 91
“The IPD approach has been introduced in the medical sciences (for this reason, IPD can be also be referred to Individual Patient Data; Cooper & Patall, 2009).”
MethodLau, Y., Chee, D. G. H., Chow, X. P., Cheng, L. J., & Wong, S. N. (2020) · Preventive Medicine · doi:10.1016/j.ypmed.2020.106001 · Cited by 78
Bradley, L. D., Pasic, R. P., & Miller, L. E. (2019) · Journal of Laparoendoscopic & Advanced Surgical Techniques · doi:10.1089/lap.2019.0550 · Cited by 100
“14 Finally, we planned to report the frequency of complications in this meta-analysis.”
Yanagisawa, A., Webb, S., & Uchihara, T. (2020) · Studies in Second Language Acquisition · doi:10.1017/s0272263119000688 · Cited by 118
“Having access to open materials and datasets helps future meta-analyses to provide a clearer picture of the effects of glossed reading, as well as make more accurate and robust estimations by taking advantage of individual participant data (see e.g., Cooper & Patall, 2009).”
MethodMartone, M. E., García, A., & VandenBos, G. R. (2018) · American Psychologist · doi:10.1037/amp0000242 · Cited by 99
Publish (your data) or (let the data) perish! Why not publish your data too?
Wicherts, J. M., & Bakker, M. (2012) · Intelligence · doi:10.1016/j.intell.2012.01.004 · Cited by 85
“Proper data archiving is essential for the quality of meta-analyses (Cooper & Patall, 2009).”
BackgroundToward an Integrative Science of Life-Span Development and Aging
Hofer, S. M., & Piccinin, A. M. (2010) · The Journals of Gerontology Series B · doi:10.1093/geronb/gbq017 · Cited by 92
Kelley, G. A., Kelley, K. S., & Pate, R. R. (2014) · BMC Pediatrics · doi:10.1186/1471-2431-14-225 · Cited by 77
“Ninth, like any meta-analysis, the results of the current investigation may be prone to both ecological fallacy and/or Simpson’s Paradox [80].”
BackgroundSchultz, K., Jelusic, D., Wittmann, M., Krämer, B., Huber, V., Fuchs, S. P., Lehbert, N., Wingart, S., Stojanovic, D., Göhl, O., et al. (2018) · European Respiratory Journal · doi:10.1183/13993003.02000-2017 · Cited by 88
Methods for Synthesizing Findings on Moderation Effects Across Multiple Randomized Trials
Prevention Science and Methodology Group, Brown, C. H., Sloboda, Z., Faggiano, F., Teasdale, B., Keller, F., Burkhart, G., Vigna-Taglianti, F., Howe, G., Masyn, K., et al. (2013) · Prevention Science · doi:10.1007/s11121-011-0207-8 · Cited by 75
“It has seen limited use so far in randomized trials (Berlin et al. 2002; Cooper and Patall 2009; Higgins et al. 2001) and even less in the examination of moderator effects.”
BackgroundPollet, T. V., Stulp, G., Henzi, S. P., & Barrett, L. (2015) · American Journal of Primatology · doi:10.1002/ajp.22405 · Cited by 98
“…[Tu et al., 2008], measurement science [D’Errico, 2014], evolutionary ecology [Scheiner et al., 2000], and those involved in meta-analyses [Cooper & Patall, 2009; Hanley & Th eriault, 2000], all of which argue that the ecological fallacy and the Simpson’s paradox are more than just…”
BackgroundBeck, J. W., & Schmidt, A. M. (2012) · Journal of Applied Psychology · doi:10.1037/a0031145 · Cited by 88
Jewish Population in the United States, 2012
Sheskin, I. M., & Dashefsky, A. (2012) · American Jewish year book · doi:10.1007/978-94-007-5204-7_5 · Cited by 69
Hartwig, T. B., Sanders, T., Vasconcellos, D., Noetel, M., Parker, P. D., Lubans, D. R., Andrade, S., Ávila‐García, M., Bartholomew, J. B., Belton, S., et al. (2021) · British Journal of Sports Medicine · doi:10.1136/bjsports-2020-102740 · Cited by 88
“, where investigators have not themselves conducted and reported subgroup analysis).[28] Individual studies and meta-analyses might also be underpowered to effectively examine mediating pathways to understand causal mechanisms such as the role of physical activity intensity in improving CRF in school-based physical ac…”
BackgroundGardner, F., Leijten, P., Mann, J., Landau, S., Harris, V., Beecham, J., Bonin, E.-M., Hutchings, J., & Scott, S. (2017) · Public Health Research · doi:10.3310/phr05100 · Cited by 67
A scoping review of indirect comparison methods and applications using individual patient data
Veroniki, A. A., Straus, S. E., Soobiah, C., Elliott, M. J., & Tricco, A. C. (2016) · BMC Medical Research Methodology · doi:10.1186/s12874-016-0146-y · Cited by 64
“Although NMA is commonly performed with aggregated data, the inclusion of IPD can increase confidence in the results [5, 6], identify interactions that are otherwise undetectable [1, 7–9], and reduce variation in treatment effects both between studies within pairwise comparisons (heterogeneity) and between pairwise co…”
BackgroundMethodological quality of meta-analyses on the blood pressure response to exercise: a review
Johnson, B. T., MacDonald, H. V., Bruneau, M. L., Goldsby, T. U., Brown, J. C., Huedo–Medina, T. B., & Pescatello, L. S. (2014) · Journal of Hypertension · doi:10.1097/hjh.0000000000000097 · Cited by 64
Project INTEGRATE: An integrative study of brief alcohol interventions for college students.
Mun, E., The Project INTEGRATE Team, TORRE, J. D. L., Atkins, D. C., WHITE, H. R., Ray, A. E., Kim, S. Y., Jiao, Y., Clarke, N., Huo, Y., et al. (2014) · Psychology of Addictive Behaviors · doi:10.1037/adb0000047 · Cited by 63
“Citing the work by Steinberg et al. (1997) and personal communication with one of the investigators, Cooper and Patall (2009) noted that IPD meta-analysis probably costs 5-8 times more than AD meta-analysis, and takes several years from start to publication in the field of medical research.”
Rossi, J. S. (2012) · Wiley eBooks · doi:10.1002/9781118133880.hop202003 · Cited by 65
Miller, R. E., & Kirschbaum, C. (2018) · Psychoneuroendocrinology · doi:10.1016/j.psyneuen.2018.12.236 · Cited by 72
Can ACE Inhibitors and Angiotensin Receptor Blockers Be Detrimental in CKD Patients?
Onuigbo, M. A. C. (2011) · Nephron Clinical Practice · doi:10.1159/000324164 · Cited by 57
Gender differences in clinical trials of binge eating disorder: An analysis of aggregated data.
Shingleton, R. M., Thompson-Brenner, H., Thompson, D. R., Pratt, E. M., & Franko, D. L. (2015) · Journal of Consulting and Clinical Psychology · doi:10.1037/a0038849 · Cited by 59
“…significant effects, we chose this method because of advantages including greater statistical power, ability to ensure uniform analytic techniques across studies, and ability to model aspects of data such as the within-study (intraclass) correlation (Cooper & Patall, 2009; Curran & Hussong, 2009).”
MethodRevisiting the Functional Properties of Self-Efficacy: A Dynamic Perspective
Yeo, G., & Neal, A. F. (2013) · Journal of Management · doi:10.1177/0149206313490027 · Cited by 78
“The problem that lord et al. (2010) referred to is known as the “ecological fallacy” (Chen, Bliese, & Mathieu, 2005; Cooper & Patall, 2009; Robinson, 1950)—it involves equating findings that are based on aggregated data to findings based on disaggregated data that ignore group membership.”
BackgroundFrom Big Data to Knowledge in the Social Sciences
Hesse, B. W., Moser, R. P., & Riley, W. T. (2015) · The Annals of the American Academy of Political and Social Science · doi:10.1177/0002716215570007 · Cited by 68
“…information as found within meta-analytic approaches, allowing researchers to answer not only what works, but for whom, and in what context (cooper and Patall 2009). combining data across studies can provide sufficient power to detect moderator and mediator effects that individual clinical…”
BackgroundUeno, T., Fastrich, G. M., & Murayama, K. (2016) · Journal of Experimental Psychology: General · doi:10.1037/xge0000159 · Cited by 56
“…in te nd ed so le ly fo r th e pe rs on al us e of th e in di vi du al us er an d is no t to be di ss em in at ed br oa dl y. account the clustering of participant data within studies (Cooper & Patall, 2009; Riley, Lambert, & Abo-Zaid, 2010), but we included this for the purpose of comparison.”
“ Historia est magistra vitae ”? The impact of historical victimhood on current conspiracy beliefs
Pantazi, M., Gkinopoulos, T., Witkowska, M., Klein, O., & Bilewicz, M. (2020) · Group Processes & Intergroup Relations · doi:10.1177/1368430220968898 · Cited by 68
“…of estimating effect sizes is preferable to simply estimating the average meta-analytic effect of the individual three studies being statistically more powerful and is thus preferable when one has access to the original datasets (Cooper & Patall, 2009; Curran & Hussong, 2009; Schimmack, 2012).”
Callan, M. J., Shead, N. W., & Olson, J. M. (2015) · Addictive Behaviors · doi:10.1016/j.addbeh.2015.01.031 · Cited by 58
“We complemented our meta-analysis of the aggregated data with an individual participant data meta-analysis (see, e.g., Cooper & Patall, 2009; Curran & Hussong, 2009; Riley, Lambert, & Abo-Zaid, 2010).”
MethodCurran, P. J. (2009) · Psychological Methods · doi:10.1037/a0015972 · Cited by 51
Marsh, H. W., Jayasinghe, U. W., & Bond, N. W. (2010) · Journal of Informetrics · doi:10.1016/j.joi.2010.10.004 · Cited by 56
Nelson, W. L., & Suls, J. M. (2013) · Journal of Pain and Symptom Management · doi:10.1016/j.jpainsymman.2012.11.005 · Cited by 60
Zhang, J., Meng, Y., He, J., Xiang, Y., Wu, C., Wang, S., & Yuan, Z. (2018) · Journal of Autism and Developmental Disorders · doi:10.1007/s10803-018-3680-0 · Cited by 54
Toolbox of Interventions Against Online MisinformationPreprint
Kozyreva, A., Lorenz-Spreen, P., Herzog, S. M., Ecker, U. K. H., Lewandowsky, S., Hertwig, R., Ali, A., Bak-Coleman, J. B., Barzilai, S., Basol, M., et al. (2022) · doi:10.31234/osf.io/x8ejt · Cited by 50
Experimentation for Chatbot Usability Evaluation: A Secondary Study
Ren, R., Zapata, M., Castro, J. W., Dieste, Ó., & Acuña, S. T. (2022) · IEEE Access · doi:10.1109/access.2022.3145323 · Cited by 56
“…families of experiments provide software engineering researchers with some advantages for evaluating the effectiveness of SE tools [28] , [29], [30], [31], [32]: (i) families of experiments provide access to raw data so that researchers can apply consistent measurements and analysis…”
MethodGriffith, L. E., van den Heuvel, E. R., Fortier, I., Sohel, N., Hofer, S. M., Payette, H., Wolfson, C. M., Belleville, S., Kenny, M., Doiron, D., et al. (2014) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2014.09.003 · Cited by 53
Luhmann, M., Necka, E. A., Schönbrodt, F. D., & Hawkley, L. C. (2015) · Journal of Research in Personality · doi:10.1016/j.jrp.2015.11.003 · Cited by 57
“…is not unidimensional, we next examined the associations between valuing happiness and well-being on the factor level using random-effects two-step meta-analysis of individual data (Cooper & Patall, 2009; Riley, Lambert, & Abo-Zaid, 2010) implemented in the R package metafor (Viechtbauer, 2010).”
MethodHedayati, S., Damghanian, H., Farhadinejad, M., & Rastgar, A. A. (2023) · International Journal of Disaster Risk Reduction · doi:10.1016/j.ijdrr.2023.103758 · Cited by 72
Miller, L. E., Bhattacharyya, R., & Miller, A. L. (2020) · Medicine · doi:10.1097/md.0000000000022439 · Cited by 56
Davis, S. M., Daily, S. M., Kristjánsson, Á. L., Kelley, G. A., Zullig, K. J., Baus, A., Davidov, D. M., & Fisher, M. (2017) · Harm Reduction Journal · doi:10.1186/s12954-017-0156-z · Cited by 48
“Finally, because the aggregate data approach for this meta-analysis was used, these results are subject to ecological fallacy [68].”
MethodCipora, K., van Dijck, J., Georges, C., Masson, N., Goebel, S. M., Willmes, K., Pesenti, M., Schiltz, C., & Nuerk, H. (2019) · PsyArXiv (OSF Preprints) · doi:10.31234/osf.io/bwyr3 · Cited by 51
Williams, R. T., Citkowicz, M., Miller, D. I., Lindsay, J., & Walters, K. (2022) · Journal of Research on Educational Effectiveness · doi:10.1080/19345747.2021.2009072 · Cited by 59
“Second, Cooper and Patall (2009) note that within-study moderators, especially those that are inherent to the individuals in the study samples, do not necessarily translate to study-level moderators, which is known as ecological fallacy.”
BackgroundLi, R., Zhang, Y., Jiang, Y., Wang, M., Ang, W. H. D., & Lau, Y. (2021) · Clinical Rehabilitation · doi:10.1177/0269215521995179 · Cited by 42
Pragmatic bias impedes women’s access to political leadership
Corbett, C., Voelkel, J. G., Cooper, M., & Willer, R. (2022) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2112616119 · Cited by 64
Lau, Y., Nyoe, R. S. S., Wong, S. N., Hamid, Z. B. A., Leong, B. S., & Lau, S. T. (2018) · Resuscitation · doi:10.1016/j.resuscitation.2018.07.033 · Cited by 49
Levine, E. E., Hart, J. L., Moore, K. A., Rubin, E. B., Yadav, K. N., & Halpern, S. D. (2017) · Journal of Personality and Social Psychology · doi:10.1037/pspa0000101 · Cited by 54
The Relationship of Jewish Community Contexts and Jewish Identity: A 22-Community Study
Hartman, H. J., & Sheskin, I. M. (2012) · Contemporary Jewry · doi:10.1007/s12397-012-9090-2 · Cited by 41
Technology Integration in Mathematics Education: Examining the Quality of Meta-Analytic Research
Young, J. (2017) · International Journal on Emerging Mathematics Education · doi:10.12928/ijeme.v1i1.5713 · Cited by 46
“Moderators are also important because they identify statistical interactions, which do not imply causation but rather add context to effect size results (Cooper & Patall, 2009).”
BackgroundInsect transformation with piggyBac : getting the number of injections just right
Gregory, M., Alphey, L., Morrison, N. I., & Shimeld, S. M. (2016) · Insect Molecular Biology · doi:10.1111/imb.12220 · Cited by 39
“Meta-analyses tend to be conducted in the medical literature, so methods were co-opted as appropriate (Reade et al., 2008; Cooper and Patall, 2009).”
MethodThe association between self-compassion and self-rated health in 26 samples
Sirois, F. M. (2020) · BMC Public Health · doi:10.1186/s12889-020-8183-1 · Cited by 52
Veroniki, A. A., Ashoor, H. M., Le, S. P. C., Rios, P. D. L., Stewart, L. A., Clarke, M. J., Mavridis, D., Straus, S. E., & Tricco, A. C. (2019) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2019.05.031 · Cited by 37
Sales, C. M. D., Faísca, L., Ashworth, M. J., & Ayis, S. A. (2021) · Journal of Clinical Psychology · doi:10.1002/jclp.23278 · Cited by 40
Metadata
- Authors
- Cooper, H., & Patall, E. A.
- Affiliations
- Duke University (US)
- Venue
- Psychological Methods
- Published
- 2009
- Volume
- 14
- Issue
- 2
- Pages
- 165-176
- ISSN
- 1082-989X (ISSN-L)
- 1939-1463
- OpenAlex
- W2003323761
- Semantic Scholar
- ea908786103694ec9857ffd3dd31530573436cdc
- Cited by
- 339
- References
- 43
- Citations by year
- 2026: 10 · 2025: 25 · 2024: 18 · 2023: 21 · 2022: 30 · 2021: 22 · 2020: 22 · 2019: 20 · 2018: 24 · 2017: 16 · 2016: 26 · 2015: 32 · 2014: 17 · 2013: 16 · 2012: 14
- FWCI
- 16.13
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Psychological Methods (publishedVersion)
- PubMed (publishedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Psychology Research and Bibliometrics
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods and Bayesian Inference
- Keywords
- individual patient data, aggregated data, meta-analysis, subgroup analysis, ecological fallacy, Simpson's paradox, causal inference, effect size, data availability
- MeSH terms
- Data Collection, Data Interpretation, Statistical, Humans, Meta-Analysis as Topic, Models, Statistical
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.