The DOI matches its Crossref record; 96 of the 96 references it cites were found in a registry or an index; 1 of the 11 sources its claims cite were checked for support.
When combinations of humans and AI are useful: A systematic review and meta-analysis
Open accessPublished version of a preprint
Abstract
Inspired by the increasing use of artificial intelligence (AI) to augment humans, researchers have studied human–AI systems involving different tasks, systems and populations. Despite such a large body of work, we lack a broad conceptual understanding of when combinations of humans and AI are better than either alone. Here we addressed this question by conducting a preregistered systematic review and meta-analysis of 106 experimental studies reporting 370 effect sizes. We searched an interdisciplinary set of databases (the Association for Computing Machinery Digital Library, the Web of Science and the Association for Information Systems eLibrary) for studies published between 1 January 2020 and 30 June 2023. Each study was required to include an original human-participants experiment that evaluated the performance of humans alone, AI alone and human–AI combinations. First, we found that, on average, human–AI combinations performed significantly worse than the best of humans or AI alone (Hedges’ g = −0.23; 95% confidence interval, −0.39 to −0.07). Second, we found performance losses in tasks that involved making decisions and significantly greater gains in tasks that involved creating content. Finally, when humans outperformed AI alone, we found performance gains in the combination, but when AI outperformed humans alone, we found losses. Limitations of the evidence assessed here include possible publication bias and variations in the study designs analysed. Overall, these findings highlight the heterogeneity of the effects of human–AI collaboration and point to promising avenues for improving human–AI systems.
Overview
AI snapshot
Written from the paper’s full text.
TL;DR
This preregistered systematic review and meta-analysis of 106 experiments (370 effect sizes) finds that, on average, human–AI combinations perform worse than the best of humans or AI alone, though they do outperform humans alone. Effects vary by task type and by whether humans or AI perform better alone.
Key findings
- Human–AI combinations showed negative overall synergy versus the better of human alone or AI alone: Hedges’ g = −0.23 (95% CI, −0.39 to −0.07; P = 0.005).
- Human–AI combinations did show human augmentation versus humans alone: g = 0.64 (95% CI, 0.53 to 0.74; P = 0.000).
- Task type moderated synergy: decision tasks had losses (g = −0.27; 95% CI, −0.44 to −0.10; P = 0.002), while creation tasks had positive but non-significant gains (g = 0.19; 95% CI, −0.09 to 0.48; P = 0.180), with a significant difference between task types.
- Relative human/AI performance moderated synergy: when humans alone outperformed AI alone, synergy was positive (g = 0.46; 95% CI, 0.28 to 0.66); when AI alone outperformed humans alone, synergy was negative (g = −0.54; 95% CI, −0.71 to −0.37). Explanation, confidence, participant type, and division of labour were not statistically significant moderators.
Methods
- Preregistered systematic review and meta-analysis following Kitchenham and PRISMA guidelines; searched ACM Digital Library, Web of Science, and AIS eLibrary for studies published 1 January 2020–30 June 2023.
- From 5,126 initial papers, 74 met inclusion criteria, yielding 106 experiments and 370 unique effect sizes; included studies had to report performance for human alone, AI alone, and human–AI combination.
- Used Hedges’ g and a three-level random-effects meta-analytic model with robust variance estimation, Knapp–Hartung adjustment, and separate meta-regressions for moderators; sensitivity analyses clustered at paper level, excluded outliers, and used leave-one-out checks.
Limitations
- The paper states that results apply only to studies reporting all three conditions, so tasks that neither humans nor AI can perform alone are excluded; samples and outcome measures varied across studies, limiting comparability, and measurement error was not fully modelled.
- The paper states possible publication bias remains possible, though tests did not indicate it for synergy; tests did indicate potential publication bias for human augmentation. It also notes possible research topic selection bias, variation in study rigour, and high unexplained heterogeneity.
- Evident from the text: most effect sizes concerned decision tasks (~85%) and most human–AI systems had humans making final decisions after AI input (>95%), so findings may not generalize to other human–AI configurations.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures
Verification
Claims the paper makes
36 Own result · 14 Own interpretation · 7 Own method · 6 No full text of the sources · 6 No basis given · 1 Supported so far · 1 Sources not found · 1 Not checked yet
The sources with a verdict support this claim; the rest weren’t checked yet, have no full text we could read, or their verdicts aren’t rated yet.
In the paper
“We thus adopted a three-level meta-analytic model in which effect sizes are nested within experiments, so we explicitly took the dependencies in our data into account in the model63,64.”
§ When combinations of humans and AI are useful: A systematic review and meta-analysis > Methods > Data analysis
What the sources say
- Supportedcites [63] Van den Noortgate, W., López-López, J. A., Marín-Martínez, F. & Sánchez-Meca, J. Meta-analysis of multiple outcomes: a multilevel approach. Behav. Res. Methods 47, 1274–1294 (2015). source
The Abstract describes a three-level meta-analytic model for dependence among multiple effect sizes from the same study. The Conclusions likewise state that the approach accounts for dependence between effect sizes within studies. The citing authors’ adoption of the model is their own methodological choice, not a finding attributed to this source. (LLM, conf=0.92)
- Not checked — no full text of the sourcecites [64] Cheung, M. W.-L. Modeling dependent effect sizes with three-level meta-analyses: a structural equation modeling approach. Psychol. Methods 19, 211–229 (2014). source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We also found that the type of AI involved in the experiment (F2,103 = 3.77, two-tailed P = 0.026) and the year of publication (F3,102 = 3.65, two-tailed P = 0.015) moderated human–AI synergy, and the experimental design moderated human augmentation (F1,104 = 4.90, two-tailed P = 0.029).”
§ When combinations of humans and AI are useful: A systematic review and meta-analysis > Results > Heterogeneity of human–AI synergy
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Even though the average performance gains for creation tasks were not significantly different from 0 (presumably because of the relatively small sample size of n = 34), the difference between losses for decision tasks and gains for creation tasks was statistically significant.”
§ When combinations of humans and AI are useful: A systematic review and meta-analysis > Results > Heterogeneity of human–AI synergy
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Only 3 of the 100+ experiments in our analysis explore such processes with a predetermined delegation of separate subtasks to humans and AI.”
§ When combinations of humans and AI are useful: A systematic review and meta-analysis > Discussion > Moderating effect of task type
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We also found that the type of AI involved in the experiment (F2,103 = 3.77, two-tailed P = 0.026) and the year of publication (F3,102 = 3.65, two-tailed P = 0.015) moderated human–AI synergy, and the experimental design moderated human augmentation (F1,104 = 4.90, two-tailed P = 0.029).”
§ When combinations of humans and AI are useful: A systematic review and meta-analysis > Results > Heterogeneity of human–AI synergy
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“First, we found that the type of task significantly moderated human–AI synergy (F1,104 = 7.84, two-tailed P = 0.006).”
§ When combinations of humans and AI are useful: A systematic review and meta-analysis > Results > Heterogeneity of human–AI synergy
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“However, when we compared the performance of the human–AI systems to a different baseline—the humans alone—we found substantial evidence of human augmentation. The human–AI systems performed significantly better than humans alone, and this pooled effect size was positive (g = 0.64; t98 = 11.87; two-tailed P = 0.000; 95% CI, 0.53 to 0.74) and medium to large21.”
- Also stated in: § Main · plain
“On average, we found evidence of human augmentation, meaning that the average human–AI systems performed better than the human alone.”
§ When combinations of humans and AI are useful: A systematic review and meta-analysis > Results > Overall levels of human–AI synergy
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § Main · plain
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Among decision tasks—those in which participants decided between a finite set of options—the pooled effect size for human–AI synergy was significantly negative (g = −0.27; t104 = −3.20; two-tailed P = 0.002; 95% CI, −0.44 to −0.10), which indicates performance losses from combining humans and AI.”
§ When combinations of humans and AI are useful: A systematic review and meta-analysis > Results > Heterogeneity of human–AI synergy
hedgeddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“But when the AI alone outperformed the human alone, performance losses occurred in the combined system relative to the AI alone, with a negative effect size for human–AI synergy of g = −0.54 (t104 = −6.20; two-tailed P = 0.000; 95% CI, −0.71 to −0.37), a medium-sized effect21.”
§ When combinations of humans and AI are useful: A systematic review and meta-analysis > Results > Heterogeneity of human–AI synergy
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Here we addressed this question by conducting a preregistered systematic review and meta-analysis of 106 experimental studies reporting 370 effect sizes.”
- Also stated in: § Results · plain
“These papers reported the results of 106 unique experiments, and many of the experiments had multiple conditions, so we collected a total of 370 unique effect sizes measuring the impact of human–AI collaboration on task performance.”
§ When combinations of humans and AI are useful: A systematic review and meta-analysis > Abstract
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § Results · plain
- It may state two separate claims as one. 23
- Probably not a substantial claim. 16
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We also found evidence for substantial heterogeneity of effect sizes in our estimations of human–AI synergy (I2 = 97.7%) and human augmentation (I2 = 93.8%) (see Supplementary Tables 5 and 6 for more details).”
§ When combinations of humans and AI are useful: A systematic review and meta-analysis > Results > Heterogeneity of human–AI synergy
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“With the four effect sizes from these 3 experiments, we found that, on average, human–AI synergy (g = 0.22, t104 = 0.69; two-tailed P = 0.494; 95% CI, −0.42 to 0.87) occurred, but the result was not statistically significant (see Supplementary Information section 2.6 for a more detailed discussion of these experiments).”
§ When combinations of humans and AI are useful: A systematic review and meta-analysis > Discussion > Moderating effect of task type
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The remaining moderators we investigated were not statistically significant for human–AI synergy or human augmentation (explanation, confidence, participant type and division of labour).”
§ When combinations of humans and AI are useful: A systematic review and meta-analysis > Results > Heterogeneity of human–AI synergy
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Second, we found that the performance of the human and AI relative to each other impacted both human–AI synergy (F1,104 = 81.79, two-tailed P = 0.000) and human augmentation (F1,104 = 24.35, two-tailed P = 0.000).”
§ When combinations of humans and AI are useful: A systematic review and meta-analysis > Results > Heterogeneity of human–AI synergy
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Relatedly, we found that the type of data involved in the task significantly moderated both human–AI synergy (F4,101 = 15.24, two-tailed P = 0.000) and human augmentation (F4,101 = 6.52, two-tailed P = 0.000).”
§ When combinations of humans and AI are useful: A systematic review and meta-analysis > Results > Heterogeneity of human–AI synergy
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“As shown in Fig. 2, when the human alone outperformed the AI alone, the combined human–AI system outperformed both alone with an average pooled effect size for human–AI synergy of g = 0.46 (t104 = 5.06; two-tailed P = 0.000; 95% CI, 0.28 to 0.66), a medium-sized effect21.”
§ When combinations of humans and AI are useful: A systematic review and meta-analysis > Results > Heterogeneity of human–AI synergy
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“When the AI outperformed the human alone, greater performance gains tended to occur in the human–AI systems relative to the human alone, and the pooled effect size for human augmentation was positive and medium to large in magnitude (g = 0.74; t104 = 13.50; two-tailed P = 0.000; 95% CI, 0.63 to 0.85) (see Supplementary Figs. 4–7 for a more detailed visualization of this result for decision tasks).”
§ When combinations of humans and AI are useful: A systematic review and meta-analysis > Results > Heterogeneity of human–AI synergy
hedgedcorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“The measurement error can also vary across experiments.”
§ When combinations of humans and AI are useful: A systematic review and meta-analysis > Discussion > Limitations
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In contrast, among creation tasks—those in which participants created some sort of open-response content—the pooled effect size for human–AI synergy was positive (g = 0.19; t104 = 1.35; two-tailed P = 0.180; 95% CI, −0.09 to 0.48), pointing to synergy between humans and AI.”
§ When combinations of humans and AI are useful: A systematic review and meta-analysis > Results > Heterogeneity of human–AI synergy
hedgedn = 34- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“We find this assumption plausible because the experiments recruited different sets of participants and entailed different tasks or interventions.”
§ When combinations of humans and AI are useful: A systematic review and meta-analysis > Methods > Sensitivity analysis
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds. It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Supplementary Fig. 9 displays the funnel plot of the included effect sizes, and we did not observe significant asymmetry or regions of missing data in the plot for our primary outcome, human–AI synergy.”
§ When combinations of humans and AI are useful: A systematic review and meta-analysis > Methods > Bias tests
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Many of the experiments in our analysis evaluate performance according to a single measure of overall accuracy, but this measure corresponds to different things depending on the situation, and it omits other important criteria for human–AI systems.”
§ When combinations of humans and AI are useful: A systematic review and meta-analysis > Discussion > A roadmap for future work: finding human–AI synergy
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“For example, studies used different attention check mechanisms and performance incentive structures, which can both affect the quality of responses and thus introduce another source of noise into our data.”
§ When combinations of humans and AI are useful: A systematic review and meta-analysis > Discussion > Limitations
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“For example, researchers mostly used their own experimental platforms and stimuli, which naturally introduce sources of variation between their studies.”
§ When combinations of humans and AI are useful: A systematic review and meta-analysis > Discussion > Limitations
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Using this approach, we identified 11 outliers for human–AI synergy and 9 outliers for human augmentation.”
§ When combinations of humans and AI are useful: A systematic review and meta-analysis > Methods > Sensitivity analysis
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Our initial literature search yielded 5,126 papers, and, per the review process described in the Methods (see ‘Literature review’), we identified 74 that met our inclusion criteria (Supplementary Fig. 1).”
§ When combinations of humans and AI are useful: A systematic review and meta-analysis > Results
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“We searched an interdisciplinary set of databases (the Association for Computing Machinery Digital Library, the Web of Science and the Association for Information Systems eLibrary) for studies published between 1 January 2020 and 30 June 2023.”
§ When combinations of humans and AI are useful: A systematic review and meta-analysis > Abstract
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“Each study was required to include an original human-participants experiment that evaluated the performance of humans alone, AI alone and human–AI combinations.”
§ When combinations of humans and AI are useful: A systematic review and meta-analysis > Abstract
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“Here we addressed this question by conducting a preregistered systematic review and meta-analysis of 106 experimental studies reporting 370 effect sizes.”
§ When combinations of humans and AI are useful: A systematic review and meta-analysis > Abstract
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
None of the sources cited for this claim have been checked yet.
In the paper
“We conducted this meta-analysis in accord with the guidelines from Kitchenham55 on systematic reviews, and we followed the standards set forth by the Preferred Reporting Items for Systematic Reviews and Meta-Analyses56.”
§ When combinations of humans and AI are useful: A systematic review and meta-analysis > Methods
What the sources say
- Not matchedcites Kitchenham55
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not checked — no full text of the sourcecites [56] Moher, D., Liberati, A., Tetzlaff, J., Altman, D. G. & PRISMA Group Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. Ann. Intern. Med. 151, 264–269 (2009). source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
- Not matched
In the paper
“We chose Hedges’ g as our measure of effect size because it is unitless, so it allows us to compare human–AI performance across different metrics, and it corrects for upward bias in the raw standardized mean difference (Cohen’s d)61.”
§ When combinations of humans and AI are useful: A systematic review and meta-analysis > Methods > Data analysis
What the sources say
- Not checked — no full text of the sourcecites [61] Hedges, L. V. Distribution theory for Glass’s estimator of effect size and related estimators. J. Educ. Stat. 6, 107–128 (1981). source
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“We synthesized these data in a three-level meta-analytic model (see ‘Data analysis’ in Methods).”
§ When combinations of humans and AI are useful: A systematic review and meta-analysis > Results
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Third, although we did not find evidence of publication biases, it remains possible that they exist, which would impact our literature base and, by extension, our meta-analytic results.”
§ When combinations of humans and AI are useful: A systematic review and meta-analysis > Discussion > Limitations
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Fourth, our results only apply to the tasks, processes and participant pools that researchers have chosen to study, and these configurations may not be representative of the ways human–AI systems are configured in practical uses of AI outside the laboratory.”
§ When combinations of humans and AI are useful: A systematic review and meta-analysis > Discussion > Limitations
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“However, we expect that if there were a publication bias operating here, it would be a bias to publish studies that showed significant gains from combining humans and AI. And since our overall results showed the opposite, it seems unlikely that they are a result of publication bias.”
§ When combinations of humans and AI are useful: A systematic review and meta-analysis > Discussion > Limitations
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
In the paper
“As is typical for meta-analyses43, in our three-level model, we weighted effect sizes as a function of their variance across participants, so we did not account for this other source of variation in measurement.”
§ When combinations of humans and AI are useful: A systematic review and meta-analysis > Discussion > Limitations
What the sources say
- Not checked — no full text of the sourcecites [43] Borenstein, M., Hedges, L. V., Higgins, J. P. & Rothstein, H. R. Introduction to Meta-analysis (John Wiley & Sons, 2021). source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In other words, the human–AI systems we analysed were, on average, better than humans alone but not better than both humans alone and AI alone.”
- Also stated in: § Main · plain
“On average, we found evidence of human augmentation, meaning that the average human–AI systems performed better than the human alone. But we did not find human–AI synergy on average, meaning that the average human–AI systems performed worse than at least one of the human alone or the AI alone.”
§ When combinations of humans and AI are useful: A systematic review and meta-analysis > Results > Overall levels of human–AI synergy
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Also stated in: § Main · plain
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“We have made the materials required to reproduce our results publicly accessible through an Open Science Framework repository.”
§ When combinations of humans and AI are useful: A systematic review and meta-analysis > Results
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Fifth, the quality of our analysis depends on the quality of the studies we synthesized.”
§ When combinations of humans and AI are useful: A systematic review and meta-analysis > Discussion > Limitations
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The moderators we investigated account for some of this heterogeneity, but much remains unexplained.”
§ When combinations of humans and AI are useful: A systematic review and meta-analysis > Discussion > Limitations
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The discrepancy between the symmetry in the funnel plot for human–AI synergy versus asymmetry in the funnel plot for human augmentation may reflect how many researchers and journals implicitly assume an interest in human augmentation, comparing the human–AI system to the human alone.”
§ When combinations of humans and AI are useful: A systematic review and meta-analysis > Methods > Bias tests
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it.
In the paper
“Additionally, as discussed in ref. 33, human–AI synergy requires that humans be better at some parts of a task, AI be better at other parts of the task and the system as a whole be good at appropriately allocating subtasks to whichever partner is best for that subtask.”
§ When combinations of humans and AI are useful: A systematic review and meta-analysis > Discussion > A roadmap for future work: finding human–AI synergy
What the sources say
- Not checked — no full text of the sourcecites [33] Donahue, K., Chouldechova, A. & Kenthapadi, K. Human–algorithm collaboration: achieving complementarity and avoiding unfairness. In Proc. 2022 ACM Conference on Fairness, Accountability, and Transparency 1639–1656 (Association for Computing Machinery, 2022); https://doi.org/10.1145/3531146.3533221 source
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“To further ensure comprehensive coverage, we also conducted a forward and backward search on all studies we found that met our inclusion criteria.”
§ When combinations of humans and AI are useful: A systematic review and meta-analysis > Methods > Literature review
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
The paper relies on this without support.
There is a lack of broad conceptual understanding of when combinations of humans and AI are better than either alone.
“Despite such a large body of work, we lack a broad conceptual understanding of when combinations of humans and AI are better than either alone.”
The authors state this as an assumption.
The experiments in the analysis are independent from each other even if they come from the same paper.
“It does, however, consider the experiments in our analysis as independent from each other, even if they come from the same paper. We find this assumption plausible because the experiments recruited different sets of participants and entailed different tasks or interventions.”
The paper relies on this without support.
The participant samples overlapped only on the level of the experiment.
“because the participant samples overlapped only on the level of the experiment.”
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“While Gen-AI as an MKO can improve learning process, whereas over-reliance on these tools leads to logical complacency and selective cognitive atrophy (Vaccaro, , Almaatouq, & Malone, 2024).”
From The Double-Edged Sword of Automation: Impact of Generative AI Adoption on Student's Critical Thinking Skills in Higher Education Considering Cognitive Offloading as a Mediator (2026) · cites this paper as Vaccaro, , Almaatouq, & Malone, 2024
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Vaccaro et al. (2024) showed that human-AI combinations often outperform either alone in creative tasks.”
From Metadiscourse Patterns in Human-Written vs. Generative AI-Authored Research Abstracts: A Comparative Corpus-Based Analysis (2026) · cites this paper as Vaccaro et al. (2024)
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“To support the introduction of AI in clinical practice in intensive care, we propose starting with the concept of human-AI augmentation, which is more inclusive and better established according to medical literature [34].”
From Implementing Artificial Intelligence in Critical Care Medicine: a consensus of 22. (2025) · cites this paper as [34]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Recent meta-analyses of human-AI collaboration show that combinations sometimes underperform either party in isolation (Vaccaro et al., 2024)”
From Indie at Scale: How Large Language Models Make Community-Centered Game Development Economically Viable (2026) · cites this paper as Vaccaro et al., 2024
The cited source says what the claim attributes to it.
In the citing paper
“This collaboration has been described in two forms: human-AI augmentation (when human–AI interface enhances clinical performance compared to human alone) and human-AI synergy (where the combined performance exceeds that of both the human and the AI individually) [34].”
From Implementing Artificial Intelligence in Critical Care Medicine: a consensus of 22. (2025) · cites this paper as [34]
The cited source says what the claim attributes to it.
In the citing paper
“Vaccaro et al .'s meta-analysis of 106 experimental studies found that human-AI combinations performed worse than the best individual decision-maker on average for decision tasks 34 .”
From AI-induced never-skilling in medical education (2026) · cites this paper as [34]
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“Such tasks may have been part of the training data in GenAI models, which is a known issue in GenAI research [61]”
From Generative AI and Creativity: A Systematic Literature Review and Meta-Analysis (2025) · cites this paper as [61]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated major: it would touch a result — a value a result rests on, reported and recomputed p-values on opposite sides of the significance level, two passages that conflict with no sign which is right, or a cited work retracted before this paper was published.
In the paper
§ When combinations of humans and AI are useful: A systematic review and meta-analysis > Discussion > Surprisingly insignificant moderators
“the significant and less researched moderators we identified: the baseline performance of the human and AI alone, the type of task they perform, and the division of labour between them.”
§ When combinations of humans and AI are useful: A systematic review and meta-analysis > Results > Heterogeneity of human–AI synergy
“The remaining moderators we investigated were not statistically significant for human–AI synergy or human augmentation (explanation, confidence, participant type and division of labour).”
Citations
Cites
Bias in meta-analysis detected by a simple, graphical test
Egger, M., Smith, G. D., Schneider, M., & Minder, C. (1997) · BMJ · doi:10.1136/bmj.315.7109.629 · Cited by 56,722 · cited 1×
“We visually inspected the plots and performed Egger’s regression test 70 as well as the rank correlation test 71 to evaluate the results in the funnel plots.”
Preferred reporting items for systematic reviews and meta-analyses: the PRISMA Statement.
(2009) · PubMed Central · Cited by 44,859
Preferred reporting items for systematic reviews and meta-analyses: The PRISMA statement
Moher, D., Liberati, A., Tetzlaff, J., Altman, D. G., & the PRISMA Group* (2009) · Annals of Internal Medicine · doi:10.7326/0003-4819-151-4-200908180-00135 · Cited by 37,554
Quantifying heterogeneity in a meta‐analysis
Higgins, J. P. T., & Thompson, S. G. (2002) · Statistics in Medicine · doi:10.1002/sim.1186 · Cited by 36,763
Statistical Power Analysis for the Behavioral Sciences
Cohen, J. (2013) · doi:10.4324/9780203771587 · Cited by 25,373
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
Conducting Meta-Analyses in R with the metafor Package
Viechtbauer, W. (2010) · Journal of Statistical Software · doi:10.18637/jss.v036.i03 · Cited by 20,464
Operating characteristics of a rank correlation test for publication bias
Begg, C. B., & Mazumdar, M. (1994) · Biometrics · doi:10.2307/2533446 · Cited by 16,955 · cited 1×
“We visually inspected the plots and performed Egger’s regression test 70 as well as the rank correlation test 71 to evaluate the results in the funnel plots.”
Mastering the game of Go with deep neural networks and tree search
Silver, D., Huang, A., Maddison, C. J., Guez, A., Sifre, L., van den Driessche, G., Schrittwieser, J., Antonoglou, I., Panneershelvam, V., Lanctot, M., et al. (2016) · Nature · doi:10.1038/nature16961 · Cited by 15,612
Borenstein, M., Hedges, L. V., Higgins, J. P. T., & Rothstein, H. R. (2021) · doi:10.1002/9781119558378 · Cited by 9,799
The future of employment: How susceptible are jobs to computerisation?
Frey, C. B., & Osborne, M. A. (2017) · Technological Forecasting and Social Change · doi:10.1016/j.techfore.2016.08.019 · Cited by 8,727
Distribution Theory for Glass's Estimator of Effect Size and Related Estimators
Hedges, L. V. (1981) · Journal of Educational Statistics · doi:10.2307/1164588 · Cited by 5,695 · cited 2×
“We chose Hedges’ g as our measure of effect size because it is unit-less, so it allows us to compare human–AI performance across different metrics, and it corrects for upward bias in the raw standardized mean difference (Cohen’s d ) 61 .”
On the Theory of Scales of Measurement
Stevens, S. S. (1946) · Science · doi:10.1126/science.103.2684.677 · Cited by 4,761
Multiple Criteria Decision Analysis: State of the Art Surveys
Figueira, J. R., Greco, S., & Ehrogott, M. (2005) · International series in management science/operations research/International series in operations research & management science · doi:10.1007/b100605 · Cited by 4,594
Funnel plots for detecting bias in meta-analysis
Sterne, J. A. C., & Egger, M. (2001) · Journal of Clinical Epidemiology · doi:10.1016/s0895-4356(01)00377-8 · Cited by 3,779 · cited 1×
“First, we created funnel plots that graph the observed effect sizes on the x axis and the corresponding standard errors on the y axis 68 .”
Outlier and influence diagnostics for meta-analysis
Viechtbauer, W., & Cheung, M. W.-L. (2010) · Research Synthesis Methods · doi:10.1002/jrsm.11 · Cited by 2,949 · cited 1×
“To evaluate this risk, we adopted multiple diagnostic procedures out-lined by Viechtbauer and Cheung 67 .”
Building machines that learn and think like people
Lake, B. M., Ullman, T. D., Tenenbaum, J. B., & Gershman, S. J. (2017) · Behavioral and Brain Sciences · doi:10.1017/s0140525x16001837 · Cited by 2,550
Experimental evidence on the productivity effects of generative artificial intelligence
Noy, S., & Zhang, W. (2023) · Science · doi:10.1126/science.adh2586 · Cited by 2,150 · cited 1×
“Furthermore, the relatively little work that does evaluate human–AI collaboration according to quantitative performance metrics tends to report only the performance of the human alone and that of the human–AI combination (not the AI alone) 52 .”
Robust variance estimation in meta‐regression with dependent effect size estimates
Hedges, L. V., Tipton, E., & Johnson, M. C. (2010) · Research Synthesis Methods · doi:10.1002/jrsm.5 · Cited by 2,086 · cited 1×
“…estimates of the variance of our effect size estimates and, relatedly, standard errors, CIs and statistical tests, which account for the dependency of sampling errors from overlapping samples that occurred in experiments that compared multiple treatment groups to a single control group 65 .”
Peters, J. L., Sutton, A. J., Jones, D. R., Abrams, K. R., & Rushton, L. (2008) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2007.11.010 · Cited by 2,053 · cited 1×
“We enhanced these plots with colours indicating the significance level of each effect size to help distinguish publication bias from other causes of asymmetry 69 .”
Resistance to Medical Artificial Intelligence
Longoni, C., Bonezzi, A., & Morewedge, C. K. (2019) · Journal of Consumer Research · doi:10.1093/jcr/ucz013 · Cited by 2,015
Licklider, J. C. R. (1960) · IRE Transactions on Human Factors in Electronics · doi:10.1109/thfe2.1960.4503259 · Cited by 1,938
The rise of artificial intelligence in healthcare applications
Bohr, A., & Memarzadeh, K. (2020) · Artificial Intelligence in Healthcare · doi:10.1016/b978-0-12-818438-7.00002-2 · Cited by 1,636
Cheung, M. W.-L. (2014) · Psychological Methods · doi:10.1037/a0032968 · Cited by 1,342 · cited 1×
“We thus adopted a three-level meta-analytic model in which effect sizes are nested within experiments, so we explicitly took the dependencies in our data into account in the model 63,64 .”
Borenstein, M., Hedges, L. V., Higgins, J. P. T., Rothstein, H. R., & Den, A. B. V. (2020) · International Coaching Psychology Review · doi:10.53841/bpsicpr.2020.15.2.121 · Cited by 1,260
Buçinca, Z., Malaya, M. B., & Gajos, K. Z. (2021) · arXiv · doi:10.48550/arxiv.2102.09692 · Cited by 1,124
Human-computer collaboration for skin cancer recognition
Tschandl, P., Rinner, C., Apalla, Z., Argenziano, G., Codella, N., Halpern, A., Janda, M., Lallas, A., Longo, C., Malvehy, J., et al. (2020) · Nature Medicine · doi:10.1038/s41591-020-0942-0 · Cited by 1,024
Zhang, Y., Liao, Q. V., & Bellamy, R. K. E. (2020) · Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency · doi:10.1145/3351095.3372852 · Cited by 895 · cited 2×
“But some studies show that augmenting humans with AI can lead to better outcomes than humans or AI working alone 24–26 , while others show the opposite 4,7,9 .”
Does the Whole Exceed its Parts? The Effect of AI Explanations on Complementary Team Performance
Bansal, G., Wu, T., Zhou, J., Fok, R., Nushi, B., Kamar, E., Ribeiro, M. T., & Weld, D. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445717 · Cited by 757
Does the Whole Exceed its Parts? The Effect of AI Explanations on Complementary Team Performance
Bansal, G., Wu, T., Zhou, J., Fok, R., Nushi, B., Kamar, E., Ribeiro, M. T., & Weld, D. S. (2020) · arXiv · doi:10.48550/arxiv.2006.14779 · Cited by 754
Does automation bias decision-making?
SKITKA, L. J., MOSIER, K. L., & BURDICK, M. (1999) · International Journal of Human-Computer Studies · doi:10.1006/ijhc.1999.0252 · Cited by 694 · cited 1×
“For example, people often rely too much on AI systems (overreliance), using its suggestions as strong guidelines without seeking and processing more information 6,30,31 .”
Publication Bias in Meta‐Analysis
Rothstein, H. R., Sutton, A. J., & Borenstein, M. (2005) · Publication Bias in Meta‐Analysis · doi:10.1002/0470870168.ch1 · Cited by 603
A systematic evaluation of large language models of code
Xu, F. F., Alon, U., Neubig, G., & Hellendoorn, V. J. (2022) · Proceedings of the 6th ACM SIGPLAN International Symposium on Machine Programming · doi:10.1145/3520312.3534862 · Cited by 540
Nominal, Ordinal, Interval, and Ratio Typologies Are Misleading
Velleman, P. F., & Wilkinson, L. (1993) · The American Statistician · doi:10.1080/00031305.1993.10475938 · Cited by 533
Meta-analysis of multiple outcomes: a multilevel approach
Van den Noortgate, W., López-López, J. A., Marín-Martínez, F., & Sánchez-Meca, J. (2015) · Behavior Research Methods · doi:10.3758/s13428-014-0527-2 · Cited by 450 · cited 2×
“We thus adopted a three-level meta-analytic model in which effect sizes are nested within experiments, so we explicitly took the dependencies in our data into account in the model 63,64 .”
Jussupow, E., Spohrer, K., Heinzl, A., & Gawlitza, J. (2021) · Information Systems Research · doi:10.1287/isre.2020.0980 · Cited by 422 · cited 1×
“The studies in our dataset, though, come from different samples of people—some look at doctors 37–39 , others at crowdworkers 4,6,34 and still others at students 13,40,41 —and this variation can limit the comparability of the effect sizes to a degree 36 .”
Explanations Can Reduce Overreliance on AI Systems During Decision-Making
Vasconcelos, H., Jörke, M., Grunde-McLaughlin, M., Gerstenberg, T., Bernstein, M. S., & Krishna, R. (2023) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3579605 · Cited by 402 · cited 1×
“Other times, however, humans rely too little on AI (underreliance), ignoring its suggestions because of adverse attitudes towards automation 7,31,32 .”
Friedrich, J. O., Adhikari, N. K. J., & Beyene, J. (2008) · BMC Medical Research Methodology · doi:10.1186/1471-2288-8-32 · Cited by 333
Proxy tasks and subjective measures can be misleading in evaluating explainable AI systems
Buçinca, Z., Lin, P., Gajos, K. Z., & Glassman, E. L. (2020) · Proceedings of the 25th International Conference on Intelligent User Interfaces · doi:10.1145/3377325.3377498 · Cited by 307
Updates in Human-AI Teams: Understanding and Addressing the Performance/Compatibility Tradeoff
Bansal, G., Nushi, B., Kamar, E., Weld, D. S., Lasecki, W. S., & Horvitz, E. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.33012429 · Cited by 284
Experimental evidence of effective human–AI collaboration in medical decision-making
Reverberi, C., Rigon, T., Solari, A., Hassan, C., Cherubini, P., GI Genius CADx Study Group, Antonelli, G., Awadie, H., Bernhofer, S., Carballal, S., et al. (2022) · Scientific Reports · doi:10.1038/s41598-022-18751-2 · Cited by 281 · cited 1×
“For example, some studies estimate overall accuracy on the basis of the evaluation of as many as 500 distinct images 25 , whereas others estimate it on the basis of the evaluation of as few as 15 distinct ones 42 .”
Nicolescu, L., & Tudorache, M. T. (2022) · Electronics · doi:10.3390/electronics11101579 · Cited by 280
In Search of Synergy in Small Group Performance
Larson, J. (2013) · Psychology Press eBooks · doi:10.4324/9780203848784 · Cited by 276
Co-Writing with Opinionated Language Models Affects Users’ Views
Jakesch, M., Bhat, A., Buschek, D., Zalmanson, L., & Naaman, M. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581196 · Cited by 265
Deepfake detection by human crowds, machines, and machine-informed crowds
Groh, M., Epstein, Z., Firestone, C., & Picard, R. (2022) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2110013119 · Cited by 248
Jacobs, M., Pradier, M. F., McCoy, T. H., Perlis, R. H., Doshi-Velez, F., & Gajos, K. Z. (2021) · Translational Psychiatry · doi:10.1038/s41398-021-01224-x · Cited by 247 · cited 1×
“The studies in our dataset, though, come from different samples of people—some look at doctors 37–39 , others at crowdworkers 4,6,34 and still others at students 13,40,41 —and this variation can limit the comparability of the effect sizes to a degree 36 .”
Mirowski, P., Mathewson, K. W., Pittman, J., & Evans, R. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581225 · Cited by 240 · cited 1×
“Much of the recent work on generative AI with human participants, however, tends to focus on attitudes towards the tool 44,45 , interviews or think-alouds with participants 46–48 , or user experience instead of task performance 49–51 .”
Understanding the Role of Human Intuition on Reliance in Human-AI Decision-Making with Explanations
Chen, V., Liao, Q. V., Wortman Vaughan, J., & Bansal, G. (2023) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3610219 · Cited by 220 · cited 3×
“But some studies show that augmenting humans with AI can lead to better outcomes than humans or AI working alone 24–26 , while others show the opposite 4,7,9 .”
Deciding Fast and Slow: The Role of Cognitive Biases in AI-assisted Decision-making
Rastogi, C., Zhang, Y., Wei, D., Varshney, K. R., Dhurandhar, A., & Tomsett, R. (2022) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3512930 · Cited by 216 · cited 1×
“These factors have received much attention in recent years 4,24,26,35 .”
Using response ratios for meta-analyzing single-case designs with behavioral outcomes
Pustejovsky, J. E. (2018) · Journal of School Psychology · doi:10.1016/j.jsp.2018.02.003 · Cited by 214
Lai, V., Chen, C., Smith-Renner, A., Liao, Q. V., & Tan, C. (2023) · 2023 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3593013.3594087 · Cited by 204 · cited 1×
“This result complements the qualitative literature reviews on human–AI collaboration 27–29 , which highlight some of the surprising challenges that arise when integrating human intelligence and AI.”
Attewell, P. A. (1987) · Work and Occupations · doi:10.1177/0730888487014003001 · Cited by 200
Jo, E., Epstein, D. A., Jung, H., & Kim, Y.-H. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581503 · Cited by 174 · cited 1×
“Much of the recent work on generative AI with human participants, however, tends to focus on attitudes towards the tool 44,45 , interviews or think-alouds with participants 46–48 , or user experience instead of task performance 49–51 .”
De novo protein design by citizen scientists
Koepnick, B., Flatten, J., Husain, T., Ford, A., Silva, D., Bick, M. J., Bauer, A., Liu, G., Ishida, Y., Boykov, A., et al. (2019) · Nature · doi:10.1038/s41586-019-1274-4 · Cited by 169
Karinshak, E., Liu, S. X., Park, J. S., & Hancock, J. T. (2023) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3579592 · Cited by 167 · cited 1×
“Much of the recent work on generative AI with human participants, however, tends to focus on attitudes towards the tool 44,45 , interviews or think-alouds with participants 46–48 , or user experience instead of task performance 49–51 .”
AI-assistance for predictive maintenance of renewable energy systems
Shin, W., Han, J., & Rhee, W. (2021) · Energy · doi:10.1016/j.energy.2021.119775 · Cited by 155 · cited 1×
“Systems that combine human intelligence and AI tools can address multiple issues of societal importance, from how we diagnose disease to how we design complex systems 22–24 .”
Is the Most Accurate AI the Best Teammate? Optimizing AI for Teamwork
Bansal, G., Nushi, B., Kamar, E. S., Horvitz, E., & Weld, D. S. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i13.17359 · Cited by 154
A Review of Artificial Intelligence in Government and its Potential from a Public Policy Perspective
Valle-Cruz, D., Alejandro Ruvalcaba-Gomez, E., Sandoval-Almazan, R., & Ignacio Criado, J. (2019) · Proceedings of the 20th Annual International Conference on Digital Government Research · doi:10.1145/3325112.3325242 · Cited by 149
The efficacy of using judgmental versus quantitative forecasting methods in practice
Sanders, N. R., & Manrodt, K. B. (2003) · Omega · doi:10.1016/j.omega.2003.08.007 · Cited by 148
It's complicated: the relationship between user trust, model accuracy and explanations in AI
Papenmeier, A., Kern, D., Englebienne, G., & Seifert, C. (2022) · ACM Transactions on Computer-Human Interaction · doi:10.1145/3495013 · Cited by 144 · cited 1×
“For example, some studies estimate overall accuracy on the basis of the evaluation of as many as 500 distinct images 25 , whereas others estimate it on the basis of the evaluation of as few as 15 distinct ones 42 .”
A Review on Human–AI Interaction in Machine Learning and Insights for Medical Applications
Maadi, M., Khorshidi, H. A., & Aickelin, U. (2021) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph18042121 · Cited by 132 · cited 1×
“This result complements the qualitative literature reviews on human–AI collaboration 27–29 , which highlight some of the surprising challenges that arise when integrating human intelligence and AI.”
What Are Effect Sizes and Why Do We Need Them?
Hedges, L. V. (2008) · Child Development Perspectives · doi:10.1111/j.1750-8606.2008.00060.x · Cited by 123 · cited 2×
“The studies in our dataset, though, come from different samples of people—some look at doctors 37–39 , others at crowdworkers 4,6,34 and still others at students 13,40,41 —and this variation can limit the comparability of the effect sizes to a degree 36 .”
"Why is 'Chicago' deceptive?" Towards Building Model-Driven Tutorials for Humans
Lai, V., Liu, H., & Tan, C. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376873 · Cited by 123 · cited 3×
“(see Supplementary Tables 5 and 6 for more details).”
Analyzing outliers: influential or nuisance?
Altman, N., & Krzywinski, M. I. (2016) · Nature Methods · doi:10.1038/nmeth.3812 · Cited by 121
Wilder, B., Horvitz, E., & Kamar, E. (2020) · Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2020/212 · Cited by 118
Liu, H., Lai, V., & Tan, C. (2021) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3479552 · Cited by 111 · cited 2×
“But some studies show that augmenting humans with AI can lead to better outcomes than humans or AI working alone 24–26 , while others show the opposite 4,7,9 .”
Task complexity moderates group synergy
Almaatouq, A., Alsobay, M., Yin, M., & Watts, D. J. (2021) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2101062118 · Cited by 104 · cited 1×
“…where the human–AI group performs better than both the human alone and the AI alone, which is analogous to strong synergy in human groups 10,11 ; and (2) human augmentation, where the human–AI group performs better than the human alone (see Supplementary Information section 1.1 for more…”
InfluentialSelf-citationAngleKindling: Supporting Journalistic Angle Ideation with Large Language Models
Petridis, S., Diakopoulos, N., Crowston, K., Hansen, M., Henderson, K., Jastrzebski, S., Nickerson, J. V., & Chilton, L. B. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3580907 · Cited by 98 · cited 1×
“Much of the recent work on generative AI with human participants, however, tends to focus on attitudes towards the tool 44,45 , interviews or think-alouds with participants 46–48 , or user experience instead of task performance 49–51 .”
A Survey of Human‐Centered Evaluations in Human‐Centered Machine Learning
Sperrle, F., El‐Assady, M., Guo, G., Borgo, R., Chau, D. H., Endert, A., & Keim, D. A. (2021) · Computer Graphics Forum · doi:10.1111/cgf.14329 · Cited by 93
A Meta-Analysis of the Utility of Explainable Artificial Intelligence in Human-AI Decision-Making
Schemmer, M., Hemmer, P., Nitsche, M., Kühl, N., & Vössing, M. (2022) · Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society · doi:10.1145/3514094.3534128 · Cited by 91 · cited 1×
“When evaluating human–AI systems, many studies focus on human augmentation 12–15 .”
The DARPA grand challenge - development of an autonomous vehicle
Behringer, R., Sundareswaran, S., Daily, R., Bevly, D. M., Gregory, B., Elsley, R. K., Addison, B., & Guthmiller, W. (2004) · IEEE Intelligent Vehicles Symposium · doi:10.1109/ivs.2004.1336386 · Cited by 89
Improving Human-AI Collaboration With Descriptions of AI Behavior
Cabrera, Á. A., Perer, A., & Hong, J. I. (2023) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3579612 · Cited by 87 · cited 3×
“The studies in our dataset, though, come from different samples of people—some look at doctors 37–39 , others at crowdworkers 4,6,34 and still others at students 13,40,41 —and this variation can limit the comparability of the effect sizes to a degree 36 .”
Vimpari, V., Kultima, A., Hämäläinen, P., & Guckelsberger, C. (2023) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3611025 · Cited by 77 · cited 1×
“Much of the recent work on generative AI with human participants, however, tends to focus on attitudes towards the tool 44,45 , interviews or think-alouds with participants 46–48 , or user experience instead of task performance 49–51 .”
Towards a Science of Human-AI Decision Making: A Survey of Empirical Studies
Lai, V., Chen, C., Liao, Q. V., Smith-Renner, A., & Tan, C. (2021) · arXiv · doi:10.48550/arxiv.2112.11471 · Cited by 75
Toward human intervention-free clinical diagnosis of intracranial aneurysm via deep neural network
Bo, Z.-H., Qiao, H., Tian, C., Guo, Y., Li, W., Liang, T., Li, D., Liao, D., Zeng, X., Mei, L., et al. (2021) · Patterns · doi:10.1016/j.patter.2020.100197 · Cited by 71 · cited 2×
“When evaluating human–AI systems, many studies focus on human augmentation 12–15 .”
AI-Assisted Decision-making: a Cognitive Modeling Approach to Infer Latent Reliance Strategies
Tejeda, H., Kumar, A., Smyth, P., & Steyvers, M. (2022) · Computational Brain & Behavior · doi:10.1007/s42113-022-00157-y · Cited by 53 · cited 1×
“Given the multidisciplinary nature of our research question, papers published in different venues tended to refer to these components under various names 8,12,13,57,58 .”
The effects of mixing machine learning and human judgment
Vaccaro, M., & Waldo, J. (2019) · Communications of the ACM · doi:10.1145/3359338 · Cited by 49 · cited 2×
“These papers reported the results of 106 unique experiments, and many of the experiments had multiple conditions, so we collected a total of 370 unique effect sizes measuring the impact of human–AI collaboration coordination between humans and AI systems can hinder the collaborative process 4–9 .”
Self-citationA Taxonomy of Human and ML Strengths in Decision-Making to Investigate Human-ML Complementarity
Rastogi, C., Leqi, L., Holstein, K., & Heidari, H. (2023) · Proceedings of the AAAI Conference on Human Computation and Crowdsourcing · doi:10.1609/hcomp.v11i1.27554 · Cited by 46
What Question Would Turing Pose Today?
Grosz, B. J. (2012) · AI Magazine · doi:10.1609/aimag.v33i4.2441 · Cited by 44
Measuring the Efficiency of Automation-Aided Performance in a Simulated Baggage Screening Task
Boskemper, M. M., Bartlett, M. L., & McCarley, J. S. (2021) · Human Factors The Journal of the Human Factors and Ergonomics Society · doi:10.1177/0018720820983632 · Cited by 44 · cited 2×
“The studies in our dataset, though, come from different samples of people—some look at doctors 37–39 , others at crowdworkers 4,6,34 and still others at students 13,40,41 —and this variation can limit the comparability of the effect sizes to a degree 36 .”
Artificial intelligence and the future of work
Fahmy, K. I. (2018) · Undergraduate Research Journal · Cited by 40
Partial automation and the technology-enabled deskilling of routine jobs
Downey, M. (2021) · Labour Economics · doi:10.1016/j.labeco.2021.101973 · Cited by 39
AI Consent Futures: A Case Study on Voice Data Collection with Clinicians
Wilcox, L. G., Brewer, R. N., & Díaz, F. (2023) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3610107 · Cited by 38 · cited 1×
“Much of the recent work on generative AI with human participants, however, tends to focus on attitudes towards the tool 44,45 , interviews or think-alouds with participants 46–48 , or user experience instead of task performance 49–51 .”
Human-Algorithm Collaboration: Achieving Complementarity and Avoiding Unfairness
Donahue, K., Chouldechova, A., & Kenthapadi, K. (2022) · 2022 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3531146.3533221 · Cited by 33 · cited 2×
“33, human–AI synergy requires that humans be better at some parts of a task, AI be better at other parts of the task and the system as a whole be good at appropriately allocating subtasks to whichever partner is best for that subtask.”
Meta-analysis in the social sciences:
Wampold, B., Ahn, H.-N., & Kim, D. (2000) · Asia Pacific Education Review · doi:10.1007/bf03026147 · Cited by 29
Adapting to the algorithm: how accuracy comparisons promote the use of a decision aid
Liang, G., Sloane, J., Donkin, C., & Newell, B. R. (2022) · Cognitive Research Principles and Implications · doi:10.1186/s41235-022-00364-y · Cited by 28 · cited 1×
“The studies in our dataset, though, come from different samples of people—some look at doctors 37–39 , others at crowdworkers 4,6,34 and still others at students 13,40,41 —and this variation can limit the comparability of the effect sizes to a degree 36 .”
He, Z., Song, Y., Zhou, S., & Cai, Z. (2023) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3580983 · Cited by 28 · cited 1×
“The studies in our dataset, though, come from different samples of people—some look at doctors 37–39 , others at crowdworkers 4,6,34 and still others at students 13,40,41 —and this variation can limit the comparability of the effect sizes to a degree 36 .”
Learning When to Advise Human Decision Makers
Noti, G., & Chen, Y. (2023) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2023/339 · Cited by 25
Automatic discovery of interpretable planning strategies
Skirzyński, J., Becker, F., & Lieder, F. (2021) · Machine Learning · doi:10.1007/s10994-021-05963-2 · Cited by 21
A Test for Evaluating Performance in Human-Computer Systems
Campero, A., Vaccaro, M., Song, J., Wen, H., Almaatouq, A., & Malone, T. W. (2022) · arXiv · doi:10.48550/arxiv.2206.12390 · Cited by 21 · cited 1×
“In these cases, we may wish to consider a metric that applies a nonlinear scaling to the overall classification accuracy and thus takes such considerations into account 54 (Supplementary Information section 1.2).”
InfluentialSelf-citationGonzález, A. V., Bansal, G., Fan, A., Mehdad, Y., Jia, R., & Iyer, S. V. (2021) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2021.findings-acl.95 · Cited by 20
Confidence Intervals for Ratios of Means and Medians
Bonett, D. G., & Price, R. M. (2020) · Journal of Educational and Behavioral Statistics · doi:10.3102/1076998620934125 · Cited by 17
Effective Human-AI Teams via Learned Natural Language Rules and Onboarding
Mozannar, H., Lee, J., Wei, D., Sattigeri, P., Das, S., & Sontag, D. A. (2023) · neural information processing systems · doi:10.52202/075280-1328 · Cited by 12
Liu, X., Kirilyuk, V., Yuan, X., et al. (2023) · ACM Symposium on User Interface Software and Technology (UIST) · doi:10.1145/3586182.3615978 · Cited by 7
Learning to Work in a Materials Recovery Facility: Can Humans and Machines Learn from Each Other?
Kyriacou, H., Ramakrishnan, A., & Whitehill, J. (2021) · International Conference on Learning Analytics and Knowledge (LAK) · doi:10.1145/3448139.3448183 · Cited by 5
Role of Human-AI Interaction in Selective Prediction
Bondi, E., Koster, R., Sheahan, H., Chadwick, M. J., Bachrach, Y., Cemgil, A. T., Paquet, U., & Dvijotham, K. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i5.20465 · Cited by 2
Research works printed but not linked: 2
- Wilson, H. J. & Daugherty, P. R. Collaborative intelligence: humans and AI are joining forces. Harv. Bus. Rev. 96, 114–123 (2018).
- Hedges, L. V. & Olkin, I. Statistical Methods for Meta-analysis (Academic Press, 2014).
Cited by
Sorted by citations
Brynjolfsson, E., Li, D., & Raymond, L. (2025) · The Quarterly Journal of Economics · doi:10.1093/qje/qjae044 · Cited by 1,083
Dell'Acqua, F., McFowland III, E., Mollick, E. R., Lifshitz-Assaf, H., Kellogg, K., Rajendran, S., Krayer, L., Candelon, F., & Lakhani, K. R. (2023) · SSRN Electronic Journal · doi:10.2139/ssrn.4573321 · Cited by 794
Educational Strategies for Clinical Supervision of Artificial Intelligence Use
Abdulnour, R.-E. E., Gin, B., & Boscardin, C. K. (2025) · New England Journal of Medicine · doi:10.1056/nejmra2503232 · Cited by 209
Dell’Acqua, F., McFowland, E., Mollick, E., Lifshitz, H., Kellogg, K. C., Rajendran, S., Krayer, L., Candelon, F., & Lakhani, K. R. (2026) · Organization Science · doi:10.1287/orsc.2025.21838 · Cited by 1,189
“Prior work shows that AI systems tend to generate larger performance gains in ideation and content creation tasks than in judgment or decision-making tasks (e.g., Vaccaro et al. 2024).”
Complementarity in human-AI collaboration: concept, sources, and evidence
Hemmer, P., Schemmer, M., Kühl, N., Vössing, M., & Satzger, G. (2025) · European Journal of Information Systems · doi:10.1080/0960085x.2025.2475962 · Cited by 185
“…illustrates this (Vaccaro et al., 2024): While applications like ChatGPT often provide helpful, but not always correct results, a human decision-maker can collaborate with the system to, for example, override erroneous responses in order to achieve superior task performance (Vaccaro et al., 2024).”
Influential
Metadata
- Authors
- Vaccaro, M., Almaatouq, A., & Malone, T.
- Affiliations
- Massachusetts Institute of Technology (US)
- Venue
- Nature Human Behaviour
- Published
- 2024-10-28
- Volume
- 8
- Issue
- 12
- Pages
- 2293-2303
- ISSN
- 2397-3374 (ISSN-L)
- PMC
- PMC11659167
- OpenAlex
- W4403839497
- Semantic Scholar
- 02aec976011a7d595d61db9639209968f4b93177
- Cited by
- 570
- References
- 95
- Citations by year
- 2026: 404 · 2025: 156 · 2024: 9
- FWCI
- 51.29
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Locations
- Nature Human Behaviour (publishedVersion, cc-by)
- Research Square (acceptedVersion, cc-by)
- PubMed (publishedVersion)
- arXiv (Cornell University) (submittedVersion, cc-by)
- PubMed Central (submittedVersion, cc-by)
- RePEc: Research Papers in Economics (submittedVersion)
- arXiv (Cornell University) (cc-by)
- Funders
- Accenture Technology Convergence Fellowship
- Toyota Research Institute; MIT Quest for Intelligence; National Research Foundation, Prime Minister's Office, Singapore
- Topics
- Social Sciences › Human Factors and Ergonomics › Human-Automation Interaction and Safety
- Social Sciences › Psychology › Social Psychology › Social Robot Interaction and HRI
- Physical Sciences › Computer Science › Artificial Intelligence › Artificial Intelligence Applications
- Keywords
- human-AI collaboration, human-AI comparison, decision-making, content creation, performance outcomes, AI augmentation, task type effects, human-AI systems
- MeSH terms
- Artificial Intelligence, Decision Making, Humans
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-10-20, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-20 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
