The DOI matches its Crossref record; 96 of the 96 references it cites were found in a registry or an index; 1 of the 11 sources its claims cite were checked for support.
When combinations of humans and AI are useful: A systematic review and meta-analysis
Open accessPublished version of a preprint
Abstract
Inspired by the increasing use of artificial intelligence (AI) to augment humans, researchers have studied human–AI systems involving different tasks, systems and populations. Despite such a large body of work, we lack a broad conceptual understanding of when combinations of humans and AI are better than either alone. Here we addressed this question by conducting a preregistered systematic review and meta-analysis of 106 experimental studies reporting 370 effect sizes. We searched an interdisciplinary set of databases (the Association for Computing Machinery Digital Library, the Web of Science and the Association for Information Systems eLibrary) for studies published between 1 January 2020 and 30 June 2023. Each study was required to include an original human-participants experiment that evaluated the performance of humans alone, AI alone and human–AI combinations. First, we found that, on average, human–AI combinations performed significantly worse than the best of humans or AI alone (Hedges’ g = −0.23; 95% confidence interval, −0.39 to −0.07). Second, we found performance losses in tasks that involved making decisions and significantly greater gains in tasks that involved creating content. Finally, when humans outperformed AI alone, we found performance gains in the combination, but when AI outperformed humans alone, we found losses. Limitations of the evidence assessed here include possible publication bias and variations in the study designs analysed. Overall, these findings highlight the heterogeneity of the effects of human–AI collaboration and point to promising avenues for improving human–AI systems.
Overview
AI snapshot
Written from the paper’s full text.
TL;DR
This preregistered systematic review and meta-analysis of 106 experiments (370 effect sizes) finds that, on average, human–AI combinations perform worse than the best of humans or AI alone, though they do outperform humans alone. Effects vary by task type and by whether humans or AI perform better alone.
Key findings
- Human–AI combinations showed negative overall synergy versus the better of human alone or AI alone: Hedges’ g = −0.23 (95% CI, −0.39 to −0.07; P = 0.005).
- Human–AI combinations did show human augmentation versus humans alone: g = 0.64 (95% CI, 0.53 to 0.74; P = 0.000).
- Task type moderated synergy: decision tasks had losses (g = −0.27; 95% CI, −0.44 to −0.10; P = 0.002), while creation tasks had positive but non-significant gains (g = 0.19; 95% CI, −0.09 to 0.48; P = 0.180), with a significant difference between task types.
- Relative human/AI performance moderated synergy: when humans alone outperformed AI alone, synergy was positive (g = 0.46; 95% CI, 0.28 to 0.66); when AI alone outperformed humans alone, synergy was negative (g = −0.54; 95% CI, −0.71 to −0.37). Explanation, confidence, participant type, and division of labour were not statistically significant moderators.
Methods
- Preregistered systematic review and meta-analysis following Kitchenham and PRISMA guidelines; searched ACM Digital Library, Web of Science, and AIS eLibrary for studies published 1 January 2020–30 June 2023.
- From 5,126 initial papers, 74 met inclusion criteria, yielding 106 experiments and 370 unique effect sizes; included studies had to report performance for human alone, AI alone, and human–AI combination.
- Used Hedges’ g and a three-level random-effects meta-analytic model with robust variance estimation, Knapp–Hartung adjustment, and separate meta-regressions for moderators; sensitivity analyses clustered at paper level, excluded outliers, and used leave-one-out checks.
Limitations
- The paper states that results apply only to studies reporting all three conditions, so tasks that neither humans nor AI can perform alone are excluded; samples and outcome measures varied across studies, limiting comparability, and measurement error was not fully modelled.
- The paper states possible publication bias remains possible, though tests did not indicate it for synergy; tests did indicate potential publication bias for human augmentation. It also notes possible research topic selection bias, variation in study rigour, and high unexplained heterogeneity.
- Evident from the text: most effect sizes concerned decision tasks (~85%) and most human–AI systems had humans making final decisions after AI input (>95%), so findings may not generalize to other human–AI configurations.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures
Verification
Claims the paper makes
36 Own result · 14 Own interpretation · 7 Own method · 6 No full text of the sources · 6 No basis given · 1 Supported so far · 1 Sources not found · 1 Not checked yet
The sources with a verdict support this claim; the rest weren’t checked yet, have no full text we could read, or their verdicts aren’t rated yet.
In the paper
“We thus adopted a three-level meta-analytic model in which effect sizes are nested within experiments, so we explicitly took the dependencies in our data into account in the model63,64.”
§ When combinations of humans and AI are useful: A systematic review and meta-analysis > Methods > Data analysis
What the sources say
- Supportedcites [63] Van den Noortgate, W., López-López, J. A., Marín-Martínez, F. & Sánchez-Meca, J. Meta-analysis of multiple outcomes: a multilevel approach. Behav. Res. Methods 47, 1274–1294 (2015). source
The Abstract describes a three-level meta-analytic model for dependence among multiple effect sizes from the same study. The Conclusions likewise state that the approach accounts for dependence between effect sizes within studies. The citing authors’ adoption of the model is their own methodological choice, not a finding attributed to this source. (LLM, conf=0.92)
- Not checked — no full text of the sourcecites [64] Cheung, M. W.-L. Modeling dependent effect sizes with three-level meta-analyses: a structural equation modeling approach. Psychol. Methods 19, 211–229 (2014). source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We also found that the type of AI involved in the experiment (F2,103 = 3.77, two-tailed P = 0.026) and the year of publication (F3,102 = 3.65, two-tailed P = 0.015) moderated human–AI synergy, and the experimental design moderated human augmentation (F1,104 = 4.90, two-tailed P = 0.029).”
§ When combinations of humans and AI are useful: A systematic review and meta-analysis > Results > Heterogeneity of human–AI synergy
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Even though the average performance gains for creation tasks were not significantly different from 0 (presumably because of the relatively small sample size of n = 34), the difference between losses for decision tasks and gains for creation tasks was statistically significant.”
§ When combinations of humans and AI are useful: A systematic review and meta-analysis > Results > Heterogeneity of human–AI synergy
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Only 3 of the 100+ experiments in our analysis explore such processes with a predetermined delegation of separate subtasks to humans and AI.”
§ When combinations of humans and AI are useful: A systematic review and meta-analysis > Discussion > Moderating effect of task type
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We also found that the type of AI involved in the experiment (F2,103 = 3.77, two-tailed P = 0.026) and the year of publication (F3,102 = 3.65, two-tailed P = 0.015) moderated human–AI synergy, and the experimental design moderated human augmentation (F1,104 = 4.90, two-tailed P = 0.029).”
§ When combinations of humans and AI are useful: A systematic review and meta-analysis > Results > Heterogeneity of human–AI synergy
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“First, we found that the type of task significantly moderated human–AI synergy (F1,104 = 7.84, two-tailed P = 0.006).”
§ When combinations of humans and AI are useful: A systematic review and meta-analysis > Results > Heterogeneity of human–AI synergy
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“However, when we compared the performance of the human–AI systems to a different baseline—the humans alone—we found substantial evidence of human augmentation. The human–AI systems performed significantly better than humans alone, and this pooled effect size was positive (g = 0.64; t98 = 11.87; two-tailed P = 0.000; 95% CI, 0.53 to 0.74) and medium to large21.”
- Also stated in: § Main · plain
“On average, we found evidence of human augmentation, meaning that the average human–AI systems performed better than the human alone.”
§ When combinations of humans and AI are useful: A systematic review and meta-analysis > Results > Overall levels of human–AI synergy
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § Main · plain
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Among decision tasks—those in which participants decided between a finite set of options—the pooled effect size for human–AI synergy was significantly negative (g = −0.27; t104 = −3.20; two-tailed P = 0.002; 95% CI, −0.44 to −0.10), which indicates performance losses from combining humans and AI.”
§ When combinations of humans and AI are useful: A systematic review and meta-analysis > Results > Heterogeneity of human–AI synergy
hedgeddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“But when the AI alone outperformed the human alone, performance losses occurred in the combined system relative to the AI alone, with a negative effect size for human–AI synergy of g = −0.54 (t104 = −6.20; two-tailed P = 0.000; 95% CI, −0.71 to −0.37), a medium-sized effect21.”
§ When combinations of humans and AI are useful: A systematic review and meta-analysis > Results > Heterogeneity of human–AI synergy
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Here we addressed this question by conducting a preregistered systematic review and meta-analysis of 106 experimental studies reporting 370 effect sizes.”
- Also stated in: § Results · plain
“These papers reported the results of 106 unique experiments, and many of the experiments had multiple conditions, so we collected a total of 370 unique effect sizes measuring the impact of human–AI collaboration on task performance.”
§ When combinations of humans and AI are useful: A systematic review and meta-analysis > Abstract
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § Results · plain
- It may state two separate claims as one. 23
- Probably not a substantial claim. 16
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We also found evidence for substantial heterogeneity of effect sizes in our estimations of human–AI synergy (I2 = 97.7%) and human augmentation (I2 = 93.8%) (see Supplementary Tables 5 and 6 for more details).”
§ When combinations of humans and AI are useful: A systematic review and meta-analysis > Results > Heterogeneity of human–AI synergy
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“With the four effect sizes from these 3 experiments, we found that, on average, human–AI synergy (g = 0.22, t104 = 0.69; two-tailed P = 0.494; 95% CI, −0.42 to 0.87) occurred, but the result was not statistically significant (see Supplementary Information section 2.6 for a more detailed discussion of these experiments).”
§ When combinations of humans and AI are useful: A systematic review and meta-analysis > Discussion > Moderating effect of task type
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The remaining moderators we investigated were not statistically significant for human–AI synergy or human augmentation (explanation, confidence, participant type and division of labour).”
§ When combinations of humans and AI are useful: A systematic review and meta-analysis > Results > Heterogeneity of human–AI synergy
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Second, we found that the performance of the human and AI relative to each other impacted both human–AI synergy (F1,104 = 81.79, two-tailed P = 0.000) and human augmentation (F1,104 = 24.35, two-tailed P = 0.000).”
§ When combinations of humans and AI are useful: A systematic review and meta-analysis > Results > Heterogeneity of human–AI synergy
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Relatedly, we found that the type of data involved in the task significantly moderated both human–AI synergy (F4,101 = 15.24, two-tailed P = 0.000) and human augmentation (F4,101 = 6.52, two-tailed P = 0.000).”
§ When combinations of humans and AI are useful: A systematic review and meta-analysis > Results > Heterogeneity of human–AI synergy
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“As shown in Fig. 2, when the human alone outperformed the AI alone, the combined human–AI system outperformed both alone with an average pooled effect size for human–AI synergy of g = 0.46 (t104 = 5.06; two-tailed P = 0.000; 95% CI, 0.28 to 0.66), a medium-sized effect21.”
§ When combinations of humans and AI are useful: A systematic review and meta-analysis > Results > Heterogeneity of human–AI synergy
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“When the AI outperformed the human alone, greater performance gains tended to occur in the human–AI systems relative to the human alone, and the pooled effect size for human augmentation was positive and medium to large in magnitude (g = 0.74; t104 = 13.50; two-tailed P = 0.000; 95% CI, 0.63 to 0.85) (see Supplementary Figs. 4–7 for a more detailed visualization of this result for decision tasks).”
§ When combinations of humans and AI are useful: A systematic review and meta-analysis > Results > Heterogeneity of human–AI synergy
hedgedcorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“The measurement error can also vary across experiments.”
§ When combinations of humans and AI are useful: A systematic review and meta-analysis > Discussion > Limitations
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In contrast, among creation tasks—those in which participants created some sort of open-response content—the pooled effect size for human–AI synergy was positive (g = 0.19; t104 = 1.35; two-tailed P = 0.180; 95% CI, −0.09 to 0.48), pointing to synergy between humans and AI.”
§ When combinations of humans and AI are useful: A systematic review and meta-analysis > Results > Heterogeneity of human–AI synergy
hedgedn = 34- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“We find this assumption plausible because the experiments recruited different sets of participants and entailed different tasks or interventions.”
§ When combinations of humans and AI are useful: A systematic review and meta-analysis > Methods > Sensitivity analysis
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds. It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Supplementary Fig. 9 displays the funnel plot of the included effect sizes, and we did not observe significant asymmetry or regions of missing data in the plot for our primary outcome, human–AI synergy.”
§ When combinations of humans and AI are useful: A systematic review and meta-analysis > Methods > Bias tests
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Many of the experiments in our analysis evaluate performance according to a single measure of overall accuracy, but this measure corresponds to different things depending on the situation, and it omits other important criteria for human–AI systems.”
§ When combinations of humans and AI are useful: A systematic review and meta-analysis > Discussion > A roadmap for future work: finding human–AI synergy
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“For example, studies used different attention check mechanisms and performance incentive structures, which can both affect the quality of responses and thus introduce another source of noise into our data.”
§ When combinations of humans and AI are useful: A systematic review and meta-analysis > Discussion > Limitations
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“For example, researchers mostly used their own experimental platforms and stimuli, which naturally introduce sources of variation between their studies.”
§ When combinations of humans and AI are useful: A systematic review and meta-analysis > Discussion > Limitations
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Using this approach, we identified 11 outliers for human–AI synergy and 9 outliers for human augmentation.”
§ When combinations of humans and AI are useful: A systematic review and meta-analysis > Methods > Sensitivity analysis
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Our initial literature search yielded 5,126 papers, and, per the review process described in the Methods (see ‘Literature review’), we identified 74 that met our inclusion criteria (Supplementary Fig. 1).”
§ When combinations of humans and AI are useful: A systematic review and meta-analysis > Results
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“We searched an interdisciplinary set of databases (the Association for Computing Machinery Digital Library, the Web of Science and the Association for Information Systems eLibrary) for studies published between 1 January 2020 and 30 June 2023.”
§ When combinations of humans and AI are useful: A systematic review and meta-analysis > Abstract
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“Each study was required to include an original human-participants experiment that evaluated the performance of humans alone, AI alone and human–AI combinations.”
§ When combinations of humans and AI are useful: A systematic review and meta-analysis > Abstract
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“Here we addressed this question by conducting a preregistered systematic review and meta-analysis of 106 experimental studies reporting 370 effect sizes.”
§ When combinations of humans and AI are useful: A systematic review and meta-analysis > Abstract
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
None of the sources cited for this claim have been checked yet.
In the paper
“We conducted this meta-analysis in accord with the guidelines from Kitchenham55 on systematic reviews, and we followed the standards set forth by the Preferred Reporting Items for Systematic Reviews and Meta-Analyses56.”
§ When combinations of humans and AI are useful: A systematic review and meta-analysis > Methods
What the sources say
- Not matchedcites Kitchenham55
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not checked — no full text of the sourcecites [56] Moher, D., Liberati, A., Tetzlaff, J., Altman, D. G. & PRISMA Group Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. Ann. Intern. Med. 151, 264–269 (2009). source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
- Not matched
In the paper
“We chose Hedges’ g as our measure of effect size because it is unitless, so it allows us to compare human–AI performance across different metrics, and it corrects for upward bias in the raw standardized mean difference (Cohen’s d)61.”
§ When combinations of humans and AI are useful: A systematic review and meta-analysis > Methods > Data analysis
What the sources say
- Not checked — no full text of the sourcecites [61] Hedges, L. V. Distribution theory for Glass’s estimator of effect size and related estimators. J. Educ. Stat. 6, 107–128 (1981). source
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“We synthesized these data in a three-level meta-analytic model (see ‘Data analysis’ in Methods).”
§ When combinations of humans and AI are useful: A systematic review and meta-analysis > Results
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Third, although we did not find evidence of publication biases, it remains possible that they exist, which would impact our literature base and, by extension, our meta-analytic results.”
§ When combinations of humans and AI are useful: A systematic review and meta-analysis > Discussion > Limitations
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Fourth, our results only apply to the tasks, processes and participant pools that researchers have chosen to study, and these configurations may not be representative of the ways human–AI systems are configured in practical uses of AI outside the laboratory.”
§ When combinations of humans and AI are useful: A systematic review and meta-analysis > Discussion > Limitations
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“However, we expect that if there were a publication bias operating here, it would be a bias to publish studies that showed significant gains from combining humans and AI. And since our overall results showed the opposite, it seems unlikely that they are a result of publication bias.”
§ When combinations of humans and AI are useful: A systematic review and meta-analysis > Discussion > Limitations
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
In the paper
“As is typical for meta-analyses43, in our three-level model, we weighted effect sizes as a function of their variance across participants, so we did not account for this other source of variation in measurement.”
§ When combinations of humans and AI are useful: A systematic review and meta-analysis > Discussion > Limitations
What the sources say
- Not checked — no full text of the sourcecites [43] Borenstein, M., Hedges, L. V., Higgins, J. P. & Rothstein, H. R. Introduction to Meta-analysis (John Wiley & Sons, 2021). source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In other words, the human–AI systems we analysed were, on average, better than humans alone but not better than both humans alone and AI alone.”
- Also stated in: § Main · plain
“On average, we found evidence of human augmentation, meaning that the average human–AI systems performed better than the human alone. But we did not find human–AI synergy on average, meaning that the average human–AI systems performed worse than at least one of the human alone or the AI alone.”
§ When combinations of humans and AI are useful: A systematic review and meta-analysis > Results > Overall levels of human–AI synergy
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Also stated in: § Main · plain
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“We have made the materials required to reproduce our results publicly accessible through an Open Science Framework repository.”
§ When combinations of humans and AI are useful: A systematic review and meta-analysis > Results
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Fifth, the quality of our analysis depends on the quality of the studies we synthesized.”
§ When combinations of humans and AI are useful: A systematic review and meta-analysis > Discussion > Limitations
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The moderators we investigated account for some of this heterogeneity, but much remains unexplained.”
§ When combinations of humans and AI are useful: A systematic review and meta-analysis > Discussion > Limitations
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The discrepancy between the symmetry in the funnel plot for human–AI synergy versus asymmetry in the funnel plot for human augmentation may reflect how many researchers and journals implicitly assume an interest in human augmentation, comparing the human–AI system to the human alone.”
§ When combinations of humans and AI are useful: A systematic review and meta-analysis > Methods > Bias tests
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it.
In the paper
“Additionally, as discussed in ref. 33, human–AI synergy requires that humans be better at some parts of a task, AI be better at other parts of the task and the system as a whole be good at appropriately allocating subtasks to whichever partner is best for that subtask.”
§ When combinations of humans and AI are useful: A systematic review and meta-analysis > Discussion > A roadmap for future work: finding human–AI synergy
What the sources say
- Not checked — no full text of the sourcecites [33] Donahue, K., Chouldechova, A. & Kenthapadi, K. Human–algorithm collaboration: achieving complementarity and avoiding unfairness. In Proc. 2022 ACM Conference on Fairness, Accountability, and Transparency 1639–1656 (Association for Computing Machinery, 2022); https://doi.org/10.1145/3531146.3533221 source
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“To further ensure comprehensive coverage, we also conducted a forward and backward search on all studies we found that met our inclusion criteria.”
§ When combinations of humans and AI are useful: A systematic review and meta-analysis > Methods > Literature review
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
The paper relies on this without support.
There is a lack of broad conceptual understanding of when combinations of humans and AI are better than either alone.
“Despite such a large body of work, we lack a broad conceptual understanding of when combinations of humans and AI are better than either alone.”
The authors state this as an assumption.
The experiments in the analysis are independent from each other even if they come from the same paper.
“It does, however, consider the experiments in our analysis as independent from each other, even if they come from the same paper. We find this assumption plausible because the experiments recruited different sets of participants and entailed different tasks or interventions.”
The paper relies on this without support.
The participant samples overlapped only on the level of the experiment.
“because the participant samples overlapped only on the level of the experiment.”
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“While Gen-AI as an MKO can improve learning process, whereas over-reliance on these tools leads to logical complacency and selective cognitive atrophy (Vaccaro, , Almaatouq, & Malone, 2024).”
From The Double-Edged Sword of Automation: Impact of Generative AI Adoption on Student's Critical Thinking Skills in Higher Education Considering Cognitive Offloading as a Mediator (2026) · cites this paper as Vaccaro, , Almaatouq, & Malone, 2024
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Vaccaro et al. (2024) showed that human-AI combinations often outperform either alone in creative tasks.”
From Metadiscourse Patterns in Human-Written vs. Generative AI-Authored Research Abstracts: A Comparative Corpus-Based Analysis (2026) · cites this paper as Vaccaro et al. (2024)
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“To support the introduction of AI in clinical practice in intensive care, we propose starting with the concept of human-AI augmentation, which is more inclusive and better established according to medical literature [34].”
From Implementing Artificial Intelligence in Critical Care Medicine: a consensus of 22. (2025) · cites this paper as [34]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Recent meta-analyses of human-AI collaboration show that combinations sometimes underperform either party in isolation (Vaccaro et al., 2024)”
From Indie at Scale: How Large Language Models Make Community-Centered Game Development Economically Viable (2026) · cites this paper as Vaccaro et al., 2024
The cited source says what the claim attributes to it.
In the citing paper
“This collaboration has been described in two forms: human-AI augmentation (when human–AI interface enhances clinical performance compared to human alone) and human-AI synergy (where the combined performance exceeds that of both the human and the AI individually) [34].”
From Implementing Artificial Intelligence in Critical Care Medicine: a consensus of 22. (2025) · cites this paper as [34]
The cited source says what the claim attributes to it.
In the citing paper
“Vaccaro et al .'s meta-analysis of 106 experimental studies found that human-AI combinations performed worse than the best individual decision-maker on average for decision tasks 34 .”
From AI-induced never-skilling in medical education (2026) · cites this paper as [34]
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“Such tasks may have been part of the training data in GenAI models, which is a known issue in GenAI research [61]”
From Generative AI and Creativity: A Systematic Literature Review and Meta-Analysis (2025) · cites this paper as [61]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated major: it would touch a result — a value a result rests on, reported and recomputed p-values on opposite sides of the significance level, two passages that conflict with no sign which is right, or a cited work retracted before this paper was published.
In the paper
§ When combinations of humans and AI are useful: A systematic review and meta-analysis > Discussion > Surprisingly insignificant moderators
“the significant and less researched moderators we identified: the baseline performance of the human and AI alone, the type of task they perform, and the division of labour between them.”
§ When combinations of humans and AI are useful: A systematic review and meta-analysis > Results > Heterogeneity of human–AI synergy
“The remaining moderators we investigated were not statistically significant for human–AI synergy or human augmentation (explanation, confidence, participant type and division of labour).”
Citations
Cites
Bias in meta-analysis detected by a simple, graphical test
Egger, M., Smith, G. D., Schneider, M., & Minder, C. (1997) · BMJ · doi:10.1136/bmj.315.7109.629 · Cited by 56,722 · cited 1×
“We visually inspected the plots and performed Egger’s regression test 70 as well as the rank correlation test 71 to evaluate the results in the funnel plots.”
Preferred reporting items for systematic reviews and meta-analyses: the PRISMA Statement.
(2009) · PubMed Central · Cited by 44,859
Preferred reporting items for systematic reviews and meta-analyses: The PRISMA statement
Moher, D., Liberati, A., Tetzlaff, J., Altman, D. G., & the PRISMA Group* (2009) · Annals of Internal Medicine · doi:10.7326/0003-4819-151-4-200908180-00135 · Cited by 37,554
Quantifying heterogeneity in a meta‐analysis
Higgins, J. P. T., & Thompson, S. G. (2002) · Statistics in Medicine · doi:10.1002/sim.1186 · Cited by 36,763
Statistical Power Analysis for the Behavioral Sciences
Cohen, J. (2013) · doi:10.4324/9780203771587 · Cited by 25,373
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
Conducting Meta-Analyses in R with the metafor Package
Viechtbauer, W. (2010) · Journal of Statistical Software · doi:10.18637/jss.v036.i03 · Cited by 20,464
Operating characteristics of a rank correlation test for publication bias
Begg, C. B., & Mazumdar, M. (1994) · Biometrics · doi:10.2307/2533446 · Cited by 16,955 · cited 1×
“We visually inspected the plots and performed Egger’s regression test 70 as well as the rank correlation test 71 to evaluate the results in the funnel plots.”
Mastering the game of Go with deep neural networks and tree search
Silver, D., Huang, A., Maddison, C. J., Guez, A., Sifre, L., van den Driessche, G., Schrittwieser, J., Antonoglou, I., Panneershelvam, V., Lanctot, M., et al. (2016) · Nature · doi:10.1038/nature16961 · Cited by 15,612
Borenstein, M., Hedges, L. V., Higgins, J. P. T., & Rothstein, H. R. (2021) · doi:10.1002/9781119558378 · Cited by 9,799
Research works printed but not linked: 2
- Wilson, H. J. & Daugherty, P. R. Collaborative intelligence: humans and AI are joining forces. Harv. Bus. Rev. 96, 114–123 (2018).
- Hedges, L. V. & Olkin, I. Statistical Methods for Meta-analysis (Academic Press, 2014).
Cited by
Sorted by citations
Brynjolfsson, E., Li, D., & Raymond, L. (2025) · The Quarterly Journal of Economics · doi:10.1093/qje/qjae044 · Cited by 1,083
Dell'Acqua, F., McFowland III, E., Mollick, E. R., Lifshitz-Assaf, H., Kellogg, K., Rajendran, S., Krayer, L., Candelon, F., & Lakhani, K. R. (2023) · SSRN Electronic Journal · doi:10.2139/ssrn.4573321 · Cited by 794
Educational Strategies for Clinical Supervision of Artificial Intelligence Use
Abdulnour, R.-E. E., Gin, B., & Boscardin, C. K. (2025) · New England Journal of Medicine · doi:10.1056/nejmra2503232 · Cited by 209
Dell’Acqua, F., McFowland, E., Mollick, E., Lifshitz, H., Kellogg, K. C., Rajendran, S., Krayer, L., Candelon, F., & Lakhani, K. R. (2026) · Organization Science · doi:10.1287/orsc.2025.21838 · Cited by 1,189
“Prior work shows that AI systems tend to generate larger performance gains in ideation and content creation tasks than in judgment or decision-making tasks (e.g., Vaccaro et al. 2024).”
Complementarity in human-AI collaboration: concept, sources, and evidence
Hemmer, P., Schemmer, M., Kühl, N., Vössing, M., & Satzger, G. (2025) · European Journal of Information Systems · doi:10.1080/0960085x.2025.2475962 · Cited by 185
“…illustrates this (Vaccaro et al., 2024): While applications like ChatGPT often provide helpful, but not always correct results, a human decision-maker can collaborate with the system to, for example, override erroneous responses in order to achieve superior task performance (Vaccaro et al., 2024).”
InfluentialGenerative AI in Marketing and Principles for Ethical Design and Deployment
Hermann, E., & Puntoni, S. (2024) · Journal of Public Policy & Marketing · doi:10.1177/07439156241309874 · Cited by 121
Implementing Artificial Intelligence in Critical Care Medicine: a consensus of 22.
Cecconi, M., Greco, M., Shickel, B., Angus, D. C., Bailey, H., Bignami, E., Calandra, T., Celi, L. A., Einav, S., Elbers, P., et al. (2025) · Critical Care · doi:10.1186/s13054-025-05532-2 · Cited by 80
“To support the introduction of AI in clinical practice in intensive care, we propose starting with the concept of human-AI augmentation, which is more inclusive and better established according to medical literature [34].”
AI makes you smarter but none the wiser: The disconnect between performance and metacognition
Fernandes, D., Villa, S., Nicholls, S., Haavisto, O., Buschek, D., Schmidt, A., Kosch, T., Shen, C., & Welsch, R. (2025) · Computers in Human Behavior · doi:10.1016/j.chb.2025.108779 · Cited by 144
Heinrichs, H., Kies, A., Nagel, S. K., & Kiessling, F. (2025) · Journal of Medical Internet Research · doi:10.2196/74187 · Cited by 66
“However, a recent study indicates that effective collaboration between humans and AI remains elusive, especially in decision-making beyond image and text analysis [51].”
Divergent creativity in humans and large language models
Bellemare-Pepin, A., Lespinasse, F., Thölke, P., Harel, Y., Mathewson, K. W., Olson, J. A., Bengio, Y., & Jerbi, K. (2026) · Scientific Reports · doi:10.1038/s41598-025-25157-3 · Cited by 121
“More broadly, a recent meta-analysis of 106 studies revealed that while human–AI collaboration often underperforms the best of either alone, it shows notable advantages in content creation tasks, highlighting the context-dependent nature of such synergies 70 .”
Karunakaran, A., Lebovitz, S., Narayanan, D., & Rahman, H. A. (2025) · Academy of Management Annals · doi:10.5465/annals.2023.0230 · Cited by 49
A Current Review of Generative AI in Medicine: Core Concepts, Applications, and Current Limitations
Rouzrokh, P., Khosravi, B., Faghani, S., Moassefi, M., Shariatnia, M. M., Rouzrokh, P., & Erickson, B. J. (2025) · Current Reviews in Musculoskeletal Medicine · doi:10.1007/s12178-025-09961-y · Cited by 58
“However, evidence also indicates that incorporating LLMs into clinical workflows may increase clinicians’ workload, or that collaboration between LLMs and clinicians may result in worse perfor - mance than when each acts independently [ 56–59 ].”
Examining human-AI interaction in real-world healthcare beyond the laboratory
Wekenborg, M., Gilbert, S., & Kather, J. N. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-01559-5 · Cited by 54
“There are ongoing debates about how to evaluate speci fi c aspects of task performance, such as whether to focus on human augmentation (where the human-AI system outperforms humans alone) or human AI synergy (where the system performs worse than at least one of its components, either the human or the AI alone) as point…”
InfluentialA large language model improves clinicians’ diagnostic performance in complex critical illness cases
Wu, X. C., Huang, Y., & He, Q. (2025) · Critical Care · doi:10.1186/s13054-025-05468-7 · Cited by 51
“A recent study highlighted the heterogeneity of the effects of human-AI collaboration, the performance of human-AI collaboration loss when AI outperformed humans alone, and in tasks that involved making decisions [29].”
Statistical Tests for Replacing Human Decision Makers with Algorithms
Feng, K., HONG, H. P., Tang, K., & Wang, J. (2025) · Management Science · doi:10.1287/mnsc.2023.01845 · Cited by 50
A cognitive approach to human–AI complementarity in dynamic decision-making
González, C., & Heidari, H. (2025) · Nature Reviews Psychology · doi:10.1038/s44159-025-00499-x · Cited by 73
AI-induced never-skilling in medical education
Ke, Y., Jin, L., Ong, J. C. L., Thirunavukarasu, A. J., Car, J., Cheung, C. Y., Tham, Y. C., Ting, D. S. W., Ong, M. E. H., Compton, S., et al. (2026) · Nature Medicine · doi:10.1038/s41591-026-04438-y · Cited by 71
“Evidence on collaborative performance is variable 34,35 , and AI augmentation can meaningfully extend clinical capability in well-defined tasks.”
System 0: Transforming Artificial Intelligence into a Cognitive Extension
Chiriatti, M., Ganapini, M. B., Panai, E., Wiederhold, B. K., & Riva, G. (2025) · Cyberpsychology Behavior and Social Networking · doi:10.1089/cyber.2025.0201 · Cited by 53
“9 This pattern underscores that complementarity — rather than substitution — is key to successful cognitive integration.”
InfluentialChen, S., Ebrahimi, O. V., & Cheng, C. (2025) · Journal of Medical Internet Research · doi:10.2196/70483 · Cited by 53
“Despite the benefits, caution should also be practiced in designing interventions that incorporate AI-driven tools, as the effectiveness of human-AI collaboration depends heavily on the relative strengths of each contributor [89].”
Recognising errors in AI implementation in radiology: A narrative review
Stogiannos, N., Cuocolo, R., D’Antonoli, T. A., dos Santos, D. P., Harvey, H., Huisman, M., Koçak, B., Kotter, E., Lekadir, K., Shelmerdine, S. C., et al. (2025) · European Journal of Radiology · doi:10.1016/j.ejrad.2025.112311 · Cited by 31
Human–large language model collaboration in clinical medicine: a systematic review and meta-analysis
Wang, G., Zhang, K., Jiang, J., Wang, C., Bi, H., Liang, H., Qi, Z., Huang, Y., Li, Y., & Yang, X. (2026) · npj Digital Medicine · doi:10.1038/s41746-026-02382-2 · Cited by 48
“…meta-analysis synthesizing 106 experimental studies (370 effect sizes) found that, on average, human – AI teams performed signi fi cantlyworsethanthebestsingle agent — eitherhumanorAI(Hedges ’ g = − 0.23; 95 % CI − 0.39 to − 0.07) — underscoring the variability of collaborative effectiveness 42 .”
InfluentialTowards autonomous medical artificial intelligence agents
Ferber, D., Hilgers, L., Höper, C., Kinny‐Köster, B., Eckardt, J., Egger‐Heidrich, K., Bill, M., Schneider, M. M. K., Clusmann, J., Kadric, L., et al. (2026) · Nature · doi:10.1038/s41586-026-10675-5 · Cited by 57
“Finally, even though our evaluation focused on the autonomous decision-making capabilities of MIRA, we emphasize that medical AI agents currently should not be designed to replace healthcare professionals 21 .”
InfluentialLarge language models for clinical decision support in gastroenterology and hepatology
Wiest, I. C., Bhat, M., Clusmann, J., Schneider, C. V., Jiang, X., & Kather, J. N. (2025) · Nature Reviews Gastroenterology & Hepatology · doi:10.1038/s41575-025-01108-1 · Cited by 34
“Early evidence from human–AI interaction studies suggests that they can reduce perceived complexity and improve clinician performance; however, assessments using structured tools such as the NASA task load index are scarce 84–86 .”
Technology Acceptance Model in Artificial Intelligence in Education: A Meta-Analysis
Xue, L., Mahat, J., & Ghazali, N. (2026) · SAGE Open · doi:10.1177/21582440251409441 · Cited by 61
“Compared with previous meta-analyses (e.g., Scherer et al., 2019; Vaccaro et al., 2024; Yu, 2023), this study adopted a more targeted query structure—refining TAM-related terms for conceptual precision while expanding AI-related descriptors to capture a broader range of educational applications.”
Xu, F., Medappa, P. K., Tunc, M. M., Vroegindeweij, M., & Fransoo, J. C. (2025) · arXiv · doi:10.48550/arxiv.2510.10165 · Cited by 27
Navigating AI Convergence in Human–Artificial Intelligence Teams: A Signaling Theory Approach
Smith, A., Wagoner, H. P. V., Keplinger, K., & Çelebi, C. (2025) · Journal of Organizational Behavior · doi:10.1002/job.2856 · Cited by 39
Zhao, P., He, G., & Guan, J. (2025) · Journal of Business Ethics · doi:10.1007/s10551-025-06060-3 · Cited by 39
“…employees’ AI usage—defined as employing AI to perform work tasks (Tang et al., 2022)—and has uniformly indicated that AI can significantly boost employees’ performance outputs, including productivity (Noy & Zhang, 2023), creativity (Jia et al., 2024), and task performance (Vaccaro et al., 2024).”
The barriers to uptake of artificial intelligence in hepatology and how to overcome them
Clusmann, J., Balaguer‐Montero, M., Bassegoda, O., Schneider, C. V., Seraphin, T. P., Paintsil, E. K., Luedde, T., Pérez-López, R., Caldéraro, J., Gilbert, S., et al. (2025) · Journal of Hepatology · doi:10.1016/j.jhep.2025.07.003 · Cited by 30
RetractedEnhancing Intuitive Decision-Making and Reliance Through Human–AI Collaboration: A Review
Xu, G., Murthy, S. V., & Jia, B. (2025) · Informatics · doi:10.3390/informatics12040135 · Cited by 29
“Performance Paradox in HAIC Empirical evidence reveals a concerning performance paradox: human–AI combinations often perform worse than the best solo performer (human or AI) but better than humans alone [18].”
InfluentialBeyond the gender data gap: co-creating equitable digital patient twins
Weinberger, N., Hery, D., Mahr, D., Adler, S. O., Stadlbauer, J., & Ahrens, T. D. (2025) · Frontiers in Digital Health · doi:10.3389/fdgth.2025.1584415 · Cited by 24
Morey, D. A., Rayo, M. F., & Woods, D. D. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-01784-y · Cited by 28
“However,boththeimpactsofAIaugmentation 5,14 andmethodsof evaluation 15,16 have been highly variable.”
The extended hollowed mind: why foundational knowledge is indispensable in the age of AI
Klein, C. R., & Klein, R. (2025) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2025.1719019 · Cited by 33
“The study found that while teams often suffered from “negative synergy”—performing worse than their best member—this trend was inverted when the human was the more knowledgeable partner, leading to significant performance gains (Vaccaro et al., 2024).”
Toward a science of human–AI teaming for decision making: a complementarity framework
Gonzalez, C., Donahue, K., Goldstein, D. G., Heidari, H., Jalali, M. S., Schelble, B., Singh, A., & Woolley, A. W. (2026) · PNAS Nexus · doi:10.1093/pnasnexus/pgag030 · Cited by 33
Human-AI teaming in healthcare: 1 + 1 > 2?
Liu, P., Zhang, J., Chen, S., & Chen, S. (2025) · npj Artificial Intelligence · doi:10.1038/s44387-025-00052-4 · Cited by 26
“Another recent review 21 found that human-AI collaboration outperformed humans alone in tasks involving decision making and content generation in broader settings.”
InfluentialA question of alignment – AI, GenAI and applied linguistics
Curry, N., McEnery, T., & Brookes, G. (2025) · Annual Review of Applied Linguistics · doi:10.1017/s0267190525000017 · Cited by 26
“This will be a challenge, given the growing body of work revealing that AI-conducted research does not necessarily outperform research conducted by humans (e.g., Vaccaro et al., 2024).”
How ensembling AI and public managers improves decision-making
Keppeler, F., Borchert, J., Pedersen, M. J., & Nielsen, V. L. (2025) · Journal of Public Administration Research and Theory · doi:10.1093/jopart/muaf009 · Cited by 24
AI-Based EMG Reporting: A Randomized Controlled Trial
Gorenshtein, A., Weisblat, Y., Khateb, M., Kenan, G., Tsirkin, I., Fayn, G., Geller, S., & Shelly, S. (2025) · Journal of Neurology · doi:10.1007/s00415-025-13261-3 · Cited by 23
Aligning generalization between humans and machines
Ilievski, F., Hammer, B., van Harmelen, F., Paaßen, B., Saralajew, S., Schmid, U., Biehl, M. L., Bolognesi, M., Dong, X., Gashteovski, K., et al. (2025) · Nature Machine Intelligence · doi:10.1038/s42256-025-01109-4 · Cited by 37
“Recent results show that, to date, the performance of human–AI teams lags behind that of the best AI model or the best human alone in many domains 14 .”
SciSciGPT: advancing human–AI collaboration in the science of science
Shao, E., Wang, Y., Qian, Y., Pan, Z., Liu, H., & Wang, D. (2025) · Nature Computational Science · doi:10.1038/s43588-025-00906-6 · Cited by 45
“Recent advances in large language models (LLMs) and artificial intelligence (AI) agents have opened new possibilities for advancing human–AI collaboration 23–25 , offering potential tools to navigate the complex and rapidly evolving research landscape.”
Cancer workforce—a global crisis: a Lancet Oncology Commission
Hricak, H., Ward, Z. J., Moraes, F. Y., Scott, A. M., Dako, F., Ngwa, W. F., Villarreal-Garza, C. M., Abdel-Wahab, M., Avery, S. M., Addai, B. W., et al. (2026) · The Lancet Oncology · doi:10.1016/s1470-2045(26)00065-3 · Cited by 24
LLMs outperform outsourced human coders on complex textual analysis
Bermejo, V. J., Gago, A., Gálvez, R. H., & Harari, N. (2025) · Scientific Reports · doi:10.1038/s41598-025-23798-y · Cited by 27
“However, the evidence on when human-AI teaming yields net benefits remains mixed 79 .”
Adams, I. T., McLean, K. D., & Alpert, G. P. (2024) · Criminology · doi:10.1111/1745-9125.70028 · Cited by 17
Generative AI and Creativity: A Systematic Literature Review and Meta-Analysis
Holzner, N., Maier, S., & Feuerriegel, S. (2025) · arXiv · doi:10.48550/arxiv.2505.17241 · Cited by 143
High-risk artificial intelligence
Sunyaev, A., Benlian, A., Pfeiffer, J., Jussupow, E., Thiebes, S., Maedche, A., & Gawlitza, J. (2025) · Business & Information Systems Engineering · doi:10.1007/s12599-025-00942-6 · Cited by 16
“This notion has been supported by a recent meta-analysis comparing over 100 experiments to systematically test performance differences across different configurations of humans and AI systems (Vaccaro et al. 2024).”
A scoping review on pediatric sepsis prediction technologies in healthcare
Tennant, R., Graham, J., Kern, J., Mercer, K., Ansermino, J. M., & Burns, C. M. (2024) · npj Digital Medicine · doi:10.1038/s41746-024-01361-9 · Cited by 17
“Instead, human-AI performance was better when the human knew when to trust their judgment or the prediction, and vice-versa 87 .”
Lee, E. H., Yin, Y., Jia, N., & Wakslak, C. J. (2026) · Scientific Reports · doi:10.1038/s41598-026-42312-6 · Cited by 24
Cloud Security Automation Through Symmetry: Threat Detection and Response
Pitkar, H. (2025) · Symmetry · doi:10.3390/sym17060859 · Cited by 29
The EPOCH of AI: Human-Machine Complementarities at WorkPreprint
Loaiza, I., & C, R. R. B. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.5028371 · Cited by 25
Blasco, A., & Charisi, V. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.5040921 · Cited by 22
Microfoundations of rationality in the age of AI: On emotions, bodies and intelligence
Stein, M., & Shollo, A. (2025) · Information and Organization · doi:10.1016/j.infoandorg.2025.100583 · Cited by 17
The Future of Work is Blended, Not Hybrid
Constantinides, M., Verma, H., Sadeghian, S., & Ali, A. E. (2025) · Annual Symposium on Human-Computer Interaction for Work (CHIWORK) · doi:10.1145/3729176.3729202 · Cited by 19
Enhanced metagenomic strategies for elucidating the complexities of gut microbiota: a review
Li, X., & Lu, H. (2025) · Frontiers in Microbiology · doi:10.3389/fmicb.2025.1626002 · Cited by 13
“As the field advances, AI-driven metagenomics will underpin precision probiotics and microbiome-editing therapies, translating microbial ecology into clinical innovation (Vaccaro et al., 2024; Huerta-Cepas et al., 2017).”
InfluentialSafeguarding worker psychosocial well-being in the age of AI: The critical role of decision control
Passalacqua, M., Pellerin, R., Magnani, F., Joblot, L., Rosin, F., Yahia, E., & Léger, P. (2025) · International Journal of Human-Computer Studies · doi:10.1016/j.ijhcs.2025.103649 · Cited by 17
A meta-analysis of the persuasive power of large language models
Hölbling, L., Maier, S., & Feuerriegel, S. (2025) · Scientific Reports · doi:10.1038/s41598-025-30783-y · Cited by 34
“…collaboration theory, the observed heterogeneity of model capabilities, communication format, and domain aligns with frameworks emphasizing that humans and LLMs possess complementary strengths, suggesting that effective persuasion may arise when these distinct abilities are combined [63, 64].”
Singh, A. P., & Szajnfarber, Z. (2025) · Systems Engineering · doi:10.1002/sys.70024 · Cited by 20
“For example, despite the many ways in which humans and AI could collaborate, the majority of human-AI system studies are focused on AI advice being presented to a human decision maker in different ways [13, 14].”
A large-scale comparison of divergent creativity in humans and large language models.
Wang, D., Huang, D.-F., Shen, H., & Uzzi, B. (2025) · Nature Human Behaviour · doi:10.1038/s41562-025-02331-1 · Cited by 29
Enabling people-centric climate action using human-in-the-loop artificial intelligence: a review
Debnath, R., Tkachenko, N., & Bhattacharyya, M. (2025) · Current Opinion in Behavioral Sciences · doi:10.1016/j.cobeha.2025.101482 · Cited by 13
Ahn, S. (2025) · Annals of Pediatric Endocrinology & Metabolism · doi:10.6065/apem.2448278.139 · Cited by 13
“A meta-analysis of 370 human-AI collaboration experiments revealed a complex interaction pattern in which human-AI combinations generally augmented individual human capabilities but often failed to achieve synergistic benefits over AI systems alone, particularly in decision-making tasks [10].”
Designer-consumer similarity matters: The effect of AI-designed products on purchase intention
Yang, Z., & Tian, A. D. (2025) · Journal of Retailing and Consumer Services · doi:10.1016/j.jretconser.2025.104680 · Cited by 11
Convergence of machine learning and genomics for precision oncology
Reardon, B. M., Culhane, A. C., & Allen, E. M. V. (2026) · Nature reviews. Cancer · doi:10.1038/s41568-025-00897-6 · Cited by 14
“A recent meta-analysis of over 100 experiments of human and AI performance across a variety of fields and tasks has observed that the combination of AI and humans resulted in performance losses at tasks wherein AI outperform humans 135 .”
Advancing AI negotiations: A large-scale autonomous negotiation competition
Vaccaro, M. A., Caosun, M., Ju, H., Aral, S., & Curhan, J. R. (2026) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2521774123 · Cited by 14
Self-citationAI, Jobs, and the Automation Trap: Where Is HCI?
Constantinides, M., & Quercia, D. (2025) · Annual Symposium on Human-Computer Interaction for Work (CHIWORK) · doi:10.1145/3729176.3729191 · Cited by 16
Reinforcement Learning for Human-AI Collaboration: Challenges, Mechanisms, and Methods
Li, W., Liu, H., Huang, K., & Hussain, A. (2025) · Cognitive Computation · doi:10.1007/s12559-025-10500-7 · Cited by 11
Has the creativity of large-language models peaked?
Haase, J., Hanel, P. H. P., & Pokutta, S. (2025) · Journal of Creativity · doi:10.1016/j.yjoc.2025.100113 · Cited by 25
Spiegler, S. V., Hoda, R., & Pant, A. (2025) · Technology in Society · doi:10.1016/j.techsoc.2025.103109 · Cited by 12
Human–AI interaction research needs to be embedded in psychological theory
Bigman, Y., Briker, R., & Langer, M. (2026) · Nature Reviews Psychology · doi:10.1038/s44159-026-00551-4 · Cited by 10
Liang, Z., Yang, K., Sha, L., Gašević, D., Yan, L., & Chen, G. (2026) · British Journal of Educational Technology · doi:10.1111/bjet.70055 · Cited by 22
Chatbots in education: Outperforming students but perceived as less trustworthy
Laun, M., Puderbach, L., Hirt, K., Wyss, E., Friemert, D., Hartmann, U., & Wolff, F. (2025) · Contemporary Educational Psychology · doi:10.1016/j.cedpsych.2025.102373 · Cited by 11
Replace, augment, disrupt: AI & organizational decision-making
Page, S. E., & Kallapur, A. (2025) · Journal of Organization Design · doi:10.1007/s41469-025-00194-4 · Cited by 16
“As a result, even a naive averaging of the two will often result in a better prediction or decision than either alone (Vaccaro et al. 2024), though not necessarily (Hong and Page 2024).”
Recalibrating academic expertise in the age of generative AI
Lin, Z., & Sohail, A. (2026) · Patterns · doi:10.1016/j.patter.2025.101473 · Cited by 17
Kornowicz, J., & Thommes, K. (2025) · PLoS ONE · doi:10.1371/journal.pone.0318874 · Cited by 14
“As a result, the overall accuracy rate in advice-receiving situations amounted to 47.47% ( Cardio : 49.96%, Football : 45.57%).”
InfluentialAI–human collaboration in services: an integrative framework to uncover the key success factors
Jeong, E., & Lee, D. (2025) · Service Business · doi:10.1007/s11628-025-00591-5 · Cited by 8
Efstathiou, A., Charitaki, E., Triantopoulou, C., & Delis, S. (2025) · Journal of Clinical Medicine · doi:10.3390/jcm14186501 · Cited by 9
“A consistent finding, however, is that combining human expertise with AI can be synergistic—e.g., an AI model might quickly flag high-risk patients, which the surgical team can then further evaluate and proactively manage (such as closer postoperative monitoring or preemptive interventions) [43,44].”
Memmert, L., Cvetković, I., Tavanapour, N., & Bittner, E. A. C. (2025) · Business & Information Systems Engineering · doi:10.1007/s12599-025-00974-y · Cited by 7
“Such comparisons would directly respond to calls for studying human–AI complementarity (Hemmer et al. 2025; Dellermann et al. 2019) and to recent findings that challenge the notion that human input necessarily has a positive performance impact (Lee and Chung 2024; Vaccaro et al. 2024).”
Quality of human-GenAI collaboration and its driving factors: A symbiotic agency perspective
Shang, J., Huang, D., & Huang, S. (. (2025) · Information Processing & Management · doi:10.1016/j.ipm.2025.104373 · Cited by 9
Wei, Z., & Wei, R. (2025) · Expert Systems with Applications · doi:10.1016/j.eswa.2025.129617 · Cited by 9
Lee, Y., Min, B., & Han, S. (2026) · Cities · doi:10.1016/j.cities.2025.106766 · Cited by 7
Bertin, F., Lawrence, J., Niessen, S., Pinard, C. J., Reagan, K. L., & Rentko, V. T. (2026) · Journal of Veterinary Internal Medicine · doi:10.1093/jvimsj/aalaf045 · Cited by 9
“22 25 In the realm of large language modeling in clinical medicine, a randomized and single-blinded study was widely publicized, in which a commercially available large language model (LLM) outperformed physicians in diagnostic reasoning specific to 6 clinical vignettes.”
Unpacking the Paradoxes of Trust in Uncertain Times
Schilke, O. S., Bachmann, R., Blomqvist, K., Krishnan, R., & Sydow, J. (2026) · Organization Studies · doi:10.1177/01708406261430577 · Cited by 13
“4 Ultimately, configurations in which humans and intelligent systems jointly shape decisions—termed “conjoined agency” by Murray, Rhymer, et al. (2021)—may be the way forward, and more research is needed to determine how they might achieve optimal decision-making outcomes (Vaccaro et al., 2024).”
Maier, S., Schneider, M., & Feuerriegel, S. (2026) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3772318.3791185 · Cited by 26
Evaluating Trust in AI, Human, and Co-produced Feedback Among Undergraduate Students
Zhang, A., Gao, Y., Suraworachet, W., Nazaretsky, T., & Cukurova, M. (2025) · arXiv · doi:10.48550/arxiv.2504.10961 · Cited by 7
Vicente, L., Matute, H., Fregosi, C., & Cabitza, F. (2025) · International Journal of Human-Computer Studies · doi:10.1016/j.ijhcs.2025.103474 · Cited by 14
How the Role of Generative AI Shapes Perceptions of Value in Human-AI Collaborative Work
Schecter, A., & Richardson, B. (2025) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713946 · Cited by 15
“However, not all studies support this notion unequivocally; some findings indicate that collaborative efforts may not always lead to better performance, particularly when tasks involve decision-making rather than content creation [11], [12].”
Li, Y., Chang, Y., & Li, Y. (2025) · International Journal of Information Management · doi:10.1016/j.ijinfomgt.2025.102937 · Cited by 8
Schäfer, U., Sipos, L., & Müller-Birn, C. (2025) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3757451 · Cited by 10
Passi, S., Dhanorkar, S., & Vorvoreanu, M. (2025) · Springer eBooks · doi:10.1007/978-981-95-3658-0_98 · Cited by 9
Augmented Learning for Joint Creativity in Human-GenAI Co-Creation
Luan, Y. L., Kim, Y. J., & Zhou, J. (2025) · Information Systems Research · doi:10.1287/isre.2024.0984 · Cited by 12
AI in strategic alliance formation: a framework for human–AI collaboration
Sabz, A., Tabesh, P., & Yu, L. (2025) · Journal of Business Strategy · doi:10.1108/jbs-01-2025-0027 · Cited by 8
Cui, H., & Yu, P. (2026) · Information Technology and People · doi:10.1108/itp-09-2024-1184 · Cited by 10
When trust collides: Exploring human-LLM cooperation intention through the prisoner’s dilemma
Jiang, G., Yang, S., Wang, Y., & Hui, P. (2026) · International Journal of Human-Computer Studies · doi:10.1016/j.ijhcs.2026.103740 · Cited by 19
A meta-analysis on reactions to algorithmic decision-making in human resource management
Moritz, J. M., Pomrehn, L., Steinmetz, H., & Wehner, M. (2026) · Human Resource Management Review · doi:10.1016/j.hrmr.2026.101135 · Cited by 13
Li, L., & Yan, B. (2026) · Academy of Management Review · doi:10.5465/amr.2024.0546 · Cited by 15
Roemer, A., Schlicker, N., Kernder, A. L., Albe, B., Hack, J., Hirsch, M., Mayr, A. J., Kuhn, S., & Knitza, J. (2026) · BMC Medical Education · doi:10.1186/s12909-026-09079-w · Cited by 15
“This finding, though rarely demonstrated empirically [22], reinforces theoretical predictions that human–AI collaboration can achieve superior outcomes compared to either humans or AI working alone [23].”
Instant health or long-term wellness? A lay belief perspective on AI vs. human recipe recommenders
Shu, B., Mei, Y., Liu, W., & Wang, R. F. (2026) · Food Quality and Preference · doi:10.1016/j.foodqual.2026.105950 · Cited by 6
Will Autonomous AI Exceed AI-Aided Physicians as the Best Medical Care?
Emanuel, E., Baker-Butler, A., Khosla, N., & Khosla, V. (2026) · JAMA · doi:10.1001/jama.2026.15380 · Cited by 8
Park, S., Hwang, H. J., Yun, J., et al. (2025) · European Radiology · doi:10.1007/s00330-025-11689-9 · Cited by 5
Integration of Generative AI with Human Expertise in HEOR: A Hybrid Intelligence Framework
Srivastava, T. E., Irfan, H., Babiy, V. V., & Swami, S. (2025) · Advances in Therapy · doi:10.1007/s12325-025-03273-w · Cited by 8
“In niche domains that require a high degree of expertise, the involvement of human experts along with AI is shown to boost overall performance and accuracy [36].”
Human–robot collaboration in surgery at the nexus of knowledge, agency, and ownership
Merdin‐Uygur, E., Öztürkcan, S., Özbilgin, M. F., et al. (2025) · Scientific Reports · doi:10.1038/s41598-025-08437-w · Cited by 5
Mandon, P. (2025) · Digital Business · doi:10.1016/j.digbus.2025.100136 · Cited by 6
Human–AI collaboration for marketing capabilities: a meta-analysis
Ngo, V. M. (2025) · Marketing Letters · doi:10.1007/s11002-025-09783-5 · Cited by 7
“Our meta-analysis followed the structured approach outlined by Vaccaro et al. (2024) to identify and synthesize empirical studies evaluating human–AI collaboration in high-stakes decision-making contexts.”
Metadata
- Authors
- Vaccaro, M., Almaatouq, A., & Malone, T.
- Affiliations
- Massachusetts Institute of Technology (US)
- Venue
- Nature Human Behaviour
- Published
- 2024-10-28
- Volume
- 8
- Issue
- 12
- Pages
- 2293-2303
- ISSN
- 2397-3374 (ISSN-L)
- PMC
- PMC11659167
- OpenAlex
- W4403839497
- Semantic Scholar
- 02aec976011a7d595d61db9639209968f4b93177
- Cited by
- 570
- References
- 95
- Citations by year
- 2026: 404 · 2025: 156 · 2024: 9
- FWCI
- 51.29
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Locations
- Nature Human Behaviour (publishedVersion, cc-by)
- Research Square (acceptedVersion, cc-by)
- PubMed (publishedVersion)
- arXiv (Cornell University) (submittedVersion, cc-by)
- PubMed Central (submittedVersion, cc-by)
- RePEc: Research Papers in Economics (submittedVersion)
- arXiv (Cornell University) (cc-by)
- Funders
- Accenture Technology Convergence Fellowship
- Toyota Research Institute; MIT Quest for Intelligence; National Research Foundation, Prime Minister's Office, Singapore
- Topics
- Social Sciences › Human Factors and Ergonomics › Human-Automation Interaction and Safety
- Social Sciences › Psychology › Social Psychology › Social Robot Interaction and HRI
- Physical Sciences › Computer Science › Artificial Intelligence › Artificial Intelligence Applications
- Keywords
- human-AI collaboration, human-AI comparison, decision-making, content creation, performance outcomes, AI augmentation, task type effects, human-AI systems
- MeSH terms
- Artificial Intelligence, Decision Making, Humans
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-10-20, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-20 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
