Psychological Strategies for Winning a Geopolitical Forecasting Tournament
Open accessThe DOI matches its Crossref record; 43 of the 44 references it cites were found in a registry or an index; none of the 9 sources its claims cite is checked for support yet.
Abstract
Five university-based research groups competed to recruit forecasters, elicit their predictions, and aggregate those predictions to assign the most accurate probabilities to events in a 2-year geopolitical forecasting tournament. Our group tested and found support for three psychological drivers of accuracy: training, teaming, and tracking. Probability training corrected cognitive biases, encouraged forecasters to use reference classes, and provided forecasters with heuristics, such as averaging when multiple estimates were available. Teaming allowed forecasters to share information and discuss the rationales behind their beliefs. Tracking placed the highest performers (top 2% from Year 1) in elite teams that worked together. Results showed that probability training, team collaboration, and tracking improved both calibration and resolution. Forecasting is often viewed as a statistical problem, but forecasts can be improved with behavioral interventions. Training, teaming, and tracking are psychological interventions that dramatically increased the accuracy of forecasts. Statistical algorithms (reported elsewhere) improved the accuracy of the aggregation. Putting both statistics and psychology to work produced the best forecasts 2 years in a row.
Overview
CoChat Summary
## TL;DR
This paper reports a 2-year geopolitical forecasting tournament and tests three psychological interventions—training, teaming, and tracking—to improve probabilistic forecasts. All three improved accuracy, and the top 2% of Year 1 forecasters placed into elite teams (“superforecasters”) performed best in Year 2.
## Key findings
- Training improved accuracy: In Year 1, training improved Brier scores, with probability training outperforming scenario training, and scenario training outperforming no training: F(2, 1586) = 14.29, p < .001; t(1053) = 2.23, p = .026; t(1056) = 3.25, p < .001. In Year 2, probability training again beat no training: F(1, 882) = 19.12, p < .001.
- Teaming improved accuracy: Group influence was significant in both years, F(2, 1586) = 60.68, p < .001 and F(1, 882) = 62.76, p < .001. In Year 1, teams were more accurate than crowd-belief forecasters, who were more accurate than independent forecasters: t(1036) = 5.52, p < .001 and t(1120) = 5.92, p < .001.
- Tracking the top performers worked best: The best performers from Year 1, the top 2%, were placed together in elite Year 2 teams and “outperformed all other groups by a wide margin.” The paper reports no evidence of Year 2 regression to the mean.
- Both calibration and resolution improved: Calibration improved with training in Years 1 and 2, F(2, 1586) = 3.16, p = .04 and F(1, 938) = 3.78, p < .05; resolution varied with group influence and training in both years, and tracking enhanced resolution, t(293) = 4.12, p < .001.
## Methods
- The study embedded experiments in a 2-year geopolitical forecasting tournament (Year 1: September 2011–April 2012; Year 2: June 2012–April 2013) on goodjudgmentproject.com, where participants made probability estimates for 199 geopolitical outcomes.
- Year 1 used a 3 (training: no/scenario/probability) × 4 (group influence: independent/crowd-belief/team/prediction-market) factorial design; Year 2 used a 2 (training: no/probability) × 3 (group influence: independent/team/prediction-market) design plus an elite tracking condition. Results for prediction markets are noted as not discussed in the paper.
- The main outcome was Brier score; analyses also examined calibration and resolution, using statistical tests reported as F-tests, t-tests, and regressions. The paper states that findings were robust across Brier, spherical, and logarithmic scoring rules.
## Limitations
- The paper notes that results for prediction-market forecasting are not discussed here. This limits what can be concluded from the full design as presented in this article.
- The paper itself says “No one knows how much further improvement is possible” and that there are likely “pockets of inherent uncertainty” making perfect or near-perfect Brier scores impossible. This is a stated substantive limitation of forecasting accuracy.
- Evident from the text: the Year 2 no-training condition included some participants who had previously received scenario training, which the authors say “tipped the scale against finding a statistically significant effect of training.”
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures




Verification
Claims the paper makes
0 Valid · 0 Not valid · 5 Not checked · 39 Own results and interpretations
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“and in the middle period of Year 2, F (1, 882) = 6.10, p = .014.”
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For both years, training and teaming remained significant predictors even when timing and updating were included as predictors.”
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In addition to showing a main effect of group influence, Year 2 data showed an interaction between training and group influence, F (1, 882) = 5.78, p = .018.”
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In Year 2, calibration also benefited from group influence, F (1, 938) = 7.07, p = .008, and tracking.”
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“There was no evidence of Year 2 regression to the mean;”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
19 more claims
Create a free account to see all 44 claims and to verify the 1 reference still open.
- It may state two separate claims as one. 16
- Probably not a substantial claim. 6
- The quote only partly states this claim. 2
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Additionally, as reported elsewhere, training improved the calibration and resolution of forecasters by reducing overconfidence (Mellers et al., 2014; Moore et al., 2016).”
From Developing expert political judgment: The impact of training and practice on judgmental accuracy in geopolitical forecasting tournaments (2016) · cites this paper as Mellers et al., 2014
Experimental — accuracy not yet measured
Statistics that don’t add up
Each reported p-value recomputed from its test statistic and degrees of freedom, each partial η² from its F, and each ratio, mean difference or B printed with a p checked against its own confidence interval.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
“F (1, 882) = 5.78, p = .018”
Reported p = .018 · recomputed p = .016
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
“F (1, 938) = 3.78, p < .05”
Reported p < .05 · recomputed p = .052
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
“F (2, 1463) = 6.20, p < .001”
Reported p < .001 · recomputed p = .002
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
“t (1056) = 3.25, p < .001”
Reported p < .001 · recomputed p = .001
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
“t (1588) = 2.70, p < .001”
Reported p < .001 · recomputed p = .007
Statistics recomputed
Every reported test result, recomputed from its test statistic and degrees of freedom. A result shows as not adding up only once an independent check has confirmed it.
- Doesn’t add upt(1056) = 3.25reported p < .001 · recomputed p = .001
“t (1056) = 3.25, p < .001”
- Doesn’t add upF(1, 882) = 5.78reported p = .018 · recomputed p = .016
“F (1, 882) = 5.78, p = .018”
- Doesn’t add upF(2, 1463) = 6.2reported p < .001 · recomputed p = .002
“F (2, 1463) = 6.20, p < .001”
- Doesn’t add upt(1588) = 2.7reported p < .001 · recomputed p = .007
“t (1588) = 2.70, p < .001”
- Doesn’t add upF(1, 938) = 3.78reported p < .05 · recomputed p = .052
“F (1, 938) = 3.78, p < .05”
37 more recomputed tests
Create a free account to see every test the Stats check recomputed.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Self-check results appear here once an independent check has confirmed them.
Citations
Cites
Journal of The American Statistical Association
Fienberg, S. E. (2005) · Encyclopedia of Statistical Sciences · doi:10.1002/0471667196.ess1317 · Cited by 6,125
Groups: Interaction and Performance.
Polley, R. B., & McGrath, J. E. (1984) · Administrative Science Quarterly · doi:10.2307/2393041 · Cited by 3,290
Norm theory: Comparing reality to its alternatives.
Kahneman, D., & Miller, D. T. (1986) · Psychological Review · doi:10.1037/0033-295x.93.2.136 · Cited by 3,024
The difference: how the power of diversity creates better groups, firms, schools, and societies
(2007) · Choice Reviews Online · doi:10.5860/choice.45-1534 · Cited by 2,280
Many Hands Make Light the Work: The Causes and Consequences of Social Loafing
Latané, B., Williams, K., & Harkins, S. (1979) · Journal of Personality and Social Psychology · doi:10.1037/0022-3514.37.6.822 · Cited by 2,216
Stasser, G., & Titus, W. (1985) · Journal of Personality and Social Psychology · doi:10.1037/0022-3514.48.6.1467 · Cited by 1,850 · cited 1×
“Teams have a demonstrated failure to search for unique information, unshared information, or hypothesis-disconfirming information (Kerr, MacCoun, & Kramer, 1996; Stasser & Titus, 1985; Sunstein, 2006).”
BackgroundA Theory of Fads, Fashion, Custom, and Cultural Change as Informational Cascades
Bikhchandani, S., Hirshleifer, D., & Welch, I. (1992) · Journal of Political Economy · doi:10.1086/261849 · Cited by 1,805
Uncertain Judgements: Eliciting Experts' Probabilities
O'Hagan, A., Buck, C. E., Daneshkhah, A., Eiser, J. R., Garthwaite, P. H., Jenkinson, D. J., Oakley, J. E., & Rakow, T. (2006) · doi:10.1002/0470033312 · Cited by 1,550
Koriat, A., Lichtenstein, S., & Fischhoff, B. (1980) · Journal of Experimental Psychology: Human Learning and Memory · doi:10.1037/0278-7393.6.2.107 · Cited by 1,477
Group Performance and Decision Making
Kerr, N. L., & Tindale, R. S. (2004) · Annual Review of Psychology · doi:10.1146/annurev.psych.55.090902.142009 · Cited by 1,331
Research works printed but not linked: 2
- Fischhoff, B., & Chauvin, C. (Eds.). (2011). Intelligence analysis: Behavioral and social scientific foundations . Washington, DC: National Academies Press.
- Janis, I. (1982). Group-think . Boston, MA: Houghton-Mifflin.
Cited by
Algorithm appreciation: People prefer algorithmic to human judgment
Logg, J. M., Minson, J. A., & Moore, D. A. (2019) · Organizational Behavior and Human Decision Processes · doi:10.1016/j.obhdp.2018.12.005 · Cited by 2,061
“Following prior research (Mellers et al., 2014), we used Brier scores (Brier, 1950) to assess forecasting accuracy.”
Self-citationThe Bias of Crowds: How Implicit Bias Bridges Personal and Systemic Prejudice
Payne, B. K., Vuletich, H. A., & Lundberg, K. B. (2017) · Psychological Inquiry · doi:10.1080/1047840x.2017.1335568 · Cited by 535
“Research on “the wisdom of crowds” (Surowiecki, 2004) finds that, for many kinds of questions, the collective judgment of a group tends to be closer to the true answer than any one individual’s answer (Clemen, 1989; Galton, 1907; Lorge, Fox, Davitz, & Brenner, 1958; Mellers et al., 2014; Page, 2007).”
BackgroundA practical guide to structured expert elicitation using the IDEA protocol
Hemming, V., Burgman, M. A., Hanea, A. M., McBride, M. F., & Wintle, B. C. (2018) · Methods in Ecology and Evolution · doi:10.1111/2041-210x.12857 · Cited by 528
“Importantly, it should account for each step of the elicitation including the recruitment of experts, the framing of questions, the elicitation and aggregation of their judgements, using procedures that have been tested and clearly demonstrated to improve judgements (e.g. Cooke, 1991; Mellers et al., 2014).”
BackgroundMethodInfluentialNetwork dynamics of social influence in the wisdom of crowds
Becker, J., Brackbill, D., & Centola, D. (2017) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1615978114 · Cited by 397
Debiasing Decisions: Improved Decision Making With a Single Training Intervention
Morewedge, C. K., Yoon, H., Scopelliti, I., Symborski, C. W., Korris, J. H., & Kassam, K. S. (2015) · Policy Insights from the Behavioral and Brain Sciences · doi:10.1177/2372732215600886 · Cited by 384
“Researchers have produced numerous successful collaborations with government and industry partners that have reduced waste and improved the health and finances of the public (e.g., Chapman et al., 2010; Mellers et al., 2014; Schultz et al., 2007; Schwartz et al., 2014; Thaler & Benartzi, 2004).”
BackgroundA solution to the single-question crowd wisdom problem
Prelec, D., Seung, H. S., & McCoy, J. (2017) · Nature · doi:10.1038/nature21054 · Cited by 314
Philosophers’ biased judgments persist despite training, expertise and reflection
Schwitzgebel, E., & Cushman, F. (2015) · Cognition · doi:10.1016/j.cognition.2015.04.015 · Cited by 314
“Future efforts to minimize cognitive bias might more effectively focus on other means, such as alterations of choice architecture (Thaler & Sunstein, 2008) or feedback-based training and social support (Mellers et al., 2014).”
BackgroundMannes, A. E., Soll, J. B., & Larrick, R. P. (2014) · Journal of Personality and Social Psychology · doi:10.1037/a0036677 · Cited by 255
“Environments with high dispersion in expertise present an op portunity to identify and capitalize on the abilities of the more capable judges (Mellers et al., 2014).”
BackgroundMegastudies improve the impact of applied behavioural science
Milkman, K. L., Gromet, D., Ho, H., Kay, J. S., Lee, T. W., Pandiloski, P., Park, Y., Rai, A., Bazerman, M., Beshears, J., et al. (2021) · Nature · doi:10.1038/s41586-021-04128-4 · Cited by 249
Self-citationIdentifying and Cultivating Superforecasters as a Method of Improving Probabilistic Predictions
Mellers, B., Stone, E., Murray, T., Minster, A., Rohrbaugh, N., Bishop, M., Chen, E., Baker, J., Hou, Y., Horowitz, M., et al. (2015) · Perspectives on Psychological Science · doi:10.1177/1745691615577794 · Cited by 236
Self-citationThe psychology of intelligence analysis: Drivers of prediction accuracy in world politics.
Mellers, B., Stone, E., Atanasov, P., Rohrbaugh, N., Metz, S. E., Ungar, L., Bishop, M. M., Horowitz, M., Merkle, E., & Tetlock, P. (2015) · Journal of Experimental Psychology: Applied · doi:10.1037/xap0000040 · Cited by 187
“Indeed, Reyna, Chick, Corbin, and Hsia (2014) showed that intelligence analysts were more susceptible to risky-choice framing effects than either college students or postcollegiate adults, perhaps because they had developed bad habits in an “unfriendly” environment.”
BackgroundSelf-citationBoosting medical diagnostics by pooling independent judgments
Kurvers, R. H. J. M., Herzog, S. M., Hertwig, R., Krause, J., Carney, P. A., Bogart, A., Argenziano, G., Zalaudek, I., & Wolf, M. (2016) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1601827113 · Cited by 198
“However, despite its potential to boost accuracy in a wide range of contexts, including lie detection, political forecasting, investment decisions, and medical decision making (8 – 14), little is known about the conditions that underlie the emergence of collective intelligence in real-world domains.”
BackgroundAggregated knowledge from a small number of debates outperforms the wisdom of large crowds
Navajas, J., Niella, T., Garbulsky, G., Bahrami, B., & Sigman, M. (2018) · Nature Human Behaviour · doi:10.1038/s41562-017-0273-4 · Cited by 183
The Argumentative Theory: Predictions and Empirical Evidence
Mercier, H. (2016) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2016.07.001 · Cited by 184
“(C) represents five studies that used a variety of less demonstrative tasks: detecting lies [93], forecasting of economic and political events [65], solving biological problems in a classroom [94], making medical diagnoses [95], and making judicial decisions [96].”
BackgroundImplicit bias reflects systemic racism
Payne, B. K., & Hannay, J. W. (2021) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2021.08.001 · Cited by 188
Scientific Utopia III: Crowdsourcing Science
Uhlmann, E. L., Ebersole, C. R., Chartier, C. R., Errington, T. M., Kidwell, M. C., Lai, C. K., McCarthy, R. J., Riegelman, A., Silberzahn, R., & Nosek, B. A. (2019) · Perspectives on Psychological Science · doi:10.1177/1745691619850561 · Cited by 174
“The aggregated estimates of crowds perform surprisingly well at predicting future outcomes—such as election results, news and sporting events, and stock-market fluctuations—because in many cases, the aggregation cancels out individual errors (Galton, 1907; Mellers et al., 2014; Surowiecki, 2005).”
BackgroundDistilling the Wisdom of Crowds: Prediction Markets vs. Prediction Polls
Atanasov, P., Rescober, P., Stone, E., Swift, S. A., Servan-Schreiber, E., Tetlock, P., Ungar, L., & Mellers, B. (2017) · Management Science · doi:10.1287/mnsc.2015.2374 · Cited by 173
“Prior research also suggests that teaming confers a consistent advantage to the accuracy of polls (Mellers et al. 2014).”
BackgroundMethodResultInfluentialSelf-citationInformation aggregation and collective intelligence beyond the wisdom of crowds
Kameda, T., Toyokawa, W., & Tindale, R. S. (2022) · Nature Reviews Psychology · doi:10.1038/s44159-022-00054-y · Cited by 193
Hemming, V., Walshe, T. V., Hanea, A. M., Fidler, F. M., & Burgman, M. A. (2018) · PLoS ONE · doi:10.1371/journal.pone.0198468 · Cited by 141
“Many studies have found that structured group judgements usually perform as well, or better than the best credentialed expert [67, 71, 78].”
BackgroundTwo Reasons to Make Aggregated Probability Forecasts More Extreme
Baron, J., Mellers, B. A., Tetlock, P. E., Stone, E., & Ungar, L. H. (2014) · Decision Analysis · doi:10.1287/deca.2014.0293 · Cited by 130
“…a less restrictive transformation in which the level varies as well as the amount of In a study of probabilistic forecasting of international events (Mellers et al. 2014), we found empirically that an extremizing transformation in this form, with values of a greater than 1, could improve the…”
BackgroundMethodSelf-citationCulture Changes How We Think About Thinking: From “Human Inference” to “Geography of Thought”
de Oliveira, S., & Nisbett, R. E. (2017) · Perspectives on Psychological Science · doi:10.1177/1745691617702718 · Cited by 159
“A team from the University of Pennsylvania won the contest (Mellers et al., 2014).”
Background“Improving Intelligence Analysis with Decision Science.”
Dhami, M. K., Mandel, D. R., Mellers, B. A., & Tetlock, P. E. (2015) · Perspectives on Psychological Science · doi:10.1177/1745691615598511 · Cited by 104
“Forecasting can also be improved by using well-coordinated teamwork that allows forecasters to share information, debate rationales, and motivate each other to perform well (Mellers et al., 2014).”
BackgroundSelf-citation“Accuracy of Forecasts in Strategic Intelligence.”
Mandel, D. R., & Barnes, A. (2014) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1406138111 · Cited by 101
Forecasting Tournaments: Tools for Increasing Transparency and Improving the Quality of Debate
Tetlock, P. E., Mellers, B. A., Rohrbaugh, N., & Chen, E. (2014) · Current Directions in Psychological Science · doi:10.1177/0963721414534257 · Cited by 98
Self-citationBeyond collective intelligence: Collective adaptation
Galešić, M., Barkoczi, D., Berdahl, A. M., Biro, D., Carbone, G., Giannoccaro, I., Goldstone, R. L., González, C., Kandler, A., Kao, A. B., et al. (2023) · Journal of The Royal Society Interface · doi:10.1098/rsif.2022.0736 · Cited by 155
“Studies of ‘ wisdom of crowds ’ phenomena range from early seminal essays of Condorcet [61] and Galton [62] to recent studies in judgement and decision-making [63], cognitive psychology [64], political science [65], forecasting [66], economics [67], law [68], sociology [69] and biology [70].”
Chang, W., Chen, E., Mellers, B., & Tetlock, P. (2016) · Judgment and Decision Making · doi:10.1017/s1930297500004599 · Cited by 89
“Additionally, as reported elsewhere, training improved the calibration and resolution of forecasters by reducing overconfidence (Mellers et al., 2014; Moore et al., 2016).”
InfluentialSelf-citationCrowds Can Effectively Identify Misinformation at Scale
Martel, C., Allen, J., Pennycook, G., & Rand, D. G. (2024) · Perspectives on Psychological Science · doi:10.1177/17456916231190388 · Cited by 84
“Nevertheless, there are numerous examples of social feedback and group collaboration improving task performance (Granovskiy et al., 2015; Mellers et al., 2014; Moussaid, 2022; Navajas et al., 2018).”
BackgroundThe network science of collective intelligence
Centola, D. M. (2022) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2022.08.009 · Cited by 121
“…groups of experts to explore how iterative estimations among knowledgeable people [157 – 159], sometimes with voluntaryopt-inandopt-outstrategies[160 – 163],orevenamonglaypersonsengagedindirectconversation[164],canimprove upon the wisdom of the crowd that emerges from groups of independent…”
BackgroundAdvice as a form of social influence: Informational motives and the consequences for accuracy
Rader, C. A., Larrick, R. P., & Soll, J. B. (2017) · Social and Personality Psychology Compass · doi:10.1111/spc3.12329 · Cited by 73
Confidence Calibration in a Multiyear Geopolitical Forecasting Competition
Moore, D. A., Swift, S. A., Minster, A., Mellers, B., Ungar, L., Tetlock, P., Yang, H. H. J., & Tenney, E. R. (2017) · Management Science · doi:10.1287/mnsc.2016.2525 · Cited by 74
“Mellers et al. (2014) do not, however, examine confidence and accuracy over time to understand how they develop.”
BackgroundInfluentialSelf-citationCommunicating uncertainty using words and numbers
Dhami, M. K., & Mandel, D. R. (2022) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2022.03.002 · Cited by 69
The Social Context of Decisions
Larrick, R. P. (2016) · Annual Review of Organizational Psychology and Organizational Behavior · doi:10.1146/annurev-orgpsych-041015-062445 · Cited by 80
Bringing probability judgments into policy debates via forecasting tournaments
Tetlock, P. E., Mellers, B. A., & Scoblic, J. P. (2017) · Science · doi:10.1126/science.aal3147 · Cited by 67
Self-citationModeling Probability Forecasts via Information Diversity
Satopää, V., Pemantle, R., & Ungar, L. (2016) · Journal of the American Statistical Association · doi:10.1080/01621459.2015.1100621 · Cited by 69
“For instance, Ranjan and Gneiting (2010) extremized p with the CDF of a beta distribution; Satopää et al. (2014a) used a logistic regression model to derive an aggregator that extremizes plog; Baron et al.”
BackgroundMethodSelf-citationUncertainty Quantification with Experts: Present Status and Research Needs
Hanea, A. M., Hemming, V., & Nane, G. F. (2021) · Risk Analysis · doi:10.1111/risa.13718 · Cited by 79
“, Clemen & Winkler, 1999), but not always (e.g., Cooke, 2015; Hemming et al., 2018a; Mellers et al., 2014).”
BackgroundNatural-Born Arguers: Teaching How to Make the Best of Our Reasoning Abilities
Mercier, H., Boudry, M., Paglieri, F., & Trouche, E. (2016) · Educational Psychologist · doi:10.1080/00461520.2016.1207537 · Cited by 83
“For more complex tasks with less definitive arguments, discussion also allows groups to converge on the better supported answer, sometimes generating a solution superior to that of any individual member (e.g., Mellers et al., 2014; for reviews, see Laughlin, 2011; Mercier, 2011d).”
BackgroundThe Value of Performance Weights and Discussion in Aggregated Expert Judgments
Hanea, A. M., McBride, M. F., Burgman, M. A., & Wintle, B. C. (2018) · Risk Analysis · doi:10.1111/risa.12992 · Cited by 68
“As with IDEA, the GJP also made use of structured discussion between forecasters and found that it improved performance (Mellers et al., 2014).”
ResultFriedman, J. A., Baker, J. D., Mellers, B. A., Tetlock, P. E., & Zeckhauser, R. (2018) · International Studies Quarterly · doi:10.1093/isq/sqx078 · Cited by 58
Self-citationUse of expert knowledge to anticipate the future: Issues, analysis and directions
Bolger, F., & Wright, G. (2016) · International Journal of Forecasting · doi:10.1016/j.ijforecast.2016.11.001 · Cited by 65
“Prediction markets have been found to produce good forecasts in certain domains, such as prediction of geo-political events (Mellers et al., 2014).”
BackgroundAggregating predictions from experts: A review of statistical methods, experiments, and applications
McAndrew, T., Wattanachit, N., Gibson, G., & Reich, N. G. (2020) · Wiley Interdisciplinary Reviews Computational Statistics · doi:10.1002/wics.1514 · Cited by 68
“One of the most successful expert opinion forecasting systems enrolled approximately 2000 participants and was called the Good Judgment Project (GJP; Mellers et al., 2014; Satopää et al., 2014; Ungar, Mellers, Satopaa, Tetlock, & Baron, 2012).”
How generalizable is good judgment? A multi-task, multi-benchmark study
Mellers, B. A., Baker, J. D., Chen, E., Mandel, D. R., & Tetlock, P. E. (2017) · Judgment and Decision Making · doi:10.1017/s1930297500006240 · Cited by 44
Self-citationSmall steps to accuracy: Incremental belief updaters are better forecasters
Atanasov, P. D., Witkowski, J., Ungar, L., Mellers, B. A., & Tetlock, P. E. (2020) · Organizational Behavior and Human Decision Processes · doi:10.1016/j.obhdp.2020.02.001 · Cited by 50
Self-citationMartinez‐Marquez, D., Gulati, K., Carty, C. P., Stewart, R. A., & Ivanovski, S. (2020) · Materials Science and Engineering C · doi:10.1016/j.msec.2020.110995 · Cited by 51
Belief Movement, Uncertainty Reduction, and Rational Updating
Augenblick, N., & Rabin, M. (2021) · The Quarterly Journal of Economics · doi:10.1093/qje/qjaa043 · Cited by 78
Tump, A. N., Wolf, M., Krause, J., & Kurvers, R. H. J. M. (2018) · Journal of The Royal Society Interface · doi:10.1098/rsif.2018.0155 · Cited by 53
Wise teamwork: Collective confidence calibration predicts the effectiveness of group discussion
Silver, I., Mellers, B. A., & Tetlock, P. E. (2021) · Journal of Experimental Social Psychology · doi:10.1016/j.jesp.2021.104157 · Cited by 60
“Discussion can also foster engagement and learning (Smith et al., 2009), by encouraging members to articulate and stress-test their rationales (Mellers et al., 2014) and by facilitating the transfer of knowledge from more to less informed group members (Schultze, Mojzisch, & Schulz-Hardt, 2012).”
BackgroundSelf-citationGeopolitical Forecasting Skill in Strategic Intelligence
Mandel, D. R., & Barnes, A. (2017) · Journal of Behavioral Decision Making · doi:10.1002/bdm.2055 · Cited by 43
Mellers, B. A., & Tetlock, P. E. (2019) · American Psychologist · doi:10.1037/amp0000429 · Cited by 50
Self-citationExperts and laymen grossly underestimate the benefits of argumentation for reasoning
Mercier, H., Trouche, E., Yama, H., Heintz, C., & Girotto, V. (2014) · Thinking & Reasoning · doi:10.1080/13546783.2014.981582 · Cited by 51
“This improvement has been observed in a wide range of tasks in the laboratory— deductive problems (Laughlin & Ellis, 1986; Moshman & Geil, 1998; Trouche, Sander, & Mercier, in press), inductive problems (Laughlin, Bonner, & Miner, 2002), numerical estimations (Minson, Liberman, & Ross, 2011; Sniezek & Henry, 1989), an…”
BackgroundInstruction in information structuring improves Bayesian judgment in intelligence analysts
Mandel, D. R. (2015) · Frontiers in Psychology · doi:10.3389/fpsyg.2015.00387 · Cited by 42
“Forecast accuracy has also been improved by probability training that took the form of directives and rules of thumb aimed at avoiding common pitfalls, such as assigning probabilities of fifty-fifty to binary complements when forecasters are deeply unsure (Mellers et al., 2014).”
BackgroundForecasting and uncertainty: A survey
Makridakis, S., & Bakas, N. (2016) · Risk and Decision Analysis · doi:10.3233/rda-150114 · Cited by 73
“One such project is the Geopolitical Forecasting Tournament [75] whose findings indicate that forecasting accuracy and correct calibration of uncertainty can be improved by training, teaming and tracking their performance.”
Beard, S., Rowe, T. D., & Fox, J. (2019) · Futures · doi:10.1016/j.futures.2019.102469 · Cited by 56
Bias, Information, Noise: The BIN Model of Forecasting
Satopää, V., Salikhov, M., Tetlock, P. E., & Mellers, B. A. (2020) · Management Science · doi:10.1287/mnsc.2020.3882 · Cited by 65
Self-citationForecasting tournaments, epistemic humility and attitude depolarization
Mellers, B., Tetlock, P., & Arkes, H. R. (2019) · Cognition · doi:10.1016/j.cognition.2018.10.021 · Cited by 46
Self-citationAn Overview of Applications of Proper Scoring Rules
Carvalho, A. (2016) · Decision Analysis · doi:10.1287/deca.2016.0337 · Cited by 65
“…been applied to evaluate the accuracy of predictions about presidential elections (Heath and Tversky 1991, Pennock et al. 2002), in a geopolitical forecasting contest (Mellers et al. 2014), and to evaluate the opinions from policy makers regarding the occurrence of political events (Tetlock 2005).”
BackgroundMethodInfluentialQuantifying Earth system interactions for sustainable food production via expert elicitation
Chrysafi, A., Virkki, V., Jalava, M., Sandström, V., Piipponen, J., Porkka, M., Lade, S. J., La Mere, K., Wang-Erlandsson, L., Scherer, L., et al. (2022) · Nature Sustainability · doi:10.1038/s41893-022-00940-6 · Cited by 41
“It combines the benefits of Delphi 61,67 and four-step elicitation processes 67–69 , which in combination has been shown to improve judgements 65,70,71 .”
BackgroundIllegitimate Values, Confirmation Bias, and Mandevillian Cognition in Science
Peters, U. (2021) · The British Journal for the Philosophy of Science · doi:10.1093/bjps/axy079 · Cited by 45
Transforming Conservation: A Practical Guide to Evidence and Decision Making
Amano, T., Bako, L. A., Best, M., Boenisch, N., Boersch‐Supan, P. H., Browne, D., Buckley, Y. M., Burgman, M. A., Cadotte, M. W., Canessa, S., et al. (2022) · Open Book Publishers · doi:10.11647/obp.0321 · Cited by 61
Trust in Public Policy Algorithms
Kennedy, R. P., Waggoner, P. D., & Ward, M. M. (2022) · The Journal of Politics · doi:10.1086/716283 · Cited by 46
“There have also been efforts to improve human-based forecasting by applying algorithms to aggregate group forecasts (Mellers et al. 2014; Tetlock and Gardner 2016).”
BackgroundHow to detect high-performing individuals and groups: Decision similarity predicts accuracy
Kurvers, R. H. J. M., Herzog, S. M., Hertwig, R., Krause, J., Moussaïd, M., Argenziano, G., Zalaudek, I., Carney, P. A., & Wolf, M. (2019) · Science Advances · doi:10.1126/sciadv.aaw9011 · Cited by 54
Friedman, J. A., Lerner, J. S., & Zeckhauser, R. (2016) · International Organization · doi:10.1017/s0020818317000352 · Cited by 54
The Bad Thing About Good Advice: Understanding When and How Advice Exacerbates Overconfidence
Soll, J. B., Palley, A. B., & Rader, C. A. (2020) · Management Science · doi:10.1287/mnsc.2021.3987 · Cited by 63
Wisdom in a complex world: A situated account of wise reasoning and its development
Santos, H. C., Huynh, A. C., & Grossmann, I. (2017) · Social and Personality Psychology Compass · doi:10.1111/spc3.12341 · Cited by 40
Demographically diverse crowds are typically not much wiser than homogeneous crowds
de Oliveira, S., & Nisbett, R. E. (2018) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1717632115 · Cited by 43
Horowitz, M. C., Stewart, B., Tingley, D. H., Bishop, M., Samotin, L. R., Roberts, M. E., Chang, W., Mellers, B. A., & Tetlock, P. E. (2019) · The Journal of Politics · doi:10.1086/704437 · Cited by 61
“On the superiority of teams in the context of the IARPA tournament in general, see Mellers et al. (2014) andMellers, Stone, Atanasov, et al. (2015).”
BackgroundInfluentialSelf-citationDoes discussion make crowds any wiser?
Mercier, H., & Claidière, N. (2021) · Cognition · doi:10.1016/j.cognition.2021.104912 · Cited by 52
“…to improve the average performance of the group members on a wide range of problems, ranging from logical tasks to political predictions (e.g., Mellers et al., 2014; Moshman & Geil, 1998; Trouche, Sander, & Mercier, 2014; for reviews, see, Laughlin, 2011; Mercier, 2016; Mercier & Sperber,…”
BackgroundAccountability and adaptive performance under uncertainty: A long-term view
Chang, W., Atanasov, P. D., Patil, S. V., Mellers, B. A., & Tetlock, P. E. (2017) · Judgment and Decision Making · doi:10.1017/s1930297500006732 · Cited by 47
“But we do know, based on three years of study and experimental testing, that the CHAMPS KNOW training did significantly improve accuracy above that of control conditions (Mellers et al., 2014, 2015; Chang et al., 2016).”
BackgroundMethodResultInfluentialSelf-citationSocial learning strategies for matters of taste
Analytis, P. P., Barkoczi, D., & Herzog, S. M. (2018) · Nature Human Behaviour · doi:10.1038/s41562-018-0343-2 · Cited by 40
“…are expected to perform better as the dispersion in the predictive power of the cues in the environment increases because it is easier to identify highly valuable predictors and forecasters 25,36,37 and differential weighting can profit from differences in the predictive values of the cues 30 .”
BackgroundThe Impact of Overconfidence and Ambiguity Attitude on Market Entry
Gutierrez, C., Åstebro, T. B., & Obłój, T. (2018) · Organization Science · doi:10.1287/orsc.2019.1300 · Cited by 53
Improving information evaluation for intelligence production
Irwin, D., & Mandel, D. R. (2019) · Intelligence & National Security · doi:10.1080/02684527.2019.1569343 · Cited by 46
The Power of Social Influence on Estimation Accuracy
Gürçay, B., Mellers, B. A., & Baron, J. A. (2014) · Journal of Behavioral Decision Making · doi:10.1002/bdm.1843 · Cited by 49
Self-citationDuckworth, A. L., & Milkman, K. L. (2022) · PNAS Nexus · doi:10.1093/pnasnexus/pgac214 · Cited by 46
“Scientific tournaments have a similar flavor, although they do not typically involve random assignment of participants to condition (4).”
BackgroundValidating the Contribution-Weighted Model: Robustness and Cost-Benefit Analyses
Chen, E., Budescu, D. V., Lakshmikanth, S. K., Mellers, B. A., & Tetlock, P. E. (2016) · Decision Analysis · doi:10.1287/deca.2016.0329 · Cited by 40
Self-citationBarberia, I., Tubau, E., Matute, H., & Rodríguez‐Ferreiro, J. (2018) · PLoS ONE · doi:10.1371/journal.pone.0191907 · Cited by 42
Are markets more accurate than polls? The surprising informational value of “just asking”
Dana, J. D., Atanasov, P. D., Tetlock, P. E., & Mellers, B. A. (2019) · Judgment and Decision Making · doi:10.1017/s1930297500003375 · Cited by 36
“During the IARPA Aggregative Contingent Estimation (ACE) tournament (Mellers et al., 2014; Atanasov et al., 2017), our team, the Good Judgment Project, randomly assigned participants to take part in a prediction market.”
Self-citationRethinking the training of intelligence analysts
Chang, W., & Tetlock, P. E. (2016) · Intelligence & National Security · doi:10.1080/02684527.2016.1147164 · Cited by 36
Self-citationMethods for Eliciting Informative Prior Distributions: A Critical Review
Falconer, J. R., Frank, E., Polaschek, D. L. L., & Joshi, C. (2022) · Decision Analysis · doi:10.1287/deca.2022.0451 · Cited by 49
Automating hybrid collective intelligence in open-ended medical diagnostics
Kurvers, R. H. J. M., Nuzzolese, A. G., Russo, A., Barabucci, G., Herzog, S. M., & Trianni, V. (2023) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2221473120 · Cited by 40
Probability aggregation in time-series: Dynamic hierarchical modeling of sparse expert beliefs
Satopää, V. A., Jensen, S. T., Mellers, B. A., Tetlock, P. E., & Ungar, L. H. (2014) · The Annals of Applied Statistics · doi:10.1214/14-aoas739 · Cited by 30
“, Carter and Kohn (1994), Migon et al. (2005) for details on FFBS].”
BackgroundMethodInfluentialSelf-citationRace, weapons, and the perception of threat
Payne, B. K., & Correll, J. (2020) · Advances in experimental social psychology · doi:10.1016/bs.aesp.2020.04.001 · Cited by 52
Forecasting forecaster accuracy: Contributions of past performance and individual differences
Himmelstein, M., Atanasov, P. D., & Budescu, D. V. (2021) · Judgment and Decision Making · doi:10.1017/s1930297500008597 · Cited by 39
“One such research group was the Good Judgment Project (GJP, Mellers et al., 2014).”
BackgroundSelf-citationIncentive-Compatible Forecasting Competitions
Witkowski, J., Freeman, R., Vaughan, J. W., Pennock, D. M., & Krause, A. (2022) · Management Science · doi:10.1287/mnsc.2022.4410 · Cited by 31
Predicting and reasoning about replicability using structured groupsPreprint
Wintle, B. C., Mody, F., Smith, E. T., Hanea, A. M., Wilkinson, D. P., Hemming, V., Bush, M., Fraser, H. S., Thorn, F. S., McBride, M. F., et al. (2021) · doi:10.31222/osf.io/vtpmb · Cited by 28
Dunning, D. A. (2015) · Emerald Group Publishing Limited eBooks · doi:10.1108/s0882-614520150000032006 · Cited by 32
Making a Wiser Crowd: Benefits of Individual Metacognitive Control on Crowd Performance
Bennett, S. T., Benjamin, A. S., Mistry, P. K., & Mark Steyvers, B. (2018) · Computational Brain & Behavior · doi:10.1007/s42113-018-0006-4 · Cited by 30
“…in a wide variety of tasks, including complex combinatorial problems (Yi et al. 2012), recitation of lists in an appropriate order (Steyvers et al. 2009), and in predicting events with as-yet unknown outcomes (Lee and Danileiko 2014; Turner et al. 2014; Merkle et al. 2016; Mellers et al. 2014).”
BackgroundEstévez, R. A., Mardones, F. O., Alamos, F. H., Arriagada, G., Carey, J. M., Corrêa, C. T., Escobar‐Dodero, J., Gaete, Á., Gallardo, A., Ibarra, R., et al. (2019) · Aquaculture · doi:10.1016/j.aquaculture.2019.04.028 · Cited by 28
An expert judgment model to predict early stages of the COVID-19 pandemic in the United States
McAndrew, T., & Reich, N. G. (2022) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1010485 · Cited by 28
“Expert and non-expert crowds have made accurate predictions of sleep disturbances [16], geopolitical events [15, 17], and meteorologic events [20, 21], and experts have successfully chosen variables that improved the prediction of clinical phenomena [18, 22].”
BackgroundItem response models of probability judgments: Application to a geopolitical forecasting tournament.
Merkle, E. C., Mark Steyvers, B., Mellers, B. A., & Tetlock, P. E. (2015) · Decision · doi:10.1037/dec0000032 · Cited by 30
“Further details on the data collection methods can be found in Mellers et al. (2014). In the current article, we focus on a subset of the full dataset: 241 participants who responded to nearly all of 133 two-alternative forecasting questions.”
BackgroundSelf-citationA crowdsourcing triage algorithm for geopolitical event forecasting
Rostami, M. G., Huber, D., & Lu, T.-C. (2018) · ACM Conference on Recommender Systems (RecSys) · doi:10.1145/3240323.3240385 · Cited by 26
“A long term study has demonstrated that this strategy can dramatically increase the accuracy of forecasts against a controlled group of randomly selected forecasters [8].”
BackgroundToward an understanding of collective intellectual humility
Krumrei-Mancuso, E. J., Pärnamets, P., Bland, S., Astola, M., Cichocka, A. K., de Ridder, J., Mercier, H., Meyer, M., O’Connor, C., Porter, T., et al. (2024) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2024.09.006 · Cited by 39
“Social groups de fi ned haphazardly, such as right-handed Europeans, are less likely to develop the sort of distinctiveness, cohesiveness, and entitativity required for the possession of collective traits.”
A neglected dimension of good forecasting judgment: The questions we choose also matter
Merkle, E. C., Steyvers, M., Mellers, B. A., & Tetlock, P. E. (2017) · International Journal of Forecasting · doi:10.1016/j.ijforecast.2017.04.002 · Cited by 22
Self-citationImproving Accuracy by Coherence Weighting of Direct and Ratio Probability Judgments
Fan, Y., Budescu, D. V., Mandel, D. R., & Himmelstein, M. (2019) · Decision Analysis · doi:10.1287/deca.2018.0388 · Cited by 20
A strategy to improve expert technology forecasts
Savage, T., Davis, A., Fischhoff, B., & Morgan, M. G. (2021) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2021558118 · Cited by 24
“…we were guided by previous studies (27, 36, 37) finding that warnings about overconfidence (e.g., “ expand those extreme fractiles ” ) have had little effect, whereas asking people why favored answers might be wrong (28, 38) or to “ actively entertain more possibilities ” (31) can have some effect.”
BackgroundMethodForecastQA: A Question Answering Challenge for Event Forecasting with Temporal Text Data
Jin, W., Khanna, R., Kim, S., Lee, D., Morstatter, F., Galstyan, A., & Ren, X. (2021) · Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2021.acl-long.357 · Cited by 51
“Unlike the accuracy metric, the Brier score penalizes over-and under-confident fore-casts (Mellers et al., 2014) — thus the best model under each metric can be different.”
MethodChang, W., Chen, E. E., Mellers, B. A., & Tetlock, P. E. (2016) · Judgment and Decision Making · Cited by 23
Self-citationExperts and Expertise in Organizations: An Integrative Review on Individual Expertise
Rousseau, D. M., & Stouten, J. (2024) · Annual Review of Organizational Psychology and Organizational Behavior · doi:10.1146/annurev-orgpsych-020323-012717 · Cited by 33
Reimagining peer review as an expert elicitation process
Marcoci, A., Vercammen, A., Bush, M., Hamilton, D. G., Hanea, A. M., Hemming, V., Wintle, B. C., Burgman, M. A., & Fidler, F. M. (2022) · BMC Research Notes · doi:10.1186/s13104-022-06016-0 · Cited by 26
Hecksteden, A., Keller, N., Zhang, G., Meyer, T., & Hauser, T. (2023) · Sports Medicine - Open · doi:10.1186/s40798-023-00641-0 · Cited by 31
“While the assets of the latter are well supported [30, 31] (particularly when complexity is combined with sparse data, as is the case in sport) the potential contribution of expert intuition and “gut feelings” is less clear.”
BackgroundUsing the Delphi Method to Establish Expert Consensus: A Practical Guide
Jorm, A. (2025) · Advancing Methods for Interdisciplinarity in the Social Sciences · doi:10.1007/978-981-96-8357-4 · Cited by 22
Wilson, P. J., & Kiely, J. (2023) · Sports Medicine - Open · doi:10.1186/s40798-023-00629-w · Cited by 30
Metadata
- Authors
- Mellers, B., Ungar, L., Baron, J., Ramos, J., Gurcay, B., Fincher, K., Scott, S. E., Moore, D., Atanasov, P., Swift, S. A., Murray, T., Stone, E., & Tetlock, P. E.
- Affiliations
- University of Pennsylvania (US)
- Rice University (US)
- Department of Psychology, University of Pennsylvania
- University of California, Berkeley (US)
- ORCID
- Venue
- Psychological Science
- Publisher
- SAGE Publications
- Published
- 2014-03-21
- Volume
- 25
- Issue
- 5
- Pages
- 1106-1115
- ISSN
- 0956-7976 (ISSN-L)
- 1467-9280
- OpenAlex
- W2131472525
- Cited by
- 312
- References
- 42
- Citations by year
- 2026: 20 · 2025: 22 · 2024: 28 · 2023: 29 · 2022: 47 · 2021: 31 · 2020: 14 · 2019: 19 · 2018: 25 · 2017: 28 · 2016: 19 · 2015: 19 · 2014: 11
- FWCI
- 32.31
- Type
- journal article
- Language
- en
- Open access
- green
- www.escholarship.org
- Licence
- journals.sagepub.com/page/policies/text-and-data-mining-license (tdm)
- Locations
- Psychological Science (publishedVersion)
- PubMed (publishedVersion)
- eScholarship (California Digital Library) (submittedVersion)
- Funders
- U.S. Department of the Interior
- IBM Center for the Business of Government
- Intelligence Advanced Research Projects Activity
- Interior Business Center
- Topics
- Social Sciences › Decision Sciences › Management Science and Operations Research › Forecasting Techniques and Applications
- Social Sciences › Decision Sciences › General Decision Sciences › Decision-Making and Behavioral Economics
- Social Sciences › Economics, Econometrics and Finance › Economics and Econometrics › Sports Analytics and Performance
- Keywords
- geopolitical forecasting, forecasting tournaments, forecast accuracy, teaming, tracking, calibration, resolution, cognitive biases, reference classes, behavioral interventions, forecast aggregation, heuristics
- MeSH terms
- Adult, Algorithms, Female, Forecasting, Humans, Interpersonal Relations, Judgment, Male, Probability, Psychological Techniques, Social Behavior, Bias
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-10-01, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-01 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.