Objective Difficulty-Skill Balance Impacts Perceived Balance but Not Behaviour: A Test of Flow and Self-Determination Theory Predictions
The DOI matches its Crossref record; 14 of the 14 references it cites were found in a registry or an index.
Abstract
Flow and self-determination theory predict that game difficulty in balance with player skill maximises enjoyment and engagement, mediated by attentive absorption or competence. Yet recent evidence and methodological concerns are challenging this view, and key theoretical predictions have remained untested, importantly which objective difficulty-skill ratio is perceived as most balanced. To test these, we ran a preregistered study (n=309) using a Go-like 2-player game with an AI opponent, randomly assigning players to one of three objective difficulty-skill ratios (AI plays to win, draw, or lose) over five matches. The AI successfully manipulated objective balance, with the draw condition perceived as most balanced. However, balance did not impact play behaviour, nor did we find the predicted uniform 'inverted-U' between balance and positive play experiences. Importantly, we found both theories too underspecified to severely test. We conclude that balance and competence likely matter less for behavioural engagement than commonly held. We propose alternative factors such as player appraisals, novelty, and progress, and debate the value and challenges of theory-testing work in games HCI.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011) · Psychological Science · doi:10.1177/0956797611417632 · Cited by 6,567
A Survey of Monte Carlo Tree Search Methods
Browne, C. B., Powley, E., Whitehouse, D., Lucas, S. M., Cowling, P. I., Rohlfshagen, P., Tavener, S., Perez, D., Samothrakis, S., & Colton, S. (2012) · IEEE Transactions on Computational Intelligence and AI in Games · doi:10.1109/tciaig.2012.2186810 · Cited by 2,970 · cited 1×
“MCTS algorithms build a branching decision tree to some n steps into the future to determine the optimal decision per its reward function [12].”
BackgroundSweetser (2005) · doi:10.1145/1077246.1077253 · Cited by 2,124 · cited 3×
“” [78] Grounded in ow theory [17], early game and media entertainment theory suggested that “it is a balance between individual di erences in cognitive abilities and media message challenges that explains enjoyment of media use” [71].”
BackgroundFlow and the Foundations of Positive Psychology: The Collected Works of Mihaly Csikszentmihalyi
Csikszentmihalyi, M. (2014) · doi:10.1007/978-94-017-9088-8 · Cited by 1,653
Measurement Schmeasurement: Questionable Measurement Practices and How to Avoid Them
Flake, J. K., & Fried, E. I. (2020) · Advances in Methods and Practices in Psychological Science · doi:10.1177/2515245920952393 · Cited by 969 · cited 3×
“The first is questionable measurement practices [29]: studies have used such a variety of scales, measures, and calculations for challenge and balance that the cited meta-analysis found them simply “too heterogeneous to meaningfully aggregate” [30].”
BackgroundMethodInfluentialInformation-seeking, curiosity, and attention: computational and neural mechanisms
Gottlieb, J., Oudeyer, P.-Y., Lopes, M., & Baranes, A. (2013) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2013.09.001 · Cited by 784
Nelson, L. D., Simmons, J., & Simonsohn, U. (2018) · Annual Review of Psychology · doi:10.1146/annurev-psych-122216-011836 · Cited by 548 · cited 1×
“Researchers have therefore been advancing methodological reforms for more severe, transparent, and reproducible testing of hypotheses [54, 57].”
BackgroundAddressing the theory crisis in psychology
Oberauer, K., & Lewandowsky, S. (2019) · Psychonomic Bulletin & Review · doi:10.3758/s13423-019-01645-2 · Cited by 462
Abeele, V. V., Spiel, K., Nacke, L., Johnson, D., & Gerling, K. (2020) · International Journal of Human-Computer Studies · doi:10.1016/j.ijhcs.2019.102370 · Cited by 411 · cited 1×
“For proximal player experiences, we used the Player Experience Inventory (PXI) as the most well-validated, publicly available, and short recent player experience instrument [1].”
MethodLikelihood of Null Effects of Large NHLBI Clinical Trials Has Increased over Time
Kaplan, R. M., & Irvin, V. L. (2015) · PLOS ONE · doi:10.1371/journal.pone.0132382 · Cited by 356 · cited 1×
“But we know empirically that introducing preregistration substantially increases the proportion of reported null results in a given body of literature [40].”
Background
Research works printed but not linked: 63
- Sami Abuhamdeh . 2020. Investigating the “ ow ” experience: Key conceptual and operational issues. Frontiers in psychology 11 ( 2020 ), 158. Sami Abuhamdeh. 2020. Investigating the “ ow” experience: Key conceptual and operational issues. Frontiers in psychology 11 ( 2020 ), 158.
- 10.1007/s11031-011-9252-7 · DOI
- 10.1177/0146167211427147 · DOI · Paper page
- Sami Abuhamdeh Mihaly Csikszentmihalyi and Baland Jalal. 2015. Enjoying the possibility of defeat: Outcome uncertainty suspense and intrinsic motivation. Motivation and Emotion 39 1 ( 2015 ) 1-10. Sami Abuhamdeh Mihaly Csikszentmihalyi and Baland Jalal. 2015. Enjoying the possibility of defeat: Outcome uncertainty suspense and intrinsic motivation. Motivation and Emotion 39 1 ( 2015 ) 1-10. · DOI
- Nick Ballou. 2023. A Manifesto for More Productive Psychological Games Research. Games: Research and Practice 1 1 ( 2023 ). Nick Ballou. 2023. A Manifesto for More Productive Psychological Games Research. Games: Research and Practice 1 1 ( 2023 ). · DOI
- 10.1145/3491101.3503702 · DOI · Paper page
- 10.55969/paradigmplus.v1n1a2 · DOI · Paper page
- D. E. Berlyne . 1960. Con ict, arousal, and curiosity . McGraw-Hill , New York . D. E. Berlyne. 1960. Con ict, arousal, and curiosity. McGraw-Hill, New York.
- Mark Blythe and Marc Hassenzahl . 2003. The semantics of fun: Di erentiating enjoyable experiences . In Funology . Springer , 91-100. Mark Blythe and Marc Hassenzahl. 2003. The semantics of fun: Di erentiating enjoyable experiences. In Funology. Springer, 91-100.
- Jenova Chen. 2007. Flow in games (and everything else). Commun. ACM 50 4 ( 2007 ) 31-34. Jenova Chen. 2007. Flow in games (and everything else). Commun. ACM 50 4 ( 2007 ) 31-34. · DOI · Paper page
- 10.1145/3173574.3173715 · DOI · Paper page
- 10.1145/2793107.2793147 · DOI · Paper page
- Mihaly Csikszentmihalyi . 1991 . Flow: The psychology of optimal experience . HarperPerennial New York . Mihaly Csikszentmihalyi. 1991. Flow: The psychology of optimal experience. HarperPerennial New York.
- Mihaly Csikszentmihalyi and Jeanne Nakamura. 2010. E ortless attention in everyday life: A systematic phenomenology. E ortless attention: A new perspective in the cognitive science of attention and action ( 2010 ) 179-189. Mihaly Csikszentmihalyi and Jeanne Nakamura. 2010. E ortless attention in everyday life: A systematic phenomenology. E ortless attention: A new perspective in the cognitive science of attention and action ( 2010 ) 179-189. · DOI · Paper page
- Joe Cutting , Sebastian Deterding , Simon Demediuk , and Nick Sephton . 2023. Di culty-skill balance does not a ect engagement and enjoyment: a pre-registered study using arti cial intelligence-controlled di culty . Royal Society Open Science 10, 2 ( 2023 ), 220274. Joe Cutting, Sebastian Deterding, Simon Demediuk, and Nick Sephton. 2023. Di culty-skill balance does not a ect engagement and enjoyment: a pre-registered study using arti cial intelligence-controlled di culty. Royal Society Open Science 10, 2 ( 2023 ), 220274.
- Joe Cutting David Gundry and Paul Cairns. 2019. Busy doing nothing? What do players do in idle games? International journal of human-computer studies 122 ( 2019 ) 133-144. Joe Cutting David Gundry and Paul Cairns. 2019. Busy doing nothing? What do players do in idle games? International journal of human-computer studies 122 ( 2019 ) 133-144. · DOI
- Edward L. Deci and Richard M . Ryan . 1985 . Intrinsic Motivation and Self-Determination in Human Behavior. Plenum Press , New York. Edward L. Deci and Richard M. Ryan. 1985. Intrinsic Motivation and Self-Determination in Human Behavior. Plenum Press, New York.
- 10.1109/CIG.2017.8080415 · DOI · Paper page
- 10.1016/j.ijhcs.2019.102383 · DOI · Paper page
- 10.1145/3027063.3053209 · DOI · Paper page
- 10.3389/fpsyg.2022.924953 · DOI · Paper page
- Florian Echtler and Maximilian Häußler. 2018. Open source open science and the replication crisis in HCI. In Extended abstracts of the 2018 CHI conference on human factors in computing systems. 1-8. Florian Echtler and Maximilian Häußler. 2018. Open source open science and the replication crisis in HCI. In Extended abstracts of the 2018 CHI conference on human factors in computing systems. 1-8. · DOI
- 10.1080/17439760.2014.967799 · DOI · Paper page
- Tracy Fullerton . 2008 . Game Design Workshop: A Playcentric Approach to Creating Innovative Games. Morgan Kaufman, Amsterdam et al. Tracy Fullerton. 2008. Game Design Workshop: A Playcentric Approach to Creating Innovative Games. Morgan Kaufman, Amsterdam et al. · DOI
- Ralf Herbrich , Tom Minka , and Thore Graepel . 2006. TrueSkill™ : A Bayesian Skill Rating System . In Advances in Neural Information Processing Systems , Vol. 19 . MIT Press . Ralf Herbrich, Tom Minka, and Thore Graepel. 2006. TrueSkill™ : A Bayesian Skill Rating System. In Advances in Neural Information Processing Systems, Vol. 19. MIT Press.
- Susan A Jackson and Robert C Eklund . 2002 . Assessing ow in physical activity: the ow state scale-2 and dispositional ow scale-2 . Journal of sport & exercise psychology 24 , 2 ( 2002 ). Susan A Jackson and Robert C Eklund. 2002. Assessing ow in physical activity: the ow state scale-2 and dispositional ow scale-2. Journal of sport & exercise psychology 24, 2 ( 2002 ).
- Susan A Jackson , Andrew J Martin , and Robert C Eklund . 2008 . Long and short measures of ow: The construct validity of the FSS-2, DFS-2, and new brief counterparts . Journal of Sport and Exercise Psychology 30 , 5 ( 2008 ), 561-587. Susan A Jackson, Andrew J Martin, and Robert C Eklund. 2008. Long and short measures of ow: The construct validity of the FSS-2, DFS-2, and new brief counterparts. Journal of Sport and Exercise Psychology 30, 5 ( 2008 ), 561-587.
- Kyros Jalife , Casper Harteveld , and Christo er Holmgård . 2021 . From Flow to Fuse: A Cognitive Perspective . Proceedings of the ACM on Human-Computer Interaction 5, CHI PLAY ( 2021 ), 1-30 . Kyros Jalife, Casper Harteveld, and Christo er Holmgård. 2021. From Flow to Fuse: A Cognitive Perspective. Proceedings of the ACM on Human-Computer Interaction 5, CHI PLAY ( 2021 ), 1-30.
- Daniel Johnson , M John Gardner , and Ryan Perry . 2018. Validation of two game experience scales: the player experience of need satisfaction (PENS) and game experience questionnaire (GEQ) . International Journal of Human-Computer Studies 118 ( 2018 ) , 38 - 46 . Daniel Johnson, M John Gardner, and Ryan Perry. 2018. Validation of two game experience scales: the player experience of need satisfaction (PENS) and game experience questionnaire (GEQ). International Journal of Human-Computer Studies 118 ( 2018 ), 38-46.
- Celeste Kidd and Benjamin Y . Hayden . 2015 . The Psychology and Neuroscience of Curiosity. Neuron 88, 3 ( 2015 ), 449-460. Celeste Kidd and Benjamin Y. Hayden. 2015. The Psychology and Neuroscience of Curiosity. Neuron 88, 3 ( 2015 ), 449-460.
- Madison Klarkowski , Daniel Johnson , Peta Wyeth , Mitchell McEwan , Cody Phillips , and Simon Smith . 2016 . Operationalising and evaluating sub-optimal and optimal play experiences through challenge-skill manipulation . In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems. ACM, 5583-5594 . Madison Klarkowski, Daniel Johnson, Peta Wyeth, Mitchell McEwan, Cody Phillips, and Simon Smith. 2016. Operationalising and evaluating sub-optimal and optimal play experiences through challenge-skill manipulation. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems. ACM, 5583-5594.
- Christoph Klimmt . 2006. Computerspielen als Handlung: Dimensionen und Determinanten des Erlebens interaktiver Unterhaltungsangebote . Vol. 2 . Herbert von Halem Verlag . Christoph Klimmt. 2006. Computerspielen als Handlung: Dimensionen und Determinanten des Erlebens interaktiver Unterhaltungsangebote. Vol. 2. Herbert von Halem Verlag.
- Bart P Knijnenburg Martijn C Willemsen Zeno Gantner Hakan Soncu and Chris Newell. 2012. Explaining the user experience of recommender systems. User modeling and user-adapted interaction 22 4 ( 2012 ) 441-504. Bart P Knijnenburg Martijn C Willemsen Zeno Gantner Hakan Soncu and Chris Newell. 2012. Explaining the user experience of recommender systems. User modeling and user-adapted interaction 22 4 ( 2012 ) 441-504. · DOI · Paper page
- 10.1145/3311350.3347148 · DOI · Paper page
- 10.1145/2470654.2470668 · DOI · Paper page
- J Derek Lomas , Kenneth Koedinger , Nirmal Patel , Sharan Shodhan , Nikhil Poonwala , and Jodi L Forlizzi . 2017 . Is di culty overrated?: The e ects of choice, novelty and suspense on intrinsic motivation in educational games . In Proceedings of the 2017 CHI conference on human factors in computing systems. ACM, 1028-1039 . J Derek Lomas, Kenneth Koedinger, Nirmal Patel, Sharan Shodhan, Nikhil Poonwala, and Jodi L Forlizzi. 2017. Is di culty overrated?: The e ects of choice, novelty and suspense on intrinsic motivation in educational games. In Proceedings of the 2017 CHI conference on human factors in computing systems. ACM, 1028-1039.
- Colleen Macklin and John Sharp . 2016. Games , Design and Play: A Detailed Approach to Iterative Game Design . Addison-Wesley , Boston, MA . Colleen Macklin and John Sharp. 2016. Games, Design and Play: A Detailed Approach to Iterative Game Design. Addison-Wesley, Boston, MA.
- Chris McEntee . 2012 . Rational Design: The Core of Rayman Origins . http://www.gamasutra.com/view/feature/167214/ rational_design_the_core_of_.php Chris McEntee. 2012. Rational Design: The Core of Rayman Origins. http://www.gamasutra.com/view/feature/167214/ rational_design_the_core_of_.php
- Talya Miron-Shatz Arthur Stone and Daniel Kahneman. 2009. Memories of yesterday's emotions: Does the valence of experience a ect the memory-experience gap? Emotion 9 6 ( 2009 ) 885. Talya Miron-Shatz Arthur Stone and Daniel Kahneman. 2009. Memories of yesterday's emotions: Does the valence of experience a ect the memory-experience gap? Emotion 9 6 ( 2009 ) 885. · DOI · Paper page
- 10.1038/s41562-016-0021 · DOI · Paper page
- Fiona Fui-Hoon Nah. 2004. A study on tolerable waiting time: how long are web users willing to wait? Behaviour & Information Technology 23 3 ( 2004 ) 153-163. Fiona Fui-Hoon Nah. 2004. A study on tolerable waiting time: how long are web users willing to wait? Behaviour & Information Technology 23 3 ( 2004 ) 153-163. · DOI · Paper page
- Nakamura, Jeanne and Csikszentmihalyi , Mihaly. 2002. The Concept of Flow . In Handbook of positive psychology . Oxford University Press , Oxford , 89-105. Nakamura, Jeanne and Csikszentmihalyi, Mihaly. 2002. The Concept of Flow. In Handbook of positive psychology. Oxford University Press, Oxford, 89-105.
- 10.1177/2515245920917961 · DOI · Paper page
- Serge Petralito , Florian Brühlmann , Glena Iten , Elisa D Mekler , and Klaus Opwis . 2017 . A good reason to die: how avatar death and high challenges enable positive experiences . In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems. ACM, 5087-5097 . Serge Petralito, Florian Brühlmann, Glena Iten, Elisa D Mekler, and Klaus Opwis. 2017. A good reason to die: how avatar death and high challenges enable positive experiences. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems. ACM, 5087-5097.
- Travis Proulx and Richard D Morey. 2021. Beyond statistical ritual: Theory in psychological science. Perspectives on Psychological Science 16 4 ( 2021 ) 671-681. Travis Proulx and Richard D Morey. 2021. Beyond statistical ritual: Theory in psychological science. Perspectives on Psychological Science 16 4 ( 2021 ) 671-681. · DOI · Paper page
- Scott Rigby and Richard M . Ryan . 2011 . Glued to Games : How Video Games Draw Us in and Hold Us Spellbound. Praeger , Santa Barbara, Denver, Oxford. Scott Rigby and Richard M. Ryan. 2011. Glued to Games: How Video Games Draw Us in and Hold Us Spellbound. Praeger, Santa Barbara, Denver, Oxford.
- Yvonne Rogers. 2012. HCI Theory: Classical Modern and Contemporary. Morgan & Claypool. Yvonne Rogers. 2012. HCI Theory: Classical Modern and Contemporary. Morgan & Claypool. · doi:10.1007/978-3-031-02197-8 · Paper page
- Richard M Ryan . 1982 . Control and information in the intrapersonal sphere: An extension of cognitive evaluation theory . Journal of personality and social psychology 43 , 3 ( 1982 ), 450. Richard M Ryan. 1982. Control and information in the intrapersonal sphere: An extension of cognitive evaluation theory. Journal of personality and social psychology 43, 3 ( 1982 ), 450.
- Richard M. Ryan and Edward L . Deci . 2017 . Self-Determination Theory: Basic Psychological Needs in Motivation, Development, and Wellness. Guilford Press , New York. Richard M. Ryan and Edward L. Deci. 2017. Self-Determination Theory: Basic Psychological Needs in Motivation, Development, and Wellness. Guilford Press, New York.
- Richard M Ryan C Scott Rigby and Andrew Przybylski. 2006. The motivational pull of video games: A self-determination theory approach. Motivation and emotion 30 4 ( 2006 ) 344-360. Richard M Ryan C Scott Rigby and Andrew Przybylski. 2006. The motivational pull of video games: A self-determination theory approach. Motivation and emotion 30 4 ( 2006 ) 344-360. · DOI · Paper page
- Ian Schreiber and Brenda Romero . 2021. Game Balance . CRC Press , Boca Raton , London, New York. Ian Schreiber and Brenda Romero. 2021. Game Balance. CRC Press, Boca Raton, London, New York.
- 10.1093/ct/14.4.328 · doi:10.1093/ct/14.4.328 · Paper page
- Patrick E Shrout and Joseph L Rodgers. 2018. Psychology science and knowledge construction: Broadening perspectives from the replication crisis. Annual review of psychology 69 ( 2018 ) 487-510. Patrick E Shrout and Joseph L Rodgers. 2018. Psychology science and knowledge construction: Broadening perspectives from the replication crisis. Annual review of psychology 69 ( 2018 ) 487-510. · DOI · Paper page
- David Silver , Aja Huang , Chris J Maddison , Arthur Guez , Laurent Sifre , George Van Den Driessche , Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, et al. 2016 . Mastering the game of Go with deep neural networks and tree search. Nature 529, 7587 ( 2016 ), 484-489. David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, et al. 2016. Mastering the game of Go with deep neural networks and tree search. Nature 529, 7587 ( 2016 ), 484-489.
- Nathan Sorenson , Philippe Pasquier , and Steve DiPaola . 2011 . A generic approach to challenge modeling for the procedural creation of video game levels . IEEE Transactions on Computational Intelligence and AI in Games 3 , 3 ( 2011 ), 229-244. Nathan Sorenson, Philippe Pasquier, and Steve DiPaola. 2011. A generic approach to challenge modeling for the procedural creation of video game levels. IEEE Transactions on Computational Intelligence and AI in Games 3, 3 ( 2011 ), 229-244.
- Alistair Sutcli e. 2009. Designing for user engagement: Aesthetic and attractive user interfaces. Synthesis lectures on human-centered informatics 2 1 ( 2009 ) 1-55. Alistair Sutcli e. 2009. Designing for user engagement: Aesthetic and attractive user interfaces. Synthesis lectures on human-centered informatics 2 1 ( 2009 ) 1-55. · DOI
- 10.1145/3034780 · DOI
- Iris van Rooij and Giosuè Baggio. 2021. Theory before the test: How to build high-verisimilitude explanatory theories in psychological science. Perspectives on Psychological Science 16 4 ( 2021 ) 682-697. Iris van Rooij and Giosuè Baggio. 2021. Theory before the test: How to build high-verisimilitude explanatory theories in psychological science. Perspectives on Psychological Science 16 4 ( 2021 ) 682-697. · DOI · Paper page
- Max L Wilson , Ed H Chi , Stuart Reeves , and David Coyle . 2014 . RepliCHI: the workshop II. In CHI'14 Extended Abstracts on Human Factors in Computing Systems. 33-36 . Max L Wilson, Ed H Chi, Stuart Reeves, and David Coyle. 2014. RepliCHI: the workshop II. In CHI'14 Extended Abstracts on Human Factors in Computing Systems. 33-36.
- Max L Wilson , Wendy Mackay , Ed Chi, Michael Bernstein , Dan Russell, and Harold Thimbleby. 2011 . RepliCHI-CHI should be replicating and validating results more: discuss. In CHI'11 Extended Abstracts on Human Factors in Computing Systems . 463-466. Max L Wilson, Wendy Mackay, Ed Chi, Michael Bernstein, Dan Russell, and Harold Thimbleby. 2011. RepliCHI-CHI should be replicating and validating results more: discuss. In CHI'11 Extended Abstracts on Human Factors in Computing Systems. 463-466.
- Georgios N Yannakakis and Julian Togelius . 2011 . Experience-driven procedural content generation . IEEE Transactions on A ective Computing 2 , 3 ( 2011 ), 147-161. Georgios N Yannakakis and Julian Togelius. 2011. Experience-driven procedural content generation. IEEE Transactions on A ective Computing 2, 3 ( 2011 ), 147-161.
- Mohammad Zohaib. 2018. Dynamic di culty adjustment (DDA) in computer games: A review. Advances in HumanComputer Interaction 2018 ( 2018 ). Mohammad Zohaib. 2018. Dynamic di culty adjustment (DDA) in computer games: A review. Advances in HumanComputer Interaction 2018 ( 2018 ). · DOI · Paper page
- Robert Zubek . 2020. Elements of Game Design . MIT Press . Robert Zubek. 2020. Elements of Game Design. MIT Press.
Cited by
Self-Determination Theory and HCI Games Research: Unfulfilled Promises and Unquestioned Paradigms
Tyack, A., & Mekler, E. D. (2024) · ACM Transactions on Computer-Human Interaction · doi:10.1145/3673230 · Cited by 57
Metadata
- Authors
- Deterding, S., & Cutting, J.
- Venue
- Proceedings of the ACM on Human-Computer Interaction
- Published
- 2023-09-29
- Volume
- 7
- Issue
- CHI PLAY
- Pages
- 1179-1205
- ISSN
- 2573-0142
- Cited by
- 12
- References
- 14
- Language
- en
- Funders
- EPSRC (EP/M023265/1)
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.