The Design of Ecological Momentary Assessment Technologies
Open accessThe DOI matches its Crossref record; 63 of the 63 references it cites were found in a registry or an index.
Abstract
Ecological Momentary Assessment (EMA) methods and technologies, designed to support the self-report of experience in the moment of daily life, have long been considered poised to revolutionize human-centred research, the practice of design and mental healthcare. The history of EMA is inextricably linked to technology, and mobile devices embody many of the characteristics required to support these methods. However, significant barriers to the design and adoption of these systems remain, including challenges of user engagement, reporting burden, data validity and honest disclosure. While prior research has examined the feasibility of a variety of EMA systems, few reviews have attended to their design. Through inter-disciplinary narrative literature review (n = 342), this paper presents a characterization of the EMA technology design space, drawing upon a diverse set of literatures, contexts, applications and demographic groups. This paper describes the options and strategies available to the EMA systems designer, with an eye towards supporting the design and deployment of EMA technologies for research and clinical practice.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
The theory of planned behavior
Ajzen, I. (1991) · Organizational Behavior and Human Decision Processes · doi:10.1016/0749-5978(91)90020-t · Cited by 90,587
Salovey, P., & Mayer, J. D. (1990) · Imagination, Cognition and Personality · doi:10.2190/dugg-p24e-52wk-6cdg · Cited by 10,484
A social-cognitive approach to motivation and personality.
Dweck, C. S., & Leggett, E. L. (1988) · Psychological Review · doi:10.1037/0033-295x.95.2.256 · Cited by 9,745 · cited 2×
“Otherpertinenttheoriesincludesocial cognitive theories, such as Dweck & Leggett’s social cognitive theory (Dweck & Leggett, 1988),Bandura’stheoryofself-efficacy(Bandura,1977)anddual-processtheoriesofcog-nition including the Heuristic-Systemic Model of Social Information Processing (HSM) (Todorov et…”
BackgroundMethodThe Elaboration Likelihood Model of Persuasion
Petty, R. E., & Cacioppo, J. T. (1986) · Advances in Experimental Social Psychology · doi:10.1016/s0065-2601(08)60214-2 · Cited by 8,822 · cited 2×
“…Heuristic-Systemic Model of Social Information Processing (HSM) (Todorov et al. , 2002) and the Elaboration-Likelihood Model of Persuasion (ELM) (Petty & Cacioppo, 1986). prompting intervals between 30 minutes and 3 hours in duration, although not the exact time of assessment, and instructed…”
BackgroundThe Health Belief Model: a decade later.
Janz, N. K., & Becker, M. H. (1984) · Health Education Quarterly · doi:10.1177/109019818401100101 · Cited by 8,437 · cited 1×
“…to choose 1 The most commonly cited behaviour change theories include Ajzen’s Theory of Planned Behaviour (Ajzen, 1991), the Transtheoretical Model of Behavioural Change (Prochaska & Velicer, 1997), the Fogg Behavior Model (FBM) (Fogg, 2009) and the HealthBeliefModel(HBM)(Janz&Becker,1984).”
BackgroundThe Transtheoretical Model of Health Behavior Change.
Prochaska, J. O., & Velicer, W. F. (1997) · American Journal of Health Promotion · doi:10.4278/0890-1171-12.1.38 · Cited by 7,782
Ecological momentary assessment
Shiffman, S., Stone, A. A., & Hufford, M. R. (2008) · Annual Review of Clinical Psychology · doi:10.1146/annurev.clinpsy.3.022806.091415 · Cited by 6,454 · cited 10×
“…the mood congruent memory effect, duration neglect (Ebner– Priemer & Trull, 2009a b), implicit theories and motivations (Barrett & Barrett, 2001, Shiffman et al. , 2008), the ‘effort-after-meaning’ effect, ‘telescoping’ bias (Stone et al. , 1999), the ‘focusing illusion’ (Kahneman & Krueger,…”
BackgroundInfluentialInclusion of Other in the Self Scale and the structure of interpersonal closeness.
Aron, A., Aron, E. N., & Smollan, D. (1992) · Journal of Personality and Social Psychology · doi:10.1037/0022-3514.63.4.596 · Cited by 5,157 · cited 1×
“…representations of scales, including the paper-based interpersonal grid (Fournier et al. , 2008) and Inclusion of Other in the Self (IOS) scale (Aron et al. , 1992), interactive digital presentation of the circumplex model of emotion (MoodMap) (Morris et al. , Interacting with Computers, Vol.…”
MethodA Survey Method for Characterizing Daily Life Experience: The Day Reconstruction Method
Kahneman, D., Krueger, A. B., Schkade, D. A., Schwarz, N., & Stone, A. A. (2004) · Science · doi:10.1126/science.1103572 · Cited by 3,422 · cited 2×
“Completed once at the end of the day, this measure combines reconstruction of the day’s events in diary form with items related to respondents’ emotions, activities and circumstances (Kahneman et al. , 2004).”
BackgroundDevelopments in the Measurement of Subjective Well-Being
Kahneman, D., & Krueger, A. B. (2006) · Journal of Economic Perspectives · doi:10.1257/089533006776526030 · Cited by 3,177
The Dynamic Self-Concept: A Social Psychological Perspective
Markus, H., & Wurf, E. (1987) · Annual Review of Psychology · doi:10.1146/annurev.ps.38.020187.001503 · Cited by 3,089
A wandering mind is an unhappy mind
Killingsworth, M. A., & Gilbert, D. T. (2010) · Science · doi:10.1126/science.1192439 · Cited by 2,575
Mediators and Mechanisms of Change in Psychotherapy Research
Kazdin, A. E. (2007) · Annual Review of Clinical Psychology · doi:10.1146/annurev.clinpsy.3.022806.091432 · Cited by 2,504 · cited 2×
“Kazdin has remarked that despite ‘decades of psychotherapy research, we cannot provide an evidence-based explanation for how or why even our most well studied interventions produce change’ (Kazdin, 2007).”
BackgroundA behavior model for persuasive design
Fogg, B. (2009) · Proceedings of the 4th International Conference on Persuasive Technology · doi:10.1145/1541948.1541999 · Cited by 2,328 · cited 2×
“Morris et al. allowed participants to choose 1 The most commonly cited behaviour change theories include Ajzen’s Theory of Planned Behaviour (Ajzen, 1991), the Transtheoretical Model of Behavioural Change (Prochaska & Velicer, 1997), the Fogg Behavior Model (FBM) (Fogg, 2009) and the HealthBeliefModel(HBM)(Janz&Becker…”
BackgroundWriting narrative literature reviews for peer-reviewed journals: secrets of the trade
Green, B. N., Johnson, C. D., & Adams, A. (2006) · Journal of Chiropractic Medicine · doi:10.1016/s0899-3467(07)60142-6 · Cited by 2,224 · cited 1×
“…methodology, enabling a broad and interdisciplinary perspective as well as insight into the history and development of EMA across research, design and clinical practice, in contrast with the precise and typically quantitative focus of other systematic review methodologies (Green et al. , 2006).”
MethodRebooting psychotherapy research and practice to reduce the burden of mental illness
Kazdin, A. E., & Blase, S. L. (2011) · Perspectives on Psychological Science · doi:10.1177/1745691610393527 · Cited by 1,254 · cited 1×
“Studies of telephone-administered psychotherapies, for example, have revealed low attrition rates (Kazdin & Blase, 2011) and a ‘dose-response relationship’ (Lazev et al. , 2004).”
BackgroundEmotional experience improves with age: Evidence based on over 10 years of experience sampling
Carstensen, L. L., Turan, B., Scheibe, S., Ram, N., Ersner-Hershfield, H., Samanez-Larkin, G. R., Brooks, K. P., & Nesselroade, J. R. (2011) · Psychology and Aging · doi:10.1037/a0021285 · Cited by 1,223 · cited 7×
“In another example, almost 200 participants were asked to complete assessments 5 times per day for 1 week every 5 years as part of a longitudinal study of emotional wellbeing (Carstensen et al., 2011).”
BackgroundInfluentialHektner, J., Schmidt, J., & Csikszentmihalyi, M. (2007) · doi:10.4135/9781412984201 · Cited by 1,174
Smartphones for smarter delivery of mental health programs: A systematic review
Donker, T., Petrie, K., Proudfoot, J., Clarke, J., Birch, M.-R., & Christensen, H. (2013) · Journal of Medical Internet Research · doi:10.2196/jmir.2791 · Cited by 1,136
Language use of depressed and depression-vulnerable college students
Rude, S., Gortner, E.-M., & Pennebaker, J. (2004) · Cognition & Emotion · doi:10.1080/02699930441000030 · Cited by 1,018
Patient non-compliance with paper diaries
Stone, A. A., Shiffman, S., Schwartz, J. E., Broderick, J. E., & Hufford, M. R. (2002) · BMJ · doi:10.1136/bmj.324.7347.1193 · Cited by 810
Capturing momentary, self-report data: A proposal for reporting guidelines
Stone, A. A., & Shiffman, S. (2002) · Annals of Behavioral Medicine · doi:10.1207/s15324796abm2403_09 · Cited by 807 · cited 4×
“Reporting guidelines for momentary assessment stress the ‘mountain of data’ that these techniques can produce (Stone & Shiffman, 2002).”
BackgroundInfluentialTrull, T. J., & Ebner-Priemer, U. (2013) · Annual Review of Clinical Psychology · doi:10.1146/annurev-clinpsy-050212-185510 · Cited by 761
Microrandomized trials: An experimental design for developing just-in-time adaptive interventions.
Klasnja, P., Hekler, E. B., Shiffman, S., Boruvka, A., Almirall, D., Tewari, A., & Murphy, S. A. (2015) · Health Psychology · doi:10.1037/hea0000305 · Cited by 671 · cited 1×
“While others have focused on the use of smartphones sensor data to facilitate ‘just-in-time interventions’ (Klasnja et al. , 2015).”
BackgroundA self-determination theory approach to psychotherapy: The motivational basis for effective change.
Ryan, R. M., & Deci, E. L. (2008) · Canadian Psychology / Psychologie canadienne · doi:10.1037/a0012753 · Cited by 629 · cited 1×
“…the collection of ecologically valid and longitudinal data, increasing patient participation, supporting honest disclosure and fostering trust between patients and health professionals (Baldassano, 2005, König et al. , 2014, Rains & Keating, 2011, Ryan & Deci, 2008, Wichers et al. , 2011).”
Background(2005) · doi:10.1093/acprof:oso/9780198567523.003.0011 · Cited by 585 · cited 1×
“…to this view, momentary reports originate with an experiencing self, ‘a qualitatively different conscious self’ which ‘barely has time to exist’ (Kahneman & Riis, 2005), and which is ‘functionally and neuroanatomically different from the “remembering” and “believing” selves measured through…”
BackgroundFroehlich (2009) · doi:10.1145/1518701.1518861 · Cited by 563
Ecological momentary assessment of mood disorders and mood dysregulation
Ebner-Priemer, U. W., & Trull, T. J. (2009) · Psychological Assessment · doi:10.1037/a0017075 · Cited by 530
Trull, T. J., Solhan, M. B., Tragesser, S. L., Jahng, S., Wood, P. K., Piasecki, T. M., & Watson, D. (2008) · Journal of Abnormal Psychology · doi:10.1037/a0012532 · Cited by 503
Ecological momentary assessment research in behavioral medicine
Smyth, J. M., & Stone, A. A. (2003) · Journal of Happiness Studies · doi:10.1023/a:1023657221954 · Cited by 439
Geschwind, N., Peeters, F., Drukker, M., van Os, J., & Wichers, M. (2011) · Journal of Consulting and Clinical Psychology · doi:10.1037/a0024595 · Cited by 433
The Experience Sampling Method on Mobile Devices
van Berkel, N., Ferreira, D., & Kostakos, V. (2018) · ACM Computing Surveys · doi:10.1145/3123988 · Cited by 432
AWARE: Mobile Context Instrumentation Framework
Ferreira, D., Kostakos, V., & Dey, A. K. (2015) · Frontiers in ICT · doi:10.3389/fict.2015.00006 · Cited by 409 · cited 1×
“Many researchers may feel that they no longer possess the time, technical knowledge or design skills to keep pace with such developments (Ferreira et al. , 2015, Schueller et al. , 2014).”
BackgroundThieme, A., Belgrave, D., & Doherty, G. (2020) · ACM Transactions on Computer-Human Interaction · doi:10.1145/3398069 · Cited by 406 · cited 1×
“Such perceptions are unlikely to be tempered by the growing adoption of adaptive, machine learning and sensor-driven systems which invite questions of trust, acceptability and the ethical deployment of technology (Thieme et al. , 2020).”
BackgroundSelf-citationSerre, F., Fatseas, M., Swendsen, J., & Auriacombe, M. (2015) · Drug and Alcohol Dependence · doi:10.1016/j.drugalcdep.2014.12.024 · Cited by 389
Mobile Therapy: Case Study Evaluations of a Cell Phone Application for Emotional Self-Awareness
Morris, M. E., Kathawala, Q., Leen, T. K., Gorenstein, E. E., Guilak, F., Labhard, M., & Deleeuw, W. (2010) · Journal of Medical Internet Research · doi:10.2196/jmir.1371 · Cited by 384 · cited 6×
“prompting intervals between 30 minutes and 3 hours in duration, although not the exact time of assessment, and instructed users to ‘ignore prompts that could disrupt their work or personal communication’ (Morris et al., 2010).”
BackgroundMethodInfluentialBeyond total capture: a constructive critique of lifelogging
Sellen, A. J., & Whittaker, S. (2010) · Communications of the ACM · doi:10.1145/1735223.1735243 · Cited by 377
Positive Computing: Technology for Wellbeing and Human Potential
Calvo, R. A., & Peters, D. (2014) · doi:10.7551/mitpress/9764.001.0001 · Cited by 376
A Review of Mobile HCI Research Methods
Kjeldskov, J., & Graham, C. (2003) · Lecture Notes in Computer Science · doi:10.1007/978-3-540-45233-1_23 · Cited by 375
Solhan, M. B., Trull, T. J., Jahng, S., & Wood, P. K. (2009) · Psychological Assessment · doi:10.1037/a0016869 · Cited by 362
MAHI: investigation of social scaffolding for reflective thinking in diabetes management
Mamykina, L., Mynatt, E., Davidson, P., & Greenblatt, D. (2008) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/1357054.1357131 · Cited by 355 · cited 1×
“Initial attempts by HCI researchers to support users’ acquisition of such skills are highly pertinent to the design of EMA technologies and merit wider consideration (Mamykina et al. , 2008, van Gennip et al. , 2015).”
BackgroundWen, C. K. F., Schneider, S., Stone, A. A., & Spruijt-Metz, D. (2017) · Journal of Medical Internet Research · doi:10.2196/jmir.6641 · Cited by 350 · cited 9×
“A final category of reviews pertains to the use of EMA among particular population groups, such as children and adolescents (Heron et al. , 2017, Wen et al. , 2017).”
BackgroundInfluentialIgnorance of Hedonic Adaptation to Hemodialysis: A Study Using Ecological Momentary Assessment.
Riis, J., Loewenstein, G., Baron, J., Jepson, C., Fagerlin, A., & Ubel, P. A. (2005) · Journal of Experimental Psychology: General · doi:10.1037/0096-3445.134.1.3 · Cited by 303 · cited 2×
“…view, momentary reports originate with an experiencing self, ‘a qualitatively different conscious self’ which ‘barely has time to exist’ (Kahneman & Riis, 2005), and which is ‘functionally and neuroanatomically different from the “remembering” and “believing” selves measured through…”
BackgroundRizvi, S. L., Dimeff, L. A., Skutch, J., Carroll, D., & Linehan, M. M. (2011) · Behavior Therapy · doi:10.1016/j.beth.2011.01.003 · Cited by 271 · cited 2×
“It has therefore long been claimed that mobile technologies are ‘poised to become the most powerful form of media to influence clinical practice’ (Rizvi et al. , 2011), research and the design of technology itself.”
BackgroundBurke, L. E., Shiffman, S., Music, E., Styn, M. A., Kriska, A., Smailagic, A., Siewiorek, D., Ewing, L. J., Chasens, E., French, B., et al. (2017) · Journal of Medical Internet Research · doi:10.2196/jmir.7138 · Cited by 267 · cited 2×
“…a new series of design challenges related to the timing and scheduling of EMA prompts, technical concerns of network connectivity and data loss (Burke, 2017), user attentiveness to mobile notifications, optimal notification delivery, interruptibility (Ghosh et al. , 2017a) and the…”
BackgroundEchoes from the past: how technology mediated reflection improves well-being
Isaacs, E., Konrad, A., Walendowski, A., Lennig, T., Hollis, V., & Whittaker, S. (2013) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/2470654.2466137 · Cited by 251
An overview of self-monitoring research in assessment and treatment.
Korotitsch, W. J., & Nelson-Gray, R. O. (1999) · Psychological Assessment · doi:10.1037/1040-3590.11.4.415 · Cited by 241
Evidence-based apps? A review of mental health mobile applications in a psychotherapy context
Lui, J. H. L., Marcus, D. K., & Barry, C. T. (2017) · Professional Psychology Research and Practice · doi:10.1037/pro0000122 · Cited by 217 · cited 2×
“, 2017), acquisition of sensitive mental health data, location tracking (Lui et al., 2017) and monitoring of personal and stigmatised behaviours (e.”
BackgroundRevere, D., & Dunbar, P. J. (2001) · Journal of the American Medical Informatics Association · doi:10.1136/jamia.2001.0080062 · Cited by 208 · cited 1×
“Major threads of research within both HCI and psychology are now centred upon the development of technologies to support positive affect, happiness, health and wellbeing through assessment (Calvo & Peters, 2014, Donker et al. , 2013, Kennedy et al. , 2012, Revere & Dunbar, 2001).”
BackgroundWhat Makes Smartphone Use Meaningful or Meaningless?
Lukoff, K., Yu, C., Kientz, J., & Hiniker, A. (2018) · Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies · doi:10.1145/3191754 · Cited by 205 · cited 1×
“Another study, of meaningful smartphone usage, tailored the delivery of EMA prompts according to the date, time and duration of participants’ mobile app usage (Lukoff et al. , 2018).”
BackgroundAsselbergs, J., Ruwaard, J., Ejdys, M., Schrader, N., Sijbrandij, M., & Riper, H. (2016) · Journal of Medical Internet Research · doi:10.2196/jmir.5505 · Cited by 192
Bell, I. H., Lim, M. H., Rossell, S. L., & Thomas, N. (2017) · Psychiatric Services · doi:10.1176/appi.ps.201600523 · Cited by 189 · cited 2×
“One recent review presents both EMA and EMI methodologies as acceptable and feasible approaches to the treatment of psychotic disorders (Bell et al. , 2017).”
BackgroundRunyan, J. D., Steenbergh, T. A., Bainbridge, C., Daugherty, D. A., Oke, L., & Fry, B. N. (2013) · PLoS ONE · doi:10.1371/journal.pone.0071325 · Cited by 179 · cited 2×
“Open-ended questions support greater freedom of expression and allow users to provide additional forms of information and sources of insight, including pertinent features of context (Runyan et al. , 2013).”
BackgroundMethodAmbulatory Assessment: An Innovative and Promising Approach for Clinical Psychology
Ebner-Priemer, U. W., & Trull, T. J. (2009) · European Psychologist · doi:10.1027/1016-9040.14.2.109 · Cited by 173
Whittaker, R., Maddison, R., McRobbie, H., Bullen, C., Denny, S., Dorey, E., Ellis-Pegler, M., van Rooyen, J., & Rodgers, A. (2008) · Journal of Medical Internet Research · doi:10.2196/jmir.1007 · Cited by 171 · cited 1×
“A video-based mobile intervention for smoking cessation led 9 of 15 participants to quit smoking during the program and half of the remainder to reduce their smoking behaviour (Whittaker et al. , 2008).”
MethodEcological momentary assessment in aging research: A critical review
Cain, A. E., Depp, C. A., & Jeste, D. V. (2009) · Journal of Psychiatric Research · doi:10.1016/j.jpsychires.2009.01.014 · Cited by 164 · cited 1×
“Both HCI and psychology researchers have previously expressed the need for, and conducted, reviews of the EMA literature (Cain et al. , 2009, Cohn et al. , 2011, Cutmore & James, 2007, Ebner-Priemer & Kubiak, 2007, Fischer, 2009, Kjeldskov & Graham, 2003, König et al. , 2014, Trull & Ebner-Priemer,…”
BackgroundUse of ecological momentary assessment (EMA) in non-suicidal self-injury (NSSI): a systematic review
Rodríguez-Blanco, L., Carballo, J. J., & Baca-García, E. (2018) · Psychiatry Research · doi:10.1016/j.psychres.2018.02.051 · Cited by 140
Gamification of Mobile Experience Sampling Improves Data Quality and Quantity
van Berkel, N., Goncalves, J., Hosio, S., & Kostakos, V. (2017) · Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies · doi:10.1145/3130972 · Cited by 95
Bos, F. M., Schoevers, R. A., & aan het Rot, M. (2015) · European Neuropsychopharmacology · doi:10.1016/j.euroneuro.2015.08.008 · Cited by 81
Mobile Web-Based Monitoring and Coaching: Feasibility in Chronic Migraine
Sorbi, M. J., Mak, S. B., Houtveen, J. H., Kleiboer, A. M., & van Doornen, L. J. (2007) · Journal of Medical Internet Research · doi:10.2196/jmir.9.5.e38 · Cited by 78 · cited 1×
“One system tailored prompting to users’ sleep cycles, according to the times at which morning and evening diaries were completed (Sorbi et al. , 2007).”
MethodDoherty, K., Barry, M., Marcano-Belisario, J., Arnaud, B., Morrison, C., Car, J., & Doherty, G. (2018) · JMIR Mental Health · doi:10.2196/10007 · Cited by 67 · cited 2×
“…include the lack of standardization and validation of EMA scales, ethical considerations, issues of workload and responsibility, and concerns regarding their potential impact on perceptions of human connectedness (Doherty et al. , 2018, Ebner-Priemer & Trull, 2009b, Fahrenberg et al. , 2001).”
BackgroundSelf-citationChorpita, B. F., Rotheram-Borus, M. J., Daleiden, E. L., Bernstein, A., Cromley, T., Swendeman, D., & Regan, J. (2011) · Perspectives on Psychological Science · doi:10.1177/1745691611418240 · Cited by 49 · cited 1×
“…for the purposes of research therefore requires acknowledging the importance of design, developing adaptable and accessible EMA technologies and making such systems available to those without the time or expertise to invest in systems development (Chorpita et al. , 2011, Schueller et al. , 2014).”
BackgroundPredicting interruptibility for manual data collection: a cluster-based user model
Visuri, A., van Berkel, N., Luo, C., Goncalves, J., Ferreira, D., & Kostakos, V. (2017) · Proceedings of the 19th International Conference on Human-Computer Interaction with Mobile Devices and Services · doi:10.1145/3098279.3098532 · Cited by 28
Cited by
Ecological momentary interventions for mental health: A scoping review
Balaskas, A., Schueller, S. M., Cox, A. L., & Doherty, G. (2021) · PLOS ONE · doi:10.1371/journal.pone.0248152 · Cited by 235
Self-citationChen, C., Mrini, K., Charles, K., Lifset, E., Hogarth, M., Moore, A., Weibel, N., & Farcas, E. (2021) · CUI 2021 - 3rd Conference on Conversational User Interfaces · doi:10.1145/3469595.3469626 · Cited by 10
Metadata
- Authors
- Doherty, K., Balaskas, A., & Doherty, G.
- Venue
- Interacting with Computers
- Publisher
- Oxford University Press (OUP)
- Published
- 2020-05
- Volume
- 32
- Issue
- 3
- Pages
- 257-278
- ISSN
- 0953-5438 (ISSN-L)
- 1873-7951
- OpenAlex
- W3081667087
- Cited by
- 187
- References
- 63
- FWCI
- 8.86
- Type
- journal article
- Language
- en
- Open access
- hybrid
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.