An Ethics Checklist for Digital Health Research in Psychiatry: Viewpoint
Open accessPublished version of a preprint
The DOI matches its Crossref record; 55 of the 56 references it cites were found in a registry or an index.
Abstract
Background: Psychiatry has long needed a better and more scalable way to capture the dynamics of behavior and its disturbances, quantitatively across multiple data channels, at high temporal resolution in real time. By combining 24/7 data—on location, movement, email and text communications, and social media—with brain scans, genetics, genomics, neuropsychological batteries, and clinical interviews, researchers will have an unprecedented amount of objective, individual-level data. Analyzing these data with ever-evolving artificial intelligence could one day include bringing interventions to patients where they are in the real world in a convenient, efficient, effective, and timely way. Yet, the road to this innovative future is fraught with ethical dilemmas as well as ethical, legal, and social implications (ELSI). Objective: The goal of the Ethics Checklist is to promote careful design and execution of research. It is not meant to mandate particular research designs; indeed, at this early stage and without consensus guidance, there are a range of reasonable choices researchers may make. However, the checklist is meant to make those ethical choices explicit, and to require researchers to give reasons for their decisions related to ELSI issues. The Ethics Checklist is primarily focused on procedural safeguards, such as consulting with experts outside the research group and documenting standard operating procedures for clearly actionable data (eg, expressed suicidality) within written research protocols. Methods: We explored the ELSI of digital health research in psychiatry, with a particular focus on what we label “deep phenotyping” psychiatric research, which combines the potential for virtually boundless data collection and increasingly sophisticated techniques to analyze those data. We convened an interdisciplinary expert stakeholder workshop in May 2020, and this checklist emerges out of that dialogue. Results: Consistent with recent ELSI analyses, we find that existing ethical guidance and legal regulations are not sufficient for deep phenotyping research in psychiatry. At present, there are regulatory gaps, inconsistencies across research teams in ethics protocols, and a lack of consensus among institutional review boards on when and how deep phenotyping research should proceed. We thus developed a new instrument, an Ethics Checklist for Digital Health Research in Psychiatry (“the Ethics Checklist”). The Ethics Checklist is composed of 20 key questions, subdivided into 6 interrelated domains: (1) informed consent; (2) equity, diversity, and access; (3) privacy and partnerships; (4) regulation and law; (5) return of results; and (6) duty to warn and duty to report. Conclusions: Deep phenotyping research offers a vision for vastly more effective care for people with, or at risk for, psychiatric disease. The potential perils en route to realizing this vision are significant; however, and researchers must be willing to address the questions in the Ethics Checklist before embarking on each leg of the journey.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“The ethical use of digital health tools remains an active area of investigation, with frameworks emerging but none yet consistently adopted55,56.”
From Selecting and describing control conditions in mobile health randomized controlled trials: a proposed typology (2023) · cites this paper as [56]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Privacy in the age of medical big data
Price, W. N., & Cohen, I. G. (2019) · Nature Medicine · doi:10.1038/s41591-018-0272-7 · Cited by 1,522 · cited 1×
“” The literature on ethics of deep phenotyping has similarly identified privacy as foundational [39,40].”
BackgroundDigital Phenotyping: Technology for a New Science of Behavior
Insel, T. R. (2017) · JAMA · doi:10.1001/jama.2017.11295 · Cited by 833 · cited 1×
“Ideally, this could one day include bringing interventions to patients where they are in the real world, in a convenient, efficient, effective, and timely way [5,6].”
BackgroundHarnessing Smartphone-Based Digital Phenotyping to Enhance Behavioral and Mental Health
Onnela, J.-P., & Rauch, S. L. (2016) · Neuropsychopharmacology · doi:10.1038/npp.2016.7 · Cited by 726
Self-citationDiversity in clinical and biomedical research: a promise yet to be fulfilled
Oh, S. S., Galanter, J., Thakur, N., Pino-Yanes, M., Barcelo, N. E., White, M. J., de Bruin, D. M., Greenblatt, R. M., Bibbins-Domingo, K., Wu, A. H. B., et al. (2015) · PLOS Medicine · doi:10.1371/journal.pmed.1001918 · Cited by 585 · cited 1×
“It is well established that biomedical research generally [28] and psychiatric research specifically struggle to enroll racially and ethnically diverse participants [29].”
BackgroundDecisional Capacity for Informed Consent in Schizophrenia Research
Carpenter, W. T. (2000) · Archives of General Psychiatry · doi:10.1001/archpsyc.57.6.533 · Cited by 506 · cited 1×
“In addition, researchers in psychiatry must address a further question that has long been challenging for the field, “how to ensure meaningful and valid informed consent with participants who have a mental illness?” [25,26].”
BackgroundAssessing Decisional Capacity for Clinical Research or Treatment: A Review of Instruments
Dunn, L. B., Nowrangi, M. A., Palmer, B. W., Jeste, D. V., & Saks, E. R. (2006) · American Journal of Psychiatry · doi:10.1176/ajp.2006.163.8.1323 · Cited by 464 · cited 1×
“The question of decisional capacity in psychiatric research has been well researched, with multiple instruments now available [27], but the field will need to revisit the effectiveness of these instruments in the new context of deep biobehavioral research.”
BackgroundDynamic Consent: a potential solution to some of the challenges of modern biomedical research
Budin-Ljøsne, I., Teare, H. J. A., Kaye, J., Beck, S., Bentzen, H. B., Caenazzo, L., Collett, C., D’Abramo, F., Felzmann, H., Finlay, T., et al. (2017) · BMC Medical Ethics · doi:10.1186/s12910-016-0162-9 · Cited by 374 · cited 1×
“They may also consider staged informed consent [21,22], dynamic consent to facilitate 2-way communication between researchers and participants [23], or a systemic oversight approach for big data research that provides flexibility for addressing uncertainty in how data will be used [24].”
BackgroundJarvik, G. P., Amendola, L. M., Berg, J. S., Brothers, K., Clayton, E. W., Chung, W., Evans, B. J., Evans, J. P., Fullerton, S. M., Gallego, C. J., et al. (2014) · The American Journal of Human Genetics · doi:10.1016/j.ajhg.2014.04.009 · Cited by 369 · cited 1×
“The Ethics Checklist includes 4 different questions on return of results, including, “Have we considered whether our study will generate any ‘actionable’ results, based on established guidelines, and how have we defined actionability?” The concept of actionability is debated across multiple fields [50,52], and in the…”
BackgroundAvailability and quality of mobile health app privacy policies
Sunyaev, A., Dehling, T., Taylor, P. L., & Mandl, K. D. (2015) · Journal of the American Medical Informatics Association · doi:10.1136/amiajnl-2013-002605 · Cited by 360 · cited 1×
“This may not be an easy task, as research on mobile health apps suggests that many vendors do not have a privacy policy publicly available [46].”
BackgroundTorous, J., Onnela, J.-P., & Keshavan, M. (2017) · Translational Psychiatry · doi:10.1038/tp.2017.25 · Cited by 347 · cited 1×
“By combining 24/7 data—on location, movement, email and text communications, and social media—with brain scans, genetics, genomics, neuropsychological batteries, and clinical interviews, researchers will have an unprecedented amount of objective, individual-level data [2,3].”
Background
Research works printed but not linked: 3
- Solove DJ Privacy self-management and the consent dilemma Harvard Law Review 2012 11 04 126 1880 1903
- Jones OD Schall JD Shen FX Law and Neuroscience, Second Edition 2020 09 15 Philadelphia, PA, US Wolters Kluwer
- Marks M Emergent Medical Data: Health Information Inferred by Artificial Intelligence UC Irvine Law Review 2021 11 4 995 1066
Cited by
Reporting guidelines in medical artificial intelligence: a systematic review and meta-analysis
Kolbinger, F. R., Veldhuizen, G. P., Zhu, J., Truhn, D., & Kather, J. N. (2024) · Communications Medicine · doi:10.1038/s43856-024-00492-0 · Cited by 160
New and emerging approaches to treat psychiatric disorders
Scangos, K. W., State, M. W., Miller, A. H., Baker, J. T., & Williams, L. M. (2023) · Nature Medicine · doi:10.1038/s41591-022-02197-0 · Cited by 179
Self-citationSeiferth, C., Vogel, L., Aas, B., Brandhorst, I., Carlbring, P., Conzelmann, A., Esfandiari, N., Finkbeiner, M., Hollmann, K., Lautenbacher, H., et al. (2023) · Nature Mental Health · doi:10.1038/s44220-023-00085-1 · Cited by 122
Goldberg, S. B., Sun, S., Carlbring, P., & Torous, J. (2023) · npj Digital Medicine · doi:10.1038/s41746-023-00923-7 · Cited by 75
Birk, R. H., & Samuel, G. N. (2022) · Current Psychiatry Reports · doi:10.1007/s11920-022-01358-9 · Cited by 63
“Digital phenotyping raises significant ethical questions, including ownership of data [11], sustainability [39], privacy [32], and more.”
BackgroundMethodReturning Individual Research Results from Digital Phenotyping in Psychiatry
Shen, F. X., Baum, M. L., Martinez‐Martin, N., Miner, A. S., Abraham, M. E., Brownstein, C. A., Cortez, N. G., Evans, B. J., Germine, L. T., Glahn, D. C., et al. (2023) · The American Journal of Bioethics · doi:10.1080/15265161.2023.2180109 · Cited by 39
“Our broad definition of “ digital phenotyping ” can be read to overlap with alternative terms such as deep phenotyping, computational phenotyping, deep biobehavioral-typing, and deep geno/- phenotyping (Baker 2019; Shen et al. 2022).”
BackgroundSelf-citationEthics of artificial intelligence in prenatal and pediatric genomic medicine
Coghlan, S., Gyngell, C., & Vears, D. F. (2023) · Journal of Community Genetics · doi:10.1007/s12687-023-00678-4 · Cited by 46
“1 This contrasts with other medical areas such as oncology, psychiatry, ophthalmology, and radiology where the ethics of using AI has already been explored at some length (Carter et al. 2020; Morgan and Mates 2023; Rogers et al. 2021; Shen et al. 2022; Shreve et al. 2022).”
BackgroundThe Use of Mobile Assessments for Monitoring Mental Health in Youth: Umbrella Review
Marciano, L., Vocaj, E., Bekalu, M. A., Tona, A. L., Rocchi, G., & Viswanath, K. (2023) · Journal of Medical Internet Research · doi:10.2196/45540 · Cited by 47
“For example, a starting point might be to follow an ethics checklist to promote an appropriate digital research design using digital health technology in the context of psychiatry [109].”
Expanding a Behavioral View on Digital Health Access: Drivers and Strategies to Promote Equity
Kepper, M. M., Fowler, L. A., Kusters, I. S., Davis, J. W., Baqer, M., Henson, S. J. S., Xiao, Y., Tarfa, A., Yi, J. C., Gibson, B., et al. (2024) · Journal of Medical Internet Research · doi:10.2196/51355 · Cited by 38
“Core elements of privacy to ensure when using personal and remote individual digital monitoring include safety and security in digital services for data collection, transfer, storage, management, and sharing [21,62,84,121].”
Burns, J., Chen, K., Flathers, M., Currey, D., Macrynikola, N., Vaidyam, A. N., Langholm, C., Barnett, I., Byun, A. J. S., Lane, E., et al. (2024) · Journal of Medical Internet Research · doi:10.2196/58502 · Cited by 22
“A 2022 paper proposed the return of results as a necessary aspect of ethical digital psychiatry research [8], and a 2023 ethical framework was proposed that specially identified the need to return results in digital phenotyping in psychiatry [9].”
Zhang, X., Gu, Y., Yin, J., Zhang, Y., Jin, C., Wang, W., Li, A. M., Wang, Y., Su, L., Xu, H., et al. (2023) · JMIR Formative Research · doi:10.2196/42202 · Cited by 14
“…justice, and fairness in AI development and implementation [22,24-27], conducted through checklists, questionnaires, or stakeholders' consultations to promote careful design and execution of medical AI research, and to assess the ethical and social implications of AI implementations [28-30].”
BackgroundDigital health technologies in the accelerating medicines Partnership® Schizophrenia Program
Wigman, J. T. W., Ching, A. E., Chung, Y., Rahimi-Eichi, H., Lane, E., Langholm, C., Vaidyam, A. N., Byun, A. J. S., Haidar, A., Hartmann, J., et al. (2025) · Schizophrenia · doi:10.1038/s41537-025-00599-w · Cited by 17
“Considering the six core domains of the Ethics Checklist for digital mental health research 61 , it is possible to consider how this protocol discussed above address (1) informed consent; (2) equity, diversity, and access; (3) privacy and partnerships 62,63 ; (4) regulation and law; (5) return of…”
Self-citationImproving mental health care in depression: A call for action
Eder, J., Dom, G., Gorwood, P. A. P. M., Kärkkäinen, H., Decraene, A., Kumpf, U., Beezhold, J., Samochowiec, J., Kurimay, T., Gäebel, W., et al. (2023) · European Psychiatry · doi:10.1192/j.eurpsy.2023.2434 · Cited by 21
“However, the further one explores this issue, the more ethical questions arise that need to be addressed [62].”
BackgroundZhang, X., Ferry, J., Hewson, D. W., Collins, G. S., Wiles, M. D., Zhao, Y., Martindale, A. P. L., Tomaschek, M., Bowness, J. S., & the GRAITE‐USRA Working Group (2025) · Anaesthesia · doi:10.1111/anae.16733 · Cited by 15
Duncan, S. F., Kidd, A. C., Perdomo-Lampignano, J. A., Cannon, P., Hall, M., Stobo, D., Maclay, J. D., Blyth, K. G., & Lowe, D. J. (2025) · European Journal of Radiology · doi:10.1016/j.ejrad.2025.112409 · Cited by 7
Cognitive modeling of real-world behavior for understanding mental health
Mirea, D.-M., Nook, E. C., & Niv, Y. (2025) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2025.07.009 · Cited by 12
Pangenomics: A new era in the field of neurodegenerative diseases
Abondio, P., Bruno, F., Passarino, G., Montesanto, A., & Luiselli, D. (2023) · Ageing Research Reviews · doi:10.1016/j.arr.2023.102180 · Cited by 7
Roberts, W., McKee, S. A., Miranda, R., & Barnett, N. P. (2024) · American Psychologist · doi:10.1037/amp0001193 · Cited by 7
What patients and caregivers want to know when consenting to the use of digital behavioral markers
Sonig, A., Deeney, C., Hurley, M. E., Storch, E. A., Herrington, J. D., Lázaro‐Muñoz, G., Zampella, C., Tunç, B., Parish‐Morris, J., Blumenthal-Barby, J. S. S., et al. (2024) · NPP—Digital Psychiatry and Neuroscience · doi:10.1038/s44277-024-00022-9 · Cited by 11
“In their “ ethics checklist ” designed to help researchers establish procedural safeguards around digital phenotyping, Shen et al. [29] include three key questions related to informed consent, including whether researchers have 1) “ appropriately adapted [their] informed consent procedures to [the]…”
Bogdanova, K., Cila, N., Kudina, O., & Bozzon, A. (2025) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3714399 · Cited by 4
“The scholarship of such researchers as [105, 123, 124, 151, 178, 208] encompasses privacy, transparency, consent, accountability, and fairness.”
Addressing the Ethical, Legal, and Social Issues of Healthtech in Education: Insights From Japan
Sumiya, M., Nishimura, T., Aizaki, K., Hirata, I., Tsukui, N., Osuka, Y., Wakuta, M., & Senju, A. (2025) · JMIR Formative Research · doi:10.2196/72781 · Cited by 3
“Shen et al [12] emphasized the importance of obtaining consent after providing participants with sufficient information.”
Kim, S., Kim, M.-J., Kim, J.-S., Jang, J., Choi, W., Lee, H., Song, J.-H., & Hoe, H. (2026) · Alzheimer s Research & Therapy · doi:10.1186/s13195-026-02013-8 · Cited by 6
“Legal frameworks also provide limited clarity on accountability when AI-generated outputs (e.g., diagnostic classifications or risk scores) are incorporated into clinical decision-making, leaving clinicians and patients uncertain regarding responsibility and liability [117].”
Dietel, F. A., & Buhlmann, U. (2023) · Journal of Obsessive-Compulsive and Related Disorders · doi:10.1016/j.jocrd.2023.100847 · Cited by 2
Sakurai, T., Birnbaum, M., Chua, Y. H. V., Isa, T., & Sawa, A. (2025) · Schizophrenia Research · doi:10.1016/j.schres.2025.07.007 · Cited by 4
Kostick, K. M., Hurley, M. E., Ayaz, S., Herrington, J. D., Zampella, C., Parish‐Morris, J., Tunç, B., Lázaro‐Muñoz, G., Blumenthal‐Barby, J., & Storch, E. A. (2025) · JMIR Mental Health · doi:10.2196/79182 · Cited by 6
“A limited number of studies [13-15] provide a foundation for understanding some of these concerns; however, no empirical research to date offers a comprehensive view of the wide-ranging perspectives held by diverse stakeholder groups regarding the benefits and risks of integrating CP into care.”
An assessment of the informativeness of clinical trials in digital mental health
O’Dea, B., Rooke, S. E., Gordon, E.-R., Lyons, F. L., Vilus, B., Paravastu, N., & Batterham, P. J. (2025) · npj Mental Health Research · doi:10.1038/s44184-025-00177-z · Cited by 3
“There are also emerging ethical issues in digital mental health, such as data privacy and the capacity to ensure informed consent remotely, whichmay compromisetheinformativenessofclinical trialsinthis fi eld 13,14 .”
Artificial Intelligence in Geriatric Psychiatry
Vahia, I. V., Gregg, A. P., Pasciak, W., Kimball, J. E., & Golden, J. (2025) · Current Treatment Options in Psychiatry · doi:10.1007/s40501-025-00366-9 · Cited by 1
“To mitigate some of this this risk, itis hoped that AI technologies and guidelines will be developed with more clearly defined informed consent processes, duties to warn, and pathways to care, as indicated in certain checklists addressing the ethical use of deep phenotyping in research [ 30 ].”
López-Medina, D. C., Oliveros, A., Moreno-Ángel, N., Arellano, A. C. H., & Henao-Pérez, M. (2026) · Journal of Medical Internet Research · doi:10.2196/78168 · Cited by 1
“Taken together, these findings suggest that current evaluation of clinical AI remains more mature at the level of methodological description than at the level of demonstrated clinical usefulness [46,48].”
Deeney, C., Sonig, A., Hurley, M. E., Tunç, B., Storch, E. A., Herrington, J. D., Blumenthal‐Barby, J., & Kostick, K. M. (2025) · doi:10.2196/preprints.78788 · Cited by 1
Bustamante, C. M. V., Alama-Maruta, K., Ng, C. K. M., & Coppersmith, D. D. L. (2022) · MIT Science Policy Review · doi:10.38105/spr.qy2iibrk72 · Cited by 2
Digital Phenotyping in Mood Disorders
Marchionatti, L. E., Mastella, N. D. S., Bouvier, V. D. A., & Passos, I. C. (2023) · Springer eBooks · doi:10.1007/978-3-031-10698-9_3
Sathyanath, G. (2024) · Advances in logistics, operations, and management science book series · doi:10.4018/979-8-3693-4288-6.ch015 · Cited by 3
Tavallaei, F. (2025) · Sustainable development · doi:10.5772/intechopen.1011670 · Cited by 1
Rethinking Ethics for an Era of Trusted Computational Tools
Kostick, K. M., Hurley, M. E., Herrington, J. D., & Storch, E. A. (2025) · Psychiatric Clinics of North America · doi:10.1016/j.psc.2025.08.016 · Cited by 2
Towards Human-Centered Digital Health Interventions
Kostick, K. M., & Storch, E. A. (2025) · Psychiatric Clinics of North America · doi:10.1016/j.psc.2025.10.004
Wang, X., Li, H., Li, T., & Zhong, S. (2026) · Frontiers in Public Health · doi:10.3389/fpubh.2026.1744330 · Cited by 1
Card, A. J., Vital, D. G., & Nebeker, C. (2026) · medRxiv · doi:10.64898/2026.02.22.26346854
Future Trends in Research Ethics: Navigating New Ethical Frontiers
Karmakar, D., Das, A., Bhutia, T. O., Govindasamy, K., & Mandal, A. (2026) · IGI Global eBooks · doi:10.4018/979-8-3693-8302-5.ch015
The Changing Concept in the History of Schizophrenia
Cavalli, E., Nicoletti, G. R. P., & Nicoletti, F. (2026) · Brain Sciences · doi:10.3390/brainsci16050447
“…instruments propose safeguards for digital phenotyping and digital mental health research covering consent, equity, privacy, data partnerships, and duty-to-warn procedures reflecting the distinctive risks of continuous monitoring and algorithmic inference in severe mental illness [114,115].”
Perceived Sensitivity of Sensor-Based Digital Health Data: Qualitative Interview Study
Deeney, C., Sonig, A., Hurley, M. E., Tunç, B., Storch, E. A., Herrington, J. D., Blumenthal‐Barby, J., & Kostick, K. M. (2026) · JMIR mhealth and uhealth · doi:10.2196/78788 · Cited by 2
“Their concerns are broadly echoed in the literature around CP technologies [53-55], as well as for nonhealth-related data [31,56-59].”
Malatyńska, N., Babicz, M., Chudzicki, K., Czyżewski, W., Danielczyk, D., Słoma, A., Szot, A., Szydełko, D., Szymczyk, M., & Żurek, P. (2026) · International Journal of Innovative Technologies in Social Science · doi:10.31435/ijitss.1(49).2026.5346
“Shen et al. (2022) translate these broad concerns into a practical ethics checklist for digital health research in psychiatry.”
InfluentialKachurin, A. (2026) · Social Science & Medicine · doi:10.1016/j.socscimed.2026.119595
Digital doppelgangers in psychiatry
Elbarbary, K., & Shoib, S. (2026) · Discover Mental Health · doi:10.1007/s44192-026-00517-1
“…require smartphone ownership, reliable internet connectivity, and compatible wearable devices, and if these tools are developed and deployed primarily for well-resourced populations, they risk widening the already substantial gap in mental health service access rather than narrowing it [11, 19].”
Influential
Metadata
- Authors
- Shen, F. X., Silverman, B. C., Monette, P., Kimble, S., Rauch, S. L., & Baker, J. T.
- Affiliations
- University of Minnesota (US)
- Harvard University (US)
- McLean Hospital (US)
- Corresponding author
- Shen, Francis X
- Venue
- Journal of Medical Internet Research
- Publisher
- JMIR Publications Inc.
- Published
- 2022-02-09
- Volume
- 24
- Issue
- 2
- Pages
- e31146
- ISSN
- 1438-8871 (ISSN-L)
- PMC
- PMC8867294
- OpenAlex
- W3211054186
- Cited by
- 43
- References
- 56
- Citations by year
- 2026: 10 · 2025: 14 · 2024: 6 · 2023: 11 · 2022: 2
- FWCI
- 4.12
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- Locations
- Journal of Medical Internet Research (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
- Funders
- National Institutes of Health
- National Institute of Mental Health (U01MH116925)
- Topics
- Health Sciences › Medicine › Public Health, Environmental and Occupational Health › Ethics in Clinical Research
- Social Sciences › Psychology › Applied Psychology › Digital Mental Health Interventions
- Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
- Keywords
- digital health research, psychiatry, research ethics, deep phenotyping, ethical, legal and social implications, informed consent, privacy, data sharing, duty to warn, reporting requirements, return of results, research ethics committees, legislative gaps, artificial intelligence in psychiatry, suicidality
- MeSH terms
- Artificial Intelligence, Humans, Informed Consent, Psychiatry, Privacy, Ethics Committees, Research, Checklist
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.