Medical Instructed Real-Time Assistant for Patient with Glaucoma and Diabetic Conditions
Open accessThe DOI matches its Crossref record; 45 of the 45 references it cites were found in a registry or an index.
Abstract
Virtual assistants are involved in the daily activities of humans such as managing calendars, making appointments, and providing wake-up calls. They provide a conversational service to customers around-the-clock and make their daily life manageable. With this emerging trend, many well-known companies launched their own virtual assistants that manage the daily routine activities of customers. In the healthcare sector, virtual medical assistants also provide a list of relevant diseases linked to a specific symptom. Due to low accuracy and uncertainty, these generated recommendations are untrusted and may lead to hypochondriasis. In this study, we proposed a Medical Instructed Real-time Assistant (MIRA) that listens to the user’s chief complaint and predicts a specific disease. Instead of informing about the medical condition, the user is referred to a nearby appropriate medical specialist. We designed an architecture for MIRA that considers the limitations of existing virtual medical assistants such as weak authentication, lack of understanding multiple intent statements about a specific medical condition, and uncertain diagnosis recommendations. To implement the designed architecture, we collected the chief complaints along with the dialogue corpora of real patients. Then, we manually validated these data under the supervision of medical specialists. We then used these data for natural language understanding, disease identification, and appropriate response generation. For the prototype version of MIRA, we considered the cases of glaucoma (eye disease) and diabetes (an autoimmune disease) only. The performance measure of MIRA was evaluated in terms of accuracy (89%), precision (90%), sensitivity (89.8%), specificity (94.9%), and F-measure (89.8%). The task completion was calculated using Cohen’s Kappa ( k = 0.848 ) that categorizes MIRA as ‘Almost Perfect’. Furthermore, the voice-based authentication identifies the user effectively and prevent against masquerading attack. Simultaneously, the user experience shows relatively good results in all aspects based on the User Experience Questionnaire (UEQ) benchmark data. The experimental results show that MIRA efficiently predicts a disease based on chief complaints and supports the user in decision making.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Among the papers developing and evaluating a VCA, 6 [59,60,62,65-67] assessed the technology acceptance of the VCA, whereas 3 [63,64,68] assessed the system accuracy. Only one [69] assessed both performance and acceptance.”
From Voice-Based Conversational Agents for the Prevention and Management of Chronic and Mental Health Conditions: Systematic Literature Review (2021) · cites this paper as [69]
In the citing paper
“The majority (n=10) of the selected papers developed and evaluated VCA [59,60,62-69], whereas 2 [61,70] aimed to report a criterion-based performance evaluation of existing commercial conversational agents (eg, Google Assistant and Apple Siri).”
From Voice-Based Conversational Agents for the Prevention and Management of Chronic and Mental Health Conditions: Systematic Literature Review (2021) · cites this paper as [69]
In the citing paper
“The majority of the studies (n=10) did not report any behavioral measures [59-63,65-67,69,70], whereas 2 papers [64,68] did.”
From Voice-Based Conversational Agents for the Prevention and Management of Chronic and Mental Health Conditions: Systematic Literature Review (2021) · cites this paper as [69]
In the citing paper
“In total, 4 of those studies [63,64,68,69] described precise speech recognition performance”
From Voice-Based Conversational Agents for the Prevention and Management of Chronic and Mental Health Conditions: Systematic Literature Review (2021) · cites this paper as [69]
In the citing paper
“Of the 12 studies, 7 [59,60,62,65-67,69] reported technology acceptance findings, whereas the others (n=5) did not [61,63,64,68,70].”
From Voice-Based Conversational Agents for the Prevention and Management of Chronic and Mental Health Conditions: Systematic Literature Review (2021) · cites this paper as [69]
29 more claims about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
The Measurement of Observer Agreement for Categorical Data
Landis, J. R., & Koch, G. G. (1977) · Biometrics · doi:10.2307/2529310 · Cited by 81,148 · cited 1×
“The interpretation of Kappa categorized MIRA as ‘Almost Perfect’ in term of task completion [57].”
MethodAllocation of Physician Time in Ambulatory Practice: A Time and Motion Study in 4 Specialties
Sinsky, C., Colligan, L., Li, L., Prgomet, M., Reynolds, S., Goeders, L., Westbrook, J., Tutty, M., & Blike, G. (2016) · Annals of Internal Medicine · doi:10.7326/m16-0961 · Cited by 1,494 · cited 1×
“According to a survey conducted in [13], primary care physicians spent more time managing electronic medical records (EMRs) than engaging with patients.”
BackgroundMetalearners for estimating heterogeneous treatment effects using machine learning
Künzel, S. R., Sekhon, J. S., Bickel, P. J., & Yu, B. (2019) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1804597116 · Cited by 1,056 · cited 1×
“In the healthcare sector, the adoption of machine learning facilitated diagnosis [8,9], treatment [10,11], and streamlining of administrative tasks [12].”
BackgroundColor Psychology: Effects of Perceiving Color on Psychological Functioning in Humans
Elliot, A. J., & Maier, M. A. (2014) · Annual Review of Psychology · doi:10.1146/annurev-psych-010213-115035 · Cited by 942 · cited 1×
“We used the color change technique because warm colors have a positive impact on the user’s emotions and behavior as per psychology [55].”
MethodEvaluation of symptom checkers for self diagnosis and triage: audit study
Semigran, H. L., Linder, J. A., Gidengil, C., & Mehrotra, A. (2015) · BMJ · doi:10.1136/bmj.h3480 · Cited by 489 · cited 1×
“Due to the low accuracy and uncertainty of the existing virtual medical assistants, the resulting list of conditions may lead to depression, anxiety, and hypochondriasis [24].”
BackgroundLearning to Respond with Deep Neural Networks for Retrieval-Based Human-Computer Conversation System
Yan, R., Song, Y., & Wu, H. (2016) · Proceedings of the 39th International ACM SIGIR conference on Research and Development in Information Retrieval · doi:10.1145/2911451.2911542 · Cited by 338 · cited 1×
“These virtual assistants provide an interactive user interface (text, speech, or both) that have the ability to understand requests, handle complex tasks, and generate an appropriate response using the machine learning model [7].”
BackgroundReporting Mental Health Symptoms: Breaking Down Barriers to Care with Virtual Human Interviewers
Lucas, G. M., Rizzo, A., Gratch, J., Scherer, S., Stratou, G., Boberg, J., & Morency, L.-P. (2017) · Frontiers in Robotics and AI · doi:10.3389/frobt.2017.00051 · Cited by 308 · cited 1×
“Lucus et al., proposed a virtual human interviewer, which conducts an interview with military service members involved in an intense situation and identifies the symptoms associated with their mental state [33].”
BackgroundBickmore, T. W., Trinh, H., Olafsson, S., O'Leary, T. K., Asadi, R., Rickles, N. M., & Cruz, R. (2018) · Journal of Medical Internet Research · doi:10.2196/11510 · Cited by 308 · cited 1×
“The recommendation is predicted based on one symptom and requires approval from the medical specialist [23].”
BackgroundPARADISE: a framework for evaluating spoken dialogue agents
Walker, M. A., Litman, D. J., Kamm, C., & Abella, A. (1997) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.3115/979617.979652 · Cited by 270
On the track of Artificial Intelligence: Learning with Intelligent Personal Assistants
Goksel Canbek, N., & Mutlu, M. E. (2016) · International Journal of Human Sciences · doi:10.14687/ijhs.v13i1.3549 · Cited by 213
Machine Learning Classification Techniques for Breast Cancer Diagnosis
Omondiagbe, D. A., Veeramani, S., & Sidhu, A. S. (2019) · IOP Conference Series: Materials Science and Engineering · doi:10.1088/1757-899x/495/1/012033 · Cited by 210 · cited 1×
“In the healthcare sector, the adoption of machine learning facilitated diagnosis [8,9], treatment [10,11], and streamlining of administrative tasks [12].”
BackgroundMachine Learning in Healthcare
Callahan, A., & Shah, N. H. (2017) · Key Advances in Clinical Informatics · doi:10.1016/b978-0-12-809523-2.00019-4 · Cited by 160
Schrepp (2015) · doi:10.13140/rg.2.1.2815.0245 · Cited by 147
Tanaka, H., Negoro, H., Iwasaka, H., & Nakamura, S. (2017) · PLOS ONE · doi:10.1371/journal.pone.0182151 · Cited by 145 · cited 1×
“Moreover, an ECA was proposed for autism spectrum disorder patients that use audiovisual features for teaching social communication skills [28].”
BackgroundHow an Artificially Intelligent Virtual Assistant Helps Students Navigate the Road to College
Page, L. C., & Gehlbach, H. (2017) · AERA Open · doi:10.1177/2332858417749220 · Cited by 142 · cited 1×
“Many applications from different domains currently have their own built-in virtual assistants such as televisions [2], mobile devices [3], vehicles [4], and the Internet of things [5,6].”
BackgroundA Survey of Password Attacks and Comparative Analysis on Methods for Secure Authentication
Raza, M., Hossain, M. I., Sharif, M. I., & Haider, W. (2012) · Cited by 136
Philip, P., Micoulaud-Franchi, J.-A., Sagaspe, P., Sevin, E. D., Olive, J., Bioulac, S., & Sauteraud, A. (2017) · Scientific Reports · doi:10.1038/srep42656 · Cited by 128 · cited 1×
“Similarly, the mental disorder diagnostic system conducts an interview based on DSM-5 criteria and identifies patients with major depressive disorders [27].”
MethodContext-Aware Symptom Checking for Disease Diagnosis Using Hierarchical Reinforcement Learning
Kao, H.-C., Tang, K.-F., & Chang, E. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.11902 · Cited by 99 · cited 1×
“• Most of the existing studies along with commercially available virtual medical assistants analyze the input symptoms, and either provide a list of specific diseases or relevant information [44].”
BackgroundAusterjost, J., Porr, M., Riedel, N., Geier, D., Becker, T. M., Scheper, T. H., Marquard, D., Lindner, P., & Beutel, S. (2018) · SLAS TECHNOLOGY · doi:10.1177/2472630318788040 · Cited by 76 · cited 1×
“Many applications from different domains currently have their own built-in virtual assistants such as televisions [2], mobile devices [3], vehicles [4], and the Internet of things [5,6].”
BackgroundAutomated spoken dialogue system for hypertensive patient home management
Giorgino, T., Azzini, I., Rognoni, C., Quaglini, S., Stefanelli, M., Gretter, R., & Falavigna, D. (2005) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2004.04.026 · Cited by 69
Acceptance of Subscriber Authentication Methods For Mobile Telephony Devices
Clarke, N., Furnell, S., Rodwell, P. M., & Reynolds, P. (2002) · Computers & Security · doi:10.1016/s0167-4048(02)00304-8 · Cited by 62 · cited 1×
“• None of the existing studies considered security as a primary factor except [30], which uses the traditional PIN-based authentication mechanism [40], and it is vulnerable to brute-force attack [41].”
BackgroundAutomatic generation of spoken dialogue from medical plans and ontologies
Beveridge, M. A., & Fox, J. (2006) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2005.12.008 · Cited by 62 · cited 1×
“In [39], the virtual medical assistant supports general practitioners by analyzing patient health conditions (using a breast cancer ontological model) and recommending an oncologist.”
BackgroundWritten versus spoken queries: A qualitative and quantitative comparative analysis
Crestani, F., & Du, H. (2006) · Journal of the American Society for Information Science and Technology · doi:10.1002/asi.20350 · Cited by 58 · cited 1×
“MIRA supports primary healthcare services and uses spoken natural language for interactive communication to achieve a high success rate on task completion [25].”
BackgroundCan Machine Learning help us in dealing with treatment resistant depression? A review
Pigoni, A., Delvecchio, G., Madonna, D., Bressi, C., Soares, J. C., & Brambilla, P. (2019) · Journal of Affective Disorders · doi:10.1016/j.jad.2019.08.009 · Cited by 57 · cited 1×
“In the healthcare sector, the adoption of machine learning facilitated diagnosis [8,9], treatment [10,11], and streamlining of administrative tasks [12].”
BackgroundHello Harlie: Enabling Speech Monitoring Through Chat-Bot Conversations
David, I., Christina, A., Jacki, L., Dana, B., Helen, L., Olivia, R., Thomas, M., Dan, A., Janet, W., Simon, M., et al. (2016) · Studies in Health Technology and Informatics · doi:10.3233/978-1-61499-666-8-55 · Cited by 56 · cited 1×
“Ireland et al., proposed ‘Harlie’, which converses with the user on a variety of topics and helps in the neurological conditions of Parkinson’s patients [36].”
BackgroundYokotani, K., Takagi, G., & Wakashima, K. (2018) · Computers in Human Behavior · doi:10.1016/j.chb.2018.03.045 · Cited by 51 · cited 1×
“A similar kind of virtual agent was proposed in [34], which interacts with the users and identifies their mental symptoms using mixed methods for triangulation of data.”
MethodDimeff, L. A., Jobes, D. A., Chalker, S. A., Piehl, B. M., Duvivier, L. L., Lok, B. C., Zalake, M. S., Chung, J., & Koerner, K. (2020) · General Hospital Psychiatry · doi:10.1016/j.genhosppsych.2018.05.005 · Cited by 49 · cited 1×
“To reduce hospitalization of suicidal patients, an e-caring avatar was proposed in [29], which involves patients in self-care conversations and recommends relevant videos.”
BackgroundSmart-bot Technology: Conversational Agents Role in Maternal Healthcare Support
Mugoye, K., Okoyo, H. O., & Mcoyowo, S. (2019) · IST-Africa Conference (IST-Africa) · doi:10.23919/istafrica.2019.8764817 · Cited by 45 · cited 1×
“proposed in [37] to support maternal healthcare and provide guidance to expectant mothers during pregnancy.”
BackgroundHow an Artificially Intelligent Virtual Assistant Helps Students Navigate the Road to CollegePreprint
Page, L. C., & Gehlbach, H. (2017) · SSRN Electronic Journal · doi:10.2139/ssrn.2940297 · Cited by 30
Automated Phone Capture of Diabetes Patients Readings with Consultant Monitoring via the Web
Harper, R., Nicholl, P., McTear, M., Wallace, J., Black, L.-A., & Kearney, P. (2008) · 15th Annual IEEE International Conference and Workshop on the Engineering of Computer Based Systems (ecbs 2008) · doi:10.1109/ecbs.2008.31 · Cited by 29 · cited 1×
“Similarly, the spoken dialogue-based diabetic monitoring system collects patient vitals and helps physicians provide recommendations remotely based on the recorded information [32].”
BackgroundMcDermott, D. S., Kamerer, J. L., & Birk, A. T. (2019) · International Journal of Cyber Research and Education · doi:10.4018/ijcre.2019070104 · Cited by 28 · cited 1×
“The leakage of such information may lead to different attacks such as masquerading, and ransomware [42,43].”
BackgroundAppraisal of a Conversational Artefact and Its Utility in Remote Patient Monitoring
Black, L. A., McTear, M., Black, N., Harper, R., & Lemon, M. (2005) · 18th IEEE Symposium on Computer-Based Medical Systems (CBMS'05) · doi:10.1109/cbms.2005.33 · Cited by 27 · cited 1×
“Moreover, the virtual agent for monitoring diabetic patients was proposed in [31], which makes a phone call once a week to collect vitals.”
BackgroundCould a virtual human be used to explore excessive daytime sleepiness in patients?
Philip, P., Bioulac, S., Sauteraud, A., Chaufton, C., & Olive, J. (2014) · Presence: Teleoperators and Virtual Environments · doi:10.1162/pres_a_00197 · Cited by 25 · cited 1×
“designed an Embodied Conversational Agent (ECA) for sleep disorder patients that ask questions using the Epworth Sleepiness Scale and identify the somnolence patients [26].”
MethodHwang, E.-J., Jung, J., Lee, S. K., Lee, S., & Jee, W.-H. (2019) · Scientific Reports · doi:10.1038/s41598-019-42579-y · Cited by 23 · cited 1×
“In the healthcare sector, the adoption of machine learning facilitated diagnosis [8,9], treatment [10,11], and streamlining of administrative tasks [12].”
BackgroundSelf-citationThe Intelligent Medical Platform: A Novel Dialogue-Based Platform for Health-Care Services
Ali, T., Hussain, J., Amin, M. B., Hussain, M., Akhtar, U., Khan, W. A., Lee, S., Kang, B. H., Hussain, M., Afzal, M., et al. (2020) · Computer · doi:10.1109/mc.2019.2924393 · Cited by 23 · cited 1×
“Moreover, a rule-based patient-centric application was proposed in [35], which provides medical coaching services.”
BackgroundSelf-citationA dialogue based mobile virtual assistant for tourists: The SpaceBook Project
Bartie, P., Mackaness, W., Lemon, O., Dalmas, T., Janarthanam, S., Hill, R. L., Dickinson, A., & Liu, X. (2017) · Computers Environment and Urban Systems · doi:10.1016/j.compenvurbsys.2017.09.010 · Cited by 21 · cited 1×
“Many applications from different domains currently have their own built-in virtual assistants such as televisions [2], mobile devices [3], vehicles [4], and the Internet of things [5,6].”
BackgroundCybersecurity and the Evolutions of Healthcare: Challenges and Threats Behind Its Evolution
Frumento, E. (2019) · EAI/Springer Innovations in Communication and Computing · doi:10.1007/978-3-030-02182-5_4 · Cited by 20 · cited 1×
“The leakage of such information may lead to different attacks such as masquerading, and ransomware [42,43].”
BackgroundHummer, M., Groll, S., Kunz, M. M., Fuchs, L., & Pernul, G. (2018) · International Conference on Information Systems Security and Privacy (ICISSP) · doi:10.5220/0006557702330240 · Cited by 14 · cited 1×
“To the best of our knowledge, MIRA is the only virtual medical assistant that uses the concept of identity and access management [51].”
BackgroundBiswas, M. (2018) · Apress eBooks · doi:10.1007/978-1-4842-3754-0 · Cited by 6
Spoken dialog system for real-time data capture
Levin, E., & Levin, A. (2005) · Interspeech (USB) · doi:10.21437/interspeech.2005-779 · Cited by 4 · cited 2×
“• None of the existing studies considered security as a primary factor except [30], which uses the traditional PIN-based authentication mechanism [40], and it is vulnerable to brute-force attack [41].”
BackgroundA dialogue management system using a corpus-based framework and a dynamic dialogue transition model
Kang, S., Ko, Y., & Seo, J. (2013) · AI Communications · doi:10.3233/aic-130552 · Cited by 4 · cited 1×
“As illustrated in Figure 2, we added the identity and access manager, session manager, and security event logging and monitoring to the existing architecture of the virtual assistant [46,47], which overcomes the identified limitation of existing literature and virtual medical assistants.”
BackgroundNatural Language Voice based Authentication Mechanism for Smartphones (poster)
Rehman, U. U., & Lee, S. (2019) · International Conference on Mobile Systems, Applications, and Services · doi:10.1145/3307334.3328645 · Cited by 4 · cited 1×
“We used our designed voice-based authentication protocol that identifies the user based on their voice samples [52].”
MethodSelf-citationA neural network algorithm for signal processing of LFMCW or IFSCW system
Li, Y., Feng, Z., Xiao, Y., & Huang, J. (2003) · Asia-Pacific Microwave Conference (APMC) · doi:10.1109/apmc.1999.833739 · Cited by 3 · cited 1×
“The speech recognition module recognizes human speech, then breaks it into voice samples, and transcribes each voice sample into text using the neural network algorithm for signal processing [48].”
MethodDetecting Masquerading Attack in Software and in Hardware
Pejović, V., Bojanic, S., Carreras, C., & Nieto-Taladriz, O. (2006) · IEEE Mediterranean Electrotechnical Conference (MELECON) · doi:10.1109/melcon.2006.1653228 · Cited by 3 · cited 1×
“The masquerading attack uses a fake identity to gain unauthorized access [58].”
BackgroundKeeping the Internet Open with an Open-Source Virtual Assistant
Lam, M. S. (2018) · ACM Annual International Conference on Mobile Computing and Networking (MobiCom) · doi:10.1145/3241539.3241586 · Cited by 3 · cited 1×
“Many applications from different domains currently have their own built-in virtual assistants such as televisions [2], mobile devices [3], vehicles [4], and the Internet of things [5,6].”
Background
Research works printed but not linked: 1
- Glass, J. (1999). Challenges for spoken dialogue systems. Proceedings of the 1999 IEEE ASRU Workshop, MIT Laboratory fot Computer Science.
Cited by
Schachner, T., Keller, R., & v Wangenheim, F. (2020) · Journal of Medical Internet Research · doi:10.2196/20701 · Cited by 267
“Other types of chronic conditions included respiratory [51,58], cardiovascular [50], nervous system [56], rheumatic [57], and autoimmune/eye conditions [59].”
BackgroundInfluentialBérubé, C., Schachner, T., Keller, R., Fleisch, E., v Wangenheim, F., Barata, F., & Kowatsch, T. (2021) · Journal of Medical Internet Research · doi:10.2196/25933 · Cited by 133
“In total, 4 of those studies [63,64,68,69] described precise speech recognition performance, whereas 3 [63,68,69] reported good or very good speech recognition performance, and 1 [64] study found mediocre recognition accuracy, with single-letter responses being slightly better recognized than word-based responses (det…”
InfluentialBin Sawad, A., Narayan, B., Alnefaie, A., Maqbool, A., Mckie, I., Smith, J., Yuksel, B., Puthal, D., Prasad, M., & Kocaballi, A. B. (2022) · Sensors · doi:10.3390/s22072625 · Cited by 113
“Two studies were for type 1 [29,39], one study was for type 2 [28], and one focused on diabetes in general, whether the patients have type 1 or 2 [40].”
BackgroundInfluentialApplication of Chatbots to Help Patients Self-Manage Diabetes: Systematic Review and Meta-Analysis
Wu, Y., Zhang, J., Ge, P., Duan, T., Zhou, J., Wu, Y., Zhang, Y.-P., Liu, S., Liu, X., Wan, E., et al. (2024) · Journal of Medical Internet Research · doi:10.2196/60380 · Cited by 84
Review of artificial intelligence‐based question‐answering systems in healthcare
Cilar, L., Gosak, L., & Štiglic, G. (2023) · Wiley Interdisciplinary Reviews Data Mining and Knowledge Discovery · doi:10.1002/widm.1487 · Cited by 86
Metadata
- Authors
- Rehman, U. U., Chang, D. J., Jung, Y., Akhtar, U., Razzaq, M. A., & Lee, S.
- Affiliations
- Kyung Hee University (KR)
- The Catholic University of Korea Yeouido St. Mary's Hospital (KR)
- Corresponding author
- Lee, Sungyoung
- Venue
- Applied Sciences
- Publisher
- MDPI AG
- Published
- 2020-03-25
- Volume
- 10
- Issue
- 7
- Pages
- 2216
- ISSN
- 2076-3417 (ISSN-L)
- OpenAlex
- W3013786371
- Cited by
- 37
- References
- 44
- Citations by year
- 2026: 3 · 2025: 4 · 2024: 5 · 2023: 10 · 2022: 5 · 2021: 9 · 2020: 1
- FWCI
- 5.45
- Type
- journal article
- Language
- en
- Open access
- gold
- dx.doi.org
- Licence
- Locations
- Applied Sciences (publishedVersion, cc-by)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Funders
- Institute for Information and Communications Technology Promotion (IITP-2017-0-01629, 2017-0-00655)
- National Research Foundation of Korea (NRF-2016K1A3A7A03951968, NRF-2019R1A2C2090504)
- Topics
- Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Physical Sciences › Computer Science › Artificial Intelligence › Speech and dialogue systems
- Keywords
- glaucoma, diabetes mellitus, chief complaint, disease prediction, voice-based authentication, natural language understanding, dialogue corpus, user experience, health anxiety, clinical decision support
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.