Factors Influencing Data Quality in Electronic Health Record Systems in 50 Health Facilities in Rwanda and the Role of Clinical Alerts: Cross-Sectional Observational Study
Open accessPublished version of a preprint
The DOI matches its Crossref record; 34 of the 35 references it cites were found in a registry or an index; 23 of the 46 sources its claims cite were checked for support.
Abstract
BACKGROUND: Electronic health records (EHRs) play an increasingly important role in delivering HIV care in low- and middle-income countries. The data collected are used for direct clinical care, quality improvement, program monitoring, public health interventions, and research. Despite widespread EHR use for HIV care in African countries, challenges remain, especially in collecting high-quality data. OBJECTIVE: We aimed to assess data completeness, accuracy, and timeliness compared to paper-based records, and factors influencing data quality in a large-scale EHR deployment in Rwanda. METHODS: We randomly selected 50 health facilities (HFs) using OpenMRS, an EHR system that supports HIV care in Rwanda, and performed a data quality evaluation. All HFs were part of a larger randomized controlled trial, with 25 HFs receiving an enhanced EHR with clinical decision support systems. Trained data collectors visited the 50 HFs to collect 28 variables from the paper charts and the EHR system using the Open Data Kit app. We measured data completeness, timeliness, and the degree of matching of the data in paper and EHR records, and calculated concordance scores. Factors potentially affecting data quality were drawn from a previous survey of users in the 50 HFs. RESULTS: We randomly selected 3467 patient records, reviewing both paper and EHR copies (194,152 total data items). Data completeness was >85% threshold for all data elements except viral load (VL) results, second-line, and third-line drug regimens. Matching scores for data values were close to or >85% threshold, except for dates, particularly for drug pickups and VL. The mean data concordance was 10.2 (SD 1.28) for 15 (68%) variables. HF and user factors (eg, years of EHR use, technology experience, EHR availability and uptime, and intervention status) were tested for correlation with data quality measures. EHR system availability and uptime was positively correlated with concordance, whereas users' experience with technology was negatively correlated with concordance. The alerts for missing VL results implemented at 11 intervention HFs showed clear evidence of improving timeliness and completeness of initially low matching of VL results in the EHRs and paper records (11.9%-26.7%; P<.001). Similar effects were seen on the completeness of the recording of medication pickups (18.7%-32.6%; P<.001). CONCLUSIONS: The EHR records in the 50 HFs generally had high levels of completeness except for VL results. Matching results were close to or >85% threshold for nondate variables. Higher EHR stability and uptime, and alerts for entering VL both strongly improved data quality. Most data were considered fit for purpose, but more regular data quality assessments, training, and technical improvements in EHR forms, data reports, and alerts are recommended. The application of quality improvement techniques described in this study should benefit a wide range of HFs and data uses for clinical care, public health, and disease surveillance.
Overview
CoChat Summary
## TL;DR
This cross-sectional study assessed data quality in OpenMRS HIV records across 50 health facilities in Rwanda by comparing paper charts with EHR data for completeness, matching, and concordance. Data were generally complete, but viral load fields and date fields were weaker, and clinical alerts for missing viral load results improved completeness and matching.
## Key findings
- The study reviewed 3,467 patient records and 194,152 total data items across 50 health facilities; the mean was 69.3 (SD 8.8) records per facility.
- Data completeness was >85% for all data elements except viral load results, second-line, and third-line drug regimens; paper viral load records were 20% to 30% more complete than EHR records.
- The mean concordance score was 10.2 (SD 1.28) out of 15 variables, and only 2 health facilities met the 85% concordance threshold.
- EHR availability/uptime was positively correlated with concordance (r=0.16; P<.001), while higher user technology experience was negatively correlated (r=−0.11; P<.001).
- In facilities with viral load alerts, viral load records matched in 26.7% (85/318) of cases versus 11.9% (120/1008) without alerts (P<.001); drug pickup matches were 36.2% (280/774) with alerts versus 18.7% (183/981) without alerts (P<.001).
## Methods
- Cross-sectional observational study using an LQAS approach in 50 of 112 health facilities from a larger randomized controlled trial of OpenMRS in Rwanda; facilities included both intervention and control sites, but not hospitals.
- Trained data collectors extracted 28 variables from both paper charts and EHRs for randomly selected HIV patient records; the sample included 48 to 76 charts per facility depending on patient volume.
- Main analyses measured completeness, matching, and concordance; correlations with facility characteristics used Pearson correlation with Bonferroni correction, and alert effects were tested with chi-square tests and Pearson contingency coefficients.
## Limitations
- The paper states the study was limited to a subset of MOH-run OpenMRS EHR systems for HIV care, and the original inclusion criteria favored facilities with better IT hardware and successful server monitoring tool implementation.
- The authors note that the apparent benefit of alerts in intervention facilities might change over time, and that differences in training emphasis on viral load could have contributed to the observed effects.
- The text makes evident that this is a cross-sectional assessment with only a 4- to 5-month window after intervention rollout for some analyses; therefore, long-term causal effects are not established.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures




Verification
Claims the paper makes
19 Valid · 1 Not valid · 13 Not checked · 52 Own results and interpretations
Every source the paper cites for this claim was checked and supports it.
In the paper
“Previous studies suggest that poor completion or accuracy of key data items required for clinical care is likely to limit the benefit of EHR systems in day-to-day patient management, such as the functioning of clinical decision support system in this study and that reported by Haskew et al [41].”
§ Factors Influencing Data Quality in Electronic Health Record Systems in 50 Health Facilities in Rwanda and the Role of Clinical Alerts: Cross-Sectional Observational Study > Discussion > Comparison to Previous Work
What the sources say
- Supportedcites [41] source
The Methods section ('Data verification and clinical decision support'; page not available) describes the clinical decision-support system reported by Haskew et al. It says the system flagged missing data and displayed reminders to clinicians until corrective action was taken, supporting the citation's use as an example of such a system. (LLM, conf=0.88)
hedgedstates a possible cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“The inclusion criteria for the 112 HFs in the larger study favored HFs with better information technology hardware and successful implementation of a server monitoring tool [4].”
§ Factors Influencing Data Quality in Electronic Health Record Systems in 50 Health Facilities in Rwanda and the Role of Clinical Alerts: Cross-Sectional Observational Study > Discussion > Limitations
What the sources say
- Supportedcites [4] source
The Methods section (“The EHR Implementation Science Study”) lists minimum computer, printer, and network requirements and successful installation of the Server Monitoring Tool among the criteria used to select the 112 sites. The Discussion (“Limitations”) also states that the selected sites had better hardware than other sites. The supplied passages do not provide page numbers. (LLM, conf=0.94)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“EHR system availability and uptime was positively correlated with concordance, whereas users’ experience with technology was negatively correlated with concordance.”
§ Factors Influencing Data Quality in Electronic Health Record Systems in 50 Health Facilities in Rwanda and the Role of Clinical Alerts: Cross-Sectional Observational Study > Abstract > Results
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption not shown here.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Data completeness was >85% threshold for all data elements except viral load (VL) results, second-line, and third-line drug regimens.”
- Also stated in: § Conclusions · plain
“The EHR records in the 50 HFs generally had high levels of completeness except for VL results.”
§ Factors Influencing Data Quality in Electronic Health Record Systems in 50 Health Facilities in Rwanda and the Role of Clinical Alerts: Cross-Sectional Observational Study > Abstract > Results
plainn = 3,467- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption not shown here.
- Also stated in: § Conclusions · plain
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“EHR system availability and uptime was positively correlated with concordance, whereas users’ experience with technology was negatively correlated with concordance.”
§ Factors Influencing Data Quality in Electronic Health Record Systems in 50 Health Facilities in Rwanda and the Role of Clinical Alerts: Cross-Sectional Observational Study > Abstract > Results
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption not shown here.
27 more claims
Create a free account to see all 85 claims and to verify the 11 references still open.
- It may state two separate claims as one. 39
- Probably not a substantial claim. 34
Citations
Cites
Addressing bias in artificial intelligence in health care
Parikh, R. B., Teeple, S., & Navathe, A. S. (2019) · JAMA · doi:10.1001/jama.2019.18058 · Cited by 764
Middleton, B., Bloomrosen, M., Dente, M. A., Hashmat, B., Koppel, R., Overhage, J. M., Payne, T. H., Rosenbloom, S. T., Weaver, C., & Zhang, J. (2013) · Journal of the American Medical Informatics Association · doi:10.1136/amiajnl-2012-001458 · Cited by 618
Uses of Electronic Health Records for Public Health Surveillance to Advance Public Health
Birkhead, G. S., Klompas, M., & Shah, N. R. (2015) · Annual Review of Public Health · doi:10.1146/annurev-publhealth-031914-122747 · Cited by 345 · cited 1×
“Routine clinical data, such as those collected from EHRs, are an important component of public health systems in high-income countries [20].”
BackgroundMobile Health (mHealth) in Low- and Middle-Income Countries
McCool, J., Dobson, R., Whittaker, R., & Paton, C. (2022) · Annual Review of Public Health · doi:10.1146/annurev-publhealth-052620-093850 · Cited by 269 · cited 2×
“FHIR-based Application Programming Interfaces and toolkits are also being built into the Apple OS and Android operating systems [36].”
MethodA review of PHR, EMR and EHR integration: A more personalized healthcare and public health policy
Heart, T., Ben-Assuli, O., & Shabtai, I. (2017) · Health Policy and Technology · doi:10.1016/j.hlpt.2016.08.002 · Cited by 251 · cited 1×
“To maximize the benefits of EHRs for clinical and public health functions, they increasingly allow data collection on behavioral, social, and environmental factors that may affect patient and population health [26].”
BackgroundEgger, M. (2008) · Bulletin of the World Health Organization · doi:10.2471/blt.07.049908 · Cited by 177
Rich, M. L., Miller, A. C., Niyigena, P., Franke, M. F., Niyonzima, J. B., Socci, A., Drobac, P. C., Hakizamungu, M., Mayfield, A., Ruhayisha, R., et al. (2012) · JAIDS Journal of Acquired Immune Deficiency Syndromes · doi:10.1097/qai.0b013e31824476c4 · Cited by 172
The OpenMRS Implementers Network
Seebregts, C. J., Mamlin, B. W., Biondich, P., Fraser, H., Wolfe, B. A., Jazayeri, D., Allen, C., Miranda, J., Baker, E., Musinguzi, N., et al. (2009) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2008.09.005 · Cited by 146
Self-citationFritz, F., Tilahun, B., & Dugas, M. (2015) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocu038 · Cited by 131 · cited 2×
“This typically leads to staff questioning the value of using the systems and keeping data up to date, which can lead to a downward cycle of deterioration and sometimes abandoning the EHR [9].”
BackgroundAn information system and medical record to support HIV treatment in rural Haiti
Fraser, H. S. F., Jazayeri, D., Nevil, P., Karacaoglu, Y., Farmer, P. E., Lyon, E., Fawzi, M. K. C. S., Leandre, F., Choi, S. S., & Mukherjee, J. S. (2004) · BMJ · doi:10.1136/bmj.329.7475.1142 · Cited by 125
Self-citationHerwehe, J., Wilbright, W., Abrams, A., Bergson, S., Foxhood, J., Kaiser, M., Smith, L., Xiao, K., Zapata, A., & Magnus, M. (2012) · Journal of the American Medical Informatics Association · doi:10.1136/amiajnl-2011-000412 · Cited by 92 · cited 1×
“Health Information Exchanges linking EHR and laboratory data, for example, have a growing role in population health and disease surveillance [25].”
MethodExperience in implementing the OpenMRS medical record system to support HIV treatment in Rwanda.
Allen, C. B., Jazayeri, D. G., Miranda, J., Biondich, P., Mamlin, B. W., Wolfe, B. A., Seebregts, C. J., Lesh, N. B., Tierney, W. M., & Fraser, H. (2007) · PubMed · Cited by 91
Self-citationIntegrating Clinical Practice and Public Health Surveillance Using Electronic Medical Record Systems
Klompas, M., McVetta, J., Lazarus, R., Eggleston, E., Haney, G., Kruskal, B. A., Yih, W. K., Daly, P., Oppedisano, P., Beagan, B., et al. (2012) · American Journal of Public Health · doi:10.2105/ajph.2012.300811 · Cited by 89
Implementing medical information systems in developing countries, what works and what doesn't.
(2010) · PubMed Central · Cited by 86
Self-citationBraunstein, M. L. (2018) · Health Informatics on FHIR: How HL7's New API is Transforming Healthcare · doi:10.1007/978-3-319-93414-3_9 · Cited by 83
Dornan, L., Pinyopornpanish, K., Jiraporncharoen, W., Hashmi, A., Dejkriengkraikul, N., & Angkurawaranon, C. (2019) · BioMed Research International · doi:10.1155/2019/7341841 · Cited by 81 · cited 1×
“Moreover, the data in EHRs can help to understand the patterns of HIV distribution, the progress in achieving national and global targets, oversight of the risks and patterns in specific population groups, and the evolution of the infection in different circumstances [21,22].”
BackgroundThe OpenMRS system: collaborating toward an open source EMR for developing countries.
(2006) · PubMed Central · Cited by 74
Self-citationThe impact of routine data quality assessments on electronic medical record data quality in Kenya
Muthee, V., Bochner, A. F., Osterman, A., Liku, N., Akhwale, W., Kwach, J., Prachi, M., Wamicwe, J., Odhiambo, J., Onyango, F., et al. (2018) · PLOS ONE · doi:10.1371/journal.pone.0195362 · Cited by 64 · cited 8×
“The concordance score was calculated per record as the number of variables that exactly matched in both the paper file and the EHR after the values had been normalized, following the approach of Muthee et al [14].”
BackgroundMethodResultInfluentialBarriers to the Adoption of Electronic Medical Record System in Ethiopia: A Systematic Review
Yehualashet, D. E., Seboka, B. T., Tesfa, G. A., Demeke, A. D., & Amede, E. S. (2021) · Journal of Multidisciplinary Healthcare · doi:10.2147/jmdh.s327539 · Cited by 56 · cited 1×
“Effective EHRs need to address important priorities of stakeholders (value of the system) [10,11] and be usable by staff as part of their routine work, which requires attention to the user interface, clinical workflow, and training [11-13] (ease of use).”
BackgroundVerma, N., Mamlin, B., Flowers, J., Acharya, S., Labrique, A., & Cullen, T. (2021) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2021.104405 · Cited by 53 · cited 5×
“Furthermore, it supports care for maternal-child health, noncommunicable diseases, and oncology in some health centers and hospitals in Rwanda and other countries [7].”
BackgroundMethodInfluentialTeklegiorgis, K., Tadesse, K., Mirutse, G., & Terefe, W. (2016) · SA Journal of Information Management · doi:10.4102/sajim.v18i1.612 · Cited by 53 · cited 2×
“Furthermore, it impacts quality of care metrics (eg, in cancer care in the United States [43]) and efficient reporting of data for clinical teams, MOH, and funders [16,42].”
BackgroundInfluential(2010) · PubMed · Cited by 52
Self-citationHaskew, J., Rø, G., Turner, K., Kimanga, D., Sirengo, M., & Sharif, S. (2015) · PLOS ONE · doi:10.1371/journal.pone.0135361 · Cited by 52
Abiy, R., Gashu, K., Asemaw, T., Mitiku, M., Fekadie, B., Abebaw, Z., Mamuye, A., Tazebew, A., Teklu, A., Nurhussien, F., et al. (2018) · Online Journal of Public Health Informatics · doi:10.5210/ojphi.v10i2.8309 · Cited by 45
Applications of Electronic Health Information in Public Health: Uses, Opportunities and Barriers
Tomines, A., Readhead, H., Readhead, A., & Teutsch, S. (2013) · eGEMs (Generating Evidence & Methods to improve patient outcomes) · doi:10.13063/2327-9214.1019 · Cited by 40 · cited 1×
“EHR data are used for public health surveillance, patient to provider communication, quality of care monitoring, linkage of primary care and public health system such as notifiable disease reporting [23], and strategic policy direction [24].”
MethodBlaya, J. A., Shin, S. S., Yagui, M., Contreras, C., Cegielski, P., Yale, G., Suarez, C., Asencios, L., Bayona, J., Kim, J., et al. (2014) · PLoS ONE · doi:10.1371/journal.pone.0090110 · Cited by 36 · cited 1×
“Clinical care requires data that are as complete and accurate as possible, but specific variables, such as medications, past medical history, or laboratory results, are often most important [15].”
BackgroundSelf-citationFraser, H. S. F., Mugisha, M., Remera, E., Ngenzi, J. L., Richards, J., Santas, X., Naidoo, W., Seebregts, C., Condo, J., & Umubyeyi, A. (2022) · JMIR Medical Informatics · doi:10.2196/32305 · Cited by 33
Self-citationComputer-Generated Reminders and Quality of Pediatric HIV Care in a Resource-Limited Setting
Were, M. C., Nyandiko, W. M., Huang, K. T. L., Slaven, J. E., Shen, C., Tierney, W. M., & Vreeman, R. C. (2013) · Pediatrics · doi:10.1542/peds.2012-2072 · Cited by 28 · cited 1×
“Effective EHRs need to address important priorities of stakeholders (value of the system) [10,11] and be usable by staff as part of their routine work, which requires attention to the user interface, clinical workflow, and training [11-13] (ease of use).”
BackgroundNgugi, P., Babić, A., & Were, M. C. (2021) · PLoS ONE · doi:10.1371/journal.pone.0256799 · Cited by 23
(2010) · PubMed · Cited by 18
Self-citationHedt-Gauthier, B. L., Tenthani, L., Mitchell, S., Chimbwandira, F. M., Makombe, S., Chirwa, Z., Schouten, E. J., Pagano, M., & Jahn, A. (2012) · BMC Health Services Research · doi:10.1186/1472-6963-12-196 · Cited by 16 · cited 5×
“At each HF, depending on the patient volume, 48 to 76 charts were randomly selected from the set of patients receiving HIV care based on the LQAS protocol used by Hedt et al [19].”
MethodInfluentialAdedimeji, A. A., Hoover, D. R., Shi, Q., Kim, H., Brazier, E., Ross, J., Murenzi, G., Twizere, C., Lelo, P., Nsonde, D., et al. (2021) · Journal of the International AIDS Society · doi:10.1002/jia2.25672 · Cited by 15 · cited 1×
“Moreover, the data in EHRs can help to understand the patterns of HIV distribution, the progress in achieving national and global targets, oversight of the risks and patterns in specific population groups, and the evolution of the infection in different circumstances [21,22].”
BackgroundChasm Between Cancer Quality Measures and Electronic Health Record Data Quality
Schorer, A. E., Moldwin, R., Koskimaki, J., Bernstam, E. V., Venepalli, N. K., Miller, R. S., & Chen, J. L. (2022) · JCO Clinical Cancer Informatics · doi:10.1200/cci.21.00128 · Cited by 12
Aldredge, N. H., Rodriguez, D., González, J., & Burt, D. R. (2020) · Annals of Global Health · doi:10.5334/aogh.3041 · Cited by 6
Research works printed but not linked: 6
- Amoroso, CL, 2010, Stud Health Technol Inform, 160, 337
- Fraser, HS, 2010, AMIA Annu Symp Proc, 2010, 232
- Blaya, JA, 2010, Int J Tuberc Lung Dis, 14, 1009
- Wolfe, BA, 2006, AMIA Annu Symp Proc, 2006, 1146
- Allen, C, 2007, Stud Health Technol Inform, 129, 382
- Fraser, HS, 2021, ICT in Health: Survey on the Use of Information and Communication Technologies in Brazil Healthcare Facilities, 273
Cited by
Common data quality elements for health information systems: a systematic review
Ghalavand, H., Shirshahi, S., Rahimi, A., Zarrinabadi, Z., & Amani, F. (2024) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-024-02644-7 · Cited by 76
Sylvain, M. H., Nyabyenda, E. C., Uwase, M., Komezusenge, I., Ndikumana, F., & Ngaruye, I. (2025) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-025-02921-z · Cited by 28
Incompleteness of Electronic Health Records: An Impending Process Problem Within Healthcare
Gurupur, V. P., Hooshmand, S., Prabhu, D. F., Trader, E. A., & Salvi, S. (2025) · Healthcare · doi:10.3390/healthcare13222900 · Cited by 29
Petit, P., & Vuillerme, N. (2025) · JMIR Public Health and Surveillance · doi:10.2196/62939 · Cited by 22
Adoption of electronic health records by nurses in Africa: A scoping review
Harerimana, A., Wicking, K., Biedermann, N., Yates, K., Pillay, J. D., & Mchunu, G. G. (2025) · Digital Health · doi:10.1177/20552076251357401 · Cited by 18
Metadata
- Authors
- Fraser, H. S. F., Mugisha, M., Bacher, I., Ngenzi, J. L., Seebregts, C., Umubyeyi, A., & Condo, J.
- Affiliations
- Brown University (US)
- University of Rwanda (RW)
- University of Cape Town (ZA)
- Tulane University (US)
- Corresponding author
- Fraser, Hamish S F
- ORCID
- Venue
- JMIR Public Health and Surveillance
- Publisher
- JMIR Publications Inc.
- Published
- 2024-07-03
- Volume
- 10
- Pages
- e49127
- ISSN
- 2369-2960 (ISSN-L)
- PMC
- PMC11255528
- OpenAlex
- W4388467758
- Semantic Scholar
- 870bf0fac2a05ac519f6e719fcd62eec6fe3cc4c
- Cited by
- 20
- References
- 35
- Citations by year
- 2026: 12 · 2025: 7 · 2024: 1
- FWCI
- 4.48
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- Locations
- JMIR Public Health and Surveillance (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
- Funders
- Brown University
- U.S. President’s Emergency Plan for AIDS Relief
- University of Rwanda
- European Commission (661289)
- Rwanda Biomedical Centre
- Centers for Disease Control and Prevention (U01GH000782)
- Topics
- Health Sciences › Health Professions › Health Information Management › Electronic Health Records Systems
- Health Sciences › Health Professions › Health Information Management › Medical Coding and Health Information
- Physical Sciences › Computer Science › Information Systems › Data Quality and Management
- Keywords
- electronic health records, data quality, clinical decision support, clinical alerts, HIV care, OpenMRS, Open Data Kit, data completeness, data timeliness, data concordance, paper records, Rwanda, low- and middle-income countries, health information system evaluation
- MeSH terms
- Data Accuracy, Cross-Sectional Studies, Health Facilities, Humans, Rwanda, HIV Infections, Electronic Health Records
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.