Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat

The DOI matches its Crossref record; 39 of the 39 references it cites were found in a registry or an index.

AutoReporter: development of an artificial intelligence tool for automated assessment of research reporting guideline adherence

Open access
Chen, David, Li, Patrick, Khoshkish, Ealia, Lee, Seungmin, Ning, Tony, Tahir, Umair, Wong, Henry C Y, Lee, Michael S F, Raman, Srinivas · 2026 · Journal of the American Medical Informatics Association

Abstract

Objectives To develop AutoReporter, a large language model (LLM) system that automates evaluation of adherence to research reporting guidelines. Materials and methods Eight prompt-engineering and retrieval strategies coupled with reasoning and general-purpose LLMs were benchmarked on the SPIRIT-CONSORT-TM corpus. The top-performing approach, AutoReporter, was validated on BenchReport, a novel benchmark dataset of expert-rated reporting guideline assessments from 10 systematic reviews. Results AutoReporter, a zero-shot, no-retrieval prompt coupled with the o3-mini reasoning LLM, demonstrated strong accuracy (CONSORT 90.09%; SPIRIT: 92.07%), substantial agreement with humans (CONSORT Cohen's κ = 0.70, SPIRIT Cohen's κ = 0.77), runtime (CONSORT: 617.26 s; SPIRIT: 544.51 s), and cost (CONSORT: 0.68 USD; SPIRIT: 0.65 USD). AutoReporter achieved a mean accuracy of 91.8% and substantial agreement (Cohen's κ > 0.6) with expert ratings from the BenchReport benchmark. Discussion Structured prompting alone can match or exceed fine-tuned domain models while forgoing manually annotated corpora and computationally intensive training. Conclusion Large language models can feasibly automate reporting guideline adherence assessments for scalable quality control in scientific research reporting. AutoReporter is publicly accessible at https://autoreporter.streamlit.app.

Overview

Integrity signals

Registry

Crossref · 10.1093/jamia/ocaf223

Cites

39 of 39 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Verification

Claims the paper makes

We couldn’t retrieve this paper’s full text, so its claims and self-check weren’t run. The registry and citation checks still apply.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Citations

Cites

Research works printed but not linked: 1

  1. Yuan, 2021, The quality of methodological and reporting in network meta-analysis of acupuncture and moxibustion: a cross-sectional survey, Evid Based Complement Alternat Med, 2021, 1

Cited by

See all 10

Metadata

Authors
Chen, D., Li, P., Khoshkish, E., Lee, S., Ning, T., Tahir, U., Wong, H. C. Y., Lee, M. S. F., & Raman, S.
Affiliations
  • University of Toronto (CA)
  • McMaster University (CA)
  • Princess Margaret Hospital (HK)
  • National University Cancer Institute, Singapore (SG)
  • BC Cancer Agency (CA)
Corresponding author
Raman, Srinivas
Venue
Journal of the American Medical Informatics Association
Published
2025-12-23
Volume
33
Issue
3
Pages
724-731
ISSN
  • 1067-5027 (ISSN-L)
  • 1527-974X
OpenAlex
W7117158505
Cited by
6
References
38
Citations by year
2026: 5 · 2025: 1
FWCI
3.84
Type
journal article
Language
en
Open access
Locations
  • Journal of the American Medical Informatics Association (publishedVersion)
  • medRxiv (acceptedVersion, cc-by-nc-nd)
  • PubMed (publishedVersion)
Topics
  • Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
Keywords
reporting guidelines, large language models, automated assessment, prompt engineering, alcoholic beverages, CONSORT, systematic reviews, o3-mini, zero-shot prompting, research quality control
MeSH terms
Artificial Intelligence, Humans, Natural Language Processing, Guideline Adherence, Biomedical Research, Research Report

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.