Cited by 341Meta-analysis and systematic reviewsArtificial Intelligence in Healthcare and EducationDelphi Technique in Research

The DOI matches its Crossref record; 18 of the 18 references it cites were found in a registry or an index; none of the 7 sources its claims cite is checked for support yet.

Developing a reporting guideline for artificial intelligence-centred diagnostic test accuracy studies: the STARD-AI protocol

Open access

Abstract

Introduction: Standards for Reporting of Diagnostic Accuracy Study (STARD) was developed to improve the completeness and transparency of reporting in studies investigating diagnostic test accuracy. However, its current form, STARD 2015 does not address the issues and challenges raised by artificial intelligence (AI)-centred interventions. As such, we propose an AI-specific version of the STARD checklist (STARD-AI), which focuses on the reporting of AI diagnostic test accuracy studies. This paper describes the methods that will be used to develop STARD-AI. Methods and analysis: The development of the STARD-AI checklist can be distilled into six stages. (1) A project organisation phase has been undertaken, during which a Project Team and a Steering Committee were established; (2) An item generation process has been completed following a literature review, a patient and public involvement and engagement exercise and an online scoping survey of international experts; (3) A three-round modified Delphi consensus methodology is underway, which will culminate in a teleconference consensus meeting of experts; (4) Thereafter, the Project Team will draft the initial STARD-AI checklist and the accompanying documents; (5) A piloting phase among expert users will be undertaken to identify items which are either unclear or missing. This process, consisting of surveys and semistructured interviews, will contribute towards the explanation and elaboration document and (6) On finalisation of the manuscripts, the group’s efforts turn towards an organised dissemination and implementation strategy to maximise end-user adoption. Ethics and dissemination: Ethical approval has been granted by the Joint Research Compliance Office at Imperial College London (reference number: 19IC5679). A dissemination strategy will be aimed towards five groups of stakeholders: (1) academia, (2) policy, (3) guidelines and regulation, (4) industry and (5) public and non-specific stakeholders. We anticipate that dissemination will take place in Q3 of 2021.

Overview

AI snapshot

Written from the paper’s full text.

TL;DR

The STARD-AI protocol describes the planned development of an AI-specific reporting guideline for diagnostic test accuracy studies, extending STARD 2015. It outlines a six-stage process including item generation and a three-round modified Delphi consensus, with publication anticipated in Q3 2021.

Key findings

  • STARD 2015 is the most widely accepted reporting standard for diagnostic test accuracy studies, consisting of 30 items and endorsed by over 200 biomedical journals, but it does not address AI-specific issues such as data preprocessing, model development choices, external validation datasets, standardised nomenclature, and unfamiliar outcome measures.
  • STARD-AI development is distilled into six stages: project organisation; item generation; a three-round modified Delphi consensus; drafting the checklist and accompanying documents; piloting among expert users; and finalisation, publication, and dissemination.
  • Item generation included a January 2020 literature review, a February 2020 online scoping survey of 80 international experts that generated over 2500 responses, and a patient and public involvement and engagement focus group.
  • The modified Delphi uses a 5-point Likert-like scale with a predefined consensus threshold of ≥75%; items rated 1 or 2 by ≥75% are deemed essential, items rated 4 or 5 by ≥75% are excluded, and non-consensus items proceed to later rounds or the virtual consensus meeting.

Methods

  • Protocol constructed in accordance with the EQUATOR Network toolkit for developing reporting guidelines, with a nine-member Project Team and a multidisciplinary Steering Committee.
  • Item generation combined a literature review of MEDLINE and EMBASE via Ovid, an online scoping survey of 80 experts, and a remote patient and public involvement and engagement focus group; findings were classified into four categories: general considerations, suggested modifications, additions, and removals to STARD 2015 items.
  • A pragmatic modified Delphi consensus process is underway: rounds 1 and 2 are online surveys using DelphiManager V.4.0, and round 3 is a virtual teleconference consensus meeting using the nominal group technique; piloting will use surveys and semistructured interviews with thematic analysis.

Limitations

  • The paper states: “Views of Delphi panellists may differ from those experts who decline participation.”
  • Evident from the text: this is a protocol, so no final STARD-AI checklist, Delphi consensus results, pilot findings, or implementation outcomes are reported.
  • Evident from the text: development relies on expert consensus and selected stakeholder groups; no formal validation or reliability testing of the eventual checklist is described in this protocol.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1136/bmjopen-2020-047709

Cites

18 of 18 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Verification

Claims the paper makes

6 Not checked yet · 6 Own method · 3 Own interpretation · 2 No basis given · 1 Own result

Verify all 7 references — free account

Show all 6 claims

  • Probably not a substantial claim. 12
  • It may state two separate claims as one. 3
12 of 12
  • The paper relies on this without support.

    A significant number of contemporary AI diagnostic studies include information related to both the development and testing (validation) of AI-centred index tests.

    “A significant number of contemporary AI diagnostic studies include information related to both the development and testing (validation) of AI-centred index tests.”
  • The paper relies on this without support.

    Much of the evidence supporting diagnostic algorithms has been disseminated in the absence of AI-specific reporting guidelines.

    “Although this paradigm shift may seem imminent, it is crucial to note that much of the evidence supporting diagnostic algorithms has been disseminated in the absence of AI-specific reporting guidelines.”
2 of 2

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

2 of 2

Citations

Cites

Show all 18 references

Cited by

Sorted by citations

Items 101–200 of 341

Metadata

Authors
Sounderajah, V., Ashrafian, H., Golub, R. M., Shetty, S., De Fauw, J., Hooft, L., Moons, K., Collins, G., Moher, D., Bossuyt, P. M., Darzi, A., Karthikesalingam, A., Denniston, A. K., Mateen, B. A., Ting, D., Treanor, D., King, D., Greaves, F., Godwin, J., Pearson-Stuttard, J., Harling, L., McInnes, M., Rifai, N., Tomasev, N., Normahani, P., Whiting, P., Aggarwal, R., Vollmer, S., Markar, S. R., Panch, T., & Liu, X.
Affiliations
  • Imperial College London (GB)
  • American Medical Association (US)
  • Google Health, Pal Alto, California, USA
  • DeepMind Technologies Ltd, London, UK
  • Utrecht University (NL)
  • Centre for Statistics in Medicine, Nuffield Department of Orthopaedics, Rheumatology and Musculoskeletal Sciences, University of Oxford, Oxford, UK
  • Ottawa Hospital Research Institute (CA)
  • Amsterdam University Medical Centers (NL)
  • Google (United Kingdom) (GB)
  • University Hospitals Birmingham NHS Foundation Trust (GB)
  • The Alan Turing Institute (GB)
  • Singapore National Eye Center (SG)
  • University of Leeds (GB)
  • Optum, Paddington, London, UK
  • Google DeepMind (United Kingdom) (GB)
  • School of Public Health, Imperial College London, London, UK
  • University of Ottawa (CA)
  • Harvard University (US)
  • University of Bristol (GB)
Corresponding author
Ashrafian, Hutan
Venue
BMJ Open
Publisher
BMJ
Published
2021-06
Volume
11
Issue
6
Pages
e047709
ISSN
2044-6055 (ISSN-L)
OpenAlex
W3173393692
Cited by
333
References
18
Citations by year
2026: 79 · 2025: 76 · 2024: 82 · 2023: 57 · 2022: 32 · 2021: 7
FWCI
25.80
Type
journal article
Language
en
Open access
Locations
  • BMJ Open (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • Pure Amsterdam UMC (publishedVersion, other-oa)
  • University of Birmingham Research Portal (University of Birmingham) (publishedVersion, cc-by)
  • Data Archiving and Networked Services (DANS) (submittedVersion, other-oa)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
  • Utrecht University Repository (Utrecht University) (submittedVersion, other-oa)
  • Spiral (Imperial College London) (submittedVersion, other-oa)
Funders
  • NIHR Applied Research Collaboration (No grant number)
  • National Pathology Imaging Co-operative (No grant number)
  • Industrial Strategy Challenge Fund, UK Research and Innovation (No grant number)
  • NIHR Imperial Biomedical Research Centre (No grant number)
  • Cancer Research UK Programme Grant (C49297/A27294)
  • NIHR Oxford Biomedical Research Centre (No grant number)
Topics
  • Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
  • Social Sciences › Sociology and Political Science › Delphi Technique in Research
Keywords
reporting guidelines, artificial intelligence, diagnostic accuracy, Delphi consensus, transparency in research reporting
MeSH terms
Artificial Intelligence, Diagnostic Tests, Routine, Humans, London, Research Design, Research Report

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review