CoChat
Sign Up
Cited by 341Meta-analysis and systematic reviewsArtificial Intelligence in Healthcare and EducationDelphi Technique in Research

Developing a reporting guideline for artificial intelligence-centred diagnostic test accuracy studies: the STARD-AI protocol

Open access
Sounderajah, Viknesh, Ashrafian, Hutan, Golub, Robert M, Shetty, Shravya, De Fauw, Jeffrey, Hooft, Lotty, Moons, Karel, Collins, Gary +23 more·2021·BMJ Open
DOI 10.1136/bmjopen-2020-047709PMC PMC8240576
Open this paper in CoChat — free account

Abstract

Introduction: Standards for Reporting of Diagnostic Accuracy Study (STARD) was developed to improve the completeness and transparency of reporting in studies investigating diagnostic test accuracy. However, its current form, STARD 2015 does not address the issues and challenges raised by artificial intelligence (AI)-centred interventions. As such, we propose an AI-specific version of the STARD checklist (STARD-AI), which focuses on the reporting of AI diagnostic test accuracy studies. This paper describes the methods that will be used to develop STARD-AI. Methods and analysis: The development of the STARD-AI checklist can be distilled into six stages. (1) A project organisation phase has been undertaken, during which a Project Team and a Steering Committee were established; (2) An item generation process has been completed following a literature review, a patient and public involvement and engagement exercise and an online scoping survey of international experts; (3) A three-round modified Delphi consensus methodology is underway, which will culminate in a teleconference consensus meeting of experts; (4) Thereafter, the Project Team will draft the initial STARD-AI checklist and the accompanying documents; (5) A piloting phase among expert users will be undertaken to identify items which are either unclear or missing. This process, consisting of surveys and semistructured interviews, will contribute towards the explanation and elaboration document and (6) On finalisation of the manuscripts, the group’s efforts turn towards an organised dissemination and implementation strategy to maximise end-user adoption. Ethics and dissemination: Ethical approval has been granted by the Joint Research Compliance Office at Imperial College London (reference number: 19IC5679). A dissemination strategy will be aimed towards five groups of stakeholders: (1) academia, (2) policy, (3) guidelines and regulation, (4) industry and (5) public and non-specific stakeholders. We anticipate that dissemination will take place in Q3 of 2021.

AI snapshot

Written from the paper’s full text.

TL;DR

The STARD-AI protocol describes the planned development of an AI-specific reporting guideline for diagnostic test accuracy studies, extending STARD 2015. It outlines a six-stage process including item generation and a three-round modified Delphi consensus, with publication anticipated in Q3 2021.

Key findings

  • STARD 2015 is the most widely accepted reporting standard for diagnostic test accuracy studies, consisting of 30 items and endorsed by over 200 biomedical journals, but it does not address AI-specific issues such as data preprocessing, model development choices, external validation datasets, standardised nomenclature, and unfamiliar outcome measures.
  • STARD-AI development is distilled into six stages: project organisation; item generation; a three-round modified Delphi consensus; drafting the checklist and accompanying documents; piloting among expert users; and finalisation, publication, and dissemination.
  • Item generation included a January 2020 literature review, a February 2020 online scoping survey of 80 international experts that generated over 2500 responses, and a patient and public involvement and engagement focus group.
  • The modified Delphi uses a 5-point Likert-like scale with a predefined consensus threshold of ≥75%; items rated 1 or 2 by ≥75% are deemed essential, items rated 4 or 5 by ≥75% are excluded, and non-consensus items proceed to later rounds or the virtual consensus meeting.

Methods

  • Protocol constructed in accordance with the EQUATOR Network toolkit for developing reporting guidelines, with a nine-member Project Team and a multidisciplinary Steering Committee.
  • Item generation combined a literature review of MEDLINE and EMBASE via Ovid, an online scoping survey of 80 experts, and a remote patient and public involvement and engagement focus group; findings were classified into four categories: general considerations, suggested modifications, additions, and removals to STARD 2015 items.
  • A pragmatic modified Delphi consensus process is underway: rounds 1 and 2 are online surveys using DelphiManager V.4.0, and round 3 is a virtual teleconference consensus meeting using the nominal group technique; piloting will use surveys and semistructured interviews with thematic analysis.

Limitations

  • The paper states: “Views of Delphi panellists may differ from those experts who decline participation.”
  • Evident from the text: this is a protocol, so no final STARD-AI checklist, Delphi consensus results, pilot findings, or implementation outcomes are reported.
  • Evident from the text: development relies on expert consensus and selected stakeholder groups; no formal validation or reliability testing of the eventual checklist is described in this protocol.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1136/bmjopen-2020-047709

Cites

18 of 18 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

0 of 7 cited sources checked · 0 supported · 0 flagged

  • Interpretationreasoning

    Without AI-specific reporting guidance, key stakeholders are poorly placed to appraise quality and compare diagnostic accuracy between scientific studies.

    “Without this guidance, and in a relatively nascent area, key stakeholders are poorly placed to appraise quality and compare diagnostic accuracy between scientific studies.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: Much of the evidence supporting diagnostic algorithms has been disseminated in the absence of AI-specific reporting guidelines.
  • Premisecitation

    STARD 2015 is the most widely accepted set of reporting standards for diagnostic test accuracy studies.

    “The Standards for Reporting of Diagnostic Accuracy Studies (STARD) 2015 statement remains the most widely accepted set of reporting standards for diagnostic test accuracy studies.5”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [5] source

      Not checked in the free run.

  • Premisereasoning

    STARD 2015 does not address AI-specific issues such as unclear methodological interpretation, lack of standardised nomenclature, and unfamiliar outcome measures.

    “Issues include unclear methodological interpretation (eg, data preprocessing steps, model development choices and the use of external validation datasets), the lack of standardised nomenclature (eg, the varying definition of the term ‘validation’), as well as the use of unfamiliar outcome measures (eg, Jaccard similarity coefficient and F-score).”
    Quote in the paperSubstantial claim
  • Premisecitation

    Adherence to the STARD checklist leads to improved reporting of key study parameters.

    “studies have shown that adherence to the STARD checklist leads to improved reporting of key study parameters.7 8”
    Quote in the paperSubstantial claim0 of 2 sources checked
    • cites [7] Korevaar, 2014, Reporting quality of diagnostic accuracy studies: a systematic review and meta-analysis of investigations on adherence to STARD, Evid Based Med source

      Not checked in the free run.

    • cites [8] source

      Not checked in the free run.

  • Premisecitation

    STARD 2015 does not address the issues and challenges raised by AI-centred interventions.

    “However, its current form, STARD 2015 does not address the issues and challenges raised by artificial intelligence (AI)-centred interventions.”
    • Also stated in: § Introduction · plain
      “However, in its current iteration, STARD 2015 is not designed to address the issues and challenges raised by AI-driven modalities.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [5] source

      Not checked in the free run.

  • Classified as probably not a substantial claim. 12
  • May combine two claims. 3

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingown data

    The online scoping survey generated over 2500 responses, which were analysed and classed into four broad categories.

    “This process generated over 2500 responses, which were analysed and classed into the aforementioned four broad categories.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Study design and conductnone given

    The threshold for consensus is predefined at ≥75%.

    “The threshold for consensus is predefined at ≥75%.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    Ensuring fair representation across geographies and demographics was a pertinent consideration during recruitment.

    “Moreover, ensuring fair representation across geographies and demographics was a pertinent consideration during recruitment.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Premisenone given

    There are no specific reporting standards for artificial intelligence (AI) diagnostic test accuracy studies.

    “There are no specific reporting standards for artificial intelligence (AI) diagnostic test accuracy studies.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    Prior to stage 2, the STARD-AI project was registered with the EQUATOR Network.

    “Prior to stage 2, the STARD-AI project was registered with the EQUATOR Network.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based).

    Interpretationnone given

    Views of Delphi panellists may differ from those experts who decline participation.

    “Views of Delphi panellists may differ from those experts who decline participation.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductown data

    Two investigators (VS and HA) independently identified common themes and subthemes from the PPIE discussion, which were classed into four broad categories.

    “Two investigators (VS and HA) independently identified common themes and subthemes from the discussion, which were classed into the aforementioned four broad categories.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Study design and conductnone given

    This study has adopted a pragmatic modified Delphi consensus methodology.

    “This study has adopted a pragmatic modified Delphi consensus methodology.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductcitation

    This protocol has been constructed in accordance with the EQUATOR Network toolkit for developing reporting guidelines.

    “This protocol has been constructed in accordance with the EQUATOR Network toolkit for developing reporting guidelines.14”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [14] The EQUATOR Network . Toolkits. Available: https://www.equator-network.org/toolkits/ [Accessed 26 Sep 2020].

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    The Delphi consensus methodology is a well-established method of obtaining a collective opinion from a group of experts through a series of questionnaires.

    “The Delphi consensus methodology is a well-established method17 of obtaining a collective opinion from a group of experts through a series of questionnaires; each one refined based on feedback from respondents.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [17] Brown BB . Delphi process: a methodology used for the elicitation of opinions of experts. published online first, 1968. Available: https://www.rand.org/pubs/papers/P3925.html [Accessed 26 Sep 2020].

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductown data

    Two individuals (VS and HA) individually screened study titles and abstracts for inclusion, and disagreements were resolved through discussion.

    “Two individuals (VS and HA) individually screened study titles and abstracts for inclusion. Disagreements were resolved through discussion.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable).

    Interpretationreasoning

    Until these issues are addressed, achieving comprehensive evaluations of these technologies and their potential translational benefits will remain limited.

    “Until these issues are addressed, achieving comprehensive evaluations of these technologies and their potential translational benefits will remain limited.”
    Quote in the paperNot a substantial claim
12 of 12

  • The paper relies on this without support.

    A significant number of contemporary AI diagnostic studies include information related to both the development and testing (validation) of AI-centred index tests.

    “A significant number of contemporary AI diagnostic studies include information related to both the development and testing (validation) of AI-centred index tests.”

  • The paper relies on this without support.

    Much of the evidence supporting diagnostic algorithms has been disseminated in the absence of AI-specific reporting guidelines.

    “Although this paradigm shift may seem imminent, it is crucial to note that much of the evidence supporting diagnostic algorithms has been disseminated in the absence of AI-specific reporting guidelines.”
2 of 2

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • STARD-AI provides a framework for reporting AI-based diagnostic accuracy studies.

    “while STARD-AI provides a framework for reporting AI-based diagnostic accuracy studies [71].”

    From Implementing Artificial Intelligence in Critical Care Medicine: a consensus of 22. (2025) · cites this paper as [71]

    The Introduction’s “Focus of STARD-AI” section says STARD-AI is intended to aid reporting of AI diagnostic test accuracy research. But the Abstract and Conclusion describe it as a proposed guideline still being developed, not a framework this protocol already provides. (LLM, conf=0.88)

    Experimental — accuracy not yet measured

  • A protocol that outlined the process for developing STARD-AI was published.

    “A protocol that outlined the process for developing STARD-AI was subsequently published 56 .”

    From The STARD-AI reporting guideline for diagnostic accuracy studies using artificial intelligence (2025) · cites this paper as [56]

  • Some existing AI tools for item generation are costly, need high technical support, and have good results.

    “Some of these are costly, need high technical support, and have good results [24–27].”

    From AI in medical education: uses of AI in construction type A MCQs (2024) · cites this paper as [25]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

  • The fifth dissemination stakeholder group differs

    Minor

    The abstract names public and non-specific stakeholders as the fifth dissemination group, whereas the detailed strategy names patient representing bodies.

  • Different initials identify the same project-team member

    Minor

    The project-team description identifies the former UK Minister for Health and chair of the NHS Accelerated Access Collaborative as AD, while the glossary identifies that person as AKD.

2 of 2

Cites

Flagged first, then by citations

Cited by

Sorted by citations

Related

Authors
Sounderajah, V., Ashrafian, H., Golub, R. M., Shetty, S., De Fauw, J., Hooft, L., Moons, K., Collins, G., Moher, D., Bossuyt, P. M., Darzi, A., Karthikesalingam, A., Denniston, A. K., Mateen, B. A., Ting, D., Treanor, D., King, D., Greaves, F., Godwin, J., Pearson-Stuttard, J., Harling, L., McInnes, M., Rifai, N., Tomasev, N., Normahani, P., Whiting, P., Aggarwal, R., Vollmer, S., Markar, S. R., Panch, T., & Liu, X.
Affiliations
  • Imperial College London (GB)
  • American Medical Association (US)
  • Google Health, Pal Alto, California, USA
  • DeepMind Technologies Ltd, London, UK
  • Utrecht University (NL)
  • Centre for Statistics in Medicine, Nuffield Department of Orthopaedics, Rheumatology and Musculoskeletal Sciences, University of Oxford, Oxford, UK
  • Ottawa Hospital Research Institute (CA)
  • Amsterdam University Medical Centers (NL)
  • Google (United Kingdom) (GB)
  • University Hospitals Birmingham NHS Foundation Trust (GB)
  • The Alan Turing Institute (GB)
  • Singapore National Eye Center (SG)
  • University of Leeds (GB)
  • Optum, Paddington, London, UK
  • Google DeepMind (United Kingdom) (GB)
  • School of Public Health, Imperial College London, London, UK
  • University of Ottawa (CA)
  • Harvard University (US)
  • University of Bristol (GB)
Corresponding author
Ashrafian, Hutan
ORCID
Venue
BMJ Open
Publisher
BMJ
Published
2021-06
Volume
11
Issue
6
Pages
e047709
ISSN
2044-6055 (ISSN-L)
DOI
10.1136/bmjopen-2020-047709
PMC
PMC8240576
OpenAlex
W3173393692
Cited by
333
References
18
Citations by year
2026: 79 · 2025: 76 · 2024: 82 · 2023: 57 · 2022: 32 · 2021: 7
FWCI
25.80
Type
journal article
Language
en
Open access
Licence
Locations
  • BMJ Open (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • Pure Amsterdam UMC (publishedVersion, other-oa)
  • University of Birmingham Research Portal (University of Birmingham) (publishedVersion, cc-by)
  • Data Archiving and Networked Services (DANS) (submittedVersion, other-oa)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
  • Utrecht University Repository (Utrecht University) (submittedVersion, other-oa)
  • Spiral (Imperial College London) (submittedVersion, other-oa)
Funders
  • NIHR Applied Research Collaboration (No grant number)
  • National Pathology Imaging Co-operative (No grant number)
  • Industrial Strategy Challenge Fund, UK Research and Innovation (No grant number)
  • NIHR Imperial Biomedical Research Centre (No grant number)
  • Cancer Research UK Programme Grant (C49297/A27294)
  • NIHR Oxford Biomedical Research Centre (No grant number)
Topics
  • Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
  • Social Sciences › Sociology and Political Science › Delphi Technique in Research
Keywords
reporting guidelines, artificial intelligence, diagnostic accuracy, Delphi consensus, transparency in research reporting
MeSH terms
Artificial Intelligence, Diagnostic Tests, Routine, Humans, London, Research Design, Research Report

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-10.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-10.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review