CoChat
Sign Up
Cited by 524Meta-analysis and systematic reviewsStatistical Methods in EpidemiologyReliability and Agreement in Measurement

The DOI matches its Crossref record; 175 of the 175 references it cites were found in a registry or an index.

Evaluation of diagnostic tests when there is no gold standard. A review of methods

Open access
Rutjes, A, Reitsma, J, Coomarasamy, A, Khan, K, Bossuyt, P·2007·Health Technology Assessment
DOI 10.3310/hta11500
Open this paper in CoChat — free accountOpen PDF

Abstract

OBJECTIVE: To generate a classification of methods to evaluate medical tests when there is no gold standard. METHODS: Multiple search strategies were employed to obtain an overview of the different methods described in the literature, including searches of electronic databases, contacting experts for papers in personal archives, exploring databases from previous methodological projects and cross-checking of reference lists of useful papers already identified. RESULTS: All methods available were classified into four main groups. The first method group, impute or adjust for missing data on reference standard, needs careful attention to the pattern and fraction of missing values. The second group, correct imperfect reference standard, can be useful if there is reliable information about the degree of imperfection of the reference standard and about the correlation of the errors between the index test and the reference standard. The third group of methods, construct reference standard, have in common that they combine multiple test results to construct a reference standard outcome including deterministic predefined rules, consensus procedures and statistical modelling (latent class analysis). In the final group, validate index test results, the diagnostic test accuracy paradigm is abandoned and research examines, using a number of different methods, whether the results of an index test are meaningful in practice, for example by relating index test results to relevant other clinical characteristics and future clinical events. CONCLUSIONS: The majority of methods try to impute, adjust or construct a reference standard in an effort to obtain the familiar diagnostic accuracy statistics, such as sensitivity and specificity. In situations that deviate only marginally from the classical diagnostic accuracy paradigm, these are valuable methods. However, in situations where an acceptable reference standard does not exist, applying the concept of clinical test validation can provide a significant methodological advance. All methods summarised in this report need further development. Some methods, such as the construction of a reference standard using panel consensus methods and validation of tests outwith the accuracy paradigm, are particularly promising but are lacking in methodological research. These methods deserve particular attention in future research.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.3310/hta11500

Cites

175 of 175 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Research works printed but not linked: 13

  1. [6]Lijmer JG, Mol BWJ, Bonsel GJ, Prins MH, Bossuyt PM. Strategies for the evaluation of diagnostic technologies. Evaluation of diagnostic tests: from accuracy to outcome (Thesis). Amsterdam; 2001. pp. 15-28.
  2. [19]Whiting P , Rutjes AW, Reitsma JB, Glas AS, Bossuyt PM, Kleijnen J. Sources of variation and bias in studies of diagnostic accuracy: a systematic review. Ann Intern Med 2004; 140 :189-202.
  3. [20]Rutjes AW, Reitsma JB, Irwig L, Bossuyt PM. Partial and differential bias in diagnostic accuracy studies. Sources of bias and variation in diagnostic accuracy studies (Thesis). Amsterdam; 2005. pp. 31-44.
  4. [27]Cochrane Handbook for Systematic Reviews of Interventions 4.2.5 [updated May 2005]. Chichester Wiley; 2006.
  5. [32]Diamond GA. Off Bayes: effect of verification bias on posterior probabilities calculated using Bayes' theorem. Med Decis Making 1992; 12 :22-31.
  6. [41]Zhou XH, Higgs RE. Assessing the relative accuracies of two screening tests in the presence of verification bias. Stat Med 2000; 19 :1697-705.
  7. [42]Molenberghs G, Goetghebeur EJ, Lipsitz SR, Kenward MG, Lesaffre E, Michiels B. Missing data perspectives of the fluvoxamine data set: a review. Stat Med 1999; 18 :2449-64.
  8. [63]Kline JA, Israel EG, Michelson EA, O'Neil BJ, Plewa MC, Portelli DC. Diagnostic accuracy of a bedside Ddimer assay and alveolar dead-space measurement for rapid exclusion of pulmonary embolism: a multicenter study. JAMA 2001; 285 :761-8.
  9. [68]Hadgu A. Bias in the evaluation of DNAamplification tests for detecting Chlamydia trachomatis. Stat Med 1997; 16 :1391-9.
  10. [87]Committee on Quality Management, Japan Society of Clinical Chemistry. Proposed standard for matrix reference materials for quantitative laboratory methods. JPN J Clin Chem 2003; 32 :180-5.
  11. [97]Bertrand P , Benichou J, Grenier P , Chastang C. Hui and Walter's latent-class reference-free approach may be more useful in assessing agreement than diagnostic performance. J Clin Epidemiol 2005; 58 :688-700.
  12. [103]Fletcher RH, Fletcher SW, Wagner EH. Abnormality. In Satterfield TS, editor. Clinical epidemiology: the essentials . Baltimore, MD: Williams and Wilkins; 1996. pp. 22-24.
  13. [107]Fryback DG, Thornbury JR. The efficacy of diagnostic imaging. Med Decis Making 1991; 11 : 88-94.

Cited by

Sorted by citations

See all 524

Related

See all

Metadata

Authors
Rutjes, A., Reitsma, J., Coomarasamy, A., Khan, K., & Bossuyt, P.
Affiliations
  • Amsterdam UMC Location University of Amsterdam (NL)
  • University of Birmingham (GB)
ORCID
Venue
Health Technology Assessment
Publisher
National Institute for Health and Care Research
Published
2007-12
Volume
11
Issue
50
Pages
iii, ix-51
ISSN
  • 1366-5278 (ISSN-L)
  • 2046-4924
DOI
10.3310/hta11500
OpenAlex
W2143712591
Cited by
524
References
175
Citations by year
2026: 21 · 2025: 21 · 2024: 20 · 2023: 23 · 2022: 36 · 2021: 34 · 2020: 39 · 2019: 38 · 2018: 57 · 2017: 31 · 2016: 33 · 2015: 27 · 2014: 29 · 2013: 27 · 2012: 30
FWCI
2.43
Type
journal article
Language
en
Open access
Licence
Locations
  • Health Technology Assessment (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • Pure Amsterdam UMC (publishedVersion)
  • IRIS UNIMORE (University of Modena and Reggio Emilia) (submittedVersion, cc-by)
  • Data Archiving and Networked Services (DANS) (submittedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
  • IRIS UNIMORE (University of Modena and Reggio Emilia) (submittedVersion)
  • University of Birmingham Research Portal (University of Birmingham) (submittedVersion)
Funders
Health Technology Assessment programme (06/90/23)
Topics
  • Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
  • Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Epidemiology
  • Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Reliability and Agreement in Measurement
Keywords
diagnostic accuracy, absence of gold standard, reference materials, imperfect reference standard, latent class analysis, sensitivity and specificity, methodological review
MeSH terms
Humans, Reference Standards, Sensitivity and Specificity, Process Assessment, Health Care, Diagnostic Techniques and Procedures

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review