The DOI matches its Crossref record; 175 of the 175 references it cites were found in a registry or an index.
Evaluation of diagnostic tests when there is no gold standard. A review of methods
Open accessAbstract
OBJECTIVE: To generate a classification of methods to evaluate medical tests when there is no gold standard. METHODS: Multiple search strategies were employed to obtain an overview of the different methods described in the literature, including searches of electronic databases, contacting experts for papers in personal archives, exploring databases from previous methodological projects and cross-checking of reference lists of useful papers already identified. RESULTS: All methods available were classified into four main groups. The first method group, impute or adjust for missing data on reference standard, needs careful attention to the pattern and fraction of missing values. The second group, correct imperfect reference standard, can be useful if there is reliable information about the degree of imperfection of the reference standard and about the correlation of the errors between the index test and the reference standard. The third group of methods, construct reference standard, have in common that they combine multiple test results to construct a reference standard outcome including deterministic predefined rules, consensus procedures and statistical modelling (latent class analysis). In the final group, validate index test results, the diagnostic test accuracy paradigm is abandoned and research examines, using a number of different methods, whether the results of an index test are meaningful in practice, for example by relating index test results to relevant other clinical characteristics and future clinical events. CONCLUSIONS: The majority of methods try to impute, adjust or construct a reference standard in an effort to obtain the familiar diagnostic accuracy statistics, such as sensitivity and specificity. In situations that deviate only marginally from the classical diagnostic accuracy paradigm, these are valuable methods. However, in situations where an acceptable reference standard does not exist, applying the concept of clinical test validation can provide a significant methodological advance. All methods summarised in this report need further development. Some methods, such as the construction of a reference standard using panel consensus methods and validation of tests outwith the accuracy paradigm, are particularly promising but are lacking in methodological research. These methods deserve particular attention in future research.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“We based our literature search on the principle of theoretical saturation [17, 18], with the aim of identifying all relevant recommendations, rather than all relevant articles.”
From Methods to systematically review and meta-analyse observational studies: a systematic scoping review of recommendations (2018) · cites this paper as [18]
In the citing paper
“Locating methodological papers in electronic databases was challenging: relevant articles may be indexed differently and there is no key word to search for this type of article [18].”
From Methods to systematically review and meta-analyse observational studies: a systematic scoping review of recommendations (2018) · cites this paper as [18]
In the citing paper
“However, our search was guided by the methodological principle of theoretical saturation [17, 18].”
From Methods to systematically review and meta-analyse observational studies: a systematic scoping review of recommendations (2018) · cites this paper as [18]
In the citing paper
“Once a set of methodological articles covering all relevant methodological aspects is identified, additional searches add little to the ideas already present in the initial set of articles.”
From Methods to systematically review and meta-analyse observational studies: a systematic scoping review of recommendations (2018) · cites this paper as [18]
In the citing paper
“Theoretical saturation suggests that identifying every article is not required in methodological research.”
From Methods to systematically review and meta-analyse observational studies: a systematic scoping review of recommendations (2018) · cites this paper as [18]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Cochrane Handbook for Systematic Reviews of Interventions: Cochrane Book Series
(2008) · doi:10.1002/9780470712184 · Cited by 25,048
Statistical Analysis With Missing Data
Lahiff, M., Little, R. J. A., & Rubin, D. B. (1989) · Journal of the American Statistical Association · doi:10.2307/2289883 · Cited by 17,385
International Organization for Standardization
(2011) · Springer eBooks · doi:10.1007/springerreference_16847 · Cited by 12,504
Construct validity in psychological tests.
Cronbach, L. J., & Meehl, P. E. (1955) · Psychological Bulletin · doi:10.1037/h0040957 · Cited by 11,195
Health Measurement Scales: A Practical Guide to Their Development and Use
Streiner, D. L., Norman, G. R., & Cairney, J. (1989) · Cited by 9,595
Multiple imputation after 18+ years
Rubin, D. B. (1996) · Journal of the American Statistical Association · doi:10.1080/01621459.1996.10476908 · Cited by 4,072
Schafer, J. L. (1999) · Statistical Methods in Medical Research · doi:10.1177/096228029900800102 · Cited by 3,949
Qualitative Research: Consensus methods for medical and health services research
Jones, J., & Hunter, D. (1995) · BMJ · doi:10.1136/bmj.311.7001.376 · Cited by 3,471
Multiple Imputation After 18+ Years
Rubin, D. B. (1996) · Journal of the American Statistical Association · doi:10.2307/2291635 · Cited by 2,686
Health Measurement Scales: A Practical Guide to Their Development and Use
Vogelzang, J. L. (2015) · Journal of Nutrition Education and Behavior · doi:10.1016/j.jneb.2015.03.007 · Cited by 2,393
Flagged first, then by citations
Research works printed but not linked: 13
- [6]Lijmer JG, Mol BWJ, Bonsel GJ, Prins MH, Bossuyt PM. Strategies for the evaluation of diagnostic technologies. Evaluation of diagnostic tests: from accuracy to outcome (Thesis). Amsterdam; 2001. pp. 15-28.
- [19]Whiting P , Rutjes AW, Reitsma JB, Glas AS, Bossuyt PM, Kleijnen J. Sources of variation and bias in studies of diagnostic accuracy: a systematic review. Ann Intern Med 2004; 140 :189-202.
- [20]Rutjes AW, Reitsma JB, Irwig L, Bossuyt PM. Partial and differential bias in diagnostic accuracy studies. Sources of bias and variation in diagnostic accuracy studies (Thesis). Amsterdam; 2005. pp. 31-44.
- [27]Cochrane Handbook for Systematic Reviews of Interventions 4.2.5 [updated May 2005]. Chichester Wiley; 2006.
- [32]Diamond GA. Off Bayes: effect of verification bias on posterior probabilities calculated using Bayes' theorem. Med Decis Making 1992; 12 :22-31.
- [41]Zhou XH, Higgs RE. Assessing the relative accuracies of two screening tests in the presence of verification bias. Stat Med 2000; 19 :1697-705.
- [42]Molenberghs G, Goetghebeur EJ, Lipsitz SR, Kenward MG, Lesaffre E, Michiels B. Missing data perspectives of the fluvoxamine data set: a review. Stat Med 1999; 18 :2449-64.
- [63]Kline JA, Israel EG, Michelson EA, O'Neil BJ, Plewa MC, Portelli DC. Diagnostic accuracy of a bedside Ddimer assay and alveolar dead-space measurement for rapid exclusion of pulmonary embolism: a multicenter study. JAMA 2001; 285 :761-8.
- [68]Hadgu A. Bias in the evaluation of DNAamplification tests for detecting Chlamydia trachomatis. Stat Med 1997; 16 :1391-9.
- [87]Committee on Quality Management, Japan Society of Clinical Chemistry. Proposed standard for matrix reference materials for quantitative laboratory methods. JPN J Clin Chem 2003; 32 :180-5.
- [97]Bertrand P , Benichou J, Grenier P , Chastang C. Hui and Walter's latent-class reference-free approach may be more useful in assessing agreement than diagnostic performance. J Clin Epidemiol 2005; 58 :688-700.
- [103]Fletcher RH, Fletcher SW, Wagner EH. Abnormality. In Satterfield TS, editor. Clinical epidemiology: the essentials . Baltimore, MD: Williams and Wilkins; 1996. pp. 22-24.
- [107]Fryback DG, Thornbury JR. The efficacy of diagnostic imaging. Med Decis Making 1991; 11 : 88-94.
Cited by
Sorted by citations
Moons, K. G. M., Altman, D. G., Reitsma, J. B., Ioannidis, J. P. A., Macaskill, P., Steyerberg, E. W., Vickers, A. J., Ransohoff, D. F., & Collins, G. S. (2015) · Annals of Internal Medicine · doi:10.7326/m14-0698 · Cited by 5,147
Self-citationMoons, K. G. M., de Groot, J. A. H., Bouwmeester, W., Vergouwe, Y., Mallett, S., Altman, D. G., Reitsma, J. B., & Collins, G. S. (2014) · PLoS Medicine · doi:10.1371/journal.pmed.1001744 · Cited by 2,018
Self-citationMoons, K. G. M., Wolff, R. F., Riley, R. D., Whiting, P. F., Westwood, M., Collins, G. S., Reitsma, J. B., Kleijnen, J., & Mallett, S. (2019) · Annals of Internal Medicine · doi:10.7326/m18-1377 · Cited by 1,669
Self-citationHeilbronner, R. L., Sweet, J. J., Morgan, J. E., Larrabee, G. J., Millis, S. R., & Participants1, C. (2009) · The Clinical Neuropsychologist · doi:10.1080/13854040903155063 · Cited by 869
Proportion of hospital readmissions deemed avoidable: a systematic review
van Walraven, C., Bennett, C. L., Jennings, A., Austin, P. C., & Forster, A. J. (2011) · Canadian Medical Association Journal · doi:10.1503/cmaj.101860 · Cited by 732
Metadata
- Authors
- Rutjes, A., Reitsma, J., Coomarasamy, A., Khan, K., & Bossuyt, P.
- Affiliations
- Amsterdam UMC Location University of Amsterdam (NL)
- University of Birmingham (GB)
- Venue
- Health Technology Assessment
- Published
- 2007-12
- Volume
- 11
- Issue
- 50
- Pages
- iii, ix-51
- ISSN
- 1366-5278 (ISSN-L)
- 2046-4924
- OpenAlex
- W2143712591
- Cited by
- 524
- References
- 175
- Citations by year
- 2026: 21 · 2025: 21 · 2024: 20 · 2023: 23 · 2022: 36 · 2021: 34 · 2020: 39 · 2019: 38 · 2018: 57 · 2017: 31 · 2016: 33 · 2015: 27 · 2014: 29 · 2013: 27 · 2012: 30
- FWCI
- 2.43
- Type
- journal article
- Language
- en
- Open access
- diamond
- doi.org
- Locations
- Health Technology Assessment (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Pure Amsterdam UMC (publishedVersion)
- IRIS UNIMORE (University of Modena and Reggio Emilia) (submittedVersion, cc-by)
- Data Archiving and Networked Services (DANS) (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
- IRIS UNIMORE (University of Modena and Reggio Emilia) (submittedVersion)
- University of Birmingham Research Portal (University of Birmingham) (submittedVersion)
- Funders
- Health Technology Assessment programme (06/90/23)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Epidemiology
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Reliability and Agreement in Measurement
- Keywords
- diagnostic accuracy, absence of gold standard, reference materials, imperfect reference standard, latent class analysis, sensitivity and specificity, methodological review
- MeSH terms
- Humans, Reference Standards, Sensitivity and Specificity, Process Assessment, Health Care, Diagnostic Techniques and Procedures
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
