CoChat
Sign Up
Cited by 59Meta-analysis and systematic reviewsStatistical Methods in EpidemiologyReliability and Agreement in Measurement

Chapter 9: Options for Summarizing Medical Test Performance in the Absence of a “Gold Standard”

Open access
Trikalinos, Thomas A., Balion, Cynthia M.·2012·Journal of General Internal Medicine
DOI 10.1007/s11606-012-2031-7PMC PMC3364362
Open this paper in CoChat — free accountOpen PDF

Abstract

The classical paradigm for evaluating test performance compares the results of an index test with a reference test. When the reference test does not mirror the “truth” adequately well (e.g. is an “imperfect” reference standard), the typical (“naïve”) estimates of sensitivity and specificity are biased. One has at least four options when performing a systematic review of test performance when the reference standard is “imperfect”: (a) to forgo the classical paradigm and assess the index test’s ability to predict patient relevant outcomes instead of test accuracy (i.e., treat the index test as a predictive instrument); (b) to assess whether the results of the two tests (index and reference) agree or disagree (i.e., treat them as two alternative measurement methods); (c) to calculate “naïve” estimates of the index test’s sensitivity and specificity from each study included in the review and discuss in which direction they are biased; (d) mathematically adjust the “naïve” estimates of sensitivity and specificity of the index test to account for the imperfect reference standard. We discuss these options and illustrate some of them through examples.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1007/s11606-012-2031-7

Cites

1 of 1 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 33

  1. [1]Bossuyt PM Interpreting diagnostic test accuracy studies Semin Hematol 2008 45 3 189 195 10.1053/j.seminhematol.2008.04.001 18582626 · DOI
  2. [2]Bossuyt PM, Reitsma JB, Bruns DE, et al. Towards complete and accurate reporting of studies of diagnostic accuracy: The STARD Initiative. Radiology. 2003;226(1):24–28. 12511664
  3. [4]Whiting P Rutjes AW Reitsma JB Glas AS Bossuyt PM Kleijnen J Sources of variation and bias in studies of diagnostic accuracy: a systematic review Ann Intern Med 2004 140 3 189 202 14757617
  4. [5]Trikalinos TA, Balion CM, Coleman CI, et al. Chapter 8: Meta-analysis of test performance when there is a "Gold Standard." J Gen Internal Med. 2012; doi: 10.1007/s11606-012-2029-1 PMC3364353 22648676 · DOI
  5. [6]Reitsma JB Rutjes AW Khan KS Coomarasamy A Bossuyt PM A review of solutions for diagnostic accuracy studies with an imperfect or missing reference standard J Clin Epidemiol 2009 62 8 797 806 10.1016/j.jclinepi.2009.02.005 19447581 · DOI · Paper page
  6. [7]Jonas DE, Wilt TJ, Taylor BC, Wilkins TM, Matchar DB. Chapter 11: Challenges in and principles for conducting systematic reviews of genetic tests used as predictive indicators. J Gen Internal Med. 2011; doi: 10.1007/s11606-011-1898-z PMC3364361 22648679 · DOI
  7. [8]Sun S Meta-analysis of Cohen's kappa Health Serv Outcomes Res Method 2011 11 145 163 10.1007/s10742-011-0077-3 · DOI · Paper page
  8. [10]Bablok W Passing H Bender R Schneider B A general regression procedure for method transformation. Application of linear regression procedures for method comparison studies in clinical chemistry, Part III J Clin Chem Clin Biochem 1988 26 11 783 790 3235954
  9. [11]Linnet K Estimation of the linear relationship between the measurements of two methods with proportional errors Stat Med 1990 9 12 1463 1473 10.1002/sim.4780091210 2281234 · DOI
  10. [12]Linnet K Performance of Deming regression analysis in case of misspecified analytical error ratio in method comparison studies Clin Chem 1998 44 5 1024 1031 9590376
  11. [13]Altman DG Bland JM Absence of evidence is not evidence of absence BMJ 1995 311 7003 485 10.1136/bmj.311.7003.485 7647644 PMC2550545 · DOI · Paper page
  12. [14]Bland JM Altman DG Measuring agreement in method comparison studies Stat Methods Med Res 1999 8 2 135 160 10.1191/096228099673819272 10501650 · DOI · Paper page
  13. [15]Bland JM Altman DG Applying the right statistics: analyses of measurement studies Ultrasound Obstet Gynecol 2003 22 1 85 93 10.1002/uog.122 12858311 · DOI
  14. [16]Trikalinos TA Ip S Raman G Cepeda MS Balk EM D'Ambrosio C Home diagnosis of obstructive sleep apnea-hypopnea syndrome. Evidence Report/Technology Assessment 2007 Rockville, MD Agency for Healthcare Research and Quality 1 127 25473717
  15. [17]Thompson IM Pauler DK Goodman PJ Tangen CM Lucia MS Parnes HL Prevalence of prostate cancer among men with a prostate-specific antigen level < or =4.0 ng per milliliter N Engl J Med 2004 350 22 2239 2246 10.1056/NEJMoa031918 15163773 · DOI
  16. [18]Vacek PM The effect of conditional dependence on the evaluation of diagnostic tests Biometrics 1985 41 4 959 968 10.2307/2530967 3830260 · DOI
  17. [19]Gart JJ Buck AA Comparison of a screening test and a reference test in epidemiologic studies. II. A probabilistic model for the comparison of diagnostic tests Am J Epidemiol 1966 83 3 593 602 5932703
  18. [20]Goldberg JD Wittes JT The estimation of false negatives in medical screening Biometrics 1978 34 1 77 86 10.2307/2529590 630038 · DOI
  19. [21]Gyorkos TW Genta RM Viens P MacLean JD Seroepidemiology of Strongyloides infection in the Southeast Asian refugee population in Canada Am J Epidemiol 1990 132 2 257 264 2196791
  20. [22]Joseph L Gyorkos TW Inferences for likelihood ratios in the absence of a "gold standard" Med Decis Making 1996 16 4 412 417 10.1177/0272989X9601600412 8912303 · DOI
  21. [23]Walter SD Irwig L Glasziou PP Meta-analysis of diagnostic tests with imperfect reference standards J Clin Epidemiol 1999 52 10 943 951 10.1016/S0895-4356(99)00086-4 10513757 · DOI
  22. [24]Walter SD Irwig LM Estimation of test error rates, disease prevalence and relative risk from misclassified data: a review J Clin Epidemiol 1988 41 9 923 937 10.1016/0895-4356(88)90110-2 3054000 · DOI
  23. [25]Dendukuri N Joseph L Bayesian approaches to modeling the conditional dependence between multiple diagnostic tests Biometrics 2001 57 1 158 167 10.1111/j.0006-341X.2001.00158.x 11252592 · DOI
  24. [26]Black MA Craig BA Estimating disease prevalence in the absence of a gold standard Stat Med 2002 21 18 2653 2669 10.1002/sim.1178 12228883 · DOI
  25. [27]Dendukuri N Hadgu A Wang L Modeling conditional dependence between diagnostic tests: a multiple latent variable model Stat Med 2009 28 3 441 461 10.1002/sim.3470 19067379 · DOI
  26. [28]Garrett ES Eaton WW Zeger S Methods for evaluating the performance of diagnostic tests in the absence of a gold standard: a latent class model approach Stat Med 2002 21 9 1289 1307 10.1002/sim.1105 12111879 · DOI
  27. [29]Hui SL Zhou XH Evaluation of diagnostic tests without gold standards Stat Methods Med Res 1998 7 4 354 370 10.1191/096228098671192352 9871952 · DOI
  28. [30]Qu Y Tan M Kutner MH Random effects models in latent class analysis for evaluating accuracy of diagnostic tests Biometrics 1996 52 3 797 810 10.2307/2533043 8805757 · DOI
  29. [31]Torrance-Rynard VL Walter SD Effects of dependent errors in the assessment of diagnostic test performance Stat Med 1997 16 19 2157 2175 10.1002/(SICI)1097-0258(19971015)16:19<2157::AID-SIM653>3.0.CO;2-X 9330426 · DOI
  30. [32]Toft N Jorgensen E Hojsgaard S Diagnosing diagnostic tests: evaluating the assumptions underlying the estimation of sensitivity and specificity in the absence of a gold standard Prev Vet Med 2005 68 1 19 33 10.1016/j.prevetmed.2005.01.006 15795013 · DOI
  31. [33]Albert PS Dodd LE A cautionary note on the robustness of latent class models for estimating diagnostic error without a gold standard Biometrics 2004 60 2 427 435 10.1111/j.0006-341X.2004.00187.x 15180668 · DOI
  32. [34]Alamanos Y Voulgari PV Drosos AA Incidence and prevalence of psoriatic arthritis: a systematic review J Rheumatol 2008 35 7 1354 1358 18464305
  33. [35]Cantor T Yang Z Caraiani N Ilamathi E Lack of comparability of intact parathyroid hormone measurements among commercial assays for end-stage renal disease patients: implication for treatment decisions Clin Chem 2006 52 9 1771 1776 10.1373/clinchem.2006.071589 16858070 · DOI

Cited by

Sorted by citations

Authors
Trikalinos, T. A., & Balion, C. M.
Affiliations
  • Tufts Medical Center (US)
  • Hamilton General Hospital (CA)
Corresponding author
Trikalinos, Thomas A.
ORCID
Trikalinos, Thomas A.
Venue
Journal of General Internal Medicine
Publisher
Springer Science and Business Media LLC
Published
2012-05-31
Volume
27
Issue
S1
Pages
67-75
ISSN
  • 0884-8734 (ISSN-L)
  • 1525-1497
DOI
10.1007/s11606-012-2031-7
PMC
PMC3364362
OpenAlex
W2004500024
Cited by
59
References
1
Citations by year
2026: 3 · 2025: 1 · 2024: 9 · 2023: 5 · 2022: 5 · 2021: 1 · 2020: 5 · 2019: 9 · 2018: 5 · 2017: 1 · 2016: 4 · 2015: 4 · 2014: 3 · 2013: 2 · 2012: 2
FWCI
3.71
Type
journal article
Language
en
Open access
Licence
Locations
  • Journal of General Internal Medicine (publishedVersion, cc-by)
  • Europe PMC (PubMed Central) (publishedVersion)
  • PubMed Central (publishedVersion)
  • PubMed (publishedVersion)
Funders
  • U.S. Department of Health and Human Services
  • Agency for Healthcare Research and Quality
Topics
  • Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
  • Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Epidemiology
  • Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Reliability and Agreement in Measurement
Keywords
imperfect reference standard, diagnostic accuracy, sensitivity and specificity, bias correction, index tests, predictive value, measurement method comparison
MeSH terms
Humans, Outcome and Process Assessment, Health Care, Reference Standards, Review Literature as Topic, Sensitivity and Specificity, Meta-Analysis as Topic, Reproducibility of Results, Guidelines as Topic, Evidence-Based Medicine, Diagnostic Techniques and Procedures

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review