CoChat
Sign Up
Cited by 27Clinical Laboratory Practices and Quality ControlClinical Reasoning and Diagnostic SkillsStatistical Methods in Epidemiology

The Clinical Relevance of Methods for Handling Inconclusive Medical Test Results: Quantification of Uncertainty in Medical Decision-Making and Screening

Open access
Landsheer, Johannes A.·2018·Diagnostics
DOI 10.3390/diagnostics8020032PMC PMC6023344
Open this paper in CoChat — free accountOpen PDF

Abstract

Background: although the existence of inconclusive medical test results or bio-markers is widely recognized, there are indications that this inherent diagnostic uncertainty is sometimes ignored. This paper discusses three methods for defining and determining inconclusive medical test results, which use different definitions and differ in clinical relevance. Methods: the TG-ROC (two graphs receiver operating characteristics) method is the easiest to use, while the grey zone method and the uncertain interval method require more extensive calculations. Results: this paper discusses the technical details of the methods, as well as advantages and disadvantages for their clinical use. TG-ROC and the grey zone method can help in the acquisition of high rates of diagnostic certainty, but can exclude large groups. The uncertain interval method can prevent decisions that are the most uncertain, invalid and unreliable, while excluding smaller groups. Conclusions: the identification of uncertain test scores is relevant, because these scores indicate the need to obtain better information or to await further developments. The methods presented help to determine inconclusive test scores and can help to reduce erroneous decisions. However, further research and development is desirable.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.3390/diagnostics8020032

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Because factuality in medicine can sometimes be a grey area, especially when clinical guidelines evolve or expert consensus varies, automated metrics alone may not suffice.

    “Because factuality in medicine can sometimes be a”

    From Trustworthy Medical Question Answering: An Evaluation-Centric Survey (2025) · cites this paper as Landsheer, 2018

    The Introduction (page unavailable) discusses inconclusive diagnostic test results, not whether medical statements are factually correct. Searches for evolving clinical guidelines, varying expert consensus, and automated factuality metrics found no passage addressing those claims. (LLM, conf=0.75)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

No references could be resolved yet.

Research works printed but not linked: 28

  1. [1]Coste J. Jourdain P. Pouchot J. A gray zone assigned to inconclusive results of quantitative diagnostic tests: Application to the use of brain natriuretic peptide for diagnosis of heart failure in acute dyspneic patients Clin. Chem. 2006 52 2229 2235 10.1373/clinchem.2006.072280 17053156 · DOI
  2. [2]Coste J. Pouchot J. A grey zone for quantitative diagnostic and screening tests Int. J. Epidemiol. 2003 32 304 313 10.1093/ije/dyg054 12714554 · DOI
  3. [3]Greiner M. Sohr D. Göbel P. A modified ROC analysis for the selection of cut-off values and the definition of intermediate results of serodiagnostic tests J. Immunol. Methods 1995 185 123 132 10.1016/0022-1759(95)00121-P 7665894 · DOI
  4. [4]Landsheer J.A. Interval of Uncertainty: An Alternative Approach for the Determination of Decision Thresholds, with an Illustrative Application for the Prediction of Prostate Cancer PLoS ONE 2016 11 e0166007 10.1371/journal.pone.0166007 27829010 PMC5102386 · DOI
  5. [5]Shinkins B. Perera R. Diagnostic uncertainty: Dichotomies are not the answer Br. J. Gen. Pract. 2013 63 122 123 10.3399/bjgp13X664090 23561757 PMC3582949 · DOI
  6. [6]Simel D.L. Feussner J.R. Delong E.R. Matchar D.B. Intermediate, indeterminate, and uninterpretable diagnostic test results Med. Decis. Mak. 1987 7 107 114 10.1177/0272989X8700700208 3574020 · DOI
  7. [7]Schuetz G.M. Schlattmann P. Dewey M. Use of 3 × 2 tables with an intention to diagnose approach to assess clinical performance of diagnostic tests: Meta-analytical evaluation of coronary CT angiography studies BMJ 2012 345 10 10.1136/bmj.e6717 23097549 PMC3480336 · DOI
  8. [8]Feinstein A.R. The inadequacy of binary models for the clinical reality of three-zone diagnostic decisions J. Clin. Epidemiol. 1990 43 109 10.1016/0895-4356(90)90064-V 2319275 · DOI
  9. [10]Bangma C.H. Bul M. van der Kwast T.H. Pickles T. Korfage I.J. Hoeks C.M. Steyerberg E.W. Jenster G. Kattan M.W. Bellardita L. Active surveillance for low-risk prostate cancer Crit. Rev. Oncol. Hematol. 2013 85 295 302 10.1016/j.critrevonc.2012.07.005 22878262 · DOI
  10. [11]Cammann H. Jung K. Meyer H.-A. Stephan C. Avoiding Pitfalls in Applying Prediction Models, As Illustrated by the Example of Prostate Cancer Diagnosis Clin. Chem. 2011 57 1490 1498 10.1373/clinchem.2011.166959 21920913 · DOI
  11. [12]Ma X. Suri M.F.K. Chu H. A trivariate meta-analysis of diagnostic studies accounting for prevalence and non-evaluable subjects: Re-evaluation of the meta-analysis of coronary CT angiography studies BMC Med. Res. Methodol. 2014 14 128 10.1186/1471-2288-14-128 25475705 PMC4280699 · DOI
  12. [13]Ratón M.L. Optimal Cutoff Points for Classification in Diagnostic Studies: New Contributions and Software Development Ph.D. Thesis Universidade de Santiago de Compostela Santiago de Compostela, A Coruña, Spain 2016
  13. [14]Greiner M. Two-graph receiver operating characteristic (TG-ROC): A Microsoft-EXCEL template for the selection of cut-off values in diagnostic tests J. Immunol. Methods 1995 185 145 146 10.1016/0022-1759(95)00078-O 7665897 · DOI
  14. [15]Greiner M. Two-graph receiver operating characteristic (TG-ROC): Update version supports optimisation of cut-off values that minimise overall misclassification costs J. Immunol. Methods 1996 191 93 94 10.1016/0022-1759(96)00013-0 8642206 · DOI
  15. [16]Nakas C.T. Alonzo T.A. Yiannoutsos C.T. Accuracy and cut-off point selection in three-class classification problems using a generalization of the Youden index Stat. Med. 2010 29 2946 2955 10.1002/sim.4044 20809485 PMC2991472 · DOI
  16. [17]Nakas C.T. Dalrymple-Alford J.C. Anderson T.J. Alonzo T.A. Generalization of Youden index for multiple-class classification problems applied to the assessment of externally validated cognition in Parkinson disease screening Stat. Med. 2013 32 995 1003 10.1002/sim.5592 22949169 · DOI
  17. [18]Brugnara C. Zurakowski D. DiCanzio J. Boyd T. Platt O. Reticulocyte Hemoglobin Content to Diagnose Iron Deficiency in Children JAMA 1999 281 2225 2230 10.1001/jama.281.23.2225 10376576 · DOI
  18. [19]Schisterman E.F. Perkins N.J. Liu A. Bondell H. Optimal cut-point and its corresponding Youden Index to discriminate individuals using pooled blood samples Epidemiology 2005 16 73 81 10.1097/01.ede.0000147512.81966.ba 15613948 · DOI
  19. [20]Coffin M. Sukhatme S. Receiver operating characteristic studies and measurement errors Biometrics 1997 53 823 837 10.2307/2533545 9333348 · DOI
  20. [21]Metz C.E. Basic principles of ROC analysis Seminars in Nuclear Medicine Elsevier New York, NY, USA 1978 8 283 298 112681
  21. [22]Perkins N.J. Schisterman E.F. The inconsistency of “optimal” cutpoints obtained using two criteria based on the receiver operating characteristic curve Am. J. Epidemiol. 2006 163 670 675 10.1093/aje/kwj063 16410346 PMC1444894 · DOI
  22. [23]Andrews D.F. Herzberg A.M. Data: A Collection of Problems from Many Fields for the Student and Research Worker Springer Science & Business Media Berlin, Germany 1985
  23. [25]Fluss R. Faraggi D. Reiser B. Estimation of the Youden Index and its associated cutoff point Biom. J. 2005 47 458 472 10.1002/bimj.200410135 16161804 · DOI
  24. [26]Almond S. Mant D. Thompson M. Diagnostic safety-netting Br. J. Gen. Pract. 2009 59 872 874 10.3399/bjgp09X472971 19861036 PMC2765844 · DOI
  25. [27]Battaglia M. Pewsner D. Commentary: Black and white or shades of grey? Int. J. Epidemiol. 2003 32 314 315 10.1093/ije/dyg065 12714555 · DOI
  26. [28]Leeflang M.M. Bossuyt P.M. Irwig L. Diagnostic test accuracy may vary with prevalence: Implications for evidence-based diagnosis J. Clin. Epidemiol. 2009 62 5 12 10.1016/j.jclinepi.2008.04.007 18778913 · DOI
  27. [29]Leeflang M.M. Rutjes A.W. Reitsma J.B. Hooft L. Bossuyt P.M. Variation of a test’s sensitivity and specificity with disease prevalence Can. Med. Assoc. J. 2013 185 E537 E544 10.1503/cmaj.121286 23798453 PMC3735771 · DOI · Paper page
  28. [30]Rutjes A.W.S. Sources of Bias and Variation in Diagnostic Accuracy Studies Ph.D. Thesis University of Amsterdam Amsterdam, The Netherlands 2004

Cited by

Sorted by citations

Authors
Landsheer, J. A.
Affiliations
Utrecht University (NL)
Corresponding author
Landsheer, Johannes A.
ORCID
Landsheer, Johannes A.
Venue
Diagnostics
Publisher
MDPI AG
Published
2018-05-09
Volume
8
Issue
2
Pages
32
ISSN
2075-4418 (ISSN-L)
DOI
10.3390/diagnostics8020032
PMC
PMC6023344
OpenAlex
W2800714951
Cited by
27
References
0
Citations by year
2026: 2 · 2025: 2 · 2024: 2 · 2023: 8 · 2022: 3 · 2021: 5 · 2020: 3 · 2019: 2
FWCI
0.33
Type
journal article
Language
en
Open access
Licence
Locations
  • Diagnostics (publishedVersion, cc-by)
  • Europe PMC (PubMed Central) (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
  • Utrecht University Repository (Utrecht University) (submittedVersion, other-oa)
Topics
  • Health Sciences › Medicine › Physiology › Clinical Laboratory Practices and Quality Control
  • Health Sciences › Medicine › Family Practice › Clinical Reasoning and Diagnostic Skills
  • Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Epidemiology
Keywords
diagnostic uncertainty, uncertainty quantification, receiver operating characteristic, diagnostic decision-making, screening, biomarkers, diagnostic certainty

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review