The DOI matches its Crossref record; 70 of the 70 references it cites were found in a registry or an index.
Systematic Reviews of Diagnostic Test Accuracy
Open accessAbstract
More and more systematic reviews of diagnostic test accuracy studies are being published, but they can be methodologically challenging. In this paper, the authors present some of the recent developments in the methodology for conducting systematic reviews of diagnostic test accuracy studies. Restrictive electronic search filters are discouraged, as is the use of summary quality scores. Methods for meta-analysis should take into account the paired nature of the estimates and their dependence on threshold. Authors of these reviews are advised to use the hierarchical summary receiver-operating characteristic or the bivariate model for the data analysis. Challenges that remain are the poor reporting of original diagnostic test accuracy studies and difficulties with the interpretation of the results of diagnostic test accuracy research.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“We developed a protocol before commencing the review, following standard guidelines [12,13].”
From Xpert MTB/RIF assay for the diagnosis of extrapulmonary tuberculosis: a systematic review and meta-analysis (2014) · cites this paper as [12]
Experimental — accuracy not yet measuredIn the citing paper
“The poor reporting of results and subsequent interpretation difficulties is a longstanding challenge of diagnostic test accuracy research [15].”
From Artificial Intelligence, Speech, and Language Processing Approaches to Monitoring Alzheimer’s Disease: A Systematic Review (2020) · cites this paper as [15]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Whiting, P., Rutjes, A. W., Reitsma, J. B., Bossuyt, P. M., & Kleijnen, J. (2003) · BMC Medical Research Methodology · doi:10.1186/1471-2288-3-25 · Cited by 3,592
Self-citationReitsma, J. B., Glas, A. S., Rutjes, A. W. S., Scholten, R. J. P. M., Bossuyt, P. M., & Zwinderman, A. H. (2005) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2005.02.022 · Cited by 3,369
Self-citationTowards complete and accurate reporting of studies of diagnostic accuracy: The STARD Initiative.
Bossuyt, P. M., Reitsma, J. B., Bruns, D. E., Gatsonis, C. A., Glasziou, P. P., Irwig, L. M., Lijmer, J. G., Moher, D., Rennie, D., de Vet, H. C. W., et al. (2003) · Annals of Internal Medicine · doi:10.7326/0003-4819-138-1-200301070-00010 · Cited by 3,219
Self-citationTowards complete and accurate reporting of studies of diagnostic accuracy: the STARD initiative
Bossuyt, P. M., Reitsma, J. B., Bruns, D. E., Gatsonis, C. A., Glasziou, P. P., Irwig, L. M., Lijmer, J. G., Moher, D., Rennie, D., & de Vet, H. C. W. (2003) · BMJ · doi:10.1136/bmj.326.7379.41 · Cited by 3,215
Self-citationDeeks, J. J., Macaskill, P., & Irwig, L. (2005) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2005.01.016 · Cited by 2,910
Self-citationSystematic Reviews: Synthesis of Best Evidence for Clinical Decisions
Cook, D. J., Mulrow, C. D., & Haynes, R. B. (1997) · Annals of Internal Medicine · doi:10.7326/0003-4819-126-5-199703010-00006 · Cited by 2,184
Understanding diagnostic tests 3: receiver operating characteristic curves
Akobeng, A. K. (2007) · Acta Paediatrica · doi:10.1111/j.1651-2227.2006.00178.x · Cited by 1,882
Empirical evidence of design-related bias in studies of diagnostic tests.
Lijmer, J. G. (1999) · JAMA · doi:10.1001/jama.282.11.1061 · Cited by 1,861
Statistical Methods in Diagnostic Medicine
Zhou, X., Sun, J., Pennello, G. A., Obuchowski, N. A., & McClish, D. K. (2011) · Wiley series in probability and statistics · doi:10.1002/9780470906514 · Cited by 1,785
A hierarchical regression approach to meta‐analysis of diagnostic test accuracy evaluations
Rutter, C. M., & Gatsonis, C. (2001) · Statistics in Medicine · doi:10.1002/sim.942 · Cited by 1,510
Self-citationMoses, L. E., Shapiro, D. E., & Littenberg, B. (1993) · Statistics in Medicine · doi:10.1002/sim.4780121403 · Cited by 1,405
Sources of Variation and Bias in Studies of Diagnostic Accuracy
Whiting, P. F., Rutjes, A. W., Reitsma, J. B., Glas, A. S., Bossuyt, P. M., & Kleijnen, J. (2004) · Annals of Internal Medicine · doi:10.7326/0003-4819-140-3-200402030-00010 · Cited by 804
Self-citationDeveloping Optimal Search Strategies for Detecting Clinically Sound Studies in MEDLINE
Haynes, R. B., Wilczynski, N. L., McKibbon, K. A., Walker, C. J., & Sinclair, J. (1994) · Journal of the American Medical Informatics Association · doi:10.1136/jamia.1994.95153434 · Cited by 762
Guidelines for Meta-analyses Evaluating Diagnostic Tests
Irwig, L., Tosteson, A. N. A., Gatsonis, C., Lau, J., Colditz, G. A., Chalmers, T. C., & Mosteller, F. (1994) · Annals of Internal Medicine · doi:10.7326/0003-4819-120-8-199404150-00008 · Cited by 754
Self-citationA unification of models for meta-analysis of diagnostic accuracy studies
Harbord, R. M., Deeks, J. J., Egger, M., Whiting, P. F., & Sterne, J. A. C. (2006) · Biostatistics · doi:10.1093/biostatistics/kxl004 · Cited by 753
Self-citationChu, H., & Cole, S. R. (2006) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2006.06.011 · Cited by 708
Evidence of bias and variation in diagnostic accuracy studies
Rutjes, A. W., Reitsma, J. B., Nisio, M. D., Smidt, N., van Rijn, J. C., & Bossuyt, P. M. (2006) · Canadian Medical Association Journal · doi:10.1503/cmaj.050090 · Cited by 623
Self-citationHaynes, R. B., & Wilczynski, N. L. (2004) · BMJ · doi:10.1136/bmj.38068.557998.ee · Cited by 549
Asymmetric funnel plots and publication bias in meta-analyses of diagnostic accuracy
Song, F., Khan, K. S., Dinnes, J., & Sutton, A. J. (2002) · International Journal of Epidemiology · doi:10.1093/ije/31.1.88 · Cited by 485
Publication and Related Biases
Song, F., Eastwood, A. J., Gilbody, S. M., Duley, L., & Sutton, A. J. (2001) · SAGE Publications Ltd eBooks · doi:10.4135/9781848608344.n21 · Cited by 469
Comparative accuracy: assessing new tests against existing diagnostic pathways
Bossuyt, P. M., Irwig, L., Craig, J. C., & Glasziou, P. (2006) · BMJ · doi:10.1136/bmj.332.7549.1089 · Cited by 464
Self-citationExploring sources of heterogeneity in systematic reviews of diagnostic tests
Lijmer, J. G., Bossuyt, P. M., & Heisterkamp, S. H. (2002) · Statistics in Medicine · doi:10.1002/sim.1185 · Cited by 446
Self-citationPublication and related biases.
Song, Eastwood, Gilbody, Duley, & Sutton (2000) · Health Technology Assessment · doi:10.3310/hta4100 · Cited by 427
Decision Making in Health and Medicine: Integrating Evidence and Values
Hunink, M. G. M., Weinstein, M. C., Wittenberg, E., Drummond, M. F., Pliskin, J. S., Wong, J. B., & Glasziou, P. P. (2014) · doi:10.1017/cbo9781139506779 · Cited by 425
Leeflang, M., Moons, K. G. M., Reitsma, J. B., & Zwinderman, A. H. (2008) · Clinical Chemistry · doi:10.1373/clinchem.2007.096032 · Cited by 351
Self-citationCOMPARISON OF SCREENING METHODS IN THE DETECTION OF BLADDER CANCER
Ramakumar, S., Bhuiyan, J., Besse, J. A., ROBERTS, S. G., Wollan, P. C., Blute, M. L., & O’Kane, D. J. (1999) · The Journal of Urology · doi:10.1016/s0022-5347(01)61899-8 · Cited by 335
No role for quality scores in systematic reviews of diagnostic accuracy studies
Whiting, P. F., Harbord, R. M., & Kleijnen, J. (2005) · BMC Medical Research Methodology · doi:10.1186/1471-2288-5-19 · Cited by 333
Bivariate Random Effects Meta-Analysis of ROC Curves
Arends, L. R., Hamza, T. H., van Houwelingen, H. C., Heijenbrok‐Kal, M. H., Hunink, M. G. M., & Stijnen, T. (2008) · Medical Decision Making · doi:10.1177/0272989x08319957 · Cited by 322
The Evidence Base of Clinical Diagnosis
(2008) · Wiley eBooks · doi:10.1002/9781444300574 · Cited by 278
Tumor Markers in the Diagnosis of Primary Bladder Cancer. A Systematic Review
Glas, A. S., Roos, D., Deutekom, M. D. L. F., Zwinderman, A. H., Bossuyt, P. M., & Kurth, K. (2003) · The Journal of Urology · doi:10.1097/01.ju.0000067461.30468.6d · Cited by 268
Self-citationLord, S. J., Irwig, L., & Simes, R. J. (2006) · Annals of Internal Medicine · doi:10.7326/0003-4819-144-11-200606060-00011 · Cited by 268
Irwig, L. (2002) · BMJ · doi:10.1136/bmj.324.7338.669 · Cited by 246
Sensitivity and specificity of NMP-22, telomerase, and BTA in the detection of human bladder cancer
Landman, J., Chang, Y., Kavaler, E., Droller, M. J., & Liu, B. C. ‐. (1998) · Urology · doi:10.1016/s0090-4295(98)00219-2 · Cited by 239
The quality of diagnostic accuracy studies since the STARD statement
Smidt, N., Rutjes, A. W., van der Windt, D., Ostelo, R., Bossuyt, P. M., Reitsma, J. B., Bouter, L. M., & de Vet, H. C. W. (2006) · Neurology · doi:10.1212/01.wnl.0000238386.41398.30 · Cited by 233
Self-citationMeta-Analysis of Diagnostic and Screening Test Accuracy Evaluations: Methodologic Primer
Gatsonis, C., & Paliwal, P. (2006) · American Journal of Roentgenology · doi:10.2214/ajr.06.0226 · Cited by 211
Self-citationWe should not pool diagnostic likelihood ratios in systematic reviews
Zwinderman, A. H., & Bossuyt, P. M. (2007) · Statistics in Medicine · doi:10.1002/sim.2992 · Cited by 208
Self-citationMacaskill, P. (2004) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2003.12.019 · Cited by 207
SHARMA, S., Zippe, C. D., PANDRANGI, L., et al. (1999) · The Journal of Urology · doi:10.1097/00005392-199907000-00014 · Cited by 200
Sample sizes of studies on diagnostic accuracy: literature survey.
Bachmann, L. M., Puhan, M. A., Riet, G. T., & Bossuyt, P. M. (2006) · BMJ · doi:10.1136/bmj.38793.637789.2f · Cited by 184
Self-citationIdentifying Diagnostic Studies in MEDLINE: Reducing the Number Needed to Read
Bachmann, L. M. (2002) · Journal of the American Medical Informatics Association · doi:10.1197/jamia.m1124 · Cited by 173
Eugene W. Caldwell Lecture. Clinical efficacy of diagnostic imaging: love it or leave it.
Thornbury, J. R. (1994) · American Journal of Roentgenology · doi:10.2214/ajr.162.1.8273645 · Cited by 165
Leeflang, M., Scholten, R. J. P. M., Rutjes, A. W., Reitsma, J. B., & Bossuyt, P. M. (2006) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2005.07.014 · Cited by 162
Self-citationPublications on diagnostic test evaluation in family medicine journals
Devillé, W. L. J. M., Bezemer, P. D., & Bouter, L. M. (2000) · Journal of Clinical Epidemiology · doi:10.1016/s0895-4356(99)00144-4 · Cited by 161
Communicating accuracy of tests to general practitioners: a controlled study
Steurer, J., Fischer, J. E., Bachmann, L. M., Koller, M., & ter Riet, G. (2002) · BMJ · doi:10.1136/bmj.324.7341.824 · Cited by 147
Search strategies to identify diagnostic accuracy studies in MEDLINE and EMBASE
Beynon, R., Leeflang, M., McDonald, S. J., Eisinga, A., Mitchell, R. L., Whiting, P. F., & Glanville, J. (2013) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.mr000022.pub3 · Cited by 146
Self-citationQuality of Reporting of Diagnostic Accuracy Studies
Smidt, N., Rutjes, A. W., van der Windt, D., Ostelo, R., Reitsma, J. B., Bossuyt, P. M., Bouter, L. M., & de Vet, H. C. W. (2005) · Radiology · doi:10.1148/radiol.2352040507 · Cited by 134
Self-citationOriginal Paper: A likelihood ratio approach to meta-analysis of diagnostic studies
Stengel, D., Bauwens, K., Sehouli, J., Ekkernkamp, A., & Porzsolt, F. (2003) · Journal of Medical Screening · doi:10.1258/096914103321610806 · Cited by 118
NMP22 IS A SENSITIVE, COST-EFFECTIVE TEST IN PATIENTS AT RISK FOR BLADDER CANCER
Zippe, C. D., PANDRANGI, L., & Agarwal, A. K. (1999) · The Journal of Urology · doi:10.1016/s0022-5347(01)62063-9 · Cited by 107
Doust, J. A., Pietrzak, E., Sanders, S., & Glasziou, P. P. (2005) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2004.09.011 · Cited by 106
Meta‐analysis of Diagnostic Test Accuracy Assessment Studies with Varying Number of Thresholds
Dukić, V., & Gatsonis, C. (2003) · Biometrics · doi:10.1111/j.0006-341x.2003.00108.x · Cited by 101
Self-citationPost hoc choice of cut points introduced bias to diagnostic research
Ewald, B. D. (2006) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2005.11.025 · Cited by 94
Systematic reviews to evaluate diagnostic tests
Khan, K. S., Dinnes, J., & Kleijnen, J. (2001) · European Journal of Obstetrics & Gynecology and Reproductive Biology · doi:10.1016/s0301-2115(00)00463-2 · Cited by 94
How does study quality affect the results of a diagnostic meta-analysis?
Westwood, M. E., Whiting, P. F., & Kleijnen, J. (2005) · BMC Medical Research Methodology · doi:10.1186/1471-2288-5-20 · Cited by 87
Giannopoulos, A., MANOUSAKAS, T., GOUNARI, A., et al. (2001) · The Journal of Urology · doi:10.1016/s0022-5347(05)65965-4 · Cited by 85
NMP22 IS A SENSITIVE, COST-EFFECTIVE TEST IN PATIENTS AT RISK FOR BLADDER CANCER
Zippe, C. D., PANDRANGI, L., & Agarwal, A. K. (1999) · The Journal of Urology · doi:10.1097/00005392-199901000-00022 · Cited by 84
Accuracy of magnetic resonance imaging for the diagnosis of multiple sclerosis: systematic review
Whiting, P. F., Harbord, R. M., Main, C., Deeks, J. J., Filippini, G., Egger, M., & Sterne, J. A. C. (2006) · BMJ · doi:10.1136/bmj.38771.583796.7c · Cited by 81
Self-citationzen, S. S., Biri, H., k, Z. S. I. N. I., et al. (1999) · European Urology · doi:10.1159/000068002 · Cited by 70
Miyanaga, N., Akaza, H., Tsukamoto, T., et al. (1999) · International Journal of Urology · doi:10.1046/j.1442-2042.1999.06437.x · Cited by 59
Puhan, M. A., Steurer, J., Bachmann, L. M., et al. (2005) · Annals of Internal Medicine · doi:10.7326/0003-4819-143-3-200508020-00004 · Cited by 58
Lokeshwar, V. B., & Selzer, M. G. (2006) · Urologic Oncology Seminars and Original Investigations · doi:10.1016/j.urolonc.2006.07.003 · Cited by 56
Impact of Adjustment for Quality on Results of Metaanalyses of Diagnostic Accuracy
Leeflang, M., Reitsma, J. B., Scholten, R. J. P. M., Rutjes, A. W., Nisio, M. D., Deeks, J. J., & Bossuyt, P. M. (2006) · Clinical Chemistry · doi:10.1373/clinchem.2006.076398 · Cited by 48
Self-citationInterpreting Diagnostic Test Accuracy Studies
Bossuyt, P. M. (2008) · Seminars in Hematology · doi:10.1053/j.seminhematol.2008.04.001 · Cited by 43
Self-citationCasella, R., Huber, P., Blöchlinger, A., et al. (2000) · The Journal of Urology · doi:10.1016/s0022-5347(05)66920-0 · Cited by 39
Abbate, I., D’Introno, A., Cardo, G., et al. (1999) · PubMed · Cited by 34
Lahme, S., Bichler, K., Feil, G., et al. (2001) · Urologia Internationalis · doi:10.1159/000056574 · Cited by 29
Chahal, R., Darshane, A., Browning, A., et al. (2001) · European Urology · doi:10.1159/000049809 · Cited by 26
Systematic reviews of diagnostic tests: a guide to methods and application
Khan, K. S. (2005) · Best Practice & Research Clinical Obstetrics & Gynaecology · doi:10.1016/j.bpobgyn.2004.10.012 · Cited by 24
Evaluation of nuclear matrix protein 22 (NMP22) as a tumor marker in the detection of bladder cancer
Öge, Ö., Atsü, N., Kendi̇, S., et al. (2001) · International Urology and Nephrology · doi:10.1023/a:1017509418440 · Cited by 22
Paoluzzi, M., Mg, C., Mugnaini, P., et al. (1999) · PubMed · Cited by 18
Lee, K. H. (2001) · Yonsei Medical Journal · doi:10.3349/ymj.2001.42.1.14 · Cited by 13
Research works printed but not linked: 2
- Abbate I, 1998, Anticancer Res, 18, 3803
- Paoluzzi M, 1999, Arch Ital Urol Androl, 71, 13
Cited by
Sorted by citations
QUADAS-2: a revised tool for the quality assessment of diagnostic accuracy studies
Whiting, P. F., Rutjes, A. W. S., Westwood, M. E., Mallett, S., Deeks, J. J., Reitsma, J. B., Leeflang, M. M. G., Sterne, J. A. C., Bossuyt, P. M. M., & the QUADAS-2 Group* (2011) · Annals of Internal Medicine · doi:10.7326/0003-4819-155-8-201110180-00009 · Cited by 14,199
Self-citationShea, B. J., Reeves, B. C., Wells, G., Thuku, M., Hamel, C., Moran, J., Moher, D., Tugwell, P., Welch, V., Kristjansson, E., et al. (2017) · BMJ · doi:10.1136/bmj.j4008 · Cited by 10,636
ROBIS: A new tool to assess risk of bias in systematic reviews was developed
Whiting, P., Savović, J., Higgins, J. P. T., Caldwell, D. M., Reeves, B. C., Shea, B., Davies, P., Kleijnen, J., & Churchill, R. (2016) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2015.06.005 · Cited by 2,272
Zeng, X., Zhang, Y., Kwong, J. S. W., Zhang, C., Li, S., Sun, F., Niu, Y., & Du, L. (2015) · Journal of Evidence-Based Medicine · doi:10.1111/jebm.12141 · Cited by 2,082
Husby, S., Koletzko, S., Korponay‐Szabó, I., Kurppa, K., Mearin, M. L., Ribes‐Koninckx, C., Shamir, R., Troncone, R., Auricchio, R., Castillejo, G., et al. (2020) · Journal of Pediatric Gastroenterology and Nutrition · doi:10.1097/mpg.0000000000002497 · Cited by 1,255
Metadata
- Authors
- Leeflang, M. M. G., Deeks, J. J., Gatsonis, C., & Bossuyt, P. M. M.
- Affiliations
- Brown University (US)
- Corresponding author
- Deeks, Jonathan J.
- Venue
- Annals of Internal Medicine
- Publisher
- American College of Physicians
- Published
- 2008-12-16
- Volume
- 149
- Issue
- 12
- Pages
- 889-897
- ISSN
- 0003-4819 (ISSN-L)
- 1539-3704
- PMC
- PMC2956514
- OpenAlex
- W1973593314
- Cited by
- 1,124
- References
- 70
- Citations by year
- 2026: 31 · 2025: 36 · 2024: 35 · 2023: 32 · 2022: 73 · 2021: 81 · 2020: 85 · 2019: 69 · 2018: 85 · 2017: 75 · 2016: 66 · 2015: 77 · 2014: 78 · 2013: 61 · 2012: 80
- FWCI
- 16.78
- Type
- journal article
- Language
- en
- Open access
- green
- www.ncbi.nlm.nih.gov
- Locations
- Annals of Internal Medicine (publishedVersion)
- PubMed (publishedVersion)
- Bristol Research (University of Bristol) (publishedVersion)
- PubMed Central (acceptedVersion)
- Data Archiving and Networked Services (DANS) (submittedVersion)
- UvA-DARE (University of Amsterdam) (submittedVersion, other-oa)
- IRIS UNIMORE (University of Modena and Reggio Emilia) (submittedVersion, cc-by)
- Pure Amsterdam UMC (submittedVersion)
- Research at the University of Copenhagen (University of Copenhagen) (submittedVersion)
- University of Birmingham Research Portal (University of Birmingham) (submittedVersion)
- Explore Bristol Research (submittedVersion)
- Data Archiving and Networked Services (DANS) (submittedVersion, other-oa)
- Funders
- National Cancer Institute (U01CA079778)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Epidemiology
- Health Sciences › Medicine › Public Health, Environmental and Occupational Health › Clinical practice guidelines implementation
- Keywords
- diagnostic accuracy, systematic review methodology, meta-analysis of diagnostic accuracy, summary receiver operating characteristic, bivariate models, threshold dependence
- MeSH terms
- Diagnostic Tests, Routine, Humans, Research Design, Review Literature as Topic, ROC Curve, Sensitivity and Specificity, Meta-Analysis as Topic, Information Storage and Retrieval
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
