The DOI matches its Crossref record; 69 of the 69 references it cites were found in a registry or an index.
Evidence of bias and variation in diagnostic accuracy studies
Open accessAbstract
BACKGROUND: Studies with methodologic shortcomings can overestimate the accuracy of a medical test. We sought to determine and compare the direction and magnitude of the effects of a number of potential sources of bias and variation in studies on estimates of diagnostic accuracy. METHODS: We identified meta-analyses of the diagnostic accuracy of tests through an electronic search of the databases MEDLINE, EMBASE, DARE and MEDION (1999-2002). We included meta-analyses with at least 10 primary studies without preselection based on design features. Pairs of reviewers independently extracted study characteristics and original data from the primary studies. We used a multivariable meta-epidemiologic regression model to investigate the direction and strength of the association between 15 study features on estimates of diagnostic accuracy. RESULTS: We selected 31 meta-analyses with 487 primary studies of test evaluations. Only 1 study had no design deficiencies. The quality of reporting was poor in most of the studies. We found significantly higher estimates of diagnostic accuracy in studies with nonconsecutive inclusion of patients (relative diagnostic odds ratio [RDOR] 1.5, 95% confidence interval [CI] 1.0-2.1) and retrospective data collection (RDOR 1.6, 95% CI 1.1-2.2). The estimates were highest in studies that had severe cases and healthy controls (RDOR 4.9, 95% CI 0.6-37.3). Studies that selected patients based on whether they had been referred for the index test, rather than on clinical symptoms, produced significantly lower estimates of diagnostic accuracy (RDOR 0.5, 95% CI 0.3-0.9). The variance between meta-analyses of the effect of design features was large to moderate for type of design (cohort v. case-control), the use of composite reference standards and the use of differential verification; the variance was close to zero for the other design features. INTERPRETATION: Shortcomings in study design can affect estimates of diagnostic accuracy, but the magnitude of the effect may vary from one situation to another. Design features and clinical characteristics of patient groups should be carefully considered by researchers when designing new studies and by readers when appraising the results of such studies. Unfortunately, incomplete reporting hampers the evaluation of potential sources of bias in diagnostic accuracy studies.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Assessing the quality of reports of randomized clinical trials: Is blinding necessary?
Jadad, A. R., Moore, R. A., Carroll, D., Jenkinson, C., Reynolds, D. J. M., Gavaghan, D. J., & McQuay, H. J. (1996) · Controlled Clinical Trials · doi:10.1016/0197-2456(95)00134-4 · Cited by 17,582
Zhang, Q., Wang, C., Zheng, M., Li, Y., Li, J., Zhang, L., Shang, X., & Yan, C. (2015) · Cerebrovascular Diseases · doi:10.1159/000369778 · Cited by 11,533
Schulz (1995) · doi:10.1001/jama.1995.03520290060030 · Cited by 5,634
Dong, Y., Hao, L., Shi, Z., Wang, G., Zhang, Z., & Han, C. (2013) · Urologia Internationalis · doi:10.1159/000351405 · Cited by 4,540
Brocken, P., van der Heijden, H. F. M., Dekhuijzen, P. N. R., Peters-Bax, L., & de Geus-Oei, L.-F. (2014) · Respiration · doi:10.1159/000347096 · Cited by 4,002
Whiting, P., Rutjes, A. W., Reitsma, J. B., Bossuyt, P. M., & Kleijnen, J. (2003) · BMC Medical Research Methodology · doi:10.1186/1471-2288-3-25 · Cited by 3,592
Self-citationMoher, D., Pham, B., Jones, A., Cook, D. J., Jadad, A. R., Moher, M., Tugwell, P., & Klassen, T. P. (1998) · The Lancet · doi:10.1016/s0140-6736(98)01085-x · Cited by 3,560
Assessing the quality of controlled clinical trials
Jüni, P., Altman, D. G., & Egger, M. (2001) · BMJ · doi:10.1136/bmj.323.7303.42 · Cited by 3,309
Towards complete and accurate reporting of studies of diagnostic accuracy: The STARD Initiative.
Bossuyt, P. M., Reitsma, J. B., Bruns, D. E., Gatsonis, C. A., Glasziou, P. P., Irwig, L. M., Lijmer, J. G., Moher, D., Rennie, D., de Vet, H. C. W., et al. (2003) · Annals of Internal Medicine · doi:10.7326/0003-4819-138-1-200301070-00010 · Cited by 3,219
Self-citationTowards complete and accurate reporting of studies of diagnostic accuracy: the STARD initiative
Bossuyt, P. M., Reitsma, J. B., Bruns, D. E., Gatsonis, C. A., Glasziou, P. P., Irwig, L. M., Lijmer, J. G., Moher, D., Rennie, D., & de Vet, H. C. W. (2003) · BMJ · doi:10.1136/bmj.326.7379.41 · Cited by 3,215
Self-citationTowards Complete and Accurate Reporting of Studies of Diagnostic Accuracy: The STARD Initiative
Bossuyt, P. M., Reitsma, J. B., Bruns, D. E., Gatsonis, C. A., Glasziou, P. P., Irwig, L. M., Lijmer, J. G., Moher, D., Rennie, D., & de Vet, H. C. W. (2003) · Radiology · doi:10.1148/radiol.2261021292 · Cited by 3,215
Self-citationVerhagen, A. P., de Vet, H. C. W., de Bie, R. A., Kessels, A. G. H., Boers, M., Bouter, L. M., & Knipschild, P. G. (1998) · Journal of Clinical Epidemiology · doi:10.1016/s0895-4356(98)00131-0 · Cited by 2,379
The diagnostic odds ratio: a single indicator of test performance
Glas, A. S., Lijmer, J. G., Prins, M. H., Bonsel, G. J., & Bossuyt, P. M. M. (2003) · Journal of Clinical Epidemiology · doi:10.1016/s0895-4356(03)00177-x · Cited by 2,261
Self-citationThe Statistical Evaluation of Medical Tests for Classification and Prediction
(2005) · Technometrics · doi:10.1198/tech.2005.s278 · Cited by 1,899
Empirical evidence of design-related bias in studies of diagnostic tests.
Lijmer, J. G. (1999) · JAMA · doi:10.1001/jama.282.11.1061 · Cited by 1,861
Advanced methods in meta‐analysis: multivariate approach and meta‐regression
van Houwelingen, H. C., Arends, L. R., & Stijnen, T. (2002) · Statistics in Medicine · doi:10.1002/sim.1040 · Cited by 1,580
Moses, L. E., Shapiro, D., & Littenberg, B. (1993) · Statistics in Medicine · doi:10.1002/sim.4780121403 · Cited by 1,405
The STARD Statement for Reporting Studies of Diagnostic Accuracy: Explanation and Elaboration
Bossuyt, P. M., Reitsma, J. B., Bruns, D. E., Gatsonis, C. A., Glasziou, P. P., Irwig, L. M., Moher, D., Rennie, D., de Vet, H. C. W., & Lijmer, J. G. (2003) · Annals of Internal Medicine · doi:10.7326/0003-4819-138-1-200301070-00012-w1 · Cited by 1,124
Self-citationGould, M. K., Maclean, C. C., Kuschner, W. G., Rydzak, C. E., & Owens, D. K. (2001) · JAMA · doi:10.1001/jama.285.7.914 · Cited by 1,094
Towards Complete and Accurate Reporting of Studies of Diagnostic Accuracy: The STARD Initiative
Bossuyt, P. M., Reitsma, J. B., Bruns, D. E., Gatsonis, C. A., Glasziou, P. P., Irwig, L. M., Lijmer, J. G., Moher, D., Rennie, D., de Vet, H. C. W., et al. (2003) · Clinical Chemistry · doi:10.1373/49.1.1 · Cited by 862
Self-citationSources of Variation and Bias in Studies of Diagnostic Accuracy: A Systematic Review
Whiting, P., Rutjes, A. W. S., Reitsma, J. B., Glas, A. S., Bossuyt, P. M. M., & Kleijnen, J. (2004) · Annals of Internal Medicine · doi:10.7326/0003-4819-140-3-200402030-00010 · Cited by 798
Self-citationEvidence base of clinical diagnosis: The architecture of diagnostic research
Sackett, D. L. (2002) · BMJ · doi:10.1136/bmj.324.7336.539 · Cited by 724
Use of methodological standards in diagnostic test research. Getting better but still not good
Reid, M. C. (1995) · JAMA · doi:10.1001/jama.274.8.645 · Cited by 712
Screening for Alcohol Problems in Primary Care: A Systematic Review
Fiellin, D. A., Reid, M. C., & O'Connor, P. G. (2000) · Archives of Internal Medicine · doi:10.1001/archinte.160.13.1977 · Cited by 642
Case–Control and Two-Gate Designs in Diagnostic Accuracy Studies
Rutjes, A. W., Reitsma, J. B., Vandenbroucke, J. P., Glas, A. S., & Bossuyt, P. M. (2005) · Clinical Chemistry · doi:10.1373/clinchem.2005.048595 · Cited by 501
Self-citationMeta-analysis of exercise testing to detect coronary artery disease in women
Kwok, Y., Kim, C., Grady, D., Segal, M., & Redberg, R. (1999) · The American Journal of Cardiology · doi:10.1016/s0002-9149(98)00963-1 · Cited by 453
Romagnuolo, J., Bardou, M., Rahme, E., Joseph, L., Reinhold, C., & Barkun, A. N. (2003) · Annals of Internal Medicine · doi:10.7326/0003-4819-139-7-200310070-00006 · Cited by 445
Pai, M., Flores, L. L., Pai, N., Hubbard, A., Riley, L. W., & Colford, J. M. (2003) · The Lancet Infectious Diseases · doi:10.1016/s1473-3099(03)00772-2 · Cited by 384
Nederkoorn, P. J., van der Graaf, Y., & Hunink, M. G. M. (2003) · Stroke · doi:10.1161/01.str.0000068367.08991.a2 · Cited by 382
Sterne, J. A. C., Jüni, P., Schulz, K. F., Altman, D. G., Bartlett, C., & Egger, M. (2002) · Statistics in Medicine · doi:10.1002/sim.1184 · Cited by 382
Kelly, S., Harris, K. M., Berry, E., Hutton, J., Roderick, P., Cullingworth, J., Gathercole, L., & Smith, M. A. (2001) · Gut · doi:10.1136/gut.49.4.534 · Cited by 366
A bivariate approach to meta-analysis
Van Houwelingen, H. C., Zwinderman, K. H., & Stijnen, T. (1993) · Statistics in Medicine · doi:10.1002/sim.4780122405 · Cited by 327
Assessment of the accuracy of diagnostic tests: the cross-sectional study
Knottnerus, J. A., & Muris, J. W. (2003) · Journal of Clinical Epidemiology · doi:10.1016/s0895-4356(03)00206-3 · Cited by 312
Second-Trimester Ultrasound to Detect Fetuses With Down Syndrome: A Meta-analysis
Smith-Bindman, R., Hosmer, W., Feldstein, V. A., Deeks, J. J., & Goldberg, J. D. (2001) · JAMA · doi:10.1001/jama.285.8.1044 · Cited by 306
The Evidence Base of Clinical Diagnosis: Theory and methods of diagnostic research
(2008) · Wiley eBooks · doi:10.1002/9781444300574 · Cited by 276
The Test of Lasègue: Systematic Review of the Accuracy in Diagnosing Herniated Discs
Devillé, W. L. J. M., van der Windt, D. A. W. M., Dzaferagić, A., Bezemer, P. D., & Bouter, L. M. (2000) · Spine · doi:10.1097/00007632-200005010-00016 · Cited by 264
Bossuyt, P. M., Reitsma, J. B., Bruns, D. E., Gatsonis, C. A., Glasziou, P., Irwig, L., Lijmer, J. G., Moher, D., Rennie, D., & de Vet, H. C. W. (2003) · PubMed · Cited by 242
Self-citationKoelemay, M. J. W., Lijmer, J. G., Stoker, J., Legemate, D. A., & Bossuyt, P. M. M. (2001) · JAMA · doi:10.1001/jama.285.10.1338 · Cited by 234
Self-citationDoes This Patient Have Abdominal Aortic Aneurysm?
Lederle, F. A. (1999) · JAMA · doi:10.1001/jama.281.1.77 · Cited by 222
Li, J. (2001) · The Journal of Emergency Medicine · doi:10.1016/s0736-4679(00)00318-8 · Cited by 222
Pharmacologic stress testing for coronary disease diagnosis: A meta-analysis
Kim, C., Kwok, Y. S., Heagerty, P., & Redberg, R. (2001) · American Heart Journal · doi:10.1067/mhj.2001.119761 · Cited by 220
Peripheral Arterial Disease: Meta-analysis of the Diagnostic Performance of MR Angiography
Nelemans, P. J., Leiner, T., de Vet, H. C. W., & van Engelshoven, J. M. A. (2000) · Radiology · doi:10.1148/radiology.217.1.r00oc11105 · Cited by 219
Visser, K., & Myriam Hunink, M. G. (2000) · Radiology · doi:10.1148/radiology.216.1.r00jl0367 · Cited by 187
A meta-analysis of the Papanicolaou smear and wet mount for the diagnosis of vaginal trichomoniasis
Wiese, W., Patel, S. R., Patel, S. C., Ohl, C. A., & Estrada, C. A. (2000) · The American Journal of Medicine · doi:10.1016/s0002-9343(99)00466-0 · Cited by 150
Verhagen, A. P., de Vet, H. C. W., de Bie, R. A., Kessels, A. G. H., Boers, M., Bouter, L. M., & Knipschild, P. G. (1998) · Digital Academic REpository of VU University Amsterdam (Vrije Universiteit Amsterdam) · Cited by 146
Abdominal Symptoms: Do They Predict Gallstones?: A Systematic Review
Berger, J. J. I. M. V. D. V. M. Y. (2000) · Scandinavian Journal of Gastroenterology · doi:10.1080/003655200750024560 · Cited by 141
Balk, E. M., Ioannidis, J. P. A., Salem, D. N., Chew, P. W., & Lau, J. (2001) · Annals of Emergency Medicine · doi:10.1067/mem.2001.114905 · Cited by 140
Lau, J., Ioannidis, J. P. A., Balk, E. M., Milch, C. E., Terrin, N. C., Chew, P. W., & Salem, D. N. (2001) · Annals of Emergency Medicine · doi:10.1067/mem.2001.114903 · Cited by 138
Magnetic Resonance Imaging of the Wrist: Diagnostic Performance Statistics
Hobby, J., Tom, B. D. M., Bearcroft, P. W. P., & Dixon, A. K. (2001) · Clinical Radiology · doi:10.1053/crad.2000.0571 · Cited by 116
Reporting of measures of accuracy in systematic reviews of diagnostic literature
Honest, H., & Khan, K. S. (2002) · BMC Health Services Research · doi:10.1186/1472-6963-2-4 · Cited by 114
Hoogendam, A., Buntinx, F., & de Vet, H. C. (1999) · Family Practice · doi:10.1093/fampra/16.6.621 · Cited by 109
CT pulmonary angiography in the detection of pulmonary emboli
Safriel, Y. I., & Zinn, H. L. (2002) · Clinical Imaging · doi:10.1016/s0899-7071(01)00366-7 · Cited by 100
Huicho, L., Campos, M. A. L., & Alamo, C. (2002) · The Pediatric Infectious Disease Journal · doi:10.1097/00006454-200201000-00002 · Cited by 97
Whiting, P. F., Rutjes, A. W., Dinnes, J., Reitsma, J. B., Bossuyt, P. M., & Kleijnen, J. (2005) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2004.04.008 · Cited by 78
Self-citationSonnad, S. S., Langlotz, C. P., & Schwartz, J. S. (2001) · Academic Radiology · doi:10.1016/s1076-6332(01)90095-9 · Cited by 77
The Architecture of Diagnostic Research
Haynes, R. B., & You, J. J. (2008) · doi:10.1002/9781444300574.ch2 · Cited by 77
Hoffman, R. M., Clanon, D. L., Littenberg, B., et al. (2000) · Journal of General Internal Medicine · doi:10.1046/j.1525-1497.2000.90907.x · Cited by 67
Diagnostic research on routine care data
Oostenbrink, R., Moons, K. G. M., Bleeker, S. E., Moll, H. A., & Grobbee, D. E. (2003) · Journal of Clinical Epidemiology · doi:10.1016/s0895-4356(03)00080-5 · Cited by 59
Screening and Syndromic Approaches to Identify Gonorrhea and Chlamydial Infection among Women
Sloan, N. L., Winikoff, B., Haberland, N. A., Coggins, C., & Elias, C. J. (2000) · Studies in Family Planning · doi:10.1111/j.1728-4465.2000.00055.x · Cited by 57
EFFECT OF STUDY DESIGN ON THE ASSOCIATION BETWEEN NUCHAL TRANSLUCENCY MEASUREMENT AND DOWN SYNDROME
Mol, B. W., Lijmer, J. G., van der Meulen, J., Pajkrt, E., Bilardo, C. M., & Bossuyt, P. M. (1999) · Obstetrics and Gynecology · doi:10.1016/s0029-7844(99)00496-2 · Cited by 51
Self-citationCompliance with methodological standards when evaluating ophthalmic diagnostic tests.
Harper, R. A., & Reeves, B. C. (1999) · PubMed · Cited by 46
Screening for squamous intraepithelial lesions with fluorescence spectroscopy*1
Mitchell, M. F. (1999) · Obstetrics and Gynecology · doi:10.1016/s0029-7844(99)00408-1 · Cited by 45
Assessment of the Accuracy of Diagnostic Tests: The Cross‐Sectional Study
Knottnerus, J. A., & Muris, J. W. M. (2008) · doi:10.1002/9781444300574.ch3 · Cited by 30
Concordance of EndotoxemiaWith Gram-Negative Bacteremia
Hurley, J. C. (2000) · Archives of Pathology & Laboratory Medicine · doi:10.5858/2000-124-1157-coegnb · Cited by 24
Estrada, C. A., Bloch, R. M., Antonacci, D. J., Basnight, L. L., Patel, S. R., Patel, S. C., & Wiese, W. (2000) · Journal of General Internal Medicine · doi:10.1046/j.1525-1497.2000.03189.x · Cited by 23
Sources of Variation and Bias in Studies of Diagnostic Accuracy: a Systematic Review
Whiting, P., Rutjes, A. W., Reitsma, J. B., Glas, A. S., Bossuyt, P. M., & Kleijnen, J. (2004) · Bristol Research (University of Bristol) · Cited by 23
Self-citationIncomplete Data And Imperfect Reference Tests
PEPE, M. S. (2003) · Oxford University Press eBooks · doi:10.1093/oso/9780198509844.003.0007 · Cited by 14
Pharmacologic stress testing for coronary disease diagnosis: a meta-analysis
Kim, C., Kwok, Y. S., Heagerty, P. J., et al. (2002) · ACC Current Journal Review · doi:10.1016/s1062-1458(02)00615-3 · Cited by 11
Efficacy of morphine sulfate-augmented hepatobiliary imaging in acute cholecystitis.
VASQUEZ, T. E., RIMKUS, D. S., Hass, M. G., et al. (2000) · PubMed · Cited by 9
Cited by
Moons, K. G. M., Altman, D. G., Reitsma, J. B., Ioannidis, J. P. A., Macaskill, P., Steyerberg, E. W., Vickers, A. J., Ransohoff, D. F., & Collins, G. S. (2015) · Annals of Internal Medicine · doi:10.7326/m14-0698 · Cited by 5,147
Self-citationSystematic reviews: CRD's guidance for undertaking reviews in health care
Tacconelli, E. (2010) · The Lancet Infectious Diseases · doi:10.1016/s1473-3099(10)70065-7 · Cited by 3,245
STARD 2015 guidelines for reporting diagnostic accuracy studies: explanation and elaboration
Cohen, J. F., Korevaar, D. A., Altman, D. G., Bruns, D. E., Gatsonis, C. A., Hooft, L., Irwig, L., Levine, D., Reitsma, J. B., de Vet, H. C. W., et al. (2016) · BMJ Open · doi:10.1136/bmjopen-2016-012799 · Cited by 2,570
Self-citationROBIS: A new tool to assess risk of bias in systematic reviews was developed
Whiting, P., Savović, J., Higgins, J. P. T., Caldwell, D. M., Reeves, B. C., Shea, B., Davies, P., Kleijnen, J., & Churchill, R. (2016) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2015.06.005 · Cited by 2,272
Avoidable waste in the production and reporting of research evidence
Chalmers, I., & Glasziou, P. (2009) · The Lancet · doi:10.1016/s0140-6736(09)60329-9 · Cited by 2,208
Metadata
- Authors
- Rutjes, A. W. S., Reitsma, J. B., Di Nisio, M., Smidt, N., van Rijn, J. C., & Bossuyt, P. M. M.
- Venue
- Canadian Medical Association Journal
- Publisher
- CMA Joule Inc.
- Published
- 2006-02-13
- Volume
- 174
- Issue
- 4
- Pages
- 469-476
- ISSN
- 0820-3946 (ISSN-L)
- 1488-2329
- OpenAlex
- W2129152887
- Cited by
- 611
- References
- 69
- FWCI
- 32.73
- Type
- journal article
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.