The DOI matches its Crossref record; 58 of the 58 references it cites were found in a registry or an index.
GRADE guidelines: 21 part 1. Study design, risk of bias, and indirectness in rating the certainty across a body of evidence for test accuracy
Open accessAbstract
OBJECTIVES This article provides updated GRADE guidance about how authors of systematic reviews and health technology assessments (HTA) and guideline developers can assess the results and the certainty of evidence (also known as quality of the evidence or confidence in the estimates) of a body of evidence addressing test accuracy (TA). STUDY DESIGN AND SETTING We present an overview of the GRADE approach and guidance for rating certainty in TA in clinical and public health and review the presentation of results of a body of evidence regarding tests. Part 1 of the two parts in this 21st guidance article about how to apply GRADE focuses on understanding study design issues in test accuracy, provide an overiew of the domains and describe risk of bias and indirectness specifically. RESULTS Supplemented by practical examples, we describe how raters of the evidence using GRADE can evaluate study designs focusing on tests and how they apply the GRADE domains risk of bias and indirectness to a body of evidence of TA studies. CONCLUSIONS Rating the certainty of a body of evidence using GRADE in Cochrane and other reviews and World Health Organization and other guidelines dealing with in TA studies helped refining our approach. The resulting guidance will help applying GRADE successfully for questions and recommendations focusing on tests.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
QUADAS-2: a revised tool for the quality assessment of diagnostic accuracy studies
Whiting, P. F., Rutjes, A. W. S., Westwood, M. E., Mallett, S., Deeks, J. J., Reitsma, J. B., Leeflang, M. M. G., Sterne, J. A. C., Bossuyt, P. M. M., & the QUADAS-2 Group* (2011) · Annals of Internal Medicine · doi:10.7326/0003-4819-155-8-201110180-00009 · Cited by 14,199
Self-citationGRADE guidelines: 1. Introduction—GRADE evidence profiles and summary of findings tables
Guyatt, G., Oxman, A. D., Akl, E. A., Kunz, R., Vist, G., Brozek, J., Norris, S., Falck-Ytter, Y., Glasziou, P., & deBeer, H. (2011) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2010.04.026 · Cited by 10,681
Self-citationWhiting, P., Rutjes, A. W., Reitsma, J. B., Bossuyt, P. M., & Kleijnen, J. (2003) · BMC Medical Research Methodology · doi:10.1186/1471-2288-3-25 · Cited by 3,592
Self-citationTowards complete and accurate reporting of studies of diagnostic accuracy: The STARD Initiative.
Bossuyt, P. M., Reitsma, J. B., Bruns, D. E., Gatsonis, C. A., Glasziou, P. P., Irwig, L. M., Lijmer, J. G., Moher, D., Rennie, D., de Vet, H. C. W., et al. (2003) · Annals of Internal Medicine · doi:10.7326/0003-4819-138-1-200301070-00010 · Cited by 3,219
Self-citationTowards complete and accurate reporting of studies of diagnostic accuracy: the STARD initiative
Bossuyt, P. M., Reitsma, J. B., Bruns, D. E., Gatsonis, C. A., Glasziou, P. P., Irwig, L. M., Lijmer, J. G., Moher, D., Rennie, D., & de Vet, H. C. W. (2003) · BMJ · doi:10.1136/bmj.326.7379.41 · Cited by 3,215
Self-citationTowards Complete and Accurate Reporting of Studies of Diagnostic Accuracy: The STARD Initiative
Bossuyt, P. M., Reitsma, J. B., Bruns, D. E., Gatsonis, C. A., Glasziou, P. P., Irwig, L. M., Lijmer, J. G., Moher, D., Rennie, D., & de Vet, H. C. W. (2003) · Radiology · doi:10.1148/radiol.2261021292 · Cited by 3,215
Self-citationGRADE guidelines: a new series of articles in the Journal of Clinical Epidemiology
Guyatt, G. H., Oxman, A. D., Schünemann, H. J., Tugwell, P., & Knottnerus, A. (2011) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2010.09.011 · Cited by 3,094
Self-citationWalker, F. O. (2007) · The Lancet · doi:10.1016/s0140-6736(07)60111-1 · Cited by 2,603
GRADE guidelines: 2. Framing the question and deciding on important outcomes
Guyatt, G. H., Oxman, A. D., Kunz, R., Atkins, D., Brozek, J., Vist, G., Alderson, P., Glasziou, P., Falck-Ytter, Y., & Schünemann, H. J. (2011) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2010.09.012 · Cited by 1,911
Self-citationEmpirical evidence of design-related bias in studies of diagnostic tests.
Lijmer, J. G. (1999) · JAMA · doi:10.1001/jama.282.11.1061 · Cited by 1,861
Puhan, M. A., Schunemann, H. J., Murad, M. H., Li, T., Brignardello-Petersen, R., Singh, J. A., Kessels, A. G., Guyatt, G. H., & for the GRADE Working Group (2014) · BMJ · doi:10.1136/bmj.g5630 · Cited by 1,812
Self-citationGrading quality of evidence and strength of recommendations for diagnostic tests and strategies
Schünemann, H. J., Oxman, A. D., Brozek, J., Glasziou, P., Jaeschke, R., Vist, G. E., Williams, J. W., Kunz, R., Craig, J., Montori, V. M., et al. (2008) · BMJ · doi:10.1136/bmj.39500.677199.ae · Cited by 1,782
Self-citationSystematic reviews of evaluations of diagnostic and screening tests
Deeks, J. J. (2001) · BMJ · doi:10.1136/bmj.323.7305.157 · Cited by 1,340
Alonso-Coello, P., Schünemann, H. J., Moberg, J., Brignardello-Petersen, R., Akl, E. A., Davoli, M., Treweek, S., Mustafa, R. A., Rada, G., Rosenbaum, S., et al. (2016) · BMJ · doi:10.1136/bmj.i2016 · Cited by 1,317
Self-citationUse of B-Type Natriuretic Peptide in the Evaluation and Management of Acute Dyspnea
Mueller, C., Scholer, A., Laule-Kilian, K., Martina, B., Schindler, C., Buser, P., Pfisterer, M., & Perruchoud, A. P. (2004) · New England Journal of Medicine · doi:10.1056/nejmoa031681 · Cited by 1,000
Iorio, A., Spencer, F. A., Falavigna, M., Alba, C., Lang, E., Burnand, B., McGinn, T., Hayden, J., Williams, K., Shea, B., et al. (2015) · BMJ · doi:10.1136/bmj.h870 · Cited by 880
Self-citationSystematic reviews of evaluations of diagnostic and screening tests
(2001) · Europe PMC (PubMed Central) · Cited by 850
Alonso-Coello, P., Oxman, A. D., Moberg, J., Brignardello-Petersen, R., Akl, E. A., Davoli, M., Treweek, S., Mustafa, R. A., Vandvik, P. O., Meerpohl, J., et al. (2016) · BMJ · doi:10.1136/bmj.i2089 · Cited by 814
Self-citationEvaluation of QUADAS, a tool for the quality assessment of diagnostic accuracy studies
Whiting, P. F., Weswood, M. E., Rutjes, A. W., Reitsma, J. B., Bossuyt, P. N., & Kleijnen, J. (2006) · BMC Medical Research Methodology · doi:10.1186/1471-2288-6-9 · Cited by 769
Self-citationGuyatt, G., Oxman, A. D., Sultan, S., Brozek, J., Glasziou, P., Alonso-Coello, P., Atkins, D., Kunz, R., Montori, V., Jaeschke, R., et al. (2013) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2012.01.006 · Cited by 762
Self-citationFiocchi, A., Brozek, J., Schünemann, H., Bahna, S. L., von Berg, A., Beyer, K., Bozzola, M., Bradsher, J., Compalati, E., Ebisawa, M., et al. (2010) · World Allergy Organization Journal · doi:10.1097/wox.0b013e3181defeb9 · Cited by 644
Self-citationEvidence of bias and variation in diagnostic accuracy studies
Rutjes, A. W. S., Reitsma, J. B., Di Nisio, M., Smidt, N., van Rijn, J. C., & Bossuyt, P. M. M. (2006) · Canadian Medical Association Journal · doi:10.1503/cmaj.050090 · Cited by 611
Self-citationClinical Trial Designs for Predictive Marker Validation in Cancer Treatment Trials
Sargent, D. J., Conley, B. A., Allegra, C., & Collette, L. (2005) · Journal of Clinical Oncology · doi:10.1200/jco.2005.01.112 · Cited by 499
Guidelines for reading literature reviews.
Oxman, A. D., & Guyatt, G. H. (1988) · Europe PMC (PubMed Central) · Cited by 489
Self-citationComparative accuracy: assessing new tests against existing diagnostic pathways
Bossuyt, P. M., Irwig, L., Craig, J., & Glasziou, P. (2006) · BMJ · doi:10.1136/bmj.332.7549.1089 · Cited by 458
Self-citation(2010) · Pediatric Allergy and Immunology · doi:10.1111/j.1399-3038.2010.01068.x · Cited by 425
Schünemann, H. J., Mustafa, R. A., Brozek, J., Steingart, K. R., Leeflang, M., Murad, M. H., Bossuyt, P., Glasziou, P., Jaeschke, R., Lange, S., et al. (2020) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2019.12.021 · Cited by 387
Self-citationSchünemann, H. J., Mustafa, R., Brozek, J., Santesso, N., Alonso-Coello, P., Guyatt, G., Scholten, R., Langendam, M., Leeflang, M. M., Akl, E. A., et al. (2016) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2016.01.032 · Cited by 350
Self-citationMoe, G. W., Howlett, J., Januzzi, J. L., & Zowall, H. (2007) · Circulation · doi:10.1161/circulationaha.106.666255 · Cited by 325
Letters, numbers, symbols and words: how to communicate grades of evidence and recommendations.
Schünemann, H. J., Best, D., Vist, G. E., & Oxman, A. D. (2003) · PubMed Central · Cited by 289
Self-citationZhang, Y., Coello, P. A., Guyatt, G. H., Yepes-Nuñez, J. J., Akl, E. A., Hazlewood, G., Pardo-Hernandez, H., Etxeandia-Ikobaltzeta, I., Qaseem, A., Williams, J. W., et al. (2019) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2018.05.011 · Cited by 285
Self-citationZhang, Y., Alonso-Coello, P., Guyatt, G. H., Yepes-Nuñez, J. J., Akl, E. A., Hazlewood, G., Pardo-Hernandez, H., Etxeandia-Ikobaltzeta, I., Qaseem, A., Williams, J. W., et al. (2019) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2018.01.013 · Cited by 281
Self-citationGRADE: Assessing the quality of evidence in environmental and occupational health
Morgan, R. L., Thayer, K. A., Bero, L., Bruce, N., Falck-Ytter, Y., Ghersi, D., Guyatt, G., Hooijmans, C., Langendam, M., Mandrioli, D., et al. (2016) · Environment International · doi:10.1016/j.envint.2016.01.004 · Cited by 277
Self-citationBrożek, J. L., Akl, E. A., Jaeschke, R., Lang, D. M., Bossuyt, P., Glasziou, P., Helfand, M., Ueffing, E., Alonso‐Coello, P., Meerpohl, J., et al. (2009) · Allergy · doi:10.1111/j.1398-9995.2009.02083.x · Cited by 270
Self-citationLord, S. J., Irwig, L., & Simes, R. J. (2006) · Annals of Internal Medicine · doi:10.7326/0003-4819-144-11-200606060-00011 · Cited by 268
Steingart, K. R., Flores, L. L., Dendukuri, N., Schiller, I., Laal, S., Ramsay, A., Hopewell, P. C., & Pai, M. (2011) · PLoS Medicine · doi:10.1371/journal.pmed.1001062 · Cited by 266
Self-citationRandomised comparisons of medical tests: sometimes invalid, not always efficient
Bossuyt, P. M., Lijmer, J. G., & Mol, B. W. (2000) · The Lancet · doi:10.1016/s0140-6736(00)03246-3 · Cited by 239
Self-citationGRADE: assessing the quality of evidence for diagnostic recommendations
Schünemann, H. J., Oxman, A. D., Brozek, J., Glasziou, P., Bossuyt, P., Chang, S., Muti, P., Jaeschke, R., & Guyatt, G. H. (2008) · Evidence Based Medicine · doi:10.1136/ebm.13.6.162-a · Cited by 230
Self-citationWalker, F. (2007) · Seminars in Neurology · doi:10.1055/s-2007-971176 · Cited by 229
Santesso, N., Mustafa, R. A., Wiercioch, W., Kehar, R., Gandhi, S., Chen, Y., Cheung, A., Hopkins, J., Khatib, R., Ma, B., et al. (2016) · International Journal of Gynecology & Obstetrics · doi:10.1016/j.ijgo.2015.07.026 · Cited by 210
Self-citationSantesso, N., Mustafa, R. A., Schünemann, H. J., Arbyn, M., Blumenthal, P. D., Cain, J., Chirenje, M., Denny, L., De Vuyst, H., Eckert, L. O., et al. (2016) · International Journal of Gynecology & Obstetrics · doi:10.1016/j.ijgo.2015.07.038 · Cited by 204
Self-citationWorster, A., Preyra, I., Weaver, B., & Haines, T. (2002) · Annals of Emergency Medicine · doi:10.1067/mem.2002.126170 · Cited by 173
Schünemann, H. J., Mustafa, R. A., Brożek, J., Santesso, N. A., Bossuyt, P. M., Steingart, K. R., Leeflang, M., Lange, S., Trenti, T., Langendam, M. W., et al. (2019) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2019.02.003 · Cited by 166
Self-citationGuyatt, G. H., Ebrahim, S., Alonso-Coello, P., Johnston, B. C., Mathioudakis, A. G., Briel, M., Mustafa, R. A., Sun, X., Walter, S. D., Heels-Ansdell, D., et al. (2017) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2017.05.005 · Cited by 163
Self-citationSchünemann, H. J., Mustafa, R. A., Brożek, J., Steingart, K. R., Leeflang, M., Murad, M. H., Bossuyt, P. M., Glasziou, P., Jaeschke, R., Lange, S., et al. (2020) · LSTM Online Archive (Liverpool School of Tropical Medicine) · doi:10.7892/boris.140684 · Cited by 156
Self-citationTransparent Development of the WHO Rapid Advice Guidelines
Schünemann, H. J., Hill, S. R., Kakad, M., Vist, G. E., Bellamy, R., Stockman, L. J., Wisløff, T. F., Mar, C. D., Hayden, F. G., Uyeki, T. M., et al. (2007) · PLoS Medicine · doi:10.1371/journal.pmed.0040119 · Cited by 149
Self-citationAlmqvist, E. W., Brinkman, R. R., Wiggins, S., Hayden, M. R., & Testing, T. C. C. S. O. P. (2003) · Clinical Genetics · doi:10.1034/j.1399-0004.2003.00157.x · Cited by 125
Various randomized designs can be used to evaluate medical tests
Lijmer, J. G., & Bossuyt, P. M. (2008) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2008.06.017 · Cited by 90
Self-citationXpert® MTB/RIF assay for extrapulmonary tuberculosis and rifampicin resistance
Kohli, M., Schiller, I., Dendukuri, N., Ryan, H., Dheda, K. U. J., Denkinger, C. M., Schumacher, S. G., & Steingart, K. R. (2017) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd012768 · Cited by 85
Self-citationParadox of a better test for Huntington's disease
Maat‐Kievit, A. (2000) · Journal of Neurology Neurosurgery & Psychiatry · doi:10.1136/jnnp.69.5.579 · Cited by 82
Assessing quality of a diagnostic test evaluation
Mulrow, C. D., Linn, W. D., Gaul, M. K., & Pugh, J. A. (1989) · Journal of General Internal Medicine · doi:10.1007/bf02597398 · Cited by 77
Uncertainties in baseline risk estimates and confidence in treatment effects
Spencer, F. A., Iorio, A., You, J. J., Murad, M. H., Schünemann, H. J., Vandvik, P. O., Crowther, M. A., Pottie, K. C., Lang, E., Meerpohl, J. J., et al. (2012) · BMJ · doi:10.1136/bmj.e7401 · Cited by 61
Self-citationMustafa, R. A., Wiercioch, W., Cheung, A., Prediger, B., Brożek, J., Bossuyt, P. M., Garg, A. X., Lelgemann, M., Bühler, D., & Schünemann, H. J. (2017) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2017.09.003 · Cited by 49
Self-citationUsing GRADE to respond to health questions with different levels of urgency
Thayer, K. A., & Schünemann, H. J. (2016) · Environment International · doi:10.1016/j.envint.2016.03.027 · Cited by 35
Self-citationEvidence-Based Approach to the Introduction of Positron Emission Tomography in Ontario, Canada
Evans, W. K., Laupacis, A., Gulenchyn, K. Y., Levin, L., & Levine, M. N. (2009) · Journal of Clinical Oncology · doi:10.1200/jco.2009.22.1614 · Cited by 22
Assessing evidence in public health: the added value of GRADE
Burford, B. J., Rehfuess, E. A., Schünemann, H. J., Akl, E. A., Waters, E. B., Armstrong, R., Thomson, H., Doyle, J., & Pettman, T. L. (2012) · Journal of Public Health · doi:10.1093/pubmed/fds092 · Cited by 22
Self-citationWorster, A., & Haines, T. (2002) · PubMed · Cited by 14
Schünemann, H. J., & Mustafa, R. A. (2017) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2017.09.006 · Cited by 11
Self-citation
Cited by
False-negative results of initial RT-PCR assays for COVID-19: A systematic review
Arevalo-Rodriguez, I., Buitrago-Garcia, D., Simancas-Racines, D., Zambrano-Achig, P., Del Campo, R., Ciapponi, A., Sued, O., Martinez-García, L., Rutjes, A. W., Low, N., et al. (2020) · PLOS ONE · doi:10.1371/journal.pone.0242958 · Cited by 656
Self-citationASH ISTH NHF WFH 2021 guidelines on the diagnosis of von Willebrand disease
James, P. D., Connell, N. T., Ameer, B., Di Paola, J., Eikenboom, J., Giraud, N., Haberichter, S., Jacobs-Pratt, V., Konkle, B., McLintock, C., et al. (2021) · Blood Advances · doi:10.1182/bloodadvances.2020003265 · Cited by 586
Self-citationSchünemann, H. J., Mustafa, R. A., Brozek, J., Steingart, K. R., Leeflang, M., Murad, M. H., Bossuyt, P., Glasziou, P., Jaeschke, R., Lange, S., et al. (2020) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2019.12.021 · Cited by 387
Self-citationFALSE-NEGATIVE RESULTS OF INITIAL RT-PCR ASSAYS FOR COVID-19: A SYSTEMATIC REVIEW
Arevalo-Rodriguez, I., Buitrago-Garcia, D., Simancas-Racines, D., Zambrano-Achig, P., Campo, R. D., Ciapponi, A., Sued, O., Martínez-García, L., Rutjes, A., Low, N., et al. (2020) · medRxiv · doi:10.1101/2020.04.16.20066787 · Cited by 310
Self-citationKohli, M., Schiller, I., Dendukuri, N., Yao, M., Dheda, K., Denkinger, C. M., Schumacher, S. G., & Steingart, K. R. (2021) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd012768.pub3 · Cited by 215
Self-citation
Metadata
- Authors
- Schünemann, H. J., Mustafa, R. A., Brozek, J., Steingart, K. R., Leeflang, M., Murad, M. H., Bossuyt, P., Glasziou, P., Jaeschke, R., Lange, S., Meerpohl, J., Langendam, M., Hultcrantz, M., Vist, G. E., Akl, E. A., Helfand, M., Santesso, N., Hooft, L., Scholten, R., Rosen, M., Rutjes, A., Crowther, M., Muti, P., Raatz, H., Ansari, M. T., Williams, J., Kunz, R., Harris, J., Rodriguez, I. A., Kohli, M., & Guyatt, G. H.
- Venue
- Journal of Clinical Epidemiology
- Publisher
- Elsevier BV
- Published
- 2020-06
- Volume
- 122
- Pages
- 129-141
- ISSN
- 0895-4356 (ISSN-L)
- OpenAlex
- W3005727251
- Cited by
- 405
- References
- 58
- FWCI
- 28.86
- Type
- journal article
- Language
- en
- Open access
- bronze
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.