Testing the Newcastle Ottawa Scale showed low reliability between individual reviewers
Abstract
Objectives To assess inter-rater reliability and validity of the Newcastle Ottawa Scale (NOS) used for methodological quality assessment of cohort studies included in systematic reviews. Study design and setting Two reviewers independently applied the NOS to 131 cohort studies included in eight meta-analyses. Inter-rater reliability was calculated using kappa (κ) statistics. To assess validity, within each meta-analysis, we generated a ratio of pooled estimates for each quality domain. Using a random-effects model, the ratios of odds ratios for each meta-analysis were combined to give an overall estimate of differences in effect estimates. Results Inter-rater reliability varied from substantial for length of follow-up (κ = 0.68, 95% confidence interval [CI] = 0.47, 0.89) to poor for selection of the nonexposed cohort and demonstration that the outcome was not present at the outset of the study (κ = -0.03, 95% CI = -0.06, 0.00; κ = -0.06, 95% CI = -0.20, 0.07). Reliability for overall score was fair (κ = 0.29, 95% CI = 0.10, 0.47). In general, reviewers found the tool difficult to use and the decision rules vague even with additional information provided as part of this study. We found no association between individual items or overall score and effect estimates. Conclusion Variable agreement and lack of evidence that the NOS can identify studies with biased results underscore the need for revisions and more detailed guidance for systematic reviewers using the NOS.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Cites
The Measurement of Observer Agreement for Categorical Data
Landis, J. R., & Koch, G. G. (1977) · Biometrics · doi:10.2307/2529310 · Cited by 81,148 · see citing works
Cochrane Handbook for Systematic Reviews of Interventions: Cochrane Book Series
(2008) · doi:10.1002/9780470712184 · Cited by 25,048 · see citing works
Downs, S. H., & Black, N. (1998) · Journal of Epidemiology and Community Health · doi:10.1136/jech.52.6.377 · Cited by 8,252 · see citing works
Agresti, A. (2002) · Wiley Series in Probability and Statistics · doi:10.1002/0471249688 · Cited by 6,746 · see citing works
The Hazards of Scoring the Quality of Clinical Trials for Meta-analysis
Jüni, P. (1999) · JAMA · doi:10.1001/jama.282.11.1054 · Cited by 1,981 · see citing works
Grote, N. K., Bridge, J. A., Gavin, A. R., Melville, J. L., Iyengar, S., & Katon, W. J. (2010) · Archives of General Psychiatry · doi:10.1001/archgenpsychiatry.2010.111 · Cited by 1,882 · see citing works
Moher, D., Jadad, A. R., Nichol, G., Penman, M., Tugwell, P., & Walsh, S. (1995) · Controlled Clinical Trials · doi:10.1016/0197-2456(94)00031-w · Cited by 1,308 · see citing works
Egger, M., Jüni, P., Bartlett, C., Holenstein, F., & Sterne, J. (2003) · Health Technology Assessment · doi:10.3310/hta7010 · Cited by 1,181 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Salvagioni, D. A. J., Melanda, F. N., Mesas, A. E., González, A. D., Gabani, F. L., & Andrade, S. M. D. (2017) · PLOS ONE · doi:10.1371/journal.pone.0185781 · Cited by 1,508 · see citing works
Cognitive biases associated with medical decisions: a systematic review
Saposnik, G., Redelmeier, D., Ruff, C. C., & Tobler, P. N. (2016) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-016-0377-1 · Cited by 1,001 · see citing works
Cook, D. A., & Reed, D. A. (2015) · Academic Medicine · doi:10.1097/acm.0000000000000786 · Cited by 781 · see citing works
Physical Activity and Anxiety: A Systematic Review and Meta-analysis of Prospective Cohort Studies
McDowell, C. P., Dishman, R. K., Gordon, B. R., & Herring, M. P. (2019) · American Journal of Preventive Medicine · doi:10.1016/j.amepre.2019.05.012 · Cited by 385 · see citing works
Knott, C., Bell, S., & Britton, A. (2015) · Diabetes Care · doi:10.2337/dc15-0710 · Cited by 375 · see citing works
Schwingshackl, L., Knüppel, S., Schwedhelm, C., Hoffmann, G., Missbach, B., Stelmach-Mardas, M., Dietrich, S., Eichelmann, F., Kontopanteils, E., Iqbal, K., et al. (2016) · Advances in Nutrition · doi:10.3945/an.116.013052 · Cited by 360 · see citing works
Petrelli, F., Signorelli, D., Ghidini, M., Ghidini, A., Pizzutilo, E. G., Ruggieri, L., Cabiddu, M., Borgonovo, K., Dognini, G., Brighenti, M., et al. (2020) · Cancers · doi:10.3390/cancers12030546 · Cited by 300 · see citing works
Farrah, K., Young, K., Tunis, M. C., & Zhao, L. (2019) · Systematic Reviews · doi:10.1186/s13643-019-1172-8 · Cited by 246 · see citing works
Serum Vitamin D Levels and Polycystic Ovary Syndrome: A Systematic Review and Meta-Analysis
He, C., Lin, Z., Robb, S., & Ezeamama, A. (2015) · Nutrients · doi:10.3390/nu7064555 · Cited by 242 · see citing works
Domnich, A., Arata, L., Amicizia, D., Puig-Barberà, J., Gasparini, R., & Panatto, D. (2017) · Vaccine · doi:10.1016/j.vaccine.2016.12.011 · Cited by 229 · see citing works
Blond, K., Brinkløv, C. F., Ried-Larsen, M., Crippa, A., & Grøntved, A. (2020) · British Journal of Sports Medicine · doi:10.1136/bjsports-2018-100393 · Cited by 201 · see citing works
A meta-analysis of chemokines in major depression
Eyre, H. A., Air, T., Pradhan, A., Johnston, J., Lavretsky, H., Stuart, M. J., & Baune, B. T. (2016) · Progress in Neuro-Psychopharmacology and Biological Psychiatry · doi:10.1016/j.pnpbp.2016.02.006 · Cited by 188 · see citing works
Baldwin, C., van Kessel, G., Phillips, A., & Johnston, K. (2017) · Physical Therapy · doi:10.1093/ptj/pzx076 · Cited by 111 · see citing works
Potterton, R., Austin, A., Robinson, L., Webb, H., Allen, K. L., & Schmidt, U. (2022) · Journal of Youth and Adolescence · doi:10.1007/s10964-021-01536-7 · Cited by 106 · see citing works
Zhang, X., Zhang, R., Cheng, L., Wang, Y., Ding, X., Fu, J., Dang, J., Moore, J., & Li, R. (2020) · Sleep Medicine · doi:10.1016/j.sleep.2020.05.014 · Cited by 50 · see citing works
Zuin, M., Cervellati, C., Trentini, A., Passaro, A., Rosta, V., Zimetti, F., & Zuliani, G. (2021) · Diagnostics · doi:10.3390/diagnostics11060984 · Cited by 45 · see citing works
Baldini, V., Gottardi, C., Di Stefano, R., Rindi, L. V., Pazzocco, G., Varallo, G., Purgato, M., De Ronchi, D., Barbui, C., & Ostuzzi, G. (2025) · European Psychiatry · doi:10.1192/j.eurpsy.2025.2452 · Cited by 41 · see citing works
“Included studies were evaluated with the Newcastle-Ottawa Scale (NOS), which assesses the risk of bias in observational studies on three domains (selection, comparability, and exposure) and provides an overall score ranging from 1 (highest risk of bias) to 9 (lowest risk of bias) [25].”
Long-term effects of childhood speech and language disorders: A scoping review
Langbecker, D., Snoswell, C. L., Smith, A. C., Verboom, J., & Caffery, L. J. (2020) · South African Journal of Childhood Education · doi:10.4102/sajce.v10i1.801 · Cited by 39 · see citing works
“Despite receiving criticism for rating validity and inter-rater reliability (Hartling et al. 2013; Lo, Mertz & Loeb 2014), the Newcastle–Ottawa scale has been used extensively (Abdel-Latif et al.”
MethodBaldini, V., Gnazzo, M., Varallo, G., De Ronchi, D., & Fiorillo, A. (2025) · International Journal of Social Psychiatry · doi:10.1177/00207640251339510 · Cited by 23 · see citing works
- Authors
- Hartling, L., Milne, A., Hamm, M. P., Vandermeer, B., Ansari, M., Tsertsvadze, A., & Dryden, D. M.
- Affiliations
- University of Alberta (CA)
- University of Ottawa (CA)
- Corresponding author
- Hartling, Lisa
- Venue
- Journal of Clinical Epidemiology
- Publisher
- Elsevier BV
- Published
- 2013-09
- Volume
- 66
- Issue
- 9
- Pages
- 982-993
- ISSN
- 0895-4356 (ISSN-L)
- OpenAlex
- W1993349052
- Cited by
- 436
- References
- 8
- Citations by year
- 2026: 31 · 2025: 36 · 2024: 50 · 2023: 46 · 2022: 30 · 2021: 55 · 2020: 35 · 2019: 35 · 2018: 24 · 2017: 21 · 2016: 36 · 2015: 20 · 2014: 14 · 2013: 2
- FWCI
- 26.44
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Journal of Clinical Epidemiology (publishedVersion)
- PubMed (publishedVersion)
- Funders
- University of Alberta
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Reliability and Agreement in Measurement
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Epidemiology
- Keywords
- Newcastle-Ottawa Scale, inter-rater reliability, methodological quality assessment, cohort studies, systematic reviews, quality assessment tools, validity, kappa statistic, bias in observational studies
- MeSH terms
- Humans, Research Design, Review Literature as Topic, Meta-Analysis as Topic, Reproducibility of Results, Cohort Studies, Observer Variation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
