The DOI matches its Crossref record; 43 of the 43 references it cites were found in a registry or an index.
Quality assessment of observational studies in a drug-safety systematic review, comparison of two tools: the Newcastle–Ottawa Scale and the RTI item bank
Open accessAbstract
BACKGROUND: The study objective was to compare the Newcastle-Ottawa Scale (NOS) and the RTI item bank (RTI-IB) and estimate interrater agreement using the RTI-IB within a systematic review on the cardiovascular safety of glucose-lowering drugs. METHODS: We tailored both tools and added four questions to the RTI-IB. Two reviewers assessed the quality of the 44 included studies with both tools, (independently for the RTI-IB) and agreed on which responses conveyed low, unclear, or high risk of bias. For each question in the RTI-IB (n=31), the observed interrater agreement was calculated as the percentage of studies given the same bias assessment by both reviewers; chance-adjusted interrater agreement was estimated with the first-order agreement coefficient (AC1) statistic. RESULTS: The NOS required less tailoring and was easier to use than the RTI-IB, but the RTI-IB produced a more thorough assessment. The RTI-IB includes most of the domains measured in the NOS. Median observed interrater agreement for the RTI-IB was 75% (25th percentile [p25] =61%; p75 =89%); median AC1 statistic was 0.64 (p25 =0.51; p75 =0.86). CONCLUSION: The RTI-IB facilitates a more complete quality assessment than the NOS but is more burdensome. The observed agreement and AC1 statistic in this study were higher than those reported by the RTI-IB's developers.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
The Cochrane Collaboration's tool for assessing risk of bias in randomised trials
Higgins, J. P. T., Altman, D. G., Gotzsche, P. C., Juni, P., Moher, D., Oxman, A. D., Savovic, J., Schulz, K. F., Weeks, L., Sterne, J. A. C., et al. (2011) · BMJ · doi:10.1136/bmj.d5928 · Cited by 33,817
Dermyshi, E., Wang, Y., Yan, C., Hong, W., Qiu, G., Gong, X., & Zhang, T. (2017) · Neonatology · doi:10.1159/000454668 · Cited by 24,451
(2014) · Medical Entomology and Zoology · Cited by 17,633
Stang, A. (2010) · European Journal of Epidemiology · doi:10.1007/s10654-010-9491-z · Cited by 17,396
(2005) · PubMed · Cited by 7,011
Evaluating non-randomised intervention studies
Deeks, J., Dinnes, J., D'Amico, R., Sowden, A., Sakarovitch, C., Song, F., Petticrew, M., & Altman, D. (2003) · Health Technology Assessment · doi:10.3310/hta7270 · Cited by 2,907
GRADE guidelines: 4. Rating the quality of evidence—study limitations (risk of bias)
Guyatt, G. H., Oxman, A. D., Vist, G., Kunz, R., Brozek, J., Alonso-Coello, P., Montori, V., Akl, E. A., Djulbegovic, B., & Falck-Ytter, Y. (2011) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2010.07.017 · Cited by 2,798
Computing inter‐rater reliability and its variance in the presence of high agreement
Gwet, K. L. (2008) · British Journal of Mathematical and Statistical Psychology · doi:10.1348/000711006x126600 · Cited by 2,167
Pharmaceutical industry sponsorship and research outcome and quality: systematic review
Lexchin, J., Bero, L. A., Djulbegovic, B., & Clark, O. (2003) · BMJ · doi:10.1136/bmj.326.7400.1167 · Cited by 2,147 · cited 2×
“37 Although it has been noted that industry-sponsored studies are sometimes biased toward the product manufactured by the sponsor, 39–43 there are voices favoring 44 but also opposing 45 the addition of such a question into the Cochrane risk of bias tool.”
Industry sponsorship and research outcome
Lundh, A., Lexchin, J., Mintzes, B., Schroll, J. B., & Bero, L. (2017) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.mr000033.pub3 · Cited by 1,697
Sanderson, S., Tatt, I. D., & Higgins, J. P. (2007) · International Journal of Epidemiology · doi:10.1093/ije/dym018 · Cited by 1,531
Wongpakaran, N., Wongpakaran, T., Wedding, D., & Gwet, K. L. (2013) · BMC Medical Research Methodology · doi:10.1186/1471-2288-13-61 · Cited by 1,041
Industry sponsorship and research outcome
Lundh, A., Sismondo, S., Lexchin, J., Busuioc, O. A., & Bero, L. (2012) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.mr000033.pub2 · Cited by 605
Immortal time bias in observational studies of drug effects
Suissa, S. (2007) · Pharmacoepidemiology and Drug Safety · doi:10.1002/pds.1357 · Cited by 584
Testing the Newcastle Ottawa Scale showed low reliability between individual reviewers
Hartling, L., Milne, A., Hamm, M. P., Vandermeer, B., Ansari, M., Tsertsvadze, A., & Dryden, D. M. (2013) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2013.03.003 · Cited by 436
Mustafa, R. A., Santesso, N., Brozek, J., Akl, E. A., Walter, S. D., Norman, G., Kulasegaram, M., Christensen, R., Guyatt, G. H., Falck-Ytter, Y., et al. (2013) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2013.02.004 · Cited by 370
Greenland, S. (2001) · Biostatistics · doi:10.1093/biostatistics/2.4.463 · Cited by 359 · cited 1×
“Also, it has been argued that quality summary scores may mask variations in quality by domain and use an unclear and often implicit weighting scheme.(3,5,13,36) The interrater reliability or agreement of the NOS has been previously evaluated.”
Sunkara, S. K., Khairy, M., El-Toukhy, T., Khalaf, Y., & Coomarasamy, A. (2010) · Human Reproduction · doi:10.1093/humrep/dep396 · Cited by 335
Individual NSAIDs and Upper Gastrointestinal Complications
Castellsagué, J., Riera‐Guàrdia, N., Calingaert, B., Varas‐Lorenzo, C., Fourrier‐Réglat, A., Nicotra, F., Sturkenboom, M., & Perez‐Gutthann, S. (2012) · Drug Safety · doi:10.2165/11633470-000000000-00000 · Cited by 309
Self-citationPharmaceutical company funding and its consequences: A qualitative systematic review
Sismondo, S. (2008) · Contemporary Clinical Trials · doi:10.1016/j.cct.2007.08.001 · Cited by 304
Shapiro, S. (1997) · PubMed · Cited by 299
(2013) · doi:10.1016/j.jclinepi.2011.05.008 · Cited by 264
Self-citationSystematic review with meta-analysis: cytokines in fibromyalgia syndrome
Üçeyler, N., Häuser, W., & Sommer, C. (2011) · BMC Musculoskeletal Disorders · doi:10.1186/1471-2474-12-245 · Cited by 260
Oremus, M., Oremus, C., Hall, G. B. C., McKinnon, M. C., & ECT & Cognition Systematic Review Team (2012) · BMJ Open · doi:10.1136/bmjopen-2012-001368 · Cited by 206
(2013) · Europe PMC (PubMed Central) · Cited by 200
Walker, A. M. (1996) · PubMed · Cited by 167
Industry sponsorship and research outcome
Lundh, A., Lexchin, J., Sismondo, S., Busuioc, O. A., & Bero, L. (2011) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.mr000033 · Cited by 166
Validity and Inter-Rater Reliability Testing of Quality Assessment Instruments
Hartling, L., Hamm, M. P., Milne, A., Vandermeer, B., Santaguida, P. L., Ansari, M., Tsertsvadze, A., Hempel, S., Shekelle, P., & Dryden, D. M. (2012) · Europe PMC (PubMed Central) · Cited by 158
Varas‐Lorenzo, C., Riera‐Guàrdia, N., Calingaert, B., Castellsagué, J., Salvo, F., Nicotra, F., Sturkenboom, M., & Perez‐Gutthann, S. (2013) · Pharmacoepidemiology and Drug Safety · doi:10.1002/pds.3437 · Cited by 147
Self-citationThe Financing of Drug Trials by Pharmaceutical Companies and Its Consequences
Schott, G., Pachl, H., Limbach, U., Gundert‐Remy, U., Ludwig, W.-D., & Lieb, K. (2010) · Deutsches Ärzteblatt international · doi:10.3238/arztebl.2010.0279 · Cited by 117
Why the Cochrane Risk of Bias Tool Should Include Funding Source as a Standard Item
Bero, L. A. (2013) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.ed000075 · Cited by 98
Stroke risk and NSAIDs: a systematic review of observational studies
Varas‐Lorenzo, C., Riera‐Guàrdia, N., Calingaert, B., Castellsagué, J., Pariente, A., Scotti, L., Sturkenboom, M., & Perez‐Gutthann, S. (2011) · Pharmacoepidemiology and Drug Safety · doi:10.1002/pds.2227 · Cited by 92
Self-citationPEDro's bias: summary quality scores should not be used in meta-analysis
da Costa, B. R., Hilfiker, R., & Egger, M. (2013) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2012.08.003 · Cited by 86
Golder, S., & Loke, Y. K. (2008) · British Journal of Clinical Pharmacology · doi:10.1111/j.1365-2125.2008.03272.x · Cited by 72
With Great Data Comes Great Responsibility
Hernán, M. Á. (2011) · Epidemiology · doi:10.1097/ede.0b013e3182114039 · Cited by 68
Why the Cochrane Risk of Bias Tool Should not Include Funding Source as a Standard Item
Sterne, J. A. (2013) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.ed000076 · Cited by 60
Castellsague, J., Riera‐Guàrdia, N., Calingaert, B., Varas-lorenzo, C., Fourrier-reglat, A., Nicotra, F., Sturkenboom, M., & Perez-gutthann, S. (n.d.) · Cited by 44
Self-citationImpact of H1N1 on Socially Disadvantaged Populations: Systematic Review
Tricco, A. C., Lillie, E., Soobiah, C., Perrier, L., & Straus, S. E. (2012) · PLoS ONE · doi:10.1371/journal.pone.0039437 · Cited by 34
Disagreement on Agreement: Two Alternative Agreement Coefficients
Spratt, K. F. (2007) · Cited by 30
Al-Saleh, M. A. Q., Armijo-Olivo, S., Thie, N. M. R., Seikaly, H. R., Boulanger, P., Wolfaardt, J., & Major, P. (2012) · PubMed · Cited by 25
Review of quality assessment tools for the evaluation of pharmacoepidemiological safety studies
Neyarapally, G. A., Hammad, T. A., Pinheiro, S. P., & Iyasu, S. (2012) · BMJ Open · doi:10.1136/bmjopen-2012-001362 · Cited by 20
Ruhe, A., Fejer, R., & Walker, B. F. (2013) · Chiropractic & Manual Therapies · doi:10.1186/2045-709x-21-9 · Cited by 20
Al-Harbi, K., Farrokhyar, F., Mulla, S. M., & Fitzgerald, P. (2009) · Journal of Pediatric Surgery · doi:10.1016/j.jpedsurg.2009.01.048 · Cited by 20
Cited by
Emami, A., Javanmardi, F., Pirbonyeh, N., & Akbari, A. (2020) · Archives of Academic Emergency Medicine · doi:10.22037/aaem.v8i1.600 · Cited by 765
Flouris, A. D., Dinas, P. C., Ioannou, L. G., Nybo, L., Havenith, G., Kenny, G. P., & Kjellstrom, T. (2018) · The Lancet Planetary Health · doi:10.1016/s2542-5196(18)30237-7 · Cited by 579
Prevalence of comorbidities among individuals with COVID-19: A rapid review of current literature
Bajgain, K. T., Badal, S., Bajgain, B. B., & Santana, M. J. (2021) · American Journal of Infection Control · doi:10.1016/j.ajic.2020.06.213 · Cited by 328
Farrah, K., Young, K., Tunis, M. C., & Zhao, L. (2019) · Systematic Reviews · doi:10.1186/s13643-019-1172-8 · Cited by 246
The Impact of Obesity on Thyroid Autoimmunity and Dysfunction: A Systematic Review and Meta-Analysis
Song, R.-H., Wang, B., Yao, Q.-M., Li, Q., Jia, X., & Zhang, J.-A. (2019) · Frontiers in Immunology · doi:10.3389/fimmu.2019.02349 · Cited by 257
Metadata
- Authors
- Margulis, A. V., Pladevall, M., Riera-Guardia, N., Varas-Lorenzo, C., Hazell, L., Berkman, N., Viswanathan, M., & Perez-Gutthann, S.
- Venue
- Clinical Epidemiology
- Publisher
- Informa UK Limited
- Published
- 2014-10
- Volume
- 6
- Pages
- 359
- ISSN
- 1179-1349 (ISSN-L)
- OpenAlex
- W2024530015
- Cited by
- 441
- References
- 43
- FWCI
- 4.27
- Type
- journal article
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.