The inclusion of reports of randomised trials published in languages other than English in systematic reviews
Open accessThe DOI matches its Crossref record; 235 of the 236 references it cites were found in a registry or an index.
Abstract
OBJECTIVE: To assemble a large dataset of language restricted and language inclusive systematic reviews, including both conventional medicinal (CM) and complementary and alternative medicine (CAM) interventions. To then assess the quality of these reports by considering and comparing different types of systematic reviews and their associated RCTs; CM and CAM interventions; the effect of language restrictions compared with language inclusions, and whether these results are influenced by other issues, including statistical heterogeneity and publication bias, in the systematic review process. DATA SOURCES: MEDLINE, EMBASE, the Cochrane Database of Systematic Reviews and the Centralised Information Service for Complementary Medicine. REVIEW METHODS: Three types of systematic reviews were included: language restricted; language inclusive/English language (EL) reviews that searched RCTs in languages other than English (LOE) but did not find any and, hence, could not include any, in the quantitative data synthesis; and systematic reviews that searched for RCTs in LOE and included them in the quantitative data synthesis. Fisher's exact test was applied to compare the three different types of systematic reviews with respect to their reporting characteristics and the systematic review quality assessment tool. The odds ratio of LOE trials versus EL trials was computed for each review and this information was pooled across the reviews to examine the influence that language of publication and type of intervention (CM, CAM) have on the estimates of intervention effect. Several sensitivity analyses were performed. RESULTS: The LOE RCTs were predominantly in French and German. Language inclusive/LOE systematic reviews were of the highest quality compared with the other types of reviews. The CAM reviews were of higher quality compared with the CM reviews. There were only minor differences in the quality of reports of EL RCTs compared with the eight other languages considered. However, there are inconsistent differences in the quality of LOE reports depending on the intervention type. The results, and those reported previously, suggest that excluding reports of RCTs in LOE from the analytical part of a systematic review is reasonable. Because the present research and previous efforts have not included every type of CM RCT and the resulting possibility of the uncertainty as to when bias will be present by excluding LOE, it is always prudent to perform a comprehensive search for all evidence. This result only applies to reviews investigating the benefits of CM interventions. This does not imply that systematic reviewers should neglect reports in LOE. We recommend that systematic reviewers search for reports regardless of the language. There may be merit in including them in some aspects of the review process although this decision is likely to depend on several factors, including fiscal and other resources being available. Language restrictions significantly shift the estimates of an intervention's effectiveness when the intervention is CAM. Here, excluding trials reported in LOE, compared with their inclusion, resulted in a reduced intervention effect. The present results do not appear to be influenced by statistical heterogeneity and publication bias. CONCLUSIONS: With the exception of CAM systematic reviews, the quality of recently published systematic reviews is less than optimal. Language inclusive/LOE systematic reviews appear to be a marker for a better quality systematic review. Language restrictions do not appear to bias the estimates of a conventional intervention's effectiveness. However, there is substantial bias in the results of a CAM systematic review if LOE reports are excluded from it.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“There is mixed evidence for the potential impact of not including non-English-language trials: one study found that non-English trials were more likely to produce significant results at p <0.05 (Jüni et al., 2002), whereas an earlier review concluded that excluding non-English trials did not significantly affect the results of meta-analyses (Moher et al., 2003).”
From Self-help interventions to reduce self-stigma in people with mental health problems: A systematic literature review (2020) · cites this paper as Moher et al., 2003
In the citing paper
“In addition, it has been reported that language-restricted meta-analyses only minimally overestimate treatment effects (ϳ2% on average) compared with language-inclusive meta-analyses.112”
From Effectiveness of Manual Therapy and Therapeutic Exercise for Temporomandibular Disorders: Systematic Review and Meta-Analysis (2016) · cites this paper as [112]
In the citing paper
“Therefore, language-restricted metaanalyses do not appear to lead to biased estimates of intervention effectiveness.112,113”
From Effectiveness of Manual Therapy and Therapeutic Exercise for Temporomandibular Disorders: Systematic Review and Meta-Analysis (2016) · cites this paper as [112]
In the citing paper
“Over the last two decades, empirical evidence has accumulated from studies investigating different sources of bias related to specific SR tasks.”
From How to conduct systematic reviews more expeditiously? (2015) · cites this paper as [31]
In the citing paper
“Some authors suggested that the impact of excluding non-English language studies may depend on the topic of the review and the quality of non-English language studies [29, 31, 32].”
From How to conduct systematic reviews more expeditiously? (2015) · cites this paper as [31]
2 more claims about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
3 retracted
Daya, S., Maheshwari, A., Siristatidis, C. S., Bhattacharya, S., & Gibreel, A. (2000) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd001299 · Cited by 114
RetractedRice-based oral rehydration solution for treating diarrhoea
Fontaine, O., Gore, S., & Pierce, N. (1998) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd001264 · Cited by 79
RetractedAndrogens versus placebo or no treatment for idiopathic oligo/asthenospermia
Vandekerckhove, P., Lilford, R. J., Vail, A., & Hughes, E. G. (1996) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd000150 · Cited by 30
RetractedThe Measurement of Observer Agreement for Categorical Data
Landis, J. R., & Koch, G. G. (1977) · Biometrics · doi:10.2307/2529310 · Cited by 81,148
Bias in meta-analysis detected by a simple, graphical test
Egger, M., Smith, G. D., Schneider, M., & Minder, C. (1997) · BMJ · doi:10.1136/bmj.315.7109.629 · Cited by 56,722
Meta-analysis in clinical trials
DerSimonian, R., & Laird, N. (1986) · Controlled Clinical Trials · doi:10.1016/0197-2456(86)90046-2 · Cited by 39,552
Quantifying heterogeneity in a meta‐analysis
Higgins, J. P. T., & Thompson, S. G. (2002) · Statistics in Medicine · doi:10.1002/sim.1186 · Cited by 36,763
Intraclass correlations: Uses in assessing rater reliability.
Shrout, P. E., & Fleiss, J. L. (1979) · Psychological Bulletin · doi:10.1037/0033-2909.86.2.420 · Cited by 24,532
Assessing the quality of reports of randomized clinical trials: Is blinding necessary?
Jadad, A. R., Moore, R. A., Carroll, D., Jenkinson, C., Reynolds, D. J. M., Gavaghan, D. J., & McQuay, H. J. (1996) · Controlled Clinical Trials · doi:10.1016/0197-2456(95)00134-4 · Cited by 17,582
Eisenberg, D. M., Davis, R. B., Ettner, S. L., Appel, S., Wilkey, S., Van Rompay, M., & Kessler, R. C. (1998) · JAMA · doi:10.1001/jama.280.18.1569 · Cited by 6,658
Cited by
Items 301–400 of 499
Liang, S.-C., Sun, C.-K., Chang, C.-H., Cheng, Y.-S., Tzang, R.-F., Chiu, H.-J., Wang, M. Y., Cheng, Y.-C., & Hung, K.-C. (2024) · BJPsych Open · doi:10.1192/bjo.2023.645 · Cited by 20
“…previous evidence showing a significant decrease in the effect size of meta-analyses focusing on complementary and alternative medicine after excluding reports in languages other than English, 46 our results may underestimate the effects of probiotics because of the possibility of language bias.”
Goldgrub, R., Côté, P., Sutton, D. A., Wong, J. J., Yu, H., Randhawa, K. A., Varatharajan, S., Southerst, D., Mior, S. A., Shearer, H. M., et al. (2016) · Journal of Manipulative and Physiological Therapeutics · doi:10.1016/j.jmpt.2016.01.002 · Cited by 23
Ng, J. H., Song, Y. L., & Yap, A. U. (2019) · Journal of Oral Rehabilitation · doi:10.1111/joor.12854 · Cited by 21
Systemic Immunosuppression in High-Risk Penetrating Keratoplasty: A Systematic Review
Bali, S. J., Filek, R., Si, F. F., & Hodge, W. (2016) · Journal of Clinical Medicine Research · doi:10.14740/jocmr2326w · Cited by 19
“recommended as being suitable for use in systematic reviews [23, 24].”
BackgroundDiriba, K., Awulachew, E., & Diribsa, K. (2021) · European journal of medical research · doi:10.1186/s40001-021-00532-8 · Cited by 20
Chronic Kidney Disease Stages 1–3
Ha, F., Ishani, A., Taylor, B. C., Nl, G., MacDonald, R. H., Rossini, D., Sadiq, S., Lankireddy, S., Kane, R. L., & Wilt, T. J. (2012) · Europe PMC (PubMed Central) · Cited by 18
Woitzik, E., Jacobs, C. L., Wong, J. J., Côté, P., Shearer, H. M., Randhawa, K. A., Sutton, D. A., Southerst, D., Varatharajan, S., Brison, R. J., et al. (2015) · Manual Therapy · doi:10.1016/j.math.2015.03.012 · Cited by 23
“However, previous reviews studying the impact of restriction to the English language did not find that it led to bias within reported results [65-69].”
ResultHasan, H., Muhammed, T., Yu, J., Taguchi, K., Samargandi, O. A., Howard, A. F., Lo, A., Olson, R. A., & Goddard, K. (2017) · Cancer Epidemiology · doi:10.1016/j.canep.2017.08.013 · Cited by 18
Strategies to improve the credibility of meta-analyses in spine surgery: a systematic survey
Evaniew, N., van der Watt, L., Bhandari, M., Ghert, M. A., Aleem, I. S., Drew, B. M., & Guyatt, G. H. (2015) · The Spine Journal · doi:10.1016/j.spinee.2015.05.018 · Cited by 19
“50), each of increasing publication year, increasing Journal Impact Factor, increasing number of databases, inclusion of RCTs, and inclusion of non-English studies were significantly associated with a higher solute counts of meta-analyses published from each continent and are not completeness of reporting, and industr…”
BackgroundMethodInfluentialHoshijima, H., Kuratani, N., Takeuchi, R., Shiga, T., Masaki, E., Doi, K., & Matsumoto, N. (2013) · Journal of Dental Sciences · doi:10.1016/j.jds.2012.11.004 · Cited by 21
Dillon, M. P., Quigley, M., & Fatone, S. (2017) · Systematic Reviews · doi:10.1186/s13643-017-0626-0 · Cited by 20
“All searches were limited to articles written in English given that such language restriction does not alter the outcome of systematic reviews and meta-analyses [18, 19].”
BackgroundLong, A., Xing, M., Morgan, K., & Brettle, A. (2009) · Evidence-based Complementary and Alternative Medicine · doi:10.1093/ecam/nep047 · Cited by 17
“[38] argued, substantial bias may occur in the results of a CAM systematic review if languages other than English are excluded.”
BackgroundYu, H., Côté, P., Wong, J. J., Shearer, H. M., Mior, S. A., Cancelliere, C., Randhawa, K. A., Ameis, A., Carroll, L., Nordin, M. C., et al. (2021) · European Journal of Pain · doi:10.1002/ejp.1788 · Cited by 17
Rogowski, W. H., Burch, J., Palmer, S. C., Craigs, C., Golder, S., & Woolacott, N. F. (2009) · Health Technology Assessment · doi:10.3310/hta13310 · Cited by 15
Hutton, B., Tetzlaff, J., Yazdi, F., Thielman, J., Kanji, S., Fergusson, D. A., Bjerre, L. M., Mills, E. J., Thorlund, K., Tricco, A. C., et al. (2013) · Systematic Reviews · doi:10.1186/2046-4053-2-44 · Cited by 19
“guage does not greatly affect summary estimates from meta-analyses [14,15].”
BackgroundSelf-citationDion, S., Wong, J. J., Côté, P., Yu, H., Sutton, D. A., Randhawa, K. A., Southerst, D., Varatharajan, S., Stern, P. J., Nordin, M. C., et al. (2016) · Clinical Journal of Pain · doi:10.1097/ajp.0000000000000368 · Cited by 18
Komplementäre und alternative Verfahren beim Fibromyalgiesyndrom
Langhorst, J., Häuser, W., Irnich, D., Speeck, N., Felde, E., Winkelmann, A., Lucius, H., Michalsen, A., & Musial, F. (2008) · Der Schmerz · doi:10.1007/s00482-008-0677-1 · Cited by 15
Science and art in reviewing literature
Griffiths, P., & Norman, I. J. (2005) · International Journal of Nursing Studies · doi:10.1016/j.ijnurstu.2005.02.001 · Cited by 18
Soria, C., Almirón, W. R., Ibarra, A. M. S., & Crocco, L. (2024) · American Journal of Tropical Medicine and Hygiene · doi:10.4269/ajtmh.23-0427 · Cited by 28
Use of antimicrobials in the treatment of calf diarrhea: a systematic review
Bernal-Córdoba, C., Lopes, R. B., Latorre-Segura, L., de Barros-Abreu, M., Fausak, E. D., & Silva-del-Río, N. (2022) · Animal Health Research Reviews · doi:10.1017/s1466252322000032 · Cited by 26
“AMs) does not change the estimates of the effectiveness of an intervention (Moher et al., 2003; Pham et al., 2005).”
BackgroundAlemnew, B. T. T., Gedefaw, G. A., Diress, G., & Demis, A. (2020) · BMC Infectious Diseases · doi:10.1186/s12879-020-4884-4 · Cited by 16
Biologics in professional and Olympic sport: a scoping review
Murray, I. R., Makaram, N. S., Rodeo, S. A., Safran, M. R., Sherman, S. L., McAdams, T. R., Murray, A. D., Haddad, F. S., & Abrams, G. D. (2021) · The Bone & Joint Journal · doi:10.1302/0301-620x.103b7.bjj-2020-2282.r1 · Cited by 16
The effects of biofeedback for the treatment of essential hypertension: a systematic review
Greenhalgh, J., Dickson, R., & Dundar, Y. (2009) · Health Technology Assessment · doi:10.3310/hta13460 · Cited by 15
Naderi, A., Shaabani, F., Keikha, M., & Degens, H. (2023) · Journal of Athletic Training · doi:10.4085/1062-6050-0680.22 · Cited by 20
D’Angelo, K., Sutton, D. A., Côté, P., Dion, S., Wong, J. J., Yu, H., Randhawa, K. A., Southerst, D., Varatharajan, S., Cox, J., et al. (2015) · Journal of Manipulative and Physiological Therapeutics · doi:10.1016/j.jmpt.2015.06.006 · Cited by 17
Methodological challenges when performing a systematic review
Anderson, N. K., & Jayaratne, Y. S. N. (2015) · European Journal of Orthodontics · doi:10.1093/ejo/cjv022 · Cited by 16
“Overall, there appears to be no evidence that non-English language articles are of lower quality than English only (12), however, the costs associated with translations may be prohibitive for most teams embarking on a systematic review.”
BackgroundMu, X., Peng, W., Ou, Y., Li, P., Li, Z., & Wei, J. (2022) · Heliyon · doi:10.1016/j.heliyon.2022.e09658 · Cited by 14
“Although the exclusion of non-English articles does not usually have a significant impact on the results of systematic reviews [57], we were unable to determine this definitively.”
Backgroundde Coo, A., Quintela, I., Blanco, J., Dios, P. D., & Carracedo, Á. (2018) · Archives of Oral Biology · doi:10.1016/j.archoralbio.2018.04.012 · Cited by 15
Gingivitis and oral health-related quality of life: a literature review
Barbosa, T. D. S., Gavião, M. B. D., & Mialhe, F. L. (2015) · Brazilian Dental Science · doi:10.14295/bds.2015.v18i1.1013 · Cited by 12
Sørensen, J. R., Bonnema, S. J., Godballe, C., & Hegedűs, L. D. (2018) · Frontiers in Endocrinology · doi:10.3389/fendo.2018.00679 · Cited by 13
Shah, S., Yeheskel, A., Hossain, A. K. M. M., Kerr, J., Young, K., Shakik, S., Nichols, J., & Yu, C. H. Y. (2021) · The American Journal of Medicine · doi:10.1016/j.amjmed.2021.03.004 · Cited by 13
Alborz, A., & McNally, R. (2004) · Health Information & Libraries Journal · doi:10.1111/j.1471-1842.2004.00543.x · Cited by 16
Acute stroke: evaluation and treatment.
Sharma, M. K., Clark, H. D., Armour, T., Stotts, G., Côté, R., Hill, M. D., Demchuck, A. M., Moher, D., Garritty, C., Yazdi, F., et al. (2005) · PubMed · Cited by 12
Self-citationFord, J. J., Bower, S. E., Ford, I., de Mello, M. M., Carneiro, S. R., Balasundaram, A. P., & Hahne, A. J. (2020) · Musculoskeletal Science and Practice · doi:10.1016/j.msksp.2020.102276 · Cited by 12
Language bias in orthodontic systematic reviews: A meta-epidemiological study
Mheissen, S., Spineli, L. M., Daraqel, B., & Alsafadi, A. S. (2024) · PLoS ONE · doi:10.1371/journal.pone.0300881 · Cited by 14
McLean, G., Murray, E., Band, R. J., Saunderson, K., Hanlon, P., Little, P. S., McManus, R. J., Yardley, L., & Mair, F. S. (2015) · JMIR Research Protocols · doi:10.2196/resprot.4648 · Cited by 13
“Inclusion criteria, based on participants, interventions, comparisons, outcomes, and study design (PICOS acronym) [35] include (1) adult population (aged ≥ 18 years) with hypertension (as defined by the primary authors), (2) an interactive digital intervention (as defined earlier), (3) a comparator of usual care, (4)…”
BackgroundNoelck, N. J., Papak, J., Freeman, M., Paynter, R. A., Low, A., Motu’apuaka, M. L., Kondo, K. K., & Kansagara, D. L. (2016) · Circulation Cardiovascular Quality and Outcomes · doi:10.1161/circoutcomes.115.002539 · Cited by 12
“There is empirical data, however, suggesting that reviews restricted to English language studies are largely concordant with reviews without language restrictions.(42) Second, we excluded observational studies enrolling fewer than 50 participants.”
ResultGates, M., Pillay, J., Thériault, G., Limburg, H., Grad, R. M., Klarenbach, S. W., Korownyk, C. S., Reynolds, D. L., Riva, J. J., Thombs, B. D., et al. (2019) · Systematic Reviews · doi:10.1186/s13643-019-1094-5 · Cited by 13
“to be biased by such restrictions [123, 124].”
Zhan, T., Xing, W. E. I., Chen, Z., Wang, D., & Dai, X. (2011) · 中医杂志:英文版 · doi:10.1016/s0254-6272(12)60006-5 · Cited by 11
Mao, C., & Li, M. (2020) · JAMA Network Open · doi:10.1001/jamanetworkopen.2020.6370 · Cited by 10
Akalu, T. Y., Aynalem, Y. A., Shiferaw, W. S., Alamneh, Y. M., Getnet, A., Abebaw, A., Atnaf, A., Abate, A., Tilahun, M., Kassie, B., et al. (2022) · SAGE Open Medicine · doi:10.1177/20503121221082447 · Cited by 14
Frank, R. A., Salameh, J., Islam, N., Yang, B., Murad, M. H., Mustafa, R. A., Leeflang, M., Bossuyt, P. M., Takwoingi, Y., Whiting, P. F., et al. (2023) · Radiology · doi:10.1148/radiol.221437 · Cited by 12
Discepoli, N., Mirra, R., & Ferrari, M. (2019) · Materials · doi:10.3390/ma12172790 · Cited by 11
“However, recent reviews on the analysis of the aftermath of language restriction do not appear to bias the estimates of the intervention’s effectiveness [48,49].”
BackgroundSystematic Review of the Quality of Prognosis Studies in Systemic Lupus Erythematosus
Lim, L. S. H., Lee, S., Feldman, B. M., Gladman, D. D., Pullenayegum, E., Uleryk, E. M., & Silverman, E. D. (2014) · Arthritis Care & Research · doi:10.1002/acr.22322 · Cited by 10
“However, a systematic review of randomized controlled trials reported that the language of publication did not affect significantly the quality of studies (36).”
BackgroundVerville, L., Côté, P., Grondin, D. E., Mior, S. A., Moodley, K., Kay, R. H., & Taylor-Vaisey, A. L. (2020) · Journal of Chiropractic Education · doi:10.7899/jce-19-17 · Cited by 17
Goss, C. W., Lowenstein, S. R., Roberts, I., & DiGuiseppi, C. G. (2007) · Journal of Information Science · doi:10.1177/0165551506068376 · Cited by 10
“Concerns about language bias [17, 18] and exclusion of journals from certain countries/languages in the biomedical databases MEDLINE and EMBASE [19] suggest that other sources should be consulted when including nonEnglish-language studies.”
BackgroundThe Use of Remote Presence Robotic Tele-Presentation in Rural and Remote Canada: A Systematic Review
Russell, K. F., Sandron, E., Normand, H., Ellis, M. J., Durcan, A., Mendez, I. M., Johnson, R., & Wittmeier, K. D. M. (2024) · Telemedicine Journal and e-Health · doi:10.1089/tmj.2024.0293 · Cited by 10
“39 While three studies commented on the cultural benefits of patients staying in their home community, 15,16,21 no study examined the cultural relevance or cultural safety of the robots.”
Quality of randomized clinical trials in juvenile idiopathic arthritis
Abrahamyan, L., Johnson, S. R., Beyene, J., Shah, P. S., & Feldman, B. M. (2008) · Rheumatology · doi:10.1093/rheumatology/kem366 · Cited by 11
Kaltenegger, H. C., Becker, L., Rohleder, N., Nowak, D., & Weigl, M. (2020) · Systematic Reviews · doi:10.1186/s13643-020-01463-x · Cited by 18
Photoplethysmography for the assessment of peripheral vascular disease
Allen, J. (2021) · Elsevier eBooks · doi:10.1016/b978-0-12-823374-0.00005-0 · Cited by 11
Vasopressor and inotrope treatment for septic shock: An umbrella review of reviews
Einav, S., Helviz, Y., Ippolito, M., & Cortegiani, A. (2021) · Journal of Critical Care · doi:10.1016/j.jcrc.2021.05.017 · Cited by 10
Gholami, F., António, J., Iranpour, M., Curtis, J., & Pereira, F. (2024) · Journal of the International Society of Sports Nutrition · doi:10.1080/15502783.2024.2411029 · Cited by 13
“The inclusion of English-language studies may introduce the possibility of publication bias; however, the exclusion of non-English articles usually does not have a significant effect on the results of systematic reviews [51].”
BackgroundA systematic review of shock-attenuating componentry for lower limb amputees
Farrar, M. J., & Thomas, E. E. (2018) · Prosthetics and Orthotics International · doi:10.1177/0309364618774062 · Cited by 8
Lemeunier, N., Suri-Chilana, M., Welsh, P., Shearer, H. M., Nordin, M. C., Wong, J. J., Torres, P., da Silva-Oolup, S., D’Silva, C., Jeoun, E. B., et al. (2019) · European Journal of Physiotherapy · doi:10.1080/21679169.2019.1626906 · Cited by 8
Rapid review: A review of methods and recommendations based on current evidence
Guo, Q., Jiang, G., Zhao, Q., Long, Y., Feng, K., Gu, X., Xu, Y., Li, Z., Huang, J., & Du, L. (2024) · Journal of Evidence-Based Medicine · doi:10.1111/jebm.12594 · Cited by 8
Nicol, R., Yu, H., Selb, M., Prodinger, B., Hartvigsen, J., & Côté, P. (2020) · American Journal of Physical Medicine & Rehabilitation · doi:10.1097/phm.0000000000001636 · Cited by 11
Spaces for Transitions in Intergenerational Childhood Experiences
Oropilla, C. T. (2020) · BRILL eBooks · doi:10.1163/9789004445666_005 · Cited by 8
Vitzthum, K., Spallek, M., Mache, S., Quarcoo, D., Scutaru, C., Groneberg, D. A., & Schöffel, N. (2009) · European Journal of Orthopaedic Surgery & Traumatology · doi:10.1007/s00590-009-0556-1 · Cited by 8
“This leads to the assumption that non-English studies such as French articles have problems to entering into the scientific discourse and being cited [15].”
BackgroundDillon, M. P., Fatone, S., & Quigley, M. (2015) · Systematic Reviews · doi:10.1186/s13643-015-0161-9 · Cited by 8
“Given that the authors are native English speakers and that restriction of non-English language articles does not seem to alter the outcome of systematic reviews and meta-analyses [54, 55], all searchers will be limited to the English language.”
BackgroundAcupuncture Treatment for Parkinson’s Disease
Asakawa, T., & Xia, Y. (2012) · Springer eBooks · doi:10.1007/978-1-4614-3357-6_7 · Cited by 8
Alamneh, Y. M., Getachew, M., Atnaf, A., & Abebaw, A. (2022) · SAGE Open Medicine · doi:10.1177/20503121221099019 · Cited by 11
Hugues, A., Marco, J. D., Bonan, I., Rode, G., Cucherat, M., & Gueyffier, F. (2020) · PLoS ONE · doi:10.1371/journal.pone.0229822 · Cited by 7
“methodological quality of SPNEL compared to SPEL [16,28,34].”
ResultSkakum, M., Katako, A., Mitchell-Dueck, J., Ricci, M. F., & Russell, K. F. (2023) · Paediatrics & Child Health · doi:10.1093/pch/pxad074 · Cited by 8
Randhawa, K. A., Côté, P., Gross, D. P., Wong, J. J., Yu, H., Sutton, D. A., Southerst, D., Varatharajan, S., Mior, S. A., Stupar, M., et al. (2015) · PubMed · Cited by 11
Evidence-Based Decision-Making 3: Health Technology Assessment
O’Reilly, D. J., Campbell, K., Vanstone, M. G., Bowen, J. M., Schwartz, L. J., Assasi, N., & Goeree, R. A. (2015) · Methods in molecular biology · doi:10.1007/978-1-4939-2428-8_25 · Cited by 7
Mpundu‐Kaambwa, C., Chen, G., Huynh, E. K., Russo, R. N., & Ratcliffe, J. (2017) · BMJ Open · doi:10.1136/bmjopen-2017-015924 · Cited by 8
Cathey, J. T., Hajeri, A. A., & Fedorowicz, Z. (2006) · Annals of Saudi Medicine · doi:10.5144/0256-4947.2006.49 · Cited by 7
Tricco, A. C., Tetzlaff, J., & Moher, D. (2013) · Wiley eBooks · doi:10.1002/9781118413555.ch04 · Cited by 10
Self-citationGholami, F., Naderi, A., Saeidpour, A., & Lefaucheur, J. P. (2024) · Primary care diabetes · doi:10.1016/j.pcd.2024.01.008 · Cited by 30
Gedfew, M., Getie, ፟., Akalu, T. Y., & Ayenew, T. (2024) · European Journal of Surgical Oncology · doi:10.1016/j.ejso.2024.108528 · Cited by 6
Sustainable Rice–Fish Farming Systems: A Systematic Review and Meta‐Analysis
Fernandez-Zarate, F. H., Pérez-Delgado, L. J., Coronel-Bustamante, D., Huanca-Silva, L., Taboada-Mitma, V. H., Quispe-Carhuapoma, M., Oliva-Alvarez, Y., Ramirez-Antaurco, M., Anchayhua, J., Huaccha-Castillo, A. E., et al. (2025) · Aquaculture Research · doi:10.1155/are/4029275 · Cited by 6
Practical applications of rapid review methods in the development of Australian health policy
Lambert, R., Vreugdenburg, T. D., Marlow, N. E., Scott, N. A., McGahan, L., & Tivey, D. R. (2016) · Australian Health Review · doi:10.1071/ah16041 · Cited by 8
Jackson, S. R., Hanna, B., Narroway, H. G., & Chung, A. S. J. (2021) · Neurourology and Urodynamics · doi:10.1002/nau.24710 · Cited by 7
“However, these literary forms are poorly accepted and/or cited due to inherent bias within literature, making the possibility of them significantly altering results low.(28,29) The citation metrics utilized in this study are only several of those widely used in literature—rankings provided may thus differ if applying…”
Essiet, E., Shapey, J., & Kailaya-Vasan, A. (2022) · British Journal of Neurosurgery · doi:10.1080/02688697.2022.2104217 · Cited by 6
“inclusion of other languages would not significantly affect the outcome.(21,22) Also, letters, commentaries, editorials, opinion”
BackgroundWong, J. J., DeSouza, A., Hogg‐Johnson, S., GROOTE, W. D., Varmazyar, H., Mior, S. A., Stern, P. J., Southerst, D., Alexopulos, S., Belchos, M., et al. (2022) · International Journal of Rehabilitation Research · doi:10.1097/mrr.0000000000000548 · Cited by 6
Sagittal alignment in operative degenerative lumbar spondylolisthesis: a scoping review
Thornley, P., Meade, M., Oitment, C. T. A., Fernandes, R. R., Urquhart, J. C., Singh, S., Siddiqi, F., Rasoulinejad, P., & Bailey, C. S. (2023) · Journal of Spine Surgery · doi:10.21037/jss-23-26 · Cited by 6
Development of a Guideline to Enhance the Reporting of Concept Mapping Research: Study Protocol
Pantha, S., Jones, M., & Gray, R. J. (2022) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph19127273 · Cited by 7
Review of Cochrane Reviews on Acupuncture: How Chinese Resources Contribute to Cochrane Reviews
Jiao, S., Tsutani, K., & Haga, N. (2013) · The Journal of Alternative and Complementary Medicine · doi:10.1089/acm.2012.0113 · Cited by 8
“publication bias.(4,6) Therefore, it is worth considering what impact Chinese resources have on current CSRs and how Chinese resources could better contribute to future CSRs.”
Vassou, C., D'Cunha, N. M., Naumovski, N., & Panagiotakos, D. B. (2020) · Journal of Diabetes & Metabolic Disorders · doi:10.1007/s40200-020-00551-y · Cited by 7
Can you trust systematic reviews of complementary and alternative therapies?
Linde, K. (2009) · European Journal of Integrative Medicine · doi:10.1016/j.eujim.2009.09.002 · Cited by 8
Screening for Hepatocellular Carcinoma in Chronic Liver Disease
Kansagara, D. L., Papak, J., & Jou, J. H. (2015) · Annals of Internal Medicine · doi:10.7326/l15-5049-3 · Cited by 6
Al Attar, W. S. A., Ghulam, H., Al Arifi, S., Alomar, A. I., Alhosaini, S., Alharbi, S., Alraddadi, Y., & Sanders, R. H. (2023) · Isokinetics and Exercise Science · doi:10.3233/ies-220084 · Cited by 5
“However, we did not find any non-English relevant studies, and language limits do not always affect or bias the findings of systematic reviews [56].”
Lu, X., Xia, W., Wang, X., Xie, F., & Sun, X. (2023) · Prehospital and Disaster Medicine · doi:10.1017/s1049023x23006039 · Cited by 5
Natsiou, G., & Tsitouridou, M. (2025) · Frontiers in Education · doi:10.3389/feduc.2025.1521473 · Cited by 8
Survival rate of atraumatic restorative treatment: A systematic review
Kotyal, M. H., Fareed, N., Battur, H., Khanagar, S. B., Manohar, B., & Basapathy, R. (2015) · Journal of Indian Association of Public Health Dentistry · doi:10.4103/2319-5932.171206 · Cited by 6
Yeh, E. J., Grigolon, R. B., Rodrigues, S. R., & Bueno, A. P. A. (2023) · Journal of Comparative Effectiveness Research · doi:10.57264/cer-2022-0085 · Cited by 9
“However, a publication suggested exclusion of languages other than English in SLR would not significantly affect the results of the meta-analyses [7]; and, available in full-text.”
BackgroundHow Perceptions Impact Recovery from Concussion in Childhood and Adolescence: a Systematic Review
Plourde, V., Kung, J. Y. C., Gates, A., Jun, S., Brooks, B. L., & Sebastianski, M. (2020) · Neuropsychology Review · doi:10.1007/s11065-020-09430-y · Cited by 8
The handsearching of 2 medical journals of Bahrain for reports of randomized controlled trials
Hajeri, A. A., Fedorowicz, Z., Amin, F. A., & Eisinga, A. (2006) · Saudi Medical Journal · doi:10.15537/1658-3175.3388 · Cited by 6
Use of Extracorporeal Shockwave Therapy for the Management of Bone Pathologies: A Systematic Review
Prisco, L. C., Zhao, E., Lynch, O. L., Hoo, J. S., Tenforde, A. S., & Goolsby, M. A. (2026) · Clinical Journal of Sport Medicine · doi:10.1097/jsm.0000000000001404 · Cited by 4
Li, Z., Wang, F., Kou, X., Couban, R., Busse, J. W., & Wang, L. (2025) · European Journal of Surgical Oncology · doi:10.1016/j.ejso.2025.110214 · Cited by 6
Feasibility of a Platform Trial Design for the Development of Mobile Health Applications: A Review
Aqib, A., Lebouché, B., Engler, K., & Schuster, T. (2022) · Telemedicine Journal and e-Health · doi:10.1089/tmj.2021.0620 · Cited by 4
Alpi, K. M., Vo, T. A., & Dorman, D. C. (2019) · International Journal of Toxicology · doi:10.1177/1091581819827232 · Cited by 4
lilacs search strategy for systematic reviews of diagnostic test accuracy studies
Pereira, R. A., Puga, M. E. D. S., Atallah, Á. N., de Macedo, E. C., & Macedo, C. R. (2019) · Health Information & Libraries Journal · doi:10.1111/hir.12263 · Cited by 4
Gingivitis and oral health-related quality of life: a systematic literature review
Barbosa, T. D. S., Gavião, M. B. D., & Mialhe, F. L. (2015) · Brazilian Dental Science · Cited by 4
Heart failure symptoms as predictors of hospital admission, readmission and all-cause mortality
Ali, M., Freeman, S. C., Gray, L. J., Kadam, U. T., & Lawson, C. A. (2022) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd015206 · Cited by 4
Should we be worried about glaucoma? Why the prevalence of glaucoma is variably reported
Worley, A., & Grimmer, K. A. (2009) · Primary Health Care Research & Development · doi:10.1017/s1463423609990193 · Cited by 4
“Sensitivity testing regarding information published in languages other than English has shown that English language reviews represent a robust view of the available evidence base in health areas (Moher et al., 2000; 2003; Møller and Jennions, 2001).”
BackgroundBasics of Health Technology Assessment
O’Reilly, D. J., Campbell, K., & Goeree, R. A. (2008) · Methods in molecular biology · doi:10.1007/978-1-59745-385-1_16 · Cited by 6
Assessing language bias in pediatric surgical systematic reviews: A meta-epidemiological study
Moghul, D., Guadagno, E., Kundu, S., Poenaru, D., & Baird, R. (2026) · Journal of Pediatric Surgery · doi:10.1016/j.jpedsurg.2026.162999 · Cited by 3
Conducting Systematic Reviews of Complementary and Alternative Medicine: Common Pitfalls
Wider, B., & Boddy, K. (2009) · Evaluation & the Health Professions · doi:10.1177/0163278709346818 · Cited by 3
Publication bias in otorhinolaryngology meta-analyses in 2021
Mohammadian, F., Bastaninejad, S., & Irani, S. (2024) · Systematic Reviews · doi:10.1186/s13643-023-02404-0 · Cited by 3
Metadata
- Authors
- Moher, D., Pham, B., Lawson, M., & Klassen, T.
- Affiliations
- University of Ottawa (CA)
- Venue
- Health Technology Assessment
- Published
- 2003-12
- Volume
- 7
- Issue
- 41
- Pages
- 1-90
- ISSN
- 1366-5278 (ISSN-L)
- 2046-4924
- OpenAlex
- W2119404638
- Cited by
- 500
- References
- 236
- Citations by year
- 2026: 9 · 2025: 11 · 2024: 18 · 2023: 13 · 2022: 33 · 2021: 27 · 2020: 26 · 2019: 26 · 2018: 27 · 2017: 18 · 2016: 22 · 2015: 44 · 2014: 19 · 2013: 24 · 2012: 16
- FWCI
- 2.71
- Type
- journal article
- Language
- en
- Open access
- diamond
- doi.org
- Locations
- Health Technology Assessment (publishedVersion, cc-by)
- PubMed (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Funders
- Health Technology Assessment programme (96/52/99)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Library and Information Sciences › scientometrics and bibliometrics research
- Social Sciences › Library and Information Sciences › Academic Publishing and Open Access
- Keywords
- randomized controlled trials, systematic review quality, complementary and alternative medicine, publication bias, statistical heterogeneity, language of publication, bias in systematic reviews, Cochrane Collaboration
- MeSH terms
- Complementary Therapies, United Kingdom, Humans, Language, Publishing, Randomized Controlled Trials as Topic
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.