The inclusion of reports of randomised trials published in languages other than English in systematic reviews
Open accessThe DOI matches its Crossref record; 235 of the 236 references it cites were found in a registry or an index.
Abstract
OBJECTIVE: To assemble a large dataset of language restricted and language inclusive systematic reviews, including both conventional medicinal (CM) and complementary and alternative medicine (CAM) interventions. To then assess the quality of these reports by considering and comparing different types of systematic reviews and their associated RCTs; CM and CAM interventions; the effect of language restrictions compared with language inclusions, and whether these results are influenced by other issues, including statistical heterogeneity and publication bias, in the systematic review process. DATA SOURCES: MEDLINE, EMBASE, the Cochrane Database of Systematic Reviews and the Centralised Information Service for Complementary Medicine. REVIEW METHODS: Three types of systematic reviews were included: language restricted; language inclusive/English language (EL) reviews that searched RCTs in languages other than English (LOE) but did not find any and, hence, could not include any, in the quantitative data synthesis; and systematic reviews that searched for RCTs in LOE and included them in the quantitative data synthesis. Fisher's exact test was applied to compare the three different types of systematic reviews with respect to their reporting characteristics and the systematic review quality assessment tool. The odds ratio of LOE trials versus EL trials was computed for each review and this information was pooled across the reviews to examine the influence that language of publication and type of intervention (CM, CAM) have on the estimates of intervention effect. Several sensitivity analyses were performed. RESULTS: The LOE RCTs were predominantly in French and German. Language inclusive/LOE systematic reviews were of the highest quality compared with the other types of reviews. The CAM reviews were of higher quality compared with the CM reviews. There were only minor differences in the quality of reports of EL RCTs compared with the eight other languages considered. However, there are inconsistent differences in the quality of LOE reports depending on the intervention type. The results, and those reported previously, suggest that excluding reports of RCTs in LOE from the analytical part of a systematic review is reasonable. Because the present research and previous efforts have not included every type of CM RCT and the resulting possibility of the uncertainty as to when bias will be present by excluding LOE, it is always prudent to perform a comprehensive search for all evidence. This result only applies to reviews investigating the benefits of CM interventions. This does not imply that systematic reviewers should neglect reports in LOE. We recommend that systematic reviewers search for reports regardless of the language. There may be merit in including them in some aspects of the review process although this decision is likely to depend on several factors, including fiscal and other resources being available. Language restrictions significantly shift the estimates of an intervention's effectiveness when the intervention is CAM. Here, excluding trials reported in LOE, compared with their inclusion, resulted in a reduced intervention effect. The present results do not appear to be influenced by statistical heterogeneity and publication bias. CONCLUSIONS: With the exception of CAM systematic reviews, the quality of recently published systematic reviews is less than optimal. Language inclusive/LOE systematic reviews appear to be a marker for a better quality systematic review. Language restrictions do not appear to bias the estimates of a conventional intervention's effectiveness. However, there is substantial bias in the results of a CAM systematic review if LOE reports are excluded from it.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“There is mixed evidence for the potential impact of not including non-English-language trials: one study found that non-English trials were more likely to produce significant results at p <0.05 (Jüni et al., 2002), whereas an earlier review concluded that excluding non-English trials did not significantly affect the results of meta-analyses (Moher et al., 2003).”
From Self-help interventions to reduce self-stigma in people with mental health problems: A systematic literature review (2020) · cites this paper as Moher et al., 2003
In the citing paper
“In addition, it has been reported that language-restricted meta-analyses only minimally overestimate treatment effects (ϳ2% on average) compared with language-inclusive meta-analyses.112”
From Effectiveness of Manual Therapy and Therapeutic Exercise for Temporomandibular Disorders: Systematic Review and Meta-Analysis (2016) · cites this paper as [112]
In the citing paper
“Therefore, language-restricted metaanalyses do not appear to lead to biased estimates of intervention effectiveness.112,113”
From Effectiveness of Manual Therapy and Therapeutic Exercise for Temporomandibular Disorders: Systematic Review and Meta-Analysis (2016) · cites this paper as [112]
In the citing paper
“Over the last two decades, empirical evidence has accumulated from studies investigating different sources of bias related to specific SR tasks.”
From How to conduct systematic reviews more expeditiously? (2015) · cites this paper as [31]
In the citing paper
“Some authors suggested that the impact of excluding non-English language studies may depend on the topic of the review and the quality of non-English language studies [29, 31, 32].”
From How to conduct systematic reviews more expeditiously? (2015) · cites this paper as [31]
2 more claims about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
3 retracted
Daya, S., Maheshwari, A., Siristatidis, C. S., Bhattacharya, S., & Gibreel, A. (2000) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd001299 · Cited by 114
RetractedRice-based oral rehydration solution for treating diarrhoea
Fontaine, O., Gore, S., & Pierce, N. (1998) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd001264 · Cited by 79
RetractedAndrogens versus placebo or no treatment for idiopathic oligo/asthenospermia
Vandekerckhove, P., Lilford, R. J., Vail, A., & Hughes, E. G. (1996) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd000150 · Cited by 30
RetractedThe Measurement of Observer Agreement for Categorical Data
Landis, J. R., & Koch, G. G. (1977) · Biometrics · doi:10.2307/2529310 · Cited by 81,148
Bias in meta-analysis detected by a simple, graphical test
Egger, M., Smith, G. D., Schneider, M., & Minder, C. (1997) · BMJ · doi:10.1136/bmj.315.7109.629 · Cited by 56,722
Meta-analysis in clinical trials
DerSimonian, R., & Laird, N. (1986) · Controlled Clinical Trials · doi:10.1016/0197-2456(86)90046-2 · Cited by 39,552
Quantifying heterogeneity in a meta‐analysis
Higgins, J. P. T., & Thompson, S. G. (2002) · Statistics in Medicine · doi:10.1002/sim.1186 · Cited by 36,763
Intraclass correlations: Uses in assessing rater reliability.
Shrout, P. E., & Fleiss, J. L. (1979) · Psychological Bulletin · doi:10.1037/0033-2909.86.2.420 · Cited by 24,532
Assessing the quality of reports of randomized clinical trials: Is blinding necessary?
Jadad, A. R., Moore, R. A., Carroll, D., Jenkinson, C., Reynolds, D. J. M., Gavaghan, D. J., & McQuay, H. J. (1996) · Controlled Clinical Trials · doi:10.1016/0197-2456(95)00134-4 · Cited by 17,582
Eisenberg, D. M., Davis, R. B., Ettner, S. L., Appel, S., Wilkey, S., Van Rompay, M., & Kessler, R. C. (1998) · JAMA · doi:10.1001/jama.280.18.1569 · Cited by 6,658
Cited by
Items 201–300 of 499
Asynchronous telehealth: a scoping review of analytic studies.
(2009) · PubMed Central · Cited by 51
Gagnier, J. J., Huang, H., Mullins, M. A., Marinac‐Dabic, D., Ghambaryan, A., Eloff, B. C., Mirza, F. M., & Bayona, M. (2018) · JBJS Reviews · doi:10.2106/jbjs.rvw.17.00038 · Cited by 53
Sørensen, J. R., Hegedűs, L. D., Kruse-Andersen, S., Godballe, C., & Bonnema, S. J. (2014) · Best Practice & Research Clinical Endocrinology & Metabolism · doi:10.1016/j.beem.2014.03.002 · Cited by 51
Wong, J. J., Côté, P., Shearer, H. M., Carroll, L., Yu, H., Varatharajan, S., Southerst, D., van der Velde, G., Jacobs, C. L., & Taylor-Vaisey, A. L. (2014) · Disability and Rehabilitation · doi:10.3109/09638288.2014.932448 · Cited by 58
Location of Hamstring Injuries Based on Magnetic Resonance Imaging: A Systematic Review
Grange, S., Reurink, G., Nguyen, A. Q., Riviera-Navarro, C., Foschia, C., Croisille, P., & Édouard, P. (2022) · Sports Health A Multidisciplinary Approach · doi:10.1177/19417381211071010 · Cited by 47
“The restriction on English language was based on findings from systematic reviews suggesting no evidence of bias for conventional medicine if studies written in languages other than English were excluded.(36) The risk of methodological bias for each study seems to be very low, even if potential confounders are possibl…”
BackgroundEffect of Yoga on Frailty in Older Adults: A Systematic Review
Loewenthal, J., Innes, K. E., Mitzner, M. G., Mita, C., & Orkaby, A. R. (2023) · Annals of Internal Medicine · doi:10.7326/m22-2553 · Cited by 69
“We excluded non-English studies during abstract and full-text review because of concerns about proper translation and because there is little effect on overall results (32).”
BackgroundLemeunier, N., Jeoun, E. B., Suri, M. F. K., Tuff, T., Shearer, H. M., Mior, S. A., Wong, J. J., da Silva-Oolup, S., Torres, P., D’Silva, C., et al. (2018) · Musculoskeletal Science and Practice · doi:10.1016/j.msksp.2018.09.013 · Cited by 50
Austin, S. L., Mattick, C. R., & Waterhouse, P. J. (2014) · Orthodontics and Craniofacial Research · doi:10.1111/ocr.12063 · Cited by 54
“In contrast, alternative research (24) has found overall the exclusion of trials not reported in English does not significantly alter the results of meta-analyses.”
ResultMURPHY, C. P., Fajt, V. R., Scott, H. M., Foster, M., WICKWIRE, P., & McEwen, S. A. (2015) · Epidemiology and Infection · doi:10.1017/s0950268815000722 · Cited by 46
“but the impact of this is difficult to determine [26, 27]; other agri-food reviews have reported that the number of relevant non-English-language citations”
BackgroundShearer, H. M., Carroll, L., Wong, J. J., Côté, P., Varatharajan, S., Southerst, D., Sutton, D. A., Randhawa, K. A., Yu, H., Mior, S. A., et al. (2015) · The Spine Journal · doi:10.1016/j.spinee.2015.08.011 · Cited by 50
Andrade, R., Duarte, H., Pereira, R., Lopes, I., Pereira, H., Rocha, R., & Espregueira‐Mendes, J. (2016) · Porto Biomedical Journal · doi:10.1016/j.pbj.2016.09.001 · Cited by 47
Dogra, S., MacIntosh, L., O’Neill, C. D., D’Silva, C., Shearer, H. M., Smith, K., & Côté, P. (2017) · Mental health and physical activity · doi:10.1016/j.mhpa.2017.11.001 · Cited by 64
A systematic review of psychological interventions for patients with head and neck cancer
Richardson, A., Broadbent, E., & Morton, R. P. (2019) · Supportive Care in Cancer · doi:10.1007/s00520-019-04768-3 · Cited by 59
Speed, C., Heaven, B., Adamson, A., Bond, J. B., Corbett, S. S., Lake, A. A., May, C. R., Vanoli, A., McMeekin, P. J., Moynihan, P. J., et al. (2010) · Health Technology Assessment · doi:10.3310/hta14520 · Cited by 46
Dillon, M. P., Quigley, M., & Fatone, S. (2017) · Systematic Reviews · doi:10.1186/s13643-017-0433-7 · Cited by 48
“All searches were restricted to the English language given that such restriction does not seem to alter the outcome of systematic reviews and meta-analyses [46, 47].”
BackgroundLemeunier, N., da Silva-Oolup, S., Chow, N., Southerst, D., Carroll, L., Wong, J. J., Shearer, H. M., Mastragostino, P., Cox, J., Côté, E., et al. (2017) · European Spine Journal · doi:10.1007/s00586-017-5153-0 · Cited by 45
Evaniew, N., Khan, M., Drew, B. M., Kwok, D., Bhandari, M., & Ghert, M. A. (2014) · CMAJ Open · doi:10.9778/cmajo.20140048 · Cited by 46
Young, I., Waddell, L. A., Sánchez, J., Wilhelm, B., McEwen, S. A., & Rajić, A. (2013) · Preventive Veterinary Medicine · doi:10.1016/j.prevetmed.2013.11.009 · Cited by 53
van den Berg, I., Bosch, J. L., Jacobs, B. Z., Bouman, I., Duvekot, J. J., & Hunink, M. G. M. (2008) · Complementary Therapies in Medicine · doi:10.1016/j.ctim.2008.01.001 · Cited by 52
Xu, K., Zha, M., Wu, X., Yu, Z., Yu, R., Xu, X., Chen, H., & Yang, T. (2010) · Diabetes Research and Clinical Practice · doi:10.1016/j.diabres.2010.11.012 · Cited by 42
“[54] Moher D, Pham B, Lawson ML, Klassen TP.”
BackgroundWilson, J. R., Patel, A. A., Brodt, E., Dettori, J. R., Brodke, D. S., & Fehlings, M. G. (2013) · Spine · doi:10.1097/brs.0b013e3182a7f478 · Cited by 45
Wu, V. S., Smith, A. B., & Girgis, A. (2022) · European Journal of Cancer Care · doi:10.1111/ecc.13269 · Cited by 40
Efficacy and harms of the hypoglycemic agent pramlintide in diabetes mellitus
Lee, N. J., Norris, S. L., & Thakurta, S. (2010) · The Annals of Family Medicine · doi:10.1370/afm.1174 · Cited by 39
Screening for depression in children and adolescents: a protocol for a systematic review update
Beck, A., LeBlanc, J. C., Morissette, K., Hamel, C., Skidmore, B., Colquhoun, H. L., Lang, E., Moore, A. E., Riva, J. J., Thombs, B. D., et al. (2021) · Systematic Reviews · doi:10.1186/s13643-020-01568-3 · Cited by 56
Self-citationAra, R., Pandor, A., Stevens, J., Rees, A., & Rafia, R. (2009) · Health Technology Assessment · doi:10.3310/hta13340 · Cited by 54
Bond, M., Wyatt, K., Lloyd, J., Welch, K., & Taylor, R. (2009) · Health Technology Assessment · doi:10.3310/hta13610 · Cited by 39
Shorter, G. W., Bray, J. W., Giles, E. L., O'Donnell, A. J., Berman, A. H., Holloway, A. S., Heather, N., Barbosa, C., Stockdale, K. J., Scott, S., et al. (2019) · Journal of Studies on Alcohol and Drugs · doi:10.15288/jsad.2019.80.286 · Cited by 46
“Although focusing on peer-reviewed literature may have also limited the number of non-English articles included, it is in keeping with our intention to focus on those articles that are likely to be most accessible and influential for many decision makers.”
BackgroundInfluentialLawson, M. L., Pham, B., Klassen, T. P., & Moher, D. (2005) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2004.08.022 · Cited by 40
Self-citationSpiral artery blood flow during pregnancy: a systematic review and meta-analysis
Schiffer, V. M. M. M., Evers, L., de Haas, S., Ghossein‐Doha, C., Al‐Nasiry, S., & Spaanderman, M. E. A. (2020) · BMC Pregnancy and Childbirth · doi:10.1186/s12884-020-03150-0 · Cited by 45
Gagnier, J. J., Moher, D., Boon, H., Beyene, J., & Bombardier, C. (2011) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2010.10.005 · Cited by 32
Self-citationRodríguez-Molina, D., Mang, P., Schmitt, H., Chifiriuc, M. C., Radon, K., & Wengenroth, L. (2019) · Systematic Reviews · doi:10.1186/s13643-019-1236-9 · Cited by 38
Wang, H., Chen, Y., Lin, Y., Abesig, J., Wu, X., & Tam, W. (2021) · BMJ · doi:10.1136/bmj.n736 · Cited by 44
Dilnessa, T., Getaneh, A., Hailu, W., Moges, F., & Gelaw, B. (2022) · PLoS ONE · doi:10.1371/journal.pone.0262597 · Cited by 38
“All searches were limited to articles written in English given that such language restriction does not alter the outcome of the systematic reviews and meta-analyses [26].”
BackgroundEffects of omega-3 fatty acids on eye health.
Hodge, W., Barnes, D., Schachter, H., Pan, Y., Lowcock, E. C., Zhang, L., Sampson, M., Morrison, A., Tran, K. T., Miguelez, M., et al. (2005) · PubMed · Cited by 38
Palla, E. S., Karaoglani, E., Νάκα, Ό., & Anastassiadou, V. (2015) · Journal of Dentistry · doi:10.1016/j.jdent.2015.09.005 · Cited by 47
“Moreover, exclusion of studies published in other languages does not significantly influence the treatment effect estimates of metaanalyses [55,56].”
Zhong, G., Ye, M.-X., Cheng, J.-H., Zhao, Y., & Gong, J. (2016) · Scientific Reports · doi:10.1038/srep24071 · Cited by 40
The Effectiveness of Physical Agents for Lower-Limb Soft Tissue Injuries: A Systematic Review
Yu, H., Randhawa, K. A., Côté, P., & Optima Collaboration (2016) · Journal of Orthopaedic and Sports Physical Therapy · doi:10.2519/jospt.2016.6521 · Cited by 41
Lemeunier, N., da Silva-Oolup, S., Olesen, K., Shearer, H. M., Carroll, L., Brady, O., Côté, E., Stern, P. J., Tuff, T., Suri-Chilana, M., et al. (2019) · European Spine Journal · doi:10.1007/s00586-019-05949-8 · Cited by 42
Lam, P. P. Y., Sardana, D., Ekambaram, M., Lee, G. H. M., & Yiu, C. K. Y. (2020) · Journal of Evidence Based Dental Practice · doi:10.1016/j.jebdp.2020.101404 · Cited by 40
Edessa, D., Assefa, N., Dessie, Y., Kumsa, F. A., Dinsa, G. D., & Oljira, L. (2022) · Journal of Pharmaceutical Policy and Practice · doi:10.1186/s40545-022-00454-8 · Cited by 43
Samargandi, O. A., Hasan, H., & Thoma, A. (2015) · Plastic & Reconstructive Surgery · doi:10.1097/prs.0000000000001898 · Cited by 38
Rathod, S., Jeremic, B. M., Dubey, A., Giuliani, M., Bashir, B., Chowdhury, A., Liang, Y., Pereira, S., Agarwal, J., & Koul, R. (2019) · European Journal of Cancer · doi:10.1016/j.ejca.2019.01.003 · Cited by 36
Alam, M., Ferdous, S., Ercümen, A., Lin, A., Kamal, A., Luies, S. K., Sharior, F., Khan, R., Rahman, M. Z., Parvez, S. M., et al. (2021) · JMIR Research Protocols · doi:10.2196/33365 · Cited by 45
“However, a certain level of bias may be introduced due to the language restriction, although few studies claim that excluding non-English literature does not have a large effect on systematic reviews and meta-analyses [43,44].”
BackgroundA Review on the Complementary Effects of Auriculotherapy in Managing Constipation
Li, M.-K., Lee, D., & Suen, L. K. P. (2010) · The Journal of Alternative and Complementary Medicine · doi:10.1089/acm.2009.0348 · Cited by 36
McIntosh, B., Yu, C., Lal, A., Chelak, K., Cameron, C. G., Singh, S. R., & Dahl, M. (2010) · PubMed Central · Cited by 40
Dormandy, E., Bryan, S., Gulliford, M. C., Roberts, T. E., Ades, A. E., Calnan, M., Atkin, K. M., Karnon, J. D., Barton, P., Logan, J., et al. (2010) · Health Technology Assessment · doi:10.3310/hta14200 · Cited by 39
Yu, H., Côté, P., Shearer, H. M., Wong, J. J., Sutton, D. A., Randhawa, K. A., Varatharajan, S., Southerst, D., Mior, S. A., Ameis, A., et al. (2014) · Physical Therapy · doi:10.2522/ptj.20140361 · Cited by 41
Varatharajan, S., Côté, P., Shearer, H. M., Loisel, P., Wong, J. J., Southerst, D., Yu, H., Randhawa, K. A., Sutton, D. A., van der Velde, G., et al. (2014) · Journal of Occupational Rehabilitation · doi:10.1007/s10926-014-9501-1 · Cited by 39
Influence of the maternal birth status on offspring: A systematic review and meta‐analysis
Shah, P. S., Shah, V. S., & on behalf of Knowledge Synthesis Group on Determinants of Preterm/LBW births (2009) · Acta Obstetricia Et Gynecologica Scandinavica · doi:10.3109/00016340903358820 · Cited by 34
Simpson, E. L., Stevenson, M., Rawdin, A., & Papaioannou, D. E. (2009) · Health Technology Assessment · doi:10.3310/hta13020 · Cited by 33
Phillips, R. S., Gopaul, S., Gibson, F., Houghton, E., Craig, J. V., Light, K., & Pizer, B. L. (2010) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd007786.pub2 · Cited by 40
Menta, R., Randhawa, K., Côté, P., Wong, J. J., Yu, H., Sutton, D., Varatharajan, S., Southerst, D., D'Angelo, K., Cox, J., et al. (2015) · Journal of Manipulative and Physiological Therapeutics · doi:10.1016/j.jmpt.2015.06.002 · Cited by 33
Bias in Randomized Controlled Trials
Jadad, A. R., & Enkin, M. W. (2007) · Wiley eBooks · doi:10.1002/9780470691922.ch3 · Cited by 36
How do we assess musicians’ musculoskeletal symptoms?: a review of outcomes and tools used
Stanhope, J., Pisaniello, D., Tooher, R. L., & Weinstein, P. (2018) · Industrial Health · doi:10.2486/indhealth.2018-0065 · Cited by 38
Shearer, H. M., Carroll, L., Côté, P., Randhawa, K. A., Southerst, D., Varatharajan, S., Wong, J. J., Yu, H., Sutton, D. A., van der Velde, G., et al. (2020) · European Journal of Physiotherapy · doi:10.1080/21679169.2020.1736150 · Cited by 35
Efficacy of periprostatic local anesthetic for prostate biopsy analgesia: A meta-analysis
Richman, J. M., Carter, H. B., Hanna, M. N., Murphy, J. D., Rowlingson, A. J., Andrews, R. A. F., & Wu, C. L. (2006) · Urology · doi:10.1016/j.urology.2005.12.030 · Cited by 32
Rodrigues, L. A., Caroba, M. E. S., Taba, F. K., Filev, R., & Gallassi, A. D. (2020) · Pharmacology Biochemistry and Behavior · doi:10.1016/j.pbb.2020.172982 · Cited by 35
"No Language Restrictions" in Database Searches: What Does This Really Mean?
Pilkington, K., Boshnakova, A., Clarke, M. J., & Richardson, J. (2005) · The Journal of Alternative and Complementary Medicine · doi:10.1089/acm.2005.11.205 · Cited by 31
“The first of these is the quality of reporting in studies published in LOE compared to those published in English, which has been addressed previously (Eg-ger et al., 2003; Moher et al., 1996, 2003).”
Peinemann, F., Grouven, U., Kröger, N. M., Bartel, C., Pittler, M. H., & Lange, S. (2011) · PLoS ONE · doi:10.1371/journal.pone.0018572 · Cited by 34
“Restricting the inclusion of studies to English articles may have little effect on summary treatment effect estimates [66,67] and German language articles may not play a preeminent role in the dissemination of medical research [68].”
BackgroundConway, A., Inglis, S. C., Chang, A. M., Horton-Breshears, M., Cleland, J. G. F., & Clark, R. A. (2013) · Journal of Telemedicine and Telecare · doi:10.1177/1357633x13503427 · Cited by 36
Parker, S. G., Oliver, P., Pennington, M., Bond, J. B., Jagger, C., Enderby, P., Curless, R., Chater, T., Vanoli, A., Fryer, K., et al. (2009) · Health Technology Assessment · doi:10.3310/hta13390 · Cited by 30
Zhang, Y., Yang, X., Geng, D., Duan, Y., & Fu, J. (2018) · World Journal of Surgical Oncology · doi:10.1186/s12957-018-1330-9 · Cited by 29
Cochrane Systematic Reviews in Acupuncture: Methodological Diversity in Database Searching
Sood, A., Sood, R., Bauer, B. A., & Ebbert, J. O. (2005) · The Journal of Alternative and Complementary Medicine · doi:10.1089/acm.2005.11.719 · Cited by 31
Citation classics: the 100 most cited articles in Urogynecology
Gupta, A., Kennedy, B., Meriwether, K. V., Francis, S. L., Cardenas-Trowers, O. O., & Stewart, J. R. (2019) · International Urogynecology Journal · doi:10.1007/s00192-019-04021-9 · Cited by 28
“That said, this same bias toward English-language articles, made clear by past research revealing that non-English-language articles are poorly accepted and cited, makes the likelihood of nonEnglish-language articles elevating to more citations very low [14, 15].”
BackgroundPapaconstantinou, E. A., Cancelliere, C., Verville, L., Wong, J. J., Connell, G. P., Yu, H., Shearer, H. M., Timperley, C., Chung, C., Porter, B. J., et al. (2021) · Chiropractic & Manual Therapies · doi:10.1186/s12998-021-00381-6 · Cited by 31
Peinemann, F., Kröger, N. M., Bartel, C., Grouven, U., Pittler, M. H., Erttmann, R., & Kulig, M. (2011) · PLoS ONE · doi:10.1371/journal.pone.0017127 · Cited by 33
“Restricting the inclusion of studies to English language may have little effect on summary treatment effect estimates [88,89] and German language articles may not play a preeminent role in the dissemination of medical research [90].”
BackgroundImpact of prenatal alcohol exposure on neurodevelopmental outcomes: a systematic review
Chu, J. T. W., McCormack, J. C., Marsh, S., Wells, A., Wilson, H. S., & Bullen, C. (2022) · Health Psychology and Behavioral Medicine · doi:10.1080/21642850.2022.2129653 · Cited by 34
“However, previous research has indicated that exclusion of non-English language studies is unlikely to introduce systematic bias (Jüni et al., 2002; Moher et al., 2003; Morrison et al., 2012).”
BackgroundJin, Z., Li, R., Gan, T. J., He, Y., & Lin, J. (2020) · PubMed · Cited by 40
Cross, J., Elender, F., Barton, G., Clark, A., Shepstone, L., Blyth, A., Bachmann, M. O., & Harvey, I. M. (2010) · Health Technology Assessment · doi:10.3310/hta14230 · Cited by 38
Diriba, K., Awulachew, E., & Anja, A. (2021) · European journal of medical research · doi:10.1186/s40001-020-00474-7 · Cited by 26
“All searches were limited to articles written in English given that such language restriction does not alter the outcome of the systematic reviews and meta-analysis [31].”
BackgroundNewman, S. P., Cooke, D. D., Casbard, A. C., Walker, S., Meredith, S. K., Nunn, A., Steed, L., Manca, A., Sculpher, M., Barnard, M., et al. (2009) · Health Technology Assessment · doi:10.3310/hta13280 · Cited by 26
Wu, H., Liu, M. B., Patel, K. R., Turner, W., Baltus, L., Caldwell, A. M., Hahn, J. C., Coughlin, R. R., Morshed, S., Miclau, T., et al. (2017) · SICOT-J · doi:10.1051/sicotj/2016042 · Cited by 27
Hodge, W., BARNES, D., Schachter, H., Pan, Y., Lowcock, E. C., Zhang, L., Sampson, M., Morrison, A., Tran, K. T., Miguelez, M., et al. (2007) · Retina · doi:10.1097/01.iae.0000233322.83713.2d · Cited by 25
Are adverse effects incorporated in economic models? An initial review of current practice
Craig, D., McDaid, C., Fonseca, T., Stock, C., Duffy, S., & Woolacott, N. F. (2009) · Health Technology Assessment · doi:10.3310/hta13620 · Cited by 26
Busse, J. W., Bruno, P. A., Malik, K., Connell, G. P., Torrance, D., Ngo, T., Kirmayr, K., Avrahami, D., Riva, J. J., Ebrahim, S., et al. (2014) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2013.07.022 · Cited by 26
“[4] found that, on average, meta-analyses excluding non-English trials resulted in a reduction of the pooled treatment effect size.”
BackgroundPires, C. W., Soldera, E. B., Bonzanini, L. I. L., Lenzi, T. L., Soares, F. Z. M., Montagner, A. F., & Rocha, R. D. O. (2018) · PubMed · doi:10.3290/j.jad.a40296 · Cited by 24
Bryant, J., Cooper, K., Picot, J., Clegg, A. J., Roderick, P., Rosenberg, W. M., & Patch, C. (2009) · Health Technology Assessment · doi:10.3310/hta13230 · Cited by 27
Husk, K., Lovell, R., Cooper, C., & Garside, R. (2013) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd010351 · Cited by 26
Corso, M., Cancelliere, C., Mior, S., Salmi, L. R., Cedraschi, C., Nordin, M., Sci, D. M., Taylor-Vaisey, A., & Côté, P. (2022) · Archives of Physical Medicine and Rehabilitation · doi:10.1016/j.apmr.2021.09.007 · Cited by 24
Zhang, X., Weakley, J. J. S., Li, H., Li, Z., & García‐Ramos, A. (2025) · Sports Medicine · doi:10.1007/s40279-025-02176-8 · Cited by 28
“Only studies that underwent peer review and were published in English were considered [35, 36].”
Diriba, K., Awulachew, E., Gemede, A., & Anja, A. (2021) · Access Microbiology · doi:10.1099/acmi.0.000195 · Cited by 28
“restriction does not alter the outcome of the systematic reviews and meta- analysis [25].”
BackgroundUse of Information Technology for Falls Detection and Prevention in the Elderly
Atoyebi, O. A., Stewart, A., & Sampson, J. (2014) · Ageing International · doi:10.1007/s12126-014-9204-0 · Cited by 27
“The search was also limited to studies published in English because it is the commonest language used in international conferences and seminars worldwide (Moher et al. 2003) and yielded adequate literature for this review.”
BackgroundPeinemann, F., McGauran, N., Sauerland, S., & Lange, S. (2008) · BMC Medical Research Methodology · doi:10.1186/1471-2288-8-41 · Cited by 24
Health behavioral theories used to explain dietary behaviors in adolescents: a systematic review
Philippi, S. T., Guerra, P. H., & Leme, A. C. B. (2016) · Nutrire · doi:10.1186/s41110-016-0023-9 · Cited by 25
“[65] observed that despite the strengths of some studies published in other languages (not English), the language restriction was not a source of bias to estimate the studies effectiveness.”
BackgroundRodgers, M., Hodges, R., Hawkins, J. E., Hollingworth, W., Duffy, S., McKibbin, M., Mansfield, M., Harbord, R. M., Sterne, J. A. C., Glasziou, P., et al. (2009) · Health Technology Assessment · doi:10.3310/hta13600 · Cited by 22
Southerst, D., Yu, H., Randhawa, K. A., Côté, P., D’Angelo, K., Shearer, H. M., Wong, J. J., Sutton, D. A., Varatharajan, S., Goldgrub, R., et al. (2015) · Chiropractic & Manual Therapies · doi:10.1186/s12998-015-0075-6 · Cited by 31
“The restriction of systematic reviews to the English language has not led to biased results in previous publications [60–62, 63, 64].”
BackgroundOrthopaedic Trauma in the Developing World: Where Are the Gaps in Research and What Can Be Done?
Challa, S., Wu, H.-H., Cunningham, B. P., Liu, M. B., Patel, K. R., Shearer, D., Morshed, S., & Miclau, T. (2018) · Journal of Orthopaedic Trauma · doi:10.1097/bot.0000000000001293 · Cited by 23
Yet another problem with systematic reviews: a living review update
Uttley, L., Weng, Y., & Falzon, L. (2024) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2024.111608 · Cited by 27
Takeda, A., Loveman, E., Harris, P. S., Hartwell, D., & Welch, K. (2008) · Health Technology Assessment · doi:10.3310/hta12320 · Cited by 23
Role of pharmacists in optimising opioid therapy for chronic non-malignant pain; A systematic review
Iqbal, A., Knaggs, R., Anderson, C., & Toh, L. S. (2020) · Research in Social and Administrative Pharmacy · doi:10.1016/j.sapharm.2020.11.014 · Cited by 24
Chow, N., Hogg‐Johnson, S., Mior, S. A., Cancelliere, C., Injeyan, H. S., Teodorczyk-Injeyan, J. A., Cassidy, J. D., Taylor-Vaisey, A. L., & Côté, P. (2021) · JAMA Network Open · doi:10.1001/jamanetworkopen.2021.5493 · Cited by 23
Getting inside acupuncture trials - Exploring intervention theory and rationale
Price, S., Long, A., Godfrey, M., & Thomas, K. J. (2011) · BMC Complementary and Alternative Medicine · doi:10.1186/1472-6882-11-22 · Cited by 21
“This is particularly important given the extensive literature on acupuncture published within the Chinese language [41,42].”
BackgroundThe surgical management of scoliosis: a scoping review of the literature
Evaniew, N., Devji, T., Drew, B. M., Peterson, D. C., Ghert, M. A., & Bhandari, M. (2015) · Scoliosis · doi:10.1186/s13013-014-0026-3 · Cited by 21
“Fifty six percent of studies were conducted in North America, followed by 23% in Europe and 18% in Asia.”
BackgroundInfluentialSutton, D. A., Gross, D. P., Côté, P., Randhawa, K. A., Yu, H., Wong, J. J., Stern, P. J., Varatharajan, S., Southerst, D., Shearer, H. M., et al. (2016) · Chiropractic & Manual Therapies · doi:10.1186/s12998-016-0089-8 · Cited by 24
“Other systematic reviews reported similar results when studying the effect of language-restrictions in conventional medicine [78–80].”
ResultThe representation of vulnerable populations in quality improvement studies
Rolnitsky, A., Kirtsman, M., Goldberg, H. R., Dunn, M., & Bell, C. M. (2018) · International Journal for Quality in Health Care · doi:10.1093/intqhc/mzy016 · Cited by 22
Blum, K., Chen, A. L., Chen, T. J. H., Prihoda, T. J., Schoolfield, J. D., DiNubile, N. A., Waite, R. L., Arcuri, V., Kerner, M., Braverman, E. R., et al. (2008) · Advances in Therapy · doi:10.1007/s12325-008-0073-3 · Cited by 25
Cancelliere, C., Sutton, D. A., Côté, P., French, S. D., Taylor-Vaisey, A. L., & Mior, S. A. (2019) · Implementation Science · doi:10.1186/s13012-019-0931-1 · Cited by 23
Liyih, M., Damtie, D., & Tegen, D. (2022) · The Scientific World JOURNAL · doi:10.1155/2022/3905963 · Cited by 29
“We searched articles written in English on international databases such as Pub Med/MEDLINE, Science Direct, Web of Science, and Google Scholar [13].”
MethodSurgery for constipation: systematic review and clinical guidance
Knowles, C. H., Grossi, U., Horrocks, E. J., Parés, D., Vollebregt, P. F., Chapman, M., Brown, S. R., Mercer-Jones, M. A., Williams, A. B., Hooper, R., et al. (2017) · Colorectal Disease · doi:10.1111/codi.13774 · Cited by 21
Sardana, D., Manchanda, S., Ekambaram, M., Yang, Y., McGrath, C. P. J., & Yiu, C. K. Y. (2019) · European Journal of Orthodontics · doi:10.1093/ejo/cjz015 · Cited by 21
Metadata
- Authors
- Moher, D., Pham, B., Lawson, M., & Klassen, T.
- Affiliations
- University of Ottawa (CA)
- Venue
- Health Technology Assessment
- Published
- 2003-12
- Volume
- 7
- Issue
- 41
- Pages
- 1-90
- ISSN
- 1366-5278 (ISSN-L)
- 2046-4924
- OpenAlex
- W2119404638
- Cited by
- 500
- References
- 236
- Citations by year
- 2026: 9 · 2025: 11 · 2024: 18 · 2023: 13 · 2022: 33 · 2021: 27 · 2020: 26 · 2019: 26 · 2018: 27 · 2017: 18 · 2016: 22 · 2015: 44 · 2014: 19 · 2013: 24 · 2012: 16
- FWCI
- 2.71
- Type
- journal article
- Language
- en
- Open access
- diamond
- doi.org
- Locations
- Health Technology Assessment (publishedVersion, cc-by)
- PubMed (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Funders
- Health Technology Assessment programme (96/52/99)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Library and Information Sciences › scientometrics and bibliometrics research
- Social Sciences › Library and Information Sciences › Academic Publishing and Open Access
- Keywords
- randomized controlled trials, systematic review quality, complementary and alternative medicine, publication bias, statistical heterogeneity, language of publication, bias in systematic reviews, Cochrane Collaboration
- MeSH terms
- Complementary Therapies, United Kingdom, Humans, Language, Publishing, Randomized Controlled Trials as Topic
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.