CoChat
Sign Up
Cited by 212Meta-analysis and systematic reviewsAdvanced Causal Inference TechniquesStatistical Methods in Epidemiology

The Evidence Project risk of bias tool: assessing study rigor for both randomized and non-randomized intervention studies

Open access
Kennedy, Caitlin E., Fonner, Virginia A., Armstrong, Kevin A., Denison, Julie A., Yeh, Ping Teresa, O’Reilly, Kevin R., Sweat, Michael D.·2019·Systematic Reviews
DOI 10.1186/s13643-018-0925-0PMC PMC6317181
Open this paper in CoChat — free account

Abstract

Different tools exist for assessing risk of bias of intervention studies for systematic reviews. We present a tool for assessing risk of bias across both randomized and non-randomized study designs. The tool was developed by the Evidence Project, which conducts systematic reviews and meta-analyses of behavioral interventions for HIV in low- and middle-income countries. We present the eight items of the tool and describe considerations for each and for the tool as a whole. We then evaluate reliability of the tool by presenting inter-rater reliability for 125 selected studies from seven published reviews, calculating a kappa for each individual item and a weighted kappa for the total count of items. The tool includes eight items, each of which is rated as being present (yes) or not present (no) and, for some items, not applicable or not reported. The items include (1) cohort, (2) control or comparison group, (3) pre-post intervention data, (4) random assignment of participants to the intervention, (5) random selection of participants for assessment, (6) follow-up rate of 80% or more, (7) comparison groups equivalent on sociodemographics, and (8) comparison groups equivalent at baseline on outcome measures. Together, items (1)–(3) summarize the study design, while the remaining items consider other common elements of study rigor. Inter-rater reliability was moderate to substantial for all items, ranging from 0.41 to 0.80 (median κ = 0.66). Agreement between raters on the total count of items endorsed was also substantial ( κ w = 0.66). Strengths of the tool include its applicability to a range of study designs, from randomized trials to various types of observational and quasi-experimental studies. It is relatively easy to use and interpret and can be applied to a range of review topics without adaptation, facilitating comparability across reviews. Limitations include the lack of potentially relevant items measured in other tools and potential threats to validity of some items. To date, the tool has been applied in over 30 reviews. We believe it is a practical option for assessing risk of bias in systematic reviews of interventions that include a range of study designs.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1186/s13643-018-0925-0

Cites

7 of 7 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • A coding matrix was modified from the Evidence Project risk of bias tool and the Critical Appraisal Skills Program checklist for qualitative studies to facilitate data extraction related to quality assessment.

    “A coding matrix was modified from the Evidence Project risk of bias tool23 and the Critical Appraisal Skills Program checklist for qualitative studies to facilitate data extraction related to quality assessment.”

    From HPV self-sampling for cervical cancer screening: a systematic review of values and preferences (2021) · cites this paper as [23]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 48

  1. [1]Bastian H Glasziou P Chalmers I Seventy-five trials and eleven systematic reviews a day: how will we ever keep up? PLoS Med 2010 7 9 e1000326 10.1371/journal.pmed.1000326 20877712 PMC2943439 · DOI · Paper page
  2. [2]Page MJ Shamseer L Altman DG Epidemiology and reporting characteristics of systematic reviews of biomedical research: a cross-sectional study PLoS Med 2016 13 5 e1002028 10.1371/journal.pmed.1002028 27218655 PMC4878797 · DOI · Paper page
  3. [3]Viswanathan M Patnode CD Berkman ND Recommendations for assessing the risk of bias in systematic reviews of health-care interventions J Clin Epidemiol 2018 97 26 34 10.1016/j.jclinepi.2017.12.004 29248724 · DOI · Paper page
  4. [4]Katrak P Bialocerkowski AE Massy-Westropp N Kumar S Grimmer KA A systematic review of the content of critical appraisal tools BMC Med Res Methodol 2004 4 22 10.1186/1471-2288-4-22 15369598 PMC521688 · DOI
  5. [6]Sanderson S Tatt ID Higgins JP Tools for assessing quality and susceptibility to bias in observational studies in epidemiology: a systematic review and annotated bibliography Int J Epidemiol 2007 36 3 666 676 10.1093/ije/dym018 17470488 · DOI · Paper page
  6. [7]Armijo-Olivo S Ospina M da Costa BR Poor reliability between Cochrane reviewers and blinded external reviewers when applying the Cochrane risk of bias tool in physical therapy trials PLoS One 2014 9 5 e96920 10.1371/journal.pone.0096920 24824199 PMC4019638 · DOI · Paper page
  7. [8]Hartling L Hamm MP Milne A Testing the risk of bias tool showed low reliability between individual reviewers and across consensus assessments of reviewer pairs J Clin Epidemiol 2013 66 9 973 981 10.1016/j.jclinepi.2012.07.005 22981249 · DOI · Paper page
  8. [9]Hartling L Milne A Hamm MP Testing the Newcastle-Ottawa Scale showed low reliability between individual reviewers J Clin Epidemiol 2013 66 9 982 993 10.1016/j.jclinepi.2013.03.003 23683848 · DOI · Paper page
  9. [10]Margulis AV Pladevall M Riera-Guardia N Quality assessment of observational studies in a drug-safety systematic review, comparison of two tools: the Newcastle-Ottawa Scale and the RTI item bank Clinical Epidemiol 2014 6 359 368 10.2147/CLEP.S66677 PMC4199858 25336990 · DOI
  10. [11]Bertrand JT O'Reilly K Denison J Anhang R Sweat M Systematic review of the effectiveness of mass communication programs to change HIV/AIDS-related behaviors in developing countries Health Educ Res 2006 21 4 567 597 10.1093/her/cyl036 16847044 · DOI
  11. [12]Denison JA O'Reilly KR Schmid GP Kennedy CE Sweat MD HIV voluntary counseling and testing and behavioral risk reduction in developing countries: a meta-analysis, 1990–2005 AIDS Behav 2008 12 3 363 373 10.1007/s10461-007-9349-x 18161018 · DOI
  12. [13]Fonner VA Armstrong KS Kennedy CE O'Reilly KR Sweat MD School based sex education and HIV prevention in low- and middle-income countries: a systematic review and meta-analysis PLoS One 2014 9 3 e89692 10.1371/journal.pone.0089692 24594648 PMC3942389 · DOI
  13. [14]Fonner VA Denison J Kennedy CE O'Reilly K Sweat M Voluntary counseling and testing (VCT) for changing HIV-related risk behavior in developing countries Cochrane Database Syst Rev 2012 9 CD001224 PMC3931252 22972050
  14. [15]Kennedy C O'Reilly K Medley A Sweat M The impact of HIV treatment on risk behaviour in developing countries: a systematic review AIDS Care 2007 19 6 707 720 10.1080/09540120701203261 17573590 · DOI
  15. [16]Kennedy CE Fonner VA Armstrong KA O'Reilly KR Sweat MD Increasing HIV serostatus disclosure in low and middle-income countries: a systematic review of intervention evaluations AIDS 2015 29 Suppl 1 S7 S23 10.1097/QAD.0000000000000671 26049541 PMC5493771 · DOI
  16. [17]Kennedy CE Fonner VA O'Reilly KR Sweat MD A systematic review of income generation interventions, including microfinance and vocational skills training, for HIV prevention AIDS Care 2014 26 6 659 673 10.1080/09540121.2013.845287 24107189 PMC3943565 · DOI
  17. [18]Kennedy CE, Fonner VA, Sweat MD, Okero FA, Baggaley R, O'Reilly KR. Provider-initiated HIV testing and counseling in low- and middle-income countries: a systematic review. AIDS Behav. 2013;17(5):1571–90. PMC3927322 22752501
  18. [19]Kennedy CE Medley AM Sweat MD O'Reilly KR Behavioural interventions for HIV positive prevention in developing countries: a systematic review and meta-analysis Bull World Health Organ 2010 88 8 615 623 10.2471/BLT.09.068213 20680127 PMC2908966 · DOI
  19. [20]Medley A Kennedy C O'Reilly K Sweat M Effectiveness of peer education interventions for HIV prevention in developing countries: a systematic review and meta-analysis AIDS Educ Prev 2009 21 3 181 206 10.1521/aeap.2009.21.3.181 19519235 PMC3927325 · DOI
  20. [21]O'Reilly KR d'Aquila E Fonner V Kennedy C Sweat M Can policy interventions affect HIV-related behaviors? A systematic review of the evidence from low- and middle-income countries AIDS Behav 2017 21 3 626 642 10.1007/s10461-016-1615-3 27864626 PMC5303549 · DOI
  21. [22]O'Reilly KR Fonner VA Kennedy CE Sweat MD Free condom distribution: what we don’t know may hurt us AIDS Behav 2014 18 11 2169 2171 10.1007/s10461-014-0742-y 24633741 PMC4729184 · DOI
  22. [23]O'Reilly KR Kennedy CE Fonner VA Sweat MD Family planning counseling for women living with HIV: a systematic review of the evidence of effectiveness on contraceptive uptake and pregnancy incidence, 1990 to 2011 BMC Public Health 2013 13 935 10.1186/1471-2458-13-935 24099177 PMC3852503 · DOI
  23. [24]Sweat M O'Reilly K Kennedy C Medley A Psychosocial support for HIV-infected populations in developing countries: a key yet understudied component of positive prevention AIDS 2007 21 8 1070 1071 10.1097/QAD.0b013e3280f774da 17457114 · DOI
  24. [25]Sweat MD, Denison J, Kennedy CE, Fonner V, O’Reilly KR. Effects of condom social marketing on condom use in developing countries: a systematic review and meta-analysis: 1990–2010. Bull World Health Organ. 2012;In Press. PMC3417793 22893745
  25. [26]Zajac K Kennedy CE Fonner VA Armstrong KS O'Reilly KR Sweat MD A systematic review of the effects of behavioral counseling on sexual risk behaviors and HIV/STI prevalence in low- and middle-income countries AIDS Behav 2015 19 7 1178 1202 10.1007/s10461-014-0893-x 25213302 PMC4362916 · DOI
  26. [27]Fonner VA Kennedy CE O'Reilly KR Sweat MD Systematic assessment of condom use measurement in evaluation of HIV prevention interventions: need for standardization of measures AIDS Behav 2014 18 12 2374 2386 10.1007/s10461-013-0655-1 24197972 PMC4013255 · DOI
  27. [29]Moher D Liberati A Tetzlaff J Altman DG Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement Ann Intern Med 2009 151 4 264 269 10.7326/0003-4819-151-4-200908180-00135 19622511 · DOI · Paper page
  28. [31]Cohen J A coefficient of agreement for nominal scales Educ Psychol Meas 1960 20 1 37 46 10.1177/001316446002000104 · DOI · Paper page
  29. [32]Cohen J Weighted kappa: nominal scale agreement with provision for scaled disagreement or partial credit Psychol Bull 1968 70 4 213 220 10.1037/h0026256 19673146 · DOI · Paper page
  30. [33]Hartling L Ospina M Liang Y Risk of bias versus quality assessment of randomised controlled trials: cross sectional study BMJ 2009 339 b4012 10.1136/bmj.b4012 19841007 PMC2764034 · DOI · Paper page
  31. [35]Efficacy of voluntary HIV-1 counselling and testing in individuals and couples in Kenya, Tanzania, and Trinidad: a randomised trial. The Voluntary HIV-1 Counseling and Testing Efficacy Study Group. Lancet. 2000;356(9224):103–112. 10963246
  32. [36]Magnani R Macintyre K Karim AM The impact of life skills education on adolescent sexual risk behaviors in KwaZulu-Natal, South Africa J Adolesc Health 2005 36 4 289 304 10.1016/j.jadohealth.2004.02.025 15780784 · DOI
  33. [39]Peinemann F Tushabe DA Kleijnen J Using multiple types of studies in systematic reviews of health care interventions--a systematic review PLoS One 2013 8 12 e85035 10.1371/journal.pone.0085035 24416098 PMC3887134 · DOI
  34. [43]Beres LK Narasimhan M Robinson J Welbourn A Kennedy CE Non-specialist psychosocial support interventions for women living with HIV: a systematic review AIDS Care 2017 29 9 1079 1087 10.1080/09540121.2017.1317324 28438030 PMC6957075 · DOI
  35. [44]Brickley DB Almers L Kennedy CE Sexual and reproductive health services for people living with HIV: a systematic review AIDS Care 2011 23 3 303 314 10.1080/09540121.2010.507746 21347893 · DOI
  36. [45]Chop E Duggaraju A Malley A Food insecurity, sexual risk behavior, and adherence to antiretroviral therapy among women living with HIV: a systematic review Health Care Women Int 2017 38 9 927 944 10.1080/07399332.2017.1337774 28586273 PMC6957078 · DOI
  37. [46]Haberlen SA Narasimhan M Beres LK Kennedy CE Integration of family planning services into HIV care and treatment services: a systematic review Stud Fam Plan 2017 48 2 153 177 10.1111/sifp.12018 PMC5516228 28337766 · DOI
  38. [47]Kennedy CE Bernard LJ Muessig KE Serosorting and HIV/STI infection among HIV-negative MSM and transgender people: a systematic review and meta-analysis to inform WHO guidelines J Sex Transm Dis 2013 2013 583627 26316960 PMC4437432
  39. [48]Kennedy CE Haberlen SA Narasimhan M Integration of sexually transmitted infection (STI) services into HIV care and treatment services for women living with HIV: a systematic review BMJ Open 2017 7 6 e015310 10.1136/bmjopen-2016-015310 PMC5734446 28637733 · DOI
  40. [49]Kennedy CE Spaulding AB Brickley DB Linking sexual and reproductive health and HIV interventions: a systematic review J Int AIDS Soc 2010 13 26 10.1186/1758-2652-13-26 20642843 PMC2918569 · DOI
  41. [51]Kerrigan DL Fonner VA Stromdahl S Kennedy CE Community empowerment among female sex workers is an effective HIV prevention intervention: a systematic review of the peer-reviewed evidence from low- and middle-income countries AIDS Behav 2013 17 6 1926 1940 10.1007/s10461-013-0458-4 23539185 · DOI
  42. [52]Lindegren ML Kennedy CE Bain-Brickley D Integration of HIV/AIDS services with maternal, neonatal and child health, nutrition, and family planning services Cochrane Database Syst Rev 2012 9 Cd010119 PMC12551653 22972150
  43. [53]Robinson JL Narasimhan M Amin A Interventions to address unequal gender and power relations and improve self-efficacy and empowerment for sexual and reproductive health decision-making for women living with HIV: a systematic review PLoS One 2017 12 8 e0180699 10.1371/journal.pone.0180699 28837562 PMC5570301 · DOI
  44. [54]Spaulding AB Brickley DB Kennedy C Linking family planning with HIV/AIDS interventions: a systematic review of the evidence AIDS 2009 23 Suppl 1 S79 S88 10.1097/01.aids.0000363780.42956.ff 20081392 · DOI
  45. [55]Lu H Liu Y Dahiya K Effectiveness of HIV risk reduction interventions among men who have sex with men in China: a systematic review and meta-analysis PLoS One 2013 8 8 e72747 10.1371/journal.pone.0072747 24137497 PMC3796948 · DOI
  46. [56]Ye S Yin L Amico R Efficacy of peer-led interventions to reduce unprotected anal intercourse among men who have sex with men: a meta-analysis PLoS One 2014 9 3 e90788 10.1371/journal.pone.0090788 24614809 PMC3948720 · DOI
  47. [57]Vrazo AC Firth J Amzel A Sedillo R Ryan J Phelps BR Interventions to significantly improve service uptake and retention of HIV-positive pregnant women and HIV-exposed infants along the prevention of mother-to-child transmission continuum of care: systematic review Trop Med Int Health 2018 23 2 136 148 10.1111/tmi.13014 29164754 · DOI
  48. [58]Xiao Z Noar SM Zeng L Systematic review of HIV prevention interventions in China: a health communication perspective Int J Public Health 2014 59 1 123 142 10.1007/s00038-013-0467-0 23604051 · DOI

Cited by

Sorted by citations

Authors
Kennedy, C. E., Fonner, V. A., Armstrong, K. A., Denison, J. A., Yeh, P. T., O’Reilly, K. R., & Sweat, M. D.
Affiliations
  • Johns Hopkins University (US)
  • Medical University of South Carolina (US)
Corresponding author
Kennedy, Caitlin E.
ORCID
Venue
Systematic Reviews
Publisher
Springer Science and Business Media LLC
Published
2019-01-03
Volume
8
Issue
1
Pages
3-3
ISSN
2046-4053 (ISSN-L)
DOI
10.1186/s13643-018-0925-0
PMC
PMC6317181
OpenAlex
W2939489030
Cited by
212
References
7
Citations by year
2026: 19 · 2025: 26 · 2024: 39 · 2023: 30 · 2022: 37 · 2021: 35 · 2020: 15 · 2019: 11
FWCI
16.39
Type
journal article
Language
en
Open access
Licence
Locations
  • Systematic Reviews (publishedVersion, cc-by)
  • Europe PMC (PubMed Central) (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
Funders
  • National Institute of Mental Health (R01MH071204, R01MH090173)
  • United States Agency for International Development (HRN-A-00-97-00012-00)
Topics
  • Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
  • Physical Sciences › Mathematics › Statistics and Probability › Advanced Causal Inference Techniques
  • Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Epidemiology
Keywords
risk of bias assessment, quasi-experimental design, observational studies, inter-rater reliability, low- and middle-income countries, systematic review methodology
MeSH terms
Systematic Reviews as Topic, Humans, Reproducibility of Results, Observer Variation, Bias, Randomized Controlled Trials as Topic, Risk Assessment, Observational Studies as Topic, Non-Randomized Controlled Trials as Topic

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review