Instruments for Assessing Risk of Bias and Other Methodological Criteria of Published Animal Studies: A Systematic Review
Open accessThe DOI matches its Crossref record; 23 of the 23 references it cites were found in a registry or an index.
Abstract
BACKGROUND: Results from animal toxicology studies are critical to evaluating the potential harm from exposure to environmental chemicals or the safety of drugs prior to human testing. However, there is significant debate about how to evaluate the methodology and potential biases of the animal studies. There is no agreed-upon approach, and a systematic evaluation of current best practices is lacking. OBJECTIVE: We performed a systematic review to identify and evaluate instruments for assessing the risk of bias and/or other methodological criteria of animal studies. METHOD: We searched Medline (January 1966-November 2011) to identify all relevant articles. We extracted data on risk of bias criteria (e.g., randomization, blinding, allocation concealment) and other study design features included in each assessment instrument. DISCUSSION: Thirty distinct instruments were identified, with the total number of assessed risk of bias, methodological, and/or reporting criteria ranging from 2 to 25. The most common criteria assessed were randomization (25/30, 83%), investigator blinding (23/30, 77%), and sample size calculation (18/30, 60%). In general, authors failed to empirically justify why these or other criteria were included. Nearly all (28/30, 93%) of the instruments have not been rigorously tested for validity or reliability. CONCLUSION: Our review highlights a number of risk of bias assessment criteria that have been empirically tested for animal research, including randomization, concealment of allocation, blinding, and accounting for all animals. In addition, there is a need for empirically testing additional methodological criteria and assessing the validity and reliability of a standard risk of bias assessment instrument.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
CONSORT 2010 Statement: updated guidelines for reporting parallel group randomised trials
the CONSORT Group, Schulz, K. F., Altman, D. G., & Moher, D. (2010) · BMC Medicine · doi:10.1186/1741-7015-8-18 · Cited by 13,371
Improving Bioscience Research Reporting: The ARRIVE Guidelines for Reporting Animal Research
Kilkenny, C., Browne, W. J., Cuthill, I. C., Emerson, M., & Altman, D. G. (2010) · PLoS Biology · doi:10.1371/journal.pbio.1000412 · Cited by 7,300 · cited 1×
“For example, Kilkenny et al. (2010) stated that the ARRIVE Guidelines is a 20criteria instrument.”
Improving bioscience research reporting: The ARRIVE guidelines for reporting animal research
Kilkenny, C., Browne, W. J., Cuthill, I. C., Emerson, M., & Altman, D. G. (2010) · Journal of Pharmacology and Pharmacotherapeutics · doi:10.4103/0976-500x.72351 · Cited by 6,277
Reliability and Validity Assessment
Carmines, E., & Zeller, R. (1979) · doi:10.4135/9781412985642 · Cited by 5,884 · cited 2×
“Finally, existing instruments (many of which were developed using stroke models) need to be tested on other animal models to ensure their rele vance and generaliz ability to other systems.”
CONSORT 2010 statement: Updated guidelines for reporting parallel group randomised trials
Schulz, K. F., Altman, D. G., & Moher, D. (2010) · Journal of Pharmacology and Pharmacotherapeutics · doi:10.4103/0976-500x.72352 · Cited by 2,165
The Hazards of Scoring the Quality of Clinical Trials for Meta-analysis
Jüni, P. (1999) · JAMA · doi:10.1001/jama.282.11.1054 · Cited by 1,981 · cited 2×
“Numerical methodological “quality” scores have been shown to be invalid for assessing risk of bias in clinical research (Jüni et al. 1999).”
Industry sponsorship and research outcome
Lundh, A., Lexchin, J., Mintzes, B., Schroll, J. B., & Bero, L. (2017) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.mr000033.pub3 · Cited by 1,697
Self-citationCan Animal Models of Disease Reliably Inform Human Studies?
van der Worp, H. B., Howells, D. W., Sena, E. S., Porritt, M. J., Rewell, S., O'Collins, V., & Macleod, M. R. (2010) · PLoS Medicine · doi:10.1371/journal.pmed.1000245 · Cited by 1,354 · cited 2×
“Timing of exposure also influences study outcome (Benatar 2007; van der Worp et al. 2010; Vesterinen et al. 2010), and some effects may be observed only for exposures that occur during certain develop mental periods (National Research Council 2009).”
Update of the Stroke Therapy Academic Industry Roundtable Preclinical Recommendations
Fisher, M., Feuerstein, G., Howells, D. W., Hurn, P. D., Kent, T. A., Savitz, S. I., & Lo, E. H. (2009) · Stroke · doi:10.1161/strokeaha.108.541128 · Cited by 1,352
A call for transparent reporting to optimize the predictive value of preclinical research
Landis, S. C., Amara, S. G., Asadullah, K., Austin, C. P., Blumenstein, R., Bradley, E. W., Crystal, R. G., Darnell, R. B., Ferrante, R. J., Fillit, H., et al. (2012) · Nature · doi:10.1038/nature11556 · Cited by 1,196
Guidelines for the Design and Statistical Analysis of Experiments Using Laboratory Animals
Festing, M. F. W., & Altman, D. G. (2002) · ILAR Journal · doi:10.1093/ilar.43.4.244 · Cited by 1,110 · cited 3×
“10 No None No General animal research Festing and Altman 2002 Developed based on consensus and citing past guidelines; derived from published guidelines for contributors to medical journals (Altman et al.”
Blinding in randomised trials: hiding who got what
Schulz, K. F., & Grimes, D. A. (2002) · The Lancet · doi:10.1016/s0140-6736(02)07816-9 · Cited by 952
Allocation concealment in randomised trials: defending against deciphering
Schulz, K. F., & Grimes, D. A. (2002) · The Lancet · doi:10.1016/s0140-6736(02)07750-4 · Cited by 923
Turner, L., Shamseer, L., Altman, D. G., Schulz, K. F., & Moher, D. (2012) · Systematic Reviews · doi:10.1186/2046-4053-1-60 · Cited by 628 · cited 2×
“Reporting of clinical research has improved because risk of bias assessments for system atic reviews and other purposes have become more prevalent and standards for reporting have been implemented by journals (Turner et al. 2012).”
Industry sponsorship and research outcome
Lundh, A., Sismondo, S., Lexchin, J., Busuioc, O. A., & Bero, L. (2012) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.mr000033.pub2 · Cited by 605
Self-citationGood Laboratory Practice: Preventing Introduction of Bias at the Bench
Macleod, M. R., Fisher, M., O'Collins, V., Sena, E. S., Dirnagl, U., Bath, P. M. W., Buchan, A., van der Worp, H. B., Traystman, R., Minematsu, K., et al. (2009) · Stroke · doi:10.1161/strokeaha.108.525386 · Cited by 331
Rising, K., Bacchetti, P., & Bero, L. (2008) · PLoS Medicine · doi:10.1371/journal.pmed.0050217 · Cited by 328 · cited 1×
“Selective reporting of outcomes and failure to publish entire studies is considered an important source of bias in clinical studies; however, little is known about the extent of this bias in animal research (Hart et al. 2012; Rising et al. 2008).”
Self-citationMacleod, M. R., van der Worp, H. B., Sena, E. S., Howells, D. W., Dirnagl, U., & Donnan, G. A. (2008) · Stroke · doi:10.1161/strokeaha.108.515957 · Cited by 319 · cited 2×
“…performance biases are relevant for all animal research, some of the pre clinical instruments contain criteria specific for assessing the qual ity of stroke research, such as the “avoidance of anesthetics with marked intrinsic neuro protective properties” (Macleod et al. 2008; Sena et al. 2007).”
Effect of reporting bias on meta-analyses of drug trials: reanalysis of meta-analyses
Hart, B., Lundh, A., & Bero, L. (2012) · BMJ · doi:10.1136/bmj.d7202 · Cited by 255 · cited 1×
“Selective reporting of outcomes and failure to publish entire studies is considered an important source of bias in clinical studies; however, little is known about the extent of this bias in animal research (Hart et al. 2012; Rising et al. 2008).”
Self-citationEmergency medicine animal research: does use of randomization and blinding affect the results?
Bebarta, V., Luyten, D., & Heard, K. (2003) · Academic Emergency Medicine · doi:10.1111/j.1553-2712.2003.tb00056.x · Cited by 244
Investigating patient exclusion bias in meta-analysis
Tierney, J. F. (2004) · International Journal of Epidemiology · doi:10.1093/ije/dyh300 · Cited by 221
Discordance between reported intention-to-treat and per protocol analyses
Porta, N., Bonet, C., & Cobo, E. (2007) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2006.09.013 · Cited by 189
Woodruff, T. J., Sutton, P., & The Navigation Guide Work Group (2011) · Health Affairs · doi:10.1377/hlthaff.2010.1219 · Cited by 134
Self-citation
Cited by
SYRCLE’s risk of bias tool for animal studies
Hooijmans, C. R., Rovers, M. M., de Vries, R. B., Leenaars, M., Ritskes-Hoitinga, M., & Langendam, M. W. (2014) · BMC Medical Research Methodology · doi:10.1186/1471-2288-14-43 · Cited by 4,415
“Discussion and conclusion In animal studies, a large variety of tools to assess study quality is currently used, but none of the tools identified so far focussed on internal validity only [11].”
BackgroundThe Mass Production of Redundant, Misleading, and Conflicted Systematic Reviews and Meta‐analyses
IOANNIDIS, J. P. A. (2016) · The Milbank Quarterly · doi:10.1111/1468-0009.12210 · Cited by 1,483
Johnson, P. I., Sutton, P., Atchley, D. S., Koustas, E., Lam, J., Sen, S., Robinson, K. A., Axelrad, D. A., & Woodruff, T. J. (2014) · Environmental Health Perspectives · doi:10.1289/ehp.1307893 · Cited by 449
“We also included the funding source and declared conflicts of interest as a potential source of bias based on empirical data from studies conducted on phar macological treatments that reported evidence of bias associated with funding source (Krauth et al. 2013; Lundh et al. 2012).”
MethodSelf-citationWoodruff, T. J., & Sutton, P. (2014) · Environmental Health Perspectives · doi:10.1289/ehp.1307175 · Cited by 427
“…are mostly composed of domains related to reporting requirements, such as compliance with regula tory requirements, description of the statistical model, and test animal details; importantly, they do not include all the risk of bias domains in use in human experimental studies (Krauth et al. 2013).”
BackgroundSelf-citationKoustas, E., Lam, J., Sutton, P., Johnson, P. I., Atchley, D. S., Sen, S., Robinson, K. A., Axelrad, D. A., & Woodruff, T. J. (2014) · Environmental Health Perspectives · doi:10.1289/ehp.1307177 · Cited by 349
“These findings underscore the urgency of calls for improved experimental-animal study design and reporting in the preclinical arena (Beronius et al. 2014; Krauth et al. 2013; Landis et al. 2012; van der Worp and Macleod 2011; Vesterinen et al. 2010, 2011).”
BackgroundSelf-citation
Metadata
- Authors
- Krauth, D., Woodruff, T. J., & Bero, L.
- Venue
- Environmental Health Perspectives
- Publisher
- American Chemical Society (ACS)
- Published
- 2013-09
- Volume
- 121
- Issue
- 9
- Pages
- 985-992
- ISSN
- 0091-6765 (ISSN-L)
- 1552-9924
- OpenAlex
- W2030520823
- Cited by
- 147
- References
- 23
- FWCI
- 4.18
- Type
- journal article
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.