Development of a quality assessment instrument for trials of treatments for depression and neurosis
Abstract
There is evidence that the quality of controlled clinical trials affects the results that are obtained. A systematic approach to the assessment of quality is required for psychiatric research. This study set out to develop an instrument for the assessment of the quality of controlled trials of interventions for depressive and non‐psychotic conditions. A pilot study led to the development of a scale containing 23 items covering a wide range of aspects of quality including objective formulation, design, presentation of results, analysis and quality of conclusions. Scoring criteria were devised and the scale was then subjected to reliability testing using a random sample of trials of treatment for depression and ‘neurosis’. The scale showed moderate inter‐rater reliability and results were comparable to those obtained with shorter instruments. It was quick and easy to use. There were significant correlations between year of publication and overall quality score with later studies showing higher quality. The instrument developed here provides a systematic approach to the assessment of quality for use in critical appraisal of individual studies and meta‐analysis. However, the scoring process should be used cautiously since inter‐rater agreement was only moderate. Copyright © 2001 Whurr Publishers Ltd.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Risk of bias within each study was assessed using an adapted Quality Rating Scale that included qualitative and uncontrolled studies.
“Risk of bias within each study was assessed using a version (Multimedia Appendix 2) of the Quality Rating Scale (QRS) [48] adapted to include qualitative and uncontrolled studies.”
From Do online mental health services improve help-seeking for young people? A systematic review (2014) · cites this paper as [48]
Cites
Assessing the quality of reports of randomized clinical trials: Is blinding necessary?
Jadad, A. R., Moore, R. A., Carroll, D., Jenkinson, C., Reynolds, D. J. M., Gavaghan, D. J., & McQuay, H. J. (1996) · Controlled Clinical Trials · doi:10.1016/0197-2456(95)00134-4 · Cited by 17,582 · see citing works
Schulz (1995) · doi:10.1001/jama.1995.03520290060030 · Cited by 5,634 · see citing works
Moher, D., Pham, B., Jones, A., Cook, D. J., Jadad, A. R., Moher, M., Tugwell, P., & Klassen, T. P. (1998) · The Lancet · doi:10.1016/s0140-6736(98)01085-x · Cited by 3,560 · see citing works
Improving the Quality of Reporting of Randomized Controlled Trials: The CONSORT Statement
Begg, C. (1996) · JAMA · doi:10.1001/jama.1996.03540080059030 · Cited by 3,127 · see citing works
Moher, D., Jadad, A. R., Nichol, G., Penman, M., Tugwell, P., & Walsh, S. (1995) · Controlled Clinical Trials · doi:10.1016/0197-2456(94)00031-w · Cited by 1,308 · see citing works
Bias in treatment assignment in controlled clinical trials
Chalmers, T. C., Celano, P., Sacks, H. S., & Smith, H. (1983) · New England Journal of Medicine · doi:10.1056/nejm198312013092204 · Cited by 864 · see citing works
How study design affects outcomes in comparisons of therapy. I: Medical
Colditz, G. A., Miller, J. N., & Mosteller, F. C. (1989) · Statistics in Medicine · doi:10.1002/sim.4780080408 · Cited by 520 · see citing works
Handbook of alcoholism treatment approaches
Edwards, G. (1990) · Behaviour Research and Therapy · doi:10.1016/0005-7967(90)90141-5 · Cited by 419 · see citing works
Bollini, P., Pampaliona, S., Tibaldi, G., Kupelnick, B., & Munizza, C. (1999) · British Journal of Psychiatry · doi:10.1192/bjp.174.4.297 · Cited by 274 · see citing works
Emerson, J. D., Burdick, E., Hoaglin, D. C., Mosteller, F. C., & Chalmers, T. C. (1990) · Controlled Clinical Trials · doi:10.1016/0197-2456(90)90175-2 · Cited by 241 · see citing works
Schulz, K. F., Grimes, D. A., Altman, D. G., & Hayes, R. J. (1996) · BMJ · doi:10.1136/bmj.312.7033.742 · Cited by 163 · see citing works
Moncrieff, J., & Drummond, C. (1998) · Addiction · doi:10.1046/j.1360-0443.1998.9368113.x · Cited by 27 · see citing works
Self-citation
Flagged first, then by citations
Cited by
Sorted by citations
Psychological debriefing for preventing post traumatic stress disorder (PTSD)
Rose, S. C., Bisson, J., Churchill, R., & Wessely, S. (2002) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd000560 · Cited by 1,250 · see citing works
Self-citationAalbers, S., Fusar-Poli, L., Freeman, R. E., Spreen, M., Ket, J. C., Vink, A. C., Maratos, A., Crawford, M., Chen, X.-J., & Gold, C. (2017) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd004517.pub3 · Cited by 838 · see citing works
Bloch, M. H., Landeros-Weisenberger, A., Kelmendi, B., Coric, V, Bracken, M. B., & Leckman, J. F. (2006) · Molecular Psychiatry · doi:10.1038/sj.mp.4001823 · Cited by 635 · see citing works
Psychological treatment of post-traumatic stress disorder (PTSD)
Bisson, J., & Andrew, M. (2007) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd003388.pub3 · Cited by 545 · see citing works
Pharmacotherapy for post traumatic stress disorder (PTSD)
Stein, D. J., Ipser, J. C., Seedat, S., Sager, C., & Amos, T. (2006) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd002795.pub2 · Cited by 519 · see citing works
Effects of psychotherapy for anxiety in children and adolescents: A meta-analytic review
Reynolds, S., Wilson, C., Austin, J., & Hooper, L. (2012) · Clinical Psychology Review · doi:10.1016/j.cpr.2012.01.005 · Cited by 483 · see citing works
Bloch, M. H., & Qawasmi, A. (2011) · Journal of the American Academy of Child & Adolescent Psychiatry · doi:10.1016/j.jaac.2011.06.008 · Cited by 417 · see citing works
Churchill, R., Hunot, V., Corney, R., Knapp, M., McGuire, H., Tylee, A., & Wessely, S. (2002) · Health Technology Assessment · doi:10.3310/hta5350 · Cited by 368 · see citing works
Self-citationRecommendations by Cochrane Review Groups for assessment of the risk of bias in studies
Lundh, A., & Gøtzsche, P. C. (2008) · BMC Medical Research Methodology · doi:10.1186/1471-2288-8-22 · Cited by 352 · see citing works
Active placebos versus antidepressants for depression
Moncrieff, J., Wessely, S., & Hardy, R. (2004) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd003012.pub2 · Cited by 342 · see citing works
Self-citationDo online mental health services improve help-seeking for young people? A systematic review
Kauer, S. D., Mangan, C., & Sanci, L. (2014) · Journal of Medical Internet Research · doi:10.2196/jmir.3103 · Cited by 339 · see citing works
A meta-analysis of behavior therapy for Tourette Syndrome
McGuire, J. F., Piacentini, J., Brennan, E. A., Lewin, A. B., Murphy, T. K., Small, B. J., & Storch, E. A. (2014) · Journal of Psychiatric Research · doi:10.1016/j.jpsychires.2013.12.009 · Cited by 334 · see citing works
Vigerland, S., Lenhard, F., Bonnert, M., Lalouni, M., Hedman, E., Ahlen, J., Olén, O., Serlachius, E., & Ljótsson, B. (2016) · Clinical Psychology Review · doi:10.1016/j.cpr.2016.09.005 · Cited by 292 · see citing works
Exercise for anxiety disorders: systematic review
Jayakody, K., Gunadasa, S., & Hosker, C. (2014) · British Journal of Sports Medicine · doi:10.1136/bjsports-2012-091287 · Cited by 310 · see citing works
Psychological therapies for generalised anxiety disorder
Hunot, V., Churchill, R., Teixeira, V., & Silva de Lima, M. (2007) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd001848.pub4 · Cited by 315 · see citing works
Self-citationPsychological treatments for irritable bowel syndrome: a systematic review and meta-analysis
Lackner, J. M., Mesmer, C., Morley, S., Dowzer, C., & Hamilton, S. (2004) · Journal of Consulting and Clinical Psychology · doi:10.1037/0022-006x.72.6.1100 · Cited by 295 · see citing works
Newer generation antidepressants for depressive disorders in children and adolescents
Hetrick, S. E., McKenzie, J. E., Cox, G. R., Simmons, M. B., & Merry, S. N. (2012) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd004851.pub3 · Cited by 291 · see citing works
Merry, S. N., McDowell, H. H., Hetrick, S. E., Bir, J. J., & Muller, N. (2004) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd003380.pub2 · Cited by 284 · see citing works
Crumlish, N., & O'Rourke, K. (2010) · Journal of Nervous & Mental Disease · doi:10.1097/nmd.0b013e3181d61258 · Cited by 264 · see citing works
A review of critical appraisal tools show they lack rigor: Alternative tool structure is proposed
Crowe, M., & Sheppard, L. (2011) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2010.02.008 · Cited by 261 · see citing works
Roberts, N. P., Kitchiner, N. J., Kenardy, J., & Bisson, J. I. (2009) · American Journal of Psychiatry · doi:10.1176/appi.ajp.2008.08040590 · Cited by 242 · see citing works
Leenarts, L. E. W., Diehle, J., Doreleijers, T. A. H., Jansma, E. P., & Lindauer, R. J. L. (2013) · European Child & Adolescent Psychiatry · doi:10.1007/s00787-012-0367-5 · Cited by 231 · see citing works
Ruhé, H. G., Huyser, J., Swinkels, J. A., & Schene, A. H. (2006) · The Journal of Clinical Psychiatry · doi:10.4088/jcp.v67n1203 · Cited by 230 · see citing works
Interventions for generalized anxiety disorder in older adults: Systematic review and meta-analysis
Gonçalves, D. C., & Byrne, G. J. (2012) · Journal of Anxiety Disorders · doi:10.1016/j.janxdis.2011.08.010 · Cited by 227 · see citing works
Roberts, N. P., Kitchiner, N. J., Kenardy, J., & Bisson, J. I. (2009) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd006869.pub2 · Cited by 218 · see citing works
Psychotherapy mediated by remote communication technologies: a meta-analytic review
Bee, P. E., Bower, P., Lovell, K., Gilbody, S., Richards, D., Gask, L., & Roach, P. (2008) · BMC Psychiatry · doi:10.1186/1471-244x-8-60 · Cited by 189 · see citing works
“The CCDAN quality rating scale [39] scores each study according to 23 elements of design and conduct including randomisation methods, sample size, follow-up period, power and appropriateness of analysis.”
MethodPsychotherapeutic treatments for older depressed people
Wilson, K., Mottram, P. G., & Vassilas, C. (2008) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd004853.pub2 · Cited by 224 · see citing works
Selective serotonin reuptake inhibitors (SSRIs) for depressive disorders in children and adolescents
Hetrick, S. E., Merry, S. N., McKenzie, J., Sindahl, P., & Proctor, M. (2007) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd004851.pub2 · Cited by 180 · see citing works
Intolerance of uncertainty, anxiety, and worry in children and adolescents: A meta-analysis
Osmanağaoğlu, N., Creswell, C., & Dodd, H. F. (2018) · Journal of Affective Disorders · doi:10.1016/j.jad.2017.07.035 · Cited by 177 · see citing works
Hall, J., Kellett, S., Berrios, R., Bains, M. K., & Scott, S. (2016) · The American Journal of Geriatric Psychiatry · doi:10.1016/j.jagp.2016.06.006 · Cited by 157 · see citing works
McGuire, J. F., Ung, D., Selles, R. R., Rahman, O., Lewin, A. B., Murphy, T. K., & Storch, E. A. (2014) · Journal of Psychiatric Research · doi:10.1016/j.jpsychires.2014.07.015 · Cited by 142 · see citing works
Bortolotti, B., Menchetti, M., Bellini, F., Montaguti, M. B., & Berardi, D. (2008) · General Hospital Psychiatry · doi:10.1016/j.genhosppsych.2008.04.001 · Cited by 136 · see citing works
Warwick, H., Reardon, T., Cooper, P., Murayama, K., Reynolds, S., Wilson, C., & Creswell, C. (2017) · Clinical Psychology Review · doi:10.1016/j.cpr.2016.12.002 · Cited by 130 · see citing works
Psychotherapies for hypochondriasis
Thomson, A., & Page, L. (2007) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd006520.pub2 · Cited by 141 · see citing works
Galloway, R., Watson, H., Greene, D., Shafran, R., & Egan, S. J. (2022) · Cognitive Behaviour Therapy · doi:10.1080/16506073.2021.1952302 · Cited by 115 · see citing works
Jónsson, H., & Hougaard, E. (2009) · Acta Psychiatrica Scandinavica · doi:10.1111/j.1600-0447.2008.01270.x · Cited by 110 · see citing works
Stephens, S., Ford, E., Paudyal, P., & Smith, H. (2016) · The Annals of Family Medicine · doi:10.1370/afm.1967 · Cited by 109 · see citing works
Taylor, M. J., Rudkin, L., & Hawton, K. (2005) · Journal of Affective Disorders · doi:10.1016/j.jad.2005.07.006 · Cited by 102 · see citing works
Romano, M., & Peters, L. (2015) · Clinical Psychology Review · doi:10.1016/j.cpr.2015.02.008 · Cited by 95 · see citing works
Selles, R. R., McGuire, J. F., Small, B. J., & Storch, E. A. (2016) · General Hospital Psychiatry · doi:10.1016/j.genhosppsych.2016.04.001 · Cited by 80 · see citing works
Exercise therapy for chronic fatigue syndrome
Larun, L., Brurberg, K. G., Odgaard-Jensen, J., & Price, J. R. (2015) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd003200.pub3 · Cited by 69 · see citing works
Armijo-Olivo, S., Fuentes, J., Ospina, M., Saltaji, H., & Hartling, L. (2013) · BMC Medical Research Methodology · doi:10.1186/1471-2288-13-116 · Cited by 60 · see citing works
Pre-treatment Predictors of Outcome in Childhood Anxiety Disorders: A Systematic Review
Knight, A., McLellan, L., Jones, M., & Hudson, J. (2014) · Psychopathology Review · doi:10.5127/pr.034613 · Cited by 57 · see citing works
Marques, M. M., De Gucht, V., Gouveia, M. J., Leal, I., & Maes, S. (2015) · Clinical Psychology Review · doi:10.1016/j.cpr.2015.05.009 · Cited by 50 · see citing works
The effects of psychotherapy on reducing depression in residential aged care: a meta-analytic review
Cody, R. A., & Drysdale, K. (2013) · Clinical Gerontologist · doi:10.1080/07317115.2012.731474 · Cited by 43 · see citing works
Esfandiari, N., Mazaheri, M. A., Akbari-Zardkhaneh, S., Sadeghi-Firoozabadi, V., & Cheraghi, M. (2021) · International Journal of Preventive Medicine · doi:10.4103/ijpvm.ijpvm_208_21 · Cited by 45 · see citing works
Randomized Controlled Trial of an Early Intervention Programme for Adolescent Anxiety Disorders
Hunt, C., Andrews, G., Sakashita, C., Crino, R., & Erskine, A. (2009) · Australian & New Zealand Journal of Psychiatry · doi:10.1080/00048670902721152 · Cited by 41 · see citing works
Roberts, N. P., Kitchiner, N. J., Kenardy, J., & Bisson, J. (2009) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd007944 · Cited by 34 · see citing works
- Authors
- Moncrieff, J., Churchill, R., Drummond, D. C., & McGuire, H.
- Affiliations
- University College London (GB)
- Corresponding author
- Moncrieff, Joanna
- Venue
- International Journal of Methods in Psychiatric Research
- Publisher
- Wiley
- Published
- 2001-08
- Volume
- 10
- Issue
- 3
- Pages
- 126-133
- ISSN
- 1049-8931 (ISSN-L)
- 1557-0657
- OpenAlex
- W2015709747
- Cited by
- 185
- References
- 12
- Citations by year
- 2026: 4 · 2025: 2 · 2024: 4 · 2023: 2 · 2022: 3 · 2021: 5 · 2020: 4 · 2019: 4 · 2018: 4 · 2017: 7 · 2016: 13 · 2015: 7 · 2014: 9 · 2013: 6 · 2012: 9
- FWCI
- 12.35
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- International Journal of Methods in Psychiatric Research (publishedVersion)
- UCL Discovery (University College London) (submittedVersion)
- Research Portal (King's College London) (submittedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Psychology Research and Bibliometrics
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Clinical Trials
- Keywords
- quality assessment tools, depression, neurosis, psychiatric research, inter-rater reliability, critical appraisal, meta-analysis
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
