Evaluation and validation of social and psychological markers in randomised trials of complex interventions in mental health: a methodological research programme
Open accessThe DOI matches its Crossref record; 30 of the 30 references it cites were found in a registry or an index.
Abstract
Background: The development of the capability and capacity to evaluate the outcomes of trials of complex interventions is a key priority of the National Institute for Health Research (NIHR) and the Medical Research Council (MRC). The evaluation of complex treatment programmes for mental illness (e.g. cognitive–behavioural therapy for depression or psychosis) not only is a vital component of this research in its own right but also provides a well-established model for the evaluation of complex interventions in other clinical areas. In the context of efficacy and mechanism evaluation (EME) there is a particular need for robust methods for making valid causal inference in explanatory analyses of the mechanisms of treatment-induced change in clinical outcomes in randomised clinical trials. Objectives: The key objective was to produce statistical methods to enable trial investigators to make valid causal inferences about the mechanisms of treatment-induced change in these clinical outcomes. The primary objective of this report is to disseminate this methodology, aiming specifically at trial practitioners. Methods: The three components of the research were (1) the extension of instrumental variable (IV) methods to latent growth curve models and growth mixture models for repeated-measures data; (2) the development of designs and regression methods for parallel trials; and (3) the evaluation of the sensitivity/robustness of findings to the assumptions necessary for model identifiability. We illustrate our methods with applications from psychological and psychosocial intervention trials, keeping the technical details to a minimum, leaving the reporting of the more theoretical and mathematically demanding results for publication in appropriate specialist journals. Results: We show how to estimate treatment effects and introduce methods for EME. We explain the use of IV methods and principal stratification to evaluate the role of putative treatment effect mediators and therapeutic process measures. These results are extended to the analysis of longitudinal data structures. We consider the design of EME trials. We focus on designs to create convincing IVs, bearing in mind assumptions needed to attain model identifiability. A key area of application that has become apparent during this work is the potential role of treatment moderators (predictive markers) in the evaluation of treatment effect mechanisms for personalised therapies (stratified medicine). We consider the role of targeted therapies and multiarm trials and the use of parallel trials to help elucidate the evaluation of mediators working in parallel. Conclusions: In order to demonstrate both efficacy and mechanism, it is necessary to (1) demonstrate a treatment effect on the primary (clinical) outcome, (2) demonstrate a treatment effect on the putative mediator (mechanism) and (3) demonstrate a causal effect from the mediator to the outcome. Appropriate regression models should be applied for (3) or alternative IV procedures, which account for unmeasured confounding, provided that a valid instrument can be identified. Stratified medicine may provide a setting where such instruments can be designed into the trial. This work could be extended by considering improved trial designs, sample size considerations and measurement properties. Funding: The project presents independent research funded under the MRC–NIHR Methodology Research Programme (grant reference G0900678).
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Baron, R. M., & Kenny, D. A. (1986) · Journal of Personality and Social Psychology · doi:10.1037/0022-3514.51.6.1173 · Cited by 91,532
Causality: Models, Reasoning, and Inference
Pearl, J. (2009) · doi:10.1017/cbo9780511803161 · Cited by 11,653
Estimating causal effects of treatments in randomized and nonrandomized studies
Rubin, D. B. (1974) · Journal of Educational Psychology · doi:10.1037/h0037350 · Cited by 9,436
Statistical Analysis with Missing Data
Little, R. J. A., & Rubin, D. B. (2002) · Wiley Series in Probability and Statistics · doi:10.1002/9781119013563 · Cited by 8,473
Davey Smith, G., & Ebrahim, S. (2003) · International Journal of Epidemiology · doi:10.1093/ije/dyg070 · Cited by 5,882
Introduction to Statistical Mediation Analysis
MacKinnon, D. P. (2012) · doi:10.4324/9780203809556 · Cited by 5,618
Mendelian randomization: Using genes as instruments for making causal inferences in epidemiology
Lawlor, D. A., Harbord, R. M., Sterne, J. A. C., Timpson, N., & Davey Smith, G. (2008) · Statistics in Medicine · doi:10.1002/sim.3034 · Cited by 4,618
Identification of Causal Effects Using Instrumental Variables
Angrist, J. D., Imbens, G. W., & Rubin, D. B. (1996) · Journal of the American Statistical Association · doi:10.1080/01621459.1996.10476902 · Cited by 4,230
Avoiding bias from weak instruments in Mendelian randomization studies
Burgess, S., Thompson, S. G., & CRP CHD Genetics Collaboration (2011) · International Journal of Epidemiology · doi:10.1093/ije/dyr036 · Cited by 3,952
Mediators and Moderators of Treatment Effects in Randomized Clinical Trials
Kraemer, H. C., Wilson, G. T., Fairburn, C. G., & Agras, W. S. (2002) · Archives of General Psychiatry · doi:10.1001/archpsyc.59.10.877 · Cited by 2,562
Spencer, S. J., Zanna, M. P., & Fong, G. T. (2005) · Journal of Personality and Social Psychology · doi:10.1037/0022-3514.89.6.845 · Cited by 2,452
Process Analysis: Estimating Mediation in Treatment Evaluations
Judd, C. M., & Kenny, D. A. (1981) · Evaluation Review · doi:10.1177/0193841x8100500502 · Cited by 2,442
Latent Variable Modeling of Differences and Changes with Longitudinal Data
McArdle, J. J. (2009) · Annual Review of Psychology · doi:10.1146/annurev.psych.60.110707.163612 · Cited by 1,952
A cognitive model of the positive symptoms of psychosis
GARETY, P. A., KUIPERS, E., FOWLER, D., FREEMAN, D., & BEBBINGTON, P. E. (2001) · Psychological Medicine · doi:10.1017/s0033291701003312 · Cited by 1,546
IMAI, K., KEELE, L., TINGLEY, D., & YAMAMOTO, T. (2011) · American Political Science Review · doi:10.1017/s0003055411000414 · Cited by 1,537
Yes, but what’s the mechanism? (don’t expect an easy answer).
Bullock, J. G., Green, D. P., & Ha, S. E. (2010) · Journal of Personality and Social Psychology · doi:10.1037/a0018933 · Cited by 1,063
Bruce, M. L., Ten Have, T. R., Reynolds III, C. F., Katz, I. I., Schulberg, H. C., Mulsant, B. H., Brown, G. K., McAvay, G. J., Pearson, J. L., & Alexopoulos, G. S. (2004) · JAMA · doi:10.1001/jama.291.9.1081 · Cited by 963
Mendelian randomization as an instrumental variable approach to causal inference
Didelez, V., & Sheehan, N. (2007) · Statistical Methods in Medical Research · doi:10.1177/0962280206077743 · Cited by 923
Green, J., Charman, T., McConachie, H., Aldred, C., Slonims, V., Howlin, P., Le Couteur, A., Leadbitter, K., Hudry, K., Byford, S., et al. (2010) · The Lancet · doi:10.1016/s0140-6736(10)60587-9 · Cited by 857
Self-citationSuspicious minds: The psychology of persecutory delusions
Freeman, D. (2007) · Clinical Psychology Review · doi:10.1016/j.cpr.2006.10.004 · Cited by 814
Odds Ratios for Mediation Analysis for a Dichotomous Outcome
VanderWeele, T. J., & Vansteelandt, S. (2010) · American Journal of Epidemiology · doi:10.1093/aje/kwq332 · Cited by 809
Cognitive approaches to delusions: a critical review of theories and evidence.
Garety, P. A., & Freeman, D. (1999) · British Journal of Clinical Psychology · doi:10.1348/014466599162700 · Cited by 794
Bounds on Treatment Effects from Studies with Imperfect Compliance
Balke, A., & Pearl, J. (1997) · Journal of the American Statistical Association · doi:10.1080/01621459.1997.10474074 · Cited by 564
Psychometric characteristics of the Scale for Suicide Ideation with psychiatric outpatients
Beck, A. T., Brown, G. K., & Steer, R. A. (1997) · Behaviour Research and Therapy · doi:10.1016/s0005-7967(97)00073-9 · Cited by 538
Bias Formulas for Sensitivity Analysis for Direct and Indirect Effects
VanderWeele, T. J. (2010) · Epidemiology · doi:10.1097/ede.0b013e3181df191c · Cited by 435
Bias in causal estimates from Mendelian randomization studies with weak instruments
Burgess, S., & Thompson, S. G. (2011) · Statistics in Medicine · doi:10.1002/sim.4197 · Cited by 421
Experimental Designs for Identifying Causal Mechanisms
Imai, K., Tingley, D., & Yamamoto, T. (2013) · Journal of the Royal Statistical Society Series A: Statistics in Society · doi:10.1111/j.1467-985x.2012.01032.x · Cited by 408
Emsley, R., Dunn, G., & White, I. R. (2010) · Statistical Methods in Medical Research · doi:10.1177/0962280209105014 · Cited by 356
Self-citationGarety, P. A., Fowler, D. G., Freeman, D., Bebbington, P., Dunn, G., & Kuipers, E. (2008) · British Journal of Psychiatry · doi:10.1192/bjp.bp.107.043570 · Cited by 296
Self-citationA review of causal estimation of effects in mediation analyses
Ten Have, T. R., & Joffe, M. M. (2012) · Statistical Methods in Medical Research · doi:10.1177/0962280210391076 · Cited by 103
Cited by
Sleep disturbance and psychiatric disorders
Freeman, D., Sheaves, B., Waite, F., Harvey, A. G., & Harrison, P. J. (2020) · The Lancet Psychiatry · doi:10.1016/s2215-0366(20)30136-x · Cited by 843
Unbiased Scene Graph Generation From Biased Training
Tang, K., Niu, Y., Huang, J., Shi, J., & Zhang, H. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00377 · Cited by 755
Freeman, D., Sheaves, B., Goodwin, G. M., Yu, L.-M., Nickless, A., Harrison, P. J., Emsley, R., Luik, A. I., Foster, R. G., Wadekar, V., et al. (2017) · The Lancet Psychiatry · doi:10.1016/s2215-0366(17)30328-0 · Cited by 652
Self-citationThe Lancet Psychiatry Commission on psychological treatments research in tomorrow's science
Holmes, E. A., Ghaderi, A., Harmer, C. J., Ramchandani, P. G., Cuijpers, P., Morrison, A. P., Roiser, J. P., Bockting, C. L. H., O'Connor, R. C., Shafran, R., et al. (2018) · The Lancet Psychiatry · doi:10.1016/s2215-0366(17)30513-8 · Cited by 624
Espie, C. A., Emsley, R., Kyle, S. D., Gordon, C., Drake, C. L., Siriwardena, A. N., Cape, J., Ong, J. C., Sheaves, B., Foster, R., et al. (2019) · JAMA Psychiatry · doi:10.1001/jamapsychiatry.2018.2745 · Cited by 466
Self-citation
Metadata
- Authors
- Dunn, G., Emsley, R., Liu, H., Landau, S., Green, J., White, I., & Pickles, A.
- Venue
- Health Technology Assessment
- Published
- 2015-11
- Volume
- 19
- Issue
- 93
- Pages
- 1-116
- ISSN
- 1366-5278 (ISSN-L)
- 2046-4924
- OpenAlex
- W2175831798
- Cited by
- 122
- References
- 30
- FWCI
- 8.76
- Type
- journal article
- Language
- en
- Open access
- diamond
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.