Analyzing multiple outcomes in clinical research using multivariate multilevel models.

Baldwin, Scott A.; Imel, Zac E.; Braithwaite, Scott R.; Atkins, David C.

doi:10.1037/a0035628

Cited by 167Statistical Methods and Bayesian InferenceMental Health Research TopicsStatistical Methods in Clinical Trials

Analyzing multiple outcomes in clinical research using multivariate multilevel models.

Open access

Baldwin, Scott A., Imel, Zac E., Braithwaite, Scott R., Atkins, David C.·2014·Journal of Consulting and Clinical Psychology

DOI 10.1037/a0035628PMC PMC4119868

Open this paper in CoChat — free account

Abstract

OBJECTIVE: Multilevel models have become a standard data analysis approach in intervention research. Although the vast majority of intervention studies involve multiple outcome measures, few studies use multivariate analysis methods. The authors discuss multivariate extensions to the multilevel model that can be used by psychotherapy researchers. METHOD AND RESULTS: Using simulated longitudinal treatment data, the authors show how multivariate models extend common univariate growth models and how the multivariate model can be used to examine multivariate hypotheses involving fixed effects (e.g., does the size of the treatment effect differ across outcomes?) and random effects (e.g., is change in one outcome related to change in the other?). An online supplemental appendix provides annotated computer code and simulated example data for implementing a multivariate model. CONCLUSIONS: Multivariate multilevel models are flexible, powerful models that can enhance clinical research.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

Integrity signals

Registry

Crossref · 10.1037/a0035628

Cites

39 of 39 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Regularization Paths for Generalized Linear Models via Coordinate Descent
Friedman, J., Hastie, T., & Tibshirani, R. (2010) · Journal of Statistical Software · doi:10.18637/jss.v033.i01 · Cited by 19,927 · see citing works
Hierarchical Linear Models: Applications and Data Analysis Methods.
Longford, N. T., Bryk, A. S., & Raudenbush, S. W. (1993) · Contemporary Sociology A Journal of Reviews · doi:10.2307/2075823 · Cited by 18,768 · see citing works
Data Analysis Using Regression and Multilevel/Hierarchical Models
Gelman, A., & Hill, J. (2006) · doi:10.1017/cbo9780511790942 · Cited by 13,597 · see citing works
Applied Longitudinal Data Analysis: Modeling Change and Event Occurrence
Singer, J. D., & Willett, J. B. (2003) · doi:10.1093/acprof:oso/9780195152968.001.0001 · Cited by 10,647 · see citing works
MCMC Methods for Multi-Response Generalized Linear Mixed Models: The MCMCglmm R Package
Hadfield, J. D. (2010) · Journal of Statistical Software · doi:10.18637/jss.v033.i02 · Cited by 4,900 · see citing works
Latent Curve Models
Bollen, K. A., & Curran, P. J. (2005) · Wiley series in probability and statistics · doi:10.1002/0471746096 · Cited by 1,731 · see citing works
Conceptualizing and testing random indirect effects and moderated mediation in multilevel models: New procedures and recommendations.
Bauer, D. J., Preacher, K. J., & Gil, K. M. (2006) · Psychological Methods · doi:10.1037/1082-989x.11.2.142 · Cited by 1,704 · see citing works
Generalized Latent Variable Modeling: Multilevel, Longitudinal, and Structural Equation Models
Skrondal, A., & Rabe‐Hesketh, S. (2004) · Thammasat University Digital Collections · Cited by 1,505 · see citing works
Erroneous analyses of interactions in neuroscience: A problem of significance
Nieuwenhuis, S., Forstmann, B. U., & Wagenmakers, E.-J. (2011) · Nature Neuroscience · doi:10.1038/nn.2886 · Cited by 864 · see citing works
Reliable Estimation of Generalized Linear Mixed Models using Adaptive Quadrature
Rabe‐Hesketh, S., Skrondal, A., & Pickles, A. R. (2002) · The Stata Journal Promoting communications on statistics and Stata · doi:10.1177/1536867x0200200101 · Cited by 614 · see citing works
People are variables too: Multilevel structural equations modeling.
Mehta, P. D., & Neale, M. C. (2005) · Psychological Methods · doi:10.1037/1082-989x.10.3.259 · Cited by 462 · see citing works
Meta‐Analysis of Therapist Effects in Psychotherapy Outcome Studies
Crits-Christoph, P. F., Baranackie, K., Kurcias, J., Beck, A. T., Carroll, K. M., Perry, K. J., Luborsky, L. B., McLellan, A. T., WOODY, G. E., Thompson, L. W., et al. (1991) · Psychotherapy Research · doi:10.1080/10503309112331335511 · Cited by 433 · see citing works
Accuracy in interpersonal perception: A social relations analysis.
Kenny, D. A., & Albright, L. (1987) · Psychological Bulletin · doi:10.1037/0033-2909.102.3.390 · Cited by 417 · see citing works
Randomized Trial of Cognitive-Behavioral Therapy for Chronic Posttraumatic Stress Disorder in Adult Female Survivors of Childhood Sexual Abuse.
McDonagh, A., Friedman, M. J., McHugo, G. J., Ford, J. D., Sengupta, A., Mueser, K. T., Demment, C. C., Fournier, D. A., Schnurr, P. P., & Descamps, M. J. (2005) · Journal of Consulting and Clinical Psychology · doi:10.1037/0022-006x.73.3.515 · Cited by 404 · see citing works
Multivariate Random-effects Meta-regression: Updates to Mvmeta
White, I. R. (2011) · The Stata Journal: Promoting communications on statistics and Stata · doi:10.1177/1536867x1101100206 · Cited by 400 · see citing works
Have Multilevel Models Been Structural Equation Models All Along?
Curran, P. J. (2003) · Multivariate Behavioral Research · doi:10.1207/s15327906mbr3804_5 · Cited by 384 · see citing works
Studying Multivariate Change Using Multilevel Models and Latent Curve Models
MacCallum, R. C., Kim, C., Malarkey, W. B., & Kiecolt‐Glaser, J. K. (1997) · Multivariate Behavioral Research · doi:10.1207/s15327906mbr3203_1 · Cited by 367 · see citing works
Implications of therapist effects for the design and analysis of comparative studies of psychotherapies.
Crits-Christoph, P. F., & Mintz, J. (1991) · Journal of Consulting and Clinical Psychology · doi:10.1037/0022-006x.59.1.20 · Cited by 352 · see citing works
Using Multilevel Models to Analyze Couple and Family Treatment Data: Basic and Advanced Issues.
Atkins, D. C. (2005) · Journal of Family Psychology · doi:10.1037/0893-3200.19.1.98 · Cited by 276 · see citing works
Self-citation
Bivariate random-effects meta-analysis and the estimation of between-study correlation
Riley, R. D., Abrams, K. R., Sutton, A. J., Lambert, P. C., & Thompson, J. R. (2007) · BMC Medical Research Methodology · doi:10.1186/1471-2288-7-3 · Cited by 243 · see citing works
Estimating Multilevel Linear Models as Structural Equation Models
Bauer, D. J. (2003) · Journal of Educational and Behavioral Statistics · doi:10.3102/10769986028002135 · Cited by 237 · see citing works
Empirically supported treatments or type I errors? Problems with the analysis of data from group-administered treatments.
Baldwin, S. A., Murray, D. M., & Shadish, W. R. (2005) · Journal of Consulting and Clinical Psychology · doi:10.1037/0022-006x.73.5.924 · Cited by 209 · see citing works
Self-citation
The consequence of ignoring a nested factor on measures of effect size in analysis of variance.
Wampold, B. E., & Serlin, R. C. (2000) · Psychological Methods · doi:10.1037/1082-989x.5.4.425 · Cited by 186 · see citing works
On the power of multivariate latent growth curve models to detect correlated change.
Hertzog, C., Lindenberger, U., Ghisletta, P., & von Oertzen, T. (2006) · Psychological Methods · doi:10.1037/1082-989x.11.3.244 · Cited by 174 · see citing works
Evaluating Group-Based Interventions When Control Participants Are Ungrouped
Bauer, D. J., Sterba, S. K., & Hallfors, D. D. (2008) · Multivariate Behavioral Research · doi:10.1080/00273170802034810 · Cited by 165 · see citing works
Patterns of therapist variability: Therapist effects and the contribution of patient severity and risk.
Saxon, D. H., & Barkham, M. (2012) · Journal of Consulting and Clinical Psychology · doi:10.1037/a0028898 · Cited by 163 · see citing works
Bayesian methods for the analysis of small sample multilevel data with a complex variance structure.
Baldwin, S., & Fellingham, G. (2013) · Psychological methods · doi:10.1037/a0030642 · Cited by 154 · see citing works
Self-citation
Reducing conduct problems among children exposed to intimate partner violence: A randomized clinical trial examining effects of Project Support.
Jouriles, E. N., McDonald, R., Rosenfield, D., Stephens, N. S., Corbitt‐Shindler, D., & Miller, P. C. (2009) · Journal of Consulting and Clinical Psychology · doi:10.1037/a0015994 · Cited by 144 · see citing works
Evaluating models for partially clustered designs.
Baldwin, S. A., Bauer, D. J., Stice, E. M., & Rohde, P. (2011) · Psychological Methods · doi:10.1037/a0023464 · Cited by 122 · see citing works
Self-citation
Joint modelling of multivariate longitudinal profiles: pitfalls of the random‐effects approach
Fieuws, S., & Verbeke, G. N. (2004) · Statistics in Medicine · doi:10.1002/sim.1885 · Cited by 120 · see citing works
Beyond the individual: Group effects in mindfulness-based stress reduction
Imel, Z., Baldwin, S., Bonus, K., & MacCoon, D. (2008) · Psychotherapy Research · doi:10.1080/10503300802326038 · Cited by 119 · see citing works
Self-citation
Implications of therapist effects for the design and analysis of comparative studies of psychotherapies.
Crits-Christoph, P. F., & Mintz, J. (1991) · Journal of Consulting and Clinical Psychology · doi:10.1037//0022-006x.59.1.20 · Cited by 113 · see citing works
Estimating Parameters of Dichotomous and Ordinal Item Response Models with Gllamm
Zheng, X., & Rabe‐Hesketh, S. (2007) · The Stata Journal Promoting communications on statistics and Stata · doi:10.1177/1536867x0700700302 · Cited by 104 · see citing works
Studying psychotherapy using the one-with-many design: The therapeutic alliance as an exemplar.
Marcus, D. K., Kashy, D. A., & Baldwin, S. A. (2009) · Journal of Counseling Psychology · doi:10.1037/a0017291 · Cited by 88 · see citing works
Self-citation
Alcohol Use, Problems, and the Course of Posttraumatic Stress Disorder: A Prospective Study of Female Crime Victims
Kaysen, D. L., Atkins, D. C., Moore, S. A., Lindgren, K. P., Dillworth, T. M., & Simpson, T. L. (2011) · Journal of Dual Diagnosis · doi:10.1080/15504263.2011.620449 · Cited by 78 · see citing works
Self-citation
The therapeutic alliance in adolescent substance abuse treatment: A one-with-many analysis.
Marcus, D. K., Kashy, D. A., Wintersteen, M. B., & Diamond, G. S. (2011) · Journal of Counseling Psychology · doi:10.1037/a0023196 · Cited by 46 · see citing works
Patient-rated alliance as a measure of therapist performance in two clinical settings.
Imel, Z. E., Hubbard, R. A., Rutter, C. M., & Simon, G. E. (2012) · Journal of Consulting and Clinical Psychology · doi:10.1037/a0030903 · Cited by 19 · see citing works
Self-citation
Multilevel regression analyses to investigate the relationship between two variables over time: Examining the longitudinal association between intrusion and avoidance
Suvak, M. K., Walling, S. M., Iverson, K. M., Taft, C. T., & Resick, P. A. (2009) · Journal of Traumatic Stress · doi:10.1002/jts.20476 · Cited by 15 · see citing works
To kill a dodo bird.
Hofmann, S. G., & Lohr, J. M. (2010) · Cited by 6 · see citing works

Flagged first, then by citations

1 of the 40 references couldn’t be matched to a known paper.

Research works printed but not linked: 4

Baldwin SA Imel ZE Lambert MJ 2013 Therapist effects: Findings and methods Bergin and Garfield’s Handbook of Psychotherapy and Behavior Change 6 258 297 New York Wiley
Hox JJ 2010 Multilevel analysis: Techniques and applications 2 New York Routledge
Kenny DA 1994 Interpersonal perception: A social relations analysis New York Guilford 3317468
Rabe-Hesketh S Skrondal A 2008 Multilevel and longitudinal modeling using Stata College Station, TX Stata Press

Sorted by citations

The association of therapist empathy and synchrony in vocally encoded arousal.
Imel, Z. E., Barco, J. S., Brown, H. J., Baucom, B. R., Baer, J. S., Kircher, J. C., & Atkins, D. C. (2014) · Journal of Counseling Psychology · doi:10.1037/a0034943 · Cited by 159 · see citing works
Self-citation
Maternal involvement and children’s academic motivation and achievement: The roles of maternal autonomy support and children’s affect
Lerner, R. E., & Grolnick, W. S. (2019) · Motivation and Emotion · doi:10.1007/s11031-019-09813-6 · Cited by 127 · see citing works
The Efficacy of Vigorous-Intensity Exercise as an Aid to Smoking Cessation in Adults With High Anxiety Sensitivity: A Randomized Controlled Trial
Smits, J. A. J., Zvolensky, M. J., Davis, M. L., Rosenfield, D., Marcus, B. H., Church, T. S., Powers, M. B., Frierson, G. M., Otto, M. W., Hopkins, L. B., et al. (2016) · Psychosomatic Medicine · doi:10.1097/psy.0000000000000264 · Cited by 103 · see citing works
Student and Teacher Perceptions of Goal Attainment During Intervention With the Self-Determined Learning Model of Instruction
Shogren, K. A., Hicks, T. A., Raley, S. K., Pace, J. R., Rifenbark, G. G., & Lane, K. L. (2020) · The Journal of Special Education · doi:10.1177/0022466920950264 · Cited by 103 · see citing works
The therapeutic alliance as a predictor of outcome in dialectical behavior therapy versus nonbehavioral psychotherapy by experts for borderline personality disorder.
Bedics, J. D., Atkins, D. C., Harned, M. S., & Linehan, M. M. (2015) · Psychotherapy · doi:10.1037/a0038457 · Cited by 88 · see citing works
Self-citation
Changes in diet, activity, weight, and wellbeing of parents during COVID-19 lockdown
Curtis, R., Olds, T., Ferguson, T., Fraysse, F., Dumuid, D., Esterman, A., Hendrie, G. A., Brown, W. J., Lagiseti, R., & Maher, C. A. (2021) · PLoS ONE · doi:10.1371/journal.pone.0248008 · Cited by 67 · see citing works
The exploratory value of cross-sectional partial correlation networks: Predicting relationships between change trajectories in borderline personality disorder
von Klipstein, L., Borsboom, D., & Arntz, A. R. (2021) · PLoS ONE · doi:10.1371/journal.pone.0254496 · Cited by 65 · see citing works
Time-series panel analysis (TSPA): Multivariate modeling of temporal associations in psychotherapy process.
Ramseyer, F., Kupper, Z., Caspar, F., Znoj, H., & Tschacher, W. (2014) · Journal of Consulting and Clinical Psychology · doi:10.1037/a0037168 · Cited by 65 · see citing works
Intimate relationships, individual adjustment, and coronary heart disease: Implications of overlapping associations in psychosocial risk.
Smith, T. W., & Baucom, B. (2017) · American Psychologist · doi:10.1037/amp0000123 · Cited by 62 · see citing works
Impact of attention‐deficit/hyperactivity disorder comorbidity on longitudinal course in Internet gaming disorder: a 3‐year clinical cohort study
Lee, J., Bae, S., Kim, B.-N., & Han, D. H. (2021) · Journal of Child Psychology and Psychiatry · doi:10.1111/jcpp.13380 · Cited by 57 · see citing works
Mindfulness and progressive muscle relaxation as standardized session‐introduction in individual therapy: A randomized controlled trial
Mander, J., Blanck, P., Neubauer, A. B., Kröger, P., Flückiger, C., Lutz, W., Barnow, S., Bents, H., & Heidenreich, T. (2019) · Journal of Clinical Psychology · doi:10.1002/jclp.22695 · Cited by 56 · see citing works
“We used a multivariate MLM to simultaneously model a change in alliance ratings from the patient and therapist perspectives (Baldwin et al., 2014; Mander et al., 2017b).”
Method
The Hitchhiker’s guide to longitudinal models: A primer on model selection for repeated-measures methods
McCormick, E. M., Byrne, M. L., Flournoy, J. C., Mills, K. L., & Pfeifer, J. H. (2023) · Developmental Cognitive Neuroscience · doi:10.1016/j.dcn.2023.101281 · Cited by 54 · see citing works
Impact of physician empathy on patient outcomes: a gender analysis
Surchat, C., Carrard, V., Gaume, J., Berney, A., & Clair, C. (2021) · British Journal of General Practice · doi:10.3399/bjgp.2021.0193 · Cited by 51 · see citing works
Enhancing the quality of cognitive behavioral therapy in community mental health through artificial intelligence generated fidelity feedback (Project AFFECT): a study protocol
Creed, T. A., Salama, L., Slevin, R., Tanana, M., Imel, Z. E., Narayanan, S. S., & Atkins, D. C. (2022) · BMC Health Services Research · doi:10.1186/s12913-022-08519-9 · Cited by 49 · see citing works
Self-citation
Effectiveness of a Disability Preventive Intervention for Minority and Immigrant Elders: The Positive Minds-Strong Bodies Randomized Clinical Trial
Alegría, M., Frontera, W., Cruz-Gonzalez, M., Markle, S. L., Trinh-Shevrin, C., Wang, Y., Herrera, L., Ishikawa, R. Z., Velazquez, E., Fuentes, L., et al. (2019) · The American Journal of Geriatric Psychiatry · doi:10.1016/j.jagp.2019.08.008 · Cited by 47 · see citing works
High-structure versus low-structure cooperative learning in introductory psychology classes for student teachers: Effects on conceptual knowledge, self-perceived competence, and subjective task values
Supanc, M., Völlinger, V. A., & Brunstein, J. C. (2017) · Learning and Instruction · doi:10.1016/j.learninstruc.2017.03.006 · Cited by 44 · see citing works
Career Calling and Task Performance: The Moderating Role of Job Demand
Vianello, M., Rosa, A. D., & Gerdel, S. (2022) · Journal of Career Assessment · doi:10.1177/10690727211039454 · Cited by 39 · see citing works
Randomized Controlled Pilot Trial of a Novel Dissonance-Based Group Treatment for Eating Disorders
Stice, E. M., Rohde, P., Butryn, M. L., Menke, K. S., & Marti, C. N. (2014) · Behaviour Research and Therapy · doi:10.1016/j.brat.2014.12.012 · Cited by 39 · see citing works
Symptom course in inpatient and day clinic treatment of depression: Results from the INDDEP-Study
Zeeck, A., von Wietersheim, J., Weiß, H., Scheidt, C. E., Völker, A., Helesic, A., Eckhardt‐Henn, A., Beutel, M. E., Endorf, K., Knoblauch, J. D., et al. (2015) · Journal of Affective Disorders · doi:10.1016/j.jad.2015.07.025 · Cited by 37 · see citing works
A closer look at US schools: What characteristics are associated with scientific literacy? A multivariate multilevel analysis using PISA 2015
You, H. S., Park, S., & Delgado, C. (2020) · Science Education · doi:10.1002/sce.21609 · Cited by 37 · see citing works
Children’s Appraisals and Involvement in Interparental Conflict: Do They Contribute Independently to Child Adjustment?
Mueller, V., Jouriles, E. N., McDonald, R., & Rosenfield, D. (2015) · Journal of Abnormal Child Psychology · doi:10.1007/s10802-014-9953-y · Cited by 36 · see citing works
Attachment‐based family therapy for sexual and gender minority young adults and their nonaccepting parents
Diamond, G. M., Boruchovitz‐Zamir, R., Nir‐Gotlieb, O., Gat, I., Bar‐Kalifa, E., Fitoussi, P., & Katz, S. (2022) · Family Process · doi:10.1111/famp.12770 · Cited by 36 · see citing works
“In addition, to reduce concerns regarding Type 1 error, we ran a secondary set of analyses: two additional multivariate models (Baldwin et al., 2014).”
Method
Improving and sustaining delivery of CPT for PTSD in mental health systems: a cluster randomized trial
Wiltsey Stirman, S., Finley, E. P., Shields, N., Cook, J., Haine-Schlagel, R., Burgess, J. F., Dimeff, L., Koerner, K., Suvak, M., Gutner, C. A., et al. (2017) · Implementation Science · doi:10.1186/s13012-017-0544-5 · Cited by 35 · see citing works
Therapists’ empathic accuracy toward their clients’ emotions.
Atzil–Slonim, D., Bar‐Kalifa, E., Fisher, H., Lazarus, G., Hasson‐Ohayon, I., Lutz, W., Rubel, J. A., & Rafaeli, E. (2018) · Journal of Consulting and Clinical Psychology · doi:10.1037/ccp0000354 · Cited by 35 · see citing works
Testing the influence of negative and positive emotion on future health-promoting behaviors in a community sample
Nylocks, K. M., Rafaeli, E., Bar‐Kalifa, E., Flynn, J. J., & Coifman, K. G. (2018) · Motivation and Emotion · doi:10.1007/s11031-018-9729-8 · Cited by 34 · see citing works
Heart Rate Variability in Individual Psychotherapy
Blanck, P., Stoffel, M., Bents, H., Ditzen, B., & Mander, J. (2019) · The Journal of Nervous and Mental Disease · doi:10.1097/nmd.0000000000000994 · Cited by 34 · see citing works
Predicting relationship satisfaction in distressed and non-distressed couples based on a stratified sample: a matter of conflict, positivity, or support?
Hilpert, P., Bodenmann, G., Nußbeck, F. W., & Bradbury, T. N. (2013) · Family Science · doi:10.1080/19424620.2013.830633 · Cited by 33 · see citing works
Bidirectional relationships between posttraumatic stress disorder and social functioning during cognitive processing therapy
Lord, K. A., Suvak, M. K., Holmes, S., Shields, N., Lane, J. E. M., Sijercic, I., Wagner, A. C., Stirman, S. W., & Monson, C. M. (2020) · Behavior Therapy · doi:10.1016/j.beth.2019.08.002 · Cited by 29 · see citing works
Well-being implications of mobility of care: Gender differences among U.S. adults
Fong, A., & Shaw, F. A. (2024) · Transportation Research Part D Transport and Environment · doi:10.1016/j.trd.2024.104109 · Cited by 28 · see citing works
The therapeutic alliance in different mental disorders: a comparison of patients with depression, somatoform, and eating disorders
Mander, J., Neubauer, A. B., Schlarb, A., Teufel, M., Bents, H., Hautzinger, M., Zipfel, S., Wittorf, A., & Sammet, I. (2017) · Psychology and Psychotherapy Theory Research and Practice · doi:10.1111/papt.12131 · Cited by 27 · see citing works
Community‐based smoking cessation treatment for adults with high anxiety sensitivity: a randomized clinical trial
Smits, J. A. J., Zvolensky, M. J., Rosenfield, D., Brown, R. A., Otto, M. W., Dutcher, C. D., Papini, S., Freeman, S. Z., DiVita, A., Perrone, A., et al. (2021) · Addiction · doi:10.1111/add.15586 · Cited by 27 · see citing works
A Daily Measure of Positive and Negative Alcohol Expectancies and Evaluations: Documenting a Two-Factor Structure and Within- and Between-Person Variability
Lee, C. M., Atkins, D. C., Cronce, J. M., Walter, T., & Leigh, B. C. (2015) · Journal of Studies on Alcohol and Drugs · doi:10.15288/jsad.2015.76.326 · Cited by 27 · see citing works
Self-citation
Use of a Portable Functional Near-Infrared Spectroscopy (fNIRS) System to Examine Team Experience During Crisis Event Management in Clinical Simulations
Xu, J., Slagle, J. M., Banerjee, A., Bracken, B. K., & Weinger, M. B. (2019) · Frontiers in Human Neuroscience · doi:10.3389/fnhum.2019.00085 · Cited by 27 · see citing works
Can cochlear implantation prevent cognitive decline in the long-term follow-up?
Völter, C., Götze, L., Kamin, S. T., Haubitz, I., Dazert, S., & Thomas, J. P. (2022) · Frontiers in Neurology · doi:10.3389/fneur.2022.1009087 · Cited by 26 · see citing works
Residential environment and subjective well-being in Beijing: A fine-grained spatial scale analysis using a bivariate response binomial multilevel model
Dang, Y., Dong, G., Chen, Y., Jones, K., & Zhang, W. (2017) · Environment and Planning B Urban Analytics and City Science · doi:10.1177/2399808317723012 · Cited by 26 · see citing works
Effects of an Ultra-brief Computer-based Mindfulness Training on Mindfulness and Self-control: a Randomised Controlled Trial Using a 40-Day Ecological Momentary Assessment
Rowland, Z., Wenzel, M., & Kubiak, T. (2019) · Mindfulness · doi:10.1007/s12671-019-01204-9 · Cited by 26 · see citing works
Sleep of mothers, fathers, and infants: a longitudinal study from pregnancy through 12 months
Horwitz, A., Bar‐Shachar, Y., Ran-Peled, D., Finkelstein, O., Ben-Zion, H., Bar‐Kalifa, E., Meiri, G., & Tikotzky, L. (2023) · SLEEP · doi:10.1093/sleep/zsad029 · Cited by 25 · see citing works
Modeling empathy as synchrony in clinician and patient vocally encoded emotional arousal: A failure to replicate.
Gaume, J., Hallgren, K. A., Clair, C., Mast, M. S., Carrard, V., & Atkins, D. C. (2019) · Journal of Counseling Psychology · doi:10.1037/cou0000322 · Cited by 25 · see citing works
Self-citation
Parenting Styles and Their Associations With Children's Body Composition, Activity Patterns, Fitness, Diet, Health, and Academic Achievement
Watson, A., Dumuid, D., Maher, C. A., Fraysse, F., Mauch, C. E., Tomkinson, G. R., Ferguson, T., & Olds, T. (2022) · Childhood Obesity · doi:10.1089/chi.2022.0054 · Cited by 24 · see citing works
Longitudinal Network Analysis Reveals Interactive Change of Schizophrenia Symptoms During Acute Antipsychotic Treatment
Sun, Y., Zhang, Y., Lu, Z., Yan, H., Guo, L., Liao, Y., Lu, T., Wang, L., Li, J., Li, W., et al. (2022) · Schizophrenia Bulletin · doi:10.1093/schbul/sbac131 · Cited by 21 · see citing works
Registered replication report of the construct validity of the error‐related negativity ( <scp>ERN</scp> ): A multi‐site study of task‐specific <scp>ERN</scp> correlations with internalizing and externalizing symptoms
Clayson, P. E., Mcdonald, J. B., Park, B., Holbrook, A., Baldwin, S. A., Riesel, A., & Larson, M. J. (2025) · Psychophysiology · doi:10.1111/psyp.14496 · Cited by 21 · see citing works
Self-citation
Increases in Exhaled Nitric Oxide After Acute Stress
Ritz, T., Trueba, A. F., Simon, E., & Auchus, R. J. (2014) · Psychosomatic Medicine · doi:10.1097/psy.0000000000000118 · Cited by 21 · see citing works
Leveraging routine clinical materials and mobile technology to assess CBT fidelity: the Innovative Methods to Assess Psychotherapy Practices (imAPP) study
Stirman, S. W., Marques, L. M., Creed, T. A., Gutner, C. A., DeRubeis, R. J., Barnett, P. G., Kuhn, E. R., Suvak, M. K., Owen, J. E., Vogt, D. S., et al. (2018) · Implementation Science · doi:10.1186/s13012-018-0756-3 · Cited by 21 · see citing works
The Influence of Peer Support for Violence and Peer Acceptance of Rape Myths on Multiple Forms of Interpersonal Violence Among Youth
Collibee, C., Rizzo, C. J., Bleiweiss, K., & Orchowski, L. M. (2019) · Journal of Interpersonal Violence · doi:10.1177/0886260519832925 · Cited by 19 · see citing works
A relational perspective on the association between working alliance and treatment outcome
Uckelstam, C.-J., Holmqvist, R., Philips, B., & Falkenström, F. (2018) · Psychotherapy Research · doi:10.1080/10503307.2018.1516306 · Cited by 19 · see citing works
The Costs and Benefits of Mindfulness and Reappraisal in Daily Life
Wenzel, M., Blanke, E. S., Rowland, Z., & Brose, A. (2023) · Affective Science · doi:10.1007/s42761-022-00178-7 · Cited by 19 · see citing works
Patients' interpersonal problems as moderators of depression outcomes in a randomized controlled trial comparing mindfulness‐based cognitive therapy and a group version of the cognitive‐behavioral analysis system of psychotherapy in chronic depression
Probst, T., Schramm, E., Heidenreich, T., Klein, J. P., & Michalak, J. (2020) · Journal of Clinical Psychology · doi:10.1002/jclp.22931 · Cited by 18 · see citing works
Personal Social Capital in Organizations: A New Scale to Assess Internal and External Personal Social Capital in Organizations
Ben‐Hador, B., Eckhaus, E., & Klein, G. (2021) · Social Indicators Research · doi:10.1007/s11205-021-02702-x · Cited by 17 · see citing works
Using Network Analysis for Examining Interpersonal Emotion Dynamics
Bar‐Kalifa, E., & Sened, H. (2019) · Multivariate Behavioral Research · doi:10.1080/00273171.2019.1624147 · Cited by 17 · see citing works
Normal congruency sequence effects in psychopathology: A behavioral and electrophysiological examination using a confound‐minimized design
Clayson, P. E., Shuford, J. L., Rast, P., Baldwin, S. A., Weissman, D. H., & Larson, M. J. (2023) · Psychophysiology · doi:10.1111/psyp.14426 · Cited by 16 · see citing works
Self-citation
The process-outcome mindfulness effects in trainees (PrOMET) study: Protocol of a pragmatic randomized controlled trial
Mander, J., Kröger, P., Heidenreich, T., Flückiger, C., Lutz, W., Bents, H., & Barnow, S. (2015) · BMC Psychology · doi:10.1186/s40359-015-0082-3 · Cited by 15 · see citing works
Comparison of progress monitoring data from general outcome measures and specific subskill mastery measures for reading
Norman, E. R. V., Maki, K. E., Burns, M. K., McComas, J. J., & Helman, L. A. (2018) · Journal of School Psychology · doi:10.1016/j.jsp.2018.02.002 · Cited by 15 · see citing works
Therapists’ interventions as a predictor of clients’ emotional experience, self-understanding, and treatment outcomes.
Fisher, H., Rafaeli, E., Bar‐Kalifa, E., Barber, J. P., Solomonov, N., Peri, T., & Atzil–Slonim, D. (2019) · Journal of Counseling Psychology · doi:10.1037/cou0000377 · Cited by 15 · see citing works
Growth and CD4 patterns of adolescents living with perinatally acquired HIV worldwide, a CIPHER cohort collaboration analysis
Jesson, J., Crichton, S. L., Quartagno, M., Yotebieng, M., Abrams, E. J., Chokephaibulkit, K., Cœur, S. L., Aké-Assi, M. H., Patel, K., Pinto, J. A., et al. (2022) · Journal of the International AIDS Society · doi:10.1002/jia2.25871 · Cited by 15 · see citing works
The impact of session-introducing mindfulness and relaxation interventions in individual psychotherapy for children and adolescents: a randomized controlled trial (MARS-CA)
Kalmar, J., Baumann, I., Gruber, E., Vonderlin, E., Bents, H., Neubauer, A., Heidenreich, T., & Mander, J. (2022) · Trials · doi:10.1186/s13063-022-06212-0 · Cited by 15 · see citing works
How are the youth? A brief‐longitudinal study on symptoms, alexithymia and expressive suppression among Italian adolescents during COVID‐19 pandemic
Pace, C. S., Rogier, G., & Muzi, S. (2022) · International Journal of Psychology · doi:10.1002/ijop.12866 · Cited by 15 · see citing works
Does intellectual property protection deliver economic benefits? A multi‐outcome meta‐regression analysis of the evidence
Churchill, S. A., Luong, H. M., & Uğur, M. (2022) · Journal of Economic Surveys · doi:10.1111/joes.12489 · Cited by 1 · see citing works
Conceptual and statistical issues in couples observational research: Rationale and methods for design decisions.
Baucom, B., Leo, K., Adamo, C., Georgiou, P., & Baucom, K. J. W. (2017) · Journal of Family Psychology · doi:10.1037/fam0000370 · Cited by 14 · see citing works
Day-to-day pattern of work and leisure time physical behaviours: are low socioeconomic status adults couch potatoes or work warriors?
Rasmussen, C. L., Dumuid, D., Hron, K., Gupta, N., Jørgensen, M. B., Nabe‐Nielsen, K., & Holtermann, A. (2021) · BMC Public Health · doi:10.1186/s12889-021-11409-0 · Cited by 14 · see citing works
Discovery of associative patterns between workplace sound level and physiological wellbeing using wearable devices and empirical Bayes modeling
Srinivasan, K. K., Currim, F., Lindberg, C. M., Razjouyan, J., Gilligan, B., Lee, H., Canada, K., Goebel, N., Mehl, M. R., Lunden, M. M., et al. (2023) · npj Digital Medicine · doi:10.1038/s41746-022-00727-1 · Cited by 14 · see citing works
Relationship satisfaction during COVID ‐19: The role of partners' perceived support and attachment
Bar‐Shachar, Y., Lopata, S., & Bar‐Kalifa, E. (2022) · Family Relations · doi:10.1111/fare.12767 · Cited by 13 · see citing works
Moderators of congruent alliance between therapists and clients: A realistic accuracy model.
Chen, R., Rafaeli, E., Bar‐Kalifa, E., Gilboa‐Schechtman, E., Lutz, W., & Atzil–Slonim, D. (2018) · Journal of Counseling Psychology · doi:10.1037/cou0000285 · Cited by 13 · see citing works
Does the two-chair dialogue intervention facilitate processing of emotions more efficiently than basic Rogerian conditions?
Stiegler, J. R., Molde, H., & Schanche, E. (2018) · European Journal of Psychotherapy & Counselling · doi:10.1080/13642537.2018.1495245 · Cited by 13 · see citing works
Are some therapists better at facilitating and consolidating sudden gains than others?
Deisenhofer, A., Rubel, J. A., Bennemann, B., Aderka, I. M., & Lutz, W. (2021) · Psychotherapy Research · doi:10.1080/10503307.2021.1921302 · Cited by 12 · see citing works
Facing Change: Using Automated Facial Expression Analysis to Examine Emotional Flexibility in the Treatment of Depression
Atzil–Slonim, D., Yehezkel, I., Paz, A., Bar‐Kalifa, E., Wolff, M., Dar, A., & Gilboa‐Schechtman, E. (2023) · Administration and Policy in Mental Health and Mental Health Services Research · doi:10.1007/s10488-023-01310-w · Cited by 12 · see citing works
Examining the time-varying association of negative affect and covariates with craving during treatment for prescription opioid dependence with two types of mixed models
Jenkins, G. J., Cleveland, M. J., Knapp, K. S., Bunce, S. C., & Cleveland, H. H. (2020) · Addictive Behaviors · doi:10.1016/j.addbeh.2020.106674 · Cited by 12 · see citing works
The effect of therapists’ enactment interventions in promoting vulnerability sharing in emotion focused couple therapy
Kula, O., Machluf, R., Shahar, B., Greenberg, L. S., & Bar‐Kalifa, E. (2023) · Psychotherapy Research · doi:10.1080/10503307.2023.2245961 · Cited by 11 · see citing works
Comparative efficacy of PsyPills and OCAT mobile psychological interventions in reducing depressive, anxiety and stress symptoms: A blinded randomized clinical trial
Sîrbu, V., David, D., Sánchez-López, Á., & Blanco, I. (2024) · Journal of Affective Disorders · doi:10.1016/j.jad.2024.10.052 · Cited by 11 · see citing works
Accuracy and Bias in Vulnerability Perceptions of Partners Undergoing Emotion‐Focused Therapy for Couples
Wiesel, I., Shahar, B., Goldman, R. N., & Bar‐Kalifa, E. (2020) · Family Process · doi:10.1111/famp.12587 · Cited by 11 · see citing works
Reciprocal association between session-by-session change in overvaluation of shape and weight and session-by-session change in bulimia nervosa symptoms during cognitive behavior therapies
Srivastava, P., Felonis, C. R., Lin, M., Clarke, K., & Juarascio, A. S. (2022) · Eating Disorders · doi:10.1080/10640266.2022.2114584 · Cited by 11 · see citing works
Cortisol response to acute stress in asthma: Moderation by depressive mood
Trueba, A. F., Simon, E., Auchus, R. J., & Ritz, T. (2016) · Physiology & Behavior · doi:10.1016/j.physbeh.2016.03.005 · Cited by 11 · see citing works
Polyunsaturated fatty acids and child neurodevelopment among a population exposed to DDT: a cohort study
Mérida-Ortega, Á., Rothenberg, S. J., Torres‐Sánchez, L., Schnaas, L., Hernández-Alcaraz, C., Cebrián, M. E., García-Hernández, R. M., Ogaz-González, R., & López‐Carrillo, L. (2019) · Environmental Health · doi:10.1186/s12940-019-0456-8 · Cited by 11 · see citing works
Evaluating the efficacy of couple and family therapy.
Baucom, B., & Crenshaw, A. O. (2018) · American Psychological Association eBooks · doi:10.1037/0000101-005 · Cited by 11 · see citing works
A Primer on Practical SignificancePreprint
Silan, M. A. A. (2019) · doi:10.31234/osf.io/zdhfe · Cited by 11 · see citing works
Developing a Multilevel Scale to Assess Retention of Workers with Disabilities
Krisi, M., Eckhaus, E., & Nagar, R. (2021) · Journal of Occupational Rehabilitation · doi:10.1007/s10926-021-09984-5 · Cited by 10 · see citing works
Promoting Healthy Attitudes and Behaviors in Youth Who Experience Homelessness: Results of a Longitudinal Intervention Study
Rew, L., Slesnick, N., Johnson, K. E., & Sales, A. (2022) · Journal of Adolescent Health · doi:10.1016/j.jadohealth.2021.12.025 · Cited by 10 · see citing works
La aplicación de modelos lineales jerárquicos para el estudio de la eficacia en psicoterapia
Gómez Penedo, J. M., Muiños, R., Hirsch, P., & Roussos, A. (2019) · Revista Argentina de Ciencias del Comportamiento · doi:10.32348/1852.4206.v11.n1.20412 · Cited by 9 · see citing works
Students' Interactional Cultural Capital and Academic Performance in Test‐ and Teacher‐Based Assessments
Geven, S., & Zwier, D. (2025) · British Journal of Sociology · doi:10.1111/1468-4446.13199 · Cited by 9 · see citing works
Relationships Between Changes in Forward Bending, Pain Catastrophizing, and Pain Self-Efficacy During Cognitive Functional Therapy for People With Chronic Low Back Pain
Chang, R., Campbell, A., Kent, P. M., O’Sullivan, P., Hancock, M. J., Hoffman, L., & Smith, A. J. (2025) · Journal of Orthopaedic and Sports Physical Therapy · doi:10.2519/jospt.2025.13114 · Cited by 9 · see citing works
Vulnerable narcissism as beneficial factor for the therapeutic alliance in borderline personality disorder
Busmann, M., Meyer, A. H., Wrege, J. S., Lang, U. E., Gaab, J., Walter, M., & Euler, S. (2021) · Clinical Psychology & Psychotherapy · doi:10.1002/cpp.2570 · Cited by 9 · see citing works
From Social Avoidance to Substance Use: Working Memory and Negative Affectivity Predict Maladaptive Regulatory Behaviors in Daily Life Across Diagnostic Groups
Coifman, K. G., & Aurora, P. (2022) · Cognitive Therapy and Research · doi:10.1007/s10608-021-10284-8 · Cited by 9 · see citing works
Improvements in Forward Bending Are Related to Improvements in Pain and Disability During Cognitive Functional Therapy for People With Chronic Low Back Pain
Chang, R., Campbell, A., Kent, P. M., O’Sullivan, P., Hancock, M. J., Hoffman, L., McGregor, A. H., Laird, R. A., & Smith, A. J. (2024) · Journal of Orthopaedic and Sports Physical Therapy · doi:10.2519/jospt.2024.12727 · Cited by 8 · see citing works
The Role of Individual and Dyadic Planning in Couples’ Daily Goal Pursuits
Katzenelenbogen, O., Knoll, N., Stadler, G., & Bar‐Kalifa, E. (2021) · Personality and Social Psychology Bulletin · doi:10.1177/0146167221997630 · Cited by 8 · see citing works
Perceived injustice after spinal cord injury: evidence for a distinct psychological construct
Monden, K. R., Philippus, A. H., Boals, A., Draganich, C., Morse, L. R., Ketchum, J. M., & Trost, Z. (2019) · Spinal Cord · doi:10.1038/s41393-019-0318-9 · Cited by 8 · see citing works
Patient and Therapist In-Session Cortisol as Predictor of Post-Session Patient Reported Affect
Levi, E., Fischer, S., Fisher, H., Admon, R., & Zilcha‐Mano, S. (2021) · Brain Sciences · doi:10.3390/brainsci11111483 · Cited by 8 · see citing works
Multivariate Models for Normal and Binary Responses in Intervention Studies
Pituch, K. A., Whittaker, T. A., & Chang, W. (2016) · American Journal of Evaluation · doi:10.1177/1098214015626297 · Cited by 7 · see citing works
The Performance of Multilevel Models When Outcome Data Are Incomplete
Chang, W., & Pituch, K. A. (2017) · The Journal of Experimental Education · doi:10.1080/00220973.2017.1377676 · Cited by 7 · see citing works
The multivariate multiple-membership random-effect model: An introduction and evaluation
Park, S., & Beretvas, S. N. (2019) · Behavior Research Methods · doi:10.3758/s13428-019-01315-0 · Cited by 7 · see citing works
Looking into the Future : How to Use Advanced Statistical Methods for Predicting Psychotherapy Outcomes in Routine Care
Uckelstam, C.-J. (2021) · Linköping studies in behavioural science · doi:10.3384/diss.diva-172754 · Cited by 6 · see citing works
Effects of social determinants on children’s health in informal settlements in Bangladesh and Kenya through an intersectionality lens: a study protocol
Kibuchi, E., Barua, P., Chumo, I., Filha, N. T. D. S., Phillips‐Howard, P. A., Mithu, M. I. H., Kabaria, C. W., Quayyum, Z., Whittaker, L., Dean, L., et al. (2022) · BMJ Open · doi:10.1136/bmjopen-2021-056494 · Cited by 6 · see citing works
Association of patients’ pre-therapy strengths and alliance in outpatient psychotherapy: A multilevel growth curve analysis
Schürmann‐Vengels, J., Appelbaum, S., Apel, M.-K., Trimpop, L. F., Muermans, M. M., Ostermann, T., Teismann, T., & Willutzki, U. (2022) · Psychotherapy Research · doi:10.1080/10503307.2022.2146543 · Cited by 6 · see citing works
Efficacy of live-supervision regarding skills, anxiety and self-efficacy: a randomized controlled trial
Maaß, U., Kühne, F., Ay-Bryson, D. S., Heinze, P. E., & Weck, F. (2023) · The Clinical Supervisor · doi:10.1080/07325223.2023.2267528 · Cited by 6 · see citing works
Exploring Variation in the Strength of Association of a Validated Recidivism Risk Score With Seven Common Measures of Juvenile Recidivism: A Research Note
Siennick, S. E., & Pupo, J. A. (2022) · Youth Violence and Juvenile Justice · doi:10.1177/15412040221115056 · Cited by 6 · see citing works
Using Peer Education to Promote Psychosocial and Occupational Health and Empowerment Among Female Sex Workers in Nepal
Menger‐Ogle, L. M., Kaufman, M. R., Fisher, G. G., Ryan, E. P., & Stallones, L. F. (2019) · Violence Against Women · doi:10.1177/1077801219861155 · Cited by 6 · see citing works
Improving the accuracy of middle school students' self-assessment, peer assessment, and mathematics achievement.
Popelka, E. (2015) · doi:10.18297/etd/2324 · Cited by 5 · see citing works
Clients’ and therapists’ parasympathetic interpersonal and intrapersonal regulation dynamics during psychotherapy for depression
Goren, O., Paz, A., Bar‐Kalifa, E., Gilboa‐Schechtman, E., Wolff, M., & Atzil–Slonim, D. (2024) · Psychotherapy Research · doi:10.1080/10503307.2024.2378038 · Cited by 5 · see citing works
A multivariate multilevel approach to unravel the associations between individual and school factors on children's motor performance in the REACT project
Maia, J. A. R., Santos, C., Pereira, S., Hedeker, D., Barreira, T. V., Garganta, R., Farias, C. F. G., Garbeloto, F., Tani, G., Cruz, H., et al. (2024) · American Journal of Human Biology · doi:10.1002/ajhb.24080 · Cited by 5 · see citing works
The use of restrictive housing with incarcerated persons with mental illness: Variation in placement disparities by restrictive housing type
Siennick, S. E., Mears, D. P., & Spies, A. (2024) · Journal of Criminal Justice · doi:10.1016/j.jcrimjus.2024.102311 · Cited by 5 · see citing works
Trajectories of change in maternal and adolescent depressive symptoms in the depression prevention initiative
Spiro, C. N., Gallop, R., & Young, J. F. (2019) · Journal of Affective Disorders · doi:10.1016/j.jad.2019.04.089 · Cited by 5 · see citing works
Associations between mindfulness and general change mechanisms in individual therapy: Secondary results of a randomised controlled trial
D’Errico, L., Call, M., Blanck, P., Vonderlin, E., Bents, H., & Mander, J. (2019) · Counselling and Psychotherapy Research · doi:10.1002/capr.12233 · Cited by 5 · see citing works

Authors: Baldwin, S. A., Imel, Z. E., Braithwaite, S. R., & Atkins, D. C.
Affiliations: Brigham Young University (US)
University of Utah (US)
University of Washington (US)
ORCID: Baldwin, Scott A.
Imel, Zac E.
Braithwaite, Scott R.
Atkins, David C.
Venue: Journal of Consulting and Clinical Psychology
Publisher: American Psychological Association (APA)
Published: 2014-10
Volume: 82
Issue: 5
Pages: 920-930
ISSN: 0022-006X (ISSN-L)
1939-2117
DOI: 10.1037/a0035628
PMC: PMC4119868
OpenAlex: W2312318302
Cited by: 167
References: 40
Citations by year: 2026: 12 · 2025: 15 · 2024: 12 · 2023: 14 · 2022: 24 · 2021: 19 · 2020: 10 · 2019: 24 · 2018: 11 · 2017: 7 · 2016: 4 · 2015: 8 · 2014: 5 · 2013: 2
FWCI: 8.46
Type: journal article
Language: en
Open access: green
www.ncbi.nlm.nih.gov
Locations: Journal of Consulting and Clinical Psychology (publishedVersion)
PubMed (publishedVersion)
PubMed Central (acceptedVersion)
ScholarsArchive (Brigham Young University) (submittedVersion)
Funders: National Institute on Alcohol Abuse and Alcoholism (R01 AA019511)
Topics: Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods and Bayesian Inference
Social Sciences › Psychology › Experimental and Cognitive Psychology › Mental Health Research Topics
Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Clinical Trials
Keywords: psychotherapy research, intervention research, growth models, random effects, fixed effects
MeSH terms: Humans, Male, Psychotherapy, Models, Statistical, Multivariate Analysis, Outcome Assessment, Health Care, Multilevel Analysis

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Analyzing multiple outcomes in clinical research using multivariate multilevel models.

Abstract

Integrity signals

Claims the paper makes

Self-check

Cites

Cited by

Related

Record

About these checks