Assessing the unwanted: A systematic review of instruments used to assess negative effects of psychotherapy
Open accessAbstract
OBJECTIVE: While the efficacy of psychotherapy in the treatment of mental disorders is well examined, systematic research into negative effects of psychotherapy seems comparatively rare. Therefore, this review evaluates instruments for assessing negative effects of psychotherapy in order to create a consensus framework and make recommendations for their assessment. METHODS: The study selection procedure follows current best-practice guidelines for conducting systematic reviews, with 10 included studies in three databases (PsycINFO, PubMed, and Web of Science). The nine instruments identified were each critically reviewed concerning the theoretical orientation, including the assessed domains of negative effects, psychometric properties, and diagnostic characteristics. RESULTS: Seventeen domains of negative effects of psychotherapy were identified but inconsistently assessed by the nine instruments. Most instruments provide some initial data on their psychometric properties. Regarding diagnostic characteristics, different item-response formats are used but often with reference to "attribution to therapy." CONCLUSION: This review indicates that the existing instruments for assessing negative effects of psychotherapy cover a wide range of relevant domains without any consensus on the most important ones and their psychometric properties are usually unsatisfactory. A framework for consensus, building on the definition and conceptualization of negative effects, is synthesized, and recommendations for improving the assessment are derived.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The INEP item construction involved a literature search, expert rating, and pilot study, which speaks for good content validity compared to similar scales.
“Item construction for the INEP involved a literature search, expert rating, and pilot study, which speaks for good content validity compared to similar scales (Herzog et al., 2019).”
From Effective—and Tolerable: Acceptance and Side Effects of Intensified Exposure for Anxiety Disorders (2023) · cites this paper as Herzog et al., 2019
In Results, under Evaluation of psychometric properties (Table 3; page unavailable), the review lists a literature review and consultation with researchers for INEP and describes its content-related validity as strong. The table lists a pilot study for NEQ, not INEP, so it does not support attributing all three item-construction steps to INEP. (LLM, conf=0.90)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Preferred Reporting Items for Systematic Reviews and Meta-Analyses: the PRISMA statement
Moher, D., Liberati, A., Tetzlaff, J., Altman, D. G., & for the PRISMA Group (2009) · BMJ · doi:10.1136/bmj.b2535 · Cited by 83,823 · see citing works
Preferred Reporting Items for Systematic Reviews and Meta-Analyses: The PRISMA Statement
Moher, D., Liberati, A., Tetzlaff, J., Altman, D. G., & The PRISMA Group (2009) · PLoS Medicine · doi:10.1371/journal.pmed.1000097 · Cited by 64,042 · see citing works
Cicchetti, D. V. (1994) · Psychological Assessment · doi:10.1037/1040-3590.6.4.284 · Cited by 9,349 · cited 2× · see citing works
“…the integration of standards and guidelines for psychometric evaluation, includ‐ ing the Journal Article Reporting Standards (APA Publications & Communications Board Working Group on Journal Article Reporting Standards, 2008) and guidelines for the evaluation of test instruments (Cicchetti, 1994).”
BackgroundDevelopment of the World Health Organization WHOQOL-BREF Quality of Life Assessment
THE WHOQOL GROUP (1998) · Psychological Medicine · doi:10.1017/s0033291798006667 · Cited by 7,933 · see citing works
Jacobson, N. S., & Truax, P. (1991) · Journal of Consulting and Clinical Psychology · doi:10.1037/0022-006x.59.1.12 · Cited by 7,770 · see citing works
The Efficacy of Cognitive Behavioral Therapy: A Review of Meta-analyses
Hofmann, S. G., Asnaani, A., Vonk, I. J. J., Sawyer, A. T., & Fang, A. (2012) · Cognitive Therapy and Research · doi:10.1007/s10608-012-9476-1 · Cited by 3,445 · see citing works
The empirical status of cognitive-behavioral therapy: A review of meta-analyses
Butler, A. C., Chapman, J. E., Forman, E. M., & Beck, A. T. (2006) · Clinical Psychology Review · doi:10.1016/j.cpr.2005.07.003 · Cited by 3,215 · see citing works
Better Reporting of Harms in Randomized Trials: An Extension of the CONSORT Statement
Ioannidis, J. P. A., Evans, S. J. W., Gøtzsche, P. C., O'Neill, R. T., Altman, D. G., Schulz, K., Moher, D., & for the CONSORT Group* (2004) · Annals of Internal Medicine · doi:10.7326/0003-4819-141-10-200411160-00009 · Cited by 1,593 · see citing works
The psychotherapeutic process: A research handbook.
Greenberg, L. S., & Pinsof, W. M. (1986) · Cited by 1,386 · see citing works
Jacobson, N. S., Follette, W. C., & Revenstorf, D. (1984) · Behavior Therapy · doi:10.1016/s0005-7894(84)80002-7 · Cited by 1,294 · see citing works
Handbook of Psychological Assessment
(2000) · Elsevier eBooks · doi:10.1016/b978-0-08-043645-6.x5078-2 · Cited by 628 · see citing works
Reporting standards for research in psychology: why do we need them? What might they be?
APA Publications and Communications Board Working Group on Journal Article Reporting Standards (2008) · American Psychologist · doi:10.1037/0003-066x.63.9.839 · Cited by 571 · see citing works
Why Cognitive Behavioral Therapy Is the Current Gold Standard of Psychotherapy
David, D., Cristea, I., & Hofmann, S. G. (2018) · Frontiers in Psychiatry · doi:10.3389/fpsyt.2018.00004 · Cited by 565 · see citing works
Feinstein, A. R. (1987) · doi:10.2307/j.ctt1xp3vbc · Cited by 538 · see citing works
The Nocebo Effect and Its Relevance for Clinical Practice
Colloca, L., & Miller, F. G. (2011) · Psychosomatic Medicine · doi:10.1097/psy.0b013e3182294a50 · Cited by 431 · see citing works
Collecting and delivering progress feedback: A meta-analysis of routine outcome monitoring.
Lambert, M. J., Whipple, J. L., & Kleinstäuber, M. (2018) · Psychotherapy · doi:10.1037/pst0000167 · Cited by 408 · see citing works
Linden, M. (2013) · Clinical Psychology & Psychotherapy · doi:10.1002/cpp.1765 · Cited by 397 · cited 20× · see citing works
“…underlying theoretical foundation. aVanderbilt Negative Indicators Scale (Suh et al., 1986). bUnwanted Effects–Adverse Treatment Reaction checklist (Linden, 2013). cInventory for the Assessment of Negative Effects of Psychotherapy (Ladwig et al., 2014). dExperiences of Therapy Questionnaire…”
BackgroundMethodInfluentialHuhn, M., Tardy, M., Spineli, L. M., Kissling, W., Förstl, H., Pitschel-Walz, G., Leucht, C., Samara, M., Dold, M., Davis, J. M., et al. (2014) · JAMA Psychiatry · doi:10.1001/jamapsychiatry.2014.112 · Cited by 333 · see citing works
Expectancies as core features of mental disorders
Rief, W., Glombiewski, J. A., Gollwitzer, M., Schubö, A., Schwarting, R. K. W., & Thorwart, A. (2015) · Current Opinion in Psychiatry · doi:10.1097/yco.0000000000000184 · Cited by 235 · see citing works
Self-citationLadwig, I., Rief, W., & Nestoriuc, Y. (2014) · Verhaltenstherapie · doi:10.1159/000367928 · Cited by 227 · see citing works
Self-citationPatient experience of negative effects of psychological treatment: results of a national survey
Crawford, M. J., Thana, L., Farquharson, L., Palmer, L., Hancock, E., Bassett, P., Clarke, J., & Parry, G. D. (2016) · British Journal of Psychiatry · doi:10.1192/bjp.bp.114.162628 · Cited by 225 · see citing works
Rozental, A., Kottorp, A., Boettcher, J., Andersson, G., & Carlbring, P. (2016) · PLOS ONE · doi:10.1371/journal.pone.0157503 · Cited by 223 · see citing works
Methodological Recommendations for Trials of Psychological Interventions
Guidi, J., Brakemeier, E.-L., Bockting, C. L. H., Cosci, F., Cuijpers, P., Jarrett, R. B., Linden, M., Marks, I., Peretti, C. S., Rafanelli, C., et al. (2018) · Psychotherapy and Psychosomatics · doi:10.1159/000490574 · Cited by 205 · see citing works
Self-citationJonsson, U., Alaie, I., Parling, T., & Arnberg, F. K. (2014) · Contemporary Clinical Trials · doi:10.1016/j.cct.2014.02.005 · Cited by 175 · see citing works
A systematic review of factors that contribute to nocebo effects
Webster, R. K., Weinman, J., & Rubin, G. J. (2016) · Health Psychology · doi:10.1037/hea0000416 · Cited by 172 · see citing works
Iatrogenic harm from psychological therapies—time to move on
Parry, G. D., Crawford, M. J., & Duggan, C. (2016) · British Journal of Psychiatry · doi:10.1192/bjp.bp.115.163618 · Cited by 146 · see citing works
The Clinical Process in Psychiatry: A Clinimetric Approach
Fava, G. A., Rafanelli, C., & Tomba, E. (2012) · The Journal of Clinical Psychiatry · doi:10.4088/jcp.10r06444 · Cited by 143 · see citing works
Understanding placebo, nocebo, and iatrogenic treatment effects
Bootzin, R. R., & Bailey, E. T. (2005) · Journal of Clinical Psychology · doi:10.1002/jclp.20131 · Cited by 98 · see citing works
The Exploitation Index: an early warning indicator of boundary violations in psychotherapy.
Epstein, R. S., & SIMON, R. I. (1990) · PubMed · Cited by 94 · see citing works
Adverse outcomes in group psychotherapy: risk factors, prevention, and research directions.
(2000) · PubMed Central · Cited by 91 · see citing works
Side effects in internet-based interventions for social anxiety disorder
Boettcher, J., Rozental, A., Andersson, G., & Carlbring, P. (2014) · Internet Interventions · doi:10.1016/j.invent.2014.02.002 · Cited by 88 · see citing works
The Nocebo Effect of Informed Consent
Cohen, S. (2012) · Bioethics · doi:10.1111/j.1467-8519.2012.01983.x · Cited by 82 · see citing works
It can’t hurt, right? Adverse effects of psychotherapy in patients with depression
Moritz, S., Nestoriuc, Y., Rief, W., Klein, J. P., Jelinek, L., & Peth, J. (2018) · European Archives of Psychiatry and Clinical Neuroscience · doi:10.1007/s00406-018-0931-1 · Cited by 77 · see citing works
Self-citationThe merits of implementing routine outcome monitoring in clinical practice.
Lambert, M. J., & Harmon, K. L. (2018) · Clinical Psychology Science and Practice · doi:10.1111/cpsp.12268 · Cited by 75 · see citing works
Abramovitch (2015) · doi:10.1016/j.jocrd.2015.02.002 · Cited by 72 · see citing works
Clinical Pharmacopsychology: Conceptual Foundations and Emerging Tasks
Fava, G. A., Tomba, E., & Bech, P. (2017) · Psychotherapy and Psychosomatics · doi:10.1159/000458458 · Cited by 52 · see citing works
Parker, G. B., Fletcher, K., Berk, M., & Paterson, A. (2013) · Psychiatry Research · doi:10.1016/j.psychres.2012.11.026 · Cited by 45 · see citing works
Meister, R., von Wolff, A., Mohr, H., Nestoriuc, Y., Härter, M. C., Hölzel, L. P., & Kriston, L. (2015) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2015.10.007 · Cited by 44 · see citing works
Rate and predictors of negative effects of psychotherapy in psychiatric and psychosomatic inpatients
Rheker, J., Beisel, S., Kräling, S., & Rief, W. (2017) · Psychiatry Research · doi:10.1016/j.psychres.2017.04.042 · Cited by 44 · see citing works
Adverse events of group psychotherapy in the in‐patient setting – results of a naturalistic trial
Schneibel, R., Wilbertz, G., Scholz, C., Becker, M., Brakemeier, E.-L., Bschor, T., Zobel, I., & Schmoll, D. (2017) · Acta Psychiatrica Scandinavica · doi:10.1111/acps.12747 · Cited by 43 · see citing works
Self-citationClinimetric Dilemmas in Outcome Scales for Mental Disorders
Bech, P. (2016) · Psychotherapy and Psychosomatics · doi:10.1159/000448810 · Cited by 42 · see citing works
Suboptimal Choice in Pigeons: Stimulus Value Predicts Choice over Frequencies
Smith, A. P., Bailey, A. R., Chow, J. J., et al. (2016) · PLoS ONE · doi:10.1371/journal.pone.0159336 · Cited by 34 · see citing works
Unerwünschte Therapiewirkungen bei verhaltenstherapeutischer Gruppentherapie
Linden, M., Walter, M., Fritz, K., & Muschalla, B. (2015) · Der Nervenarzt · doi:10.1007/s00115-015-4297-6 · Cited by 31 · see citing works
Szapocznik, J., & Prado, G. J. (2007) · Journal of Family Psychology · doi:10.1037/0893-3200.21.3.468 · Cited by 31 · see citing works
Peth, J., Jelinek, L., Nestoriuc, Y., & Moritz, S. (2018) · PPmP - Psychotherapie · Psychosomatik · Medizinische Psychologie · doi:10.1055/s-0044-101952 · Cited by 29 · see citing works
Linden, M., Strauss, B. M., Scholten, S., Nestoriuc, Y., Brakemeier, E., & Wasilewski, J. (2018) · PPmP - Psychotherapie · Psychosomatik · Medizinische Psychologie · doi:10.1055/a-0619-5949 · Cited by 27 · see citing works
Self-citationFOR BETTER OR FOR WORSE? PSYCHOTHERAPY AND VARIANCE CHANGE: A CRITICAL REVIEW OF THE LITERATURE
May, P. R. A. (1971) · The Journal of Nervous and Mental Disease · doi:10.1097/00005053-197103000-00004 · Cited by 19 · see citing works
Brakemeier, E., Herzog, P., Radtke, M., Schneibel, R., Breger, V., Becker, M., Spies, J., Jacobi, F., Heider, J., & Normann, C. (2018) · PPmP - Psychotherapie · Psychosomatik · Medizinische Psychologie · doi:10.1055/a-0629-7802 · Cited by 19 · see citing works
Self-citationSchefft, C., Guhn, A., Brakemeier, E. ‐., Sterzer, P., & Köhler, S. (2019) · Acta Psychiatrica Scandinavica · doi:10.1111/acps.12995 · Cited by 17 · see citing works
Self-citationConstruct validity of the Experiences of Therapy Questionnaire (ETQ)
Parker, G. B., Paterson, A., Fletcher, K., McClure, G., & Berk, M. (2014) · BMC Psychiatry · doi:10.1186/s12888-014-0369-6 · Cited by 13 · see citing works
„Positive Nebenwirkungen“ von Psychotherapie
Hoyer, J. (2016) · Zeitschrift für Klinische Psychologie und Psychotherapie · doi:10.1026/1616-3443/a000370 · Cited by 9 · see citing works
Scott, J. L. (2017) · Acta Psychiatrica Scandinavica · doi:10.1111/acps.12780 · Cited by 6 · see citing works
Strauss, B., Hh, S., Burgmeier-Lohse, M., et al. (1992) · PubMed · Cited by 3 · see citing works
Flagged first, then by citations
Research works printed but not linked: 5
- Burlingame, G. M., MacKenzie, K. R., & Strauß, B. (2004). Small group treatment: Evidence for effectiveness and mechanisms of change In Lambert M. J. (Ed.), Bergin & Garfield's Handbook of psychotherapy and behavior change (pp. 647– 696). New York: Wiley & Sons.
- Caspar, F., & Jacobi, F. (2007). Psychotherapieforschung In Hiller W., Leibing E., Leichsenring F., & Sulz S. K. D. (Eds.), Lehrbuch der Psychotherapie für die Ausbildung zur/zum Psychologischen PsychotherapeutIn und für ärztliche Weiterbildung, Band 1: Wissenschaftliche Grundlagen der Psychotherapie (pp. 395– 410). München, Germany: CIP‐Medien.
- Linden, M., Strauß, B., Scholten, S., Nestoriuc, Y., Brakemeier, E.‐L., & Wasilewski, J. (2018). Definition and decision‐making in the determination and detection of side effects of. psychotherapy. PPmP ‐ Psychotherapie Psychosomatik Medizinische Psychologie, 68(09/10), 377– 382. 30286505
- Strauß, B., Strupp, H. H., Burgmeier-Lohse, M., Wille, H., & Storm, S. (1992). Deutschsprachige Version der Vanderbilt-Psychotherapie-Skalen: Beschreibung und Anwendung in zwei Kurztherapien. Zeitschrift für Klinische Psychologie, Psychopathologie und Psychotherapie, 40(4), 411– 430. 1441689
- Zwingmann, C., Metzger, D., & Jäckel, W. H. (1998). Short form‐36 health survey (SF‐36): Psychometrische analysen der deutschen version bei rehabilitanden mit chronischen rückenschmerzen. Diagnostica, 44(4), 209– 219.
Cited by
Sorted by citations
Patient experiences of psychological therapy for depression: a qualitative metasynthesis
McPherson, S., Wicks, C., & Tercelli, I. (2020) · BMC Psychiatry · doi:10.1186/s12888-020-02682-1 · Cited by 52 · see citing works
Fuhr, K., Meisner, C., Broch, A., Cyrny, B., Hinkel, J., Jaberg, J., Petrasch, M., Schweizer, C., Stiegler, A., Zeep, C., et al. (2021) · Journal of Affective Disorders · doi:10.1016/j.jad.2021.02.069 · Cited by 48 · see citing works
Gerke, L., Meyrose, A.-K., Ladwig, I., Rief, W., & Nestoriuc, Y. (2020) · Frontiers in Psychology · doi:10.3389/fpsyg.2020.02144 · Cited by 33 · see citing works
Bendig, E., Bauereiss, N., Schmitt, A., Albus, P., & Baumeister, H. (2021) · BMJ Open · doi:10.1136/bmjopen-2021-049238 · Cited by 31 · see citing works
Effective—and Tolerable: Acceptance and Side Effects of Intensified Exposure for Anxiety Disorders
Heinig, I., Knappe, S., Hoyer, J., Wittchen, H.-U., Richter, J., Arolt, V., Deckert, J., Domschke, K., Hamm, A., Kircher, T., et al. (2023) · Behavior Therapy · doi:10.1016/j.beth.2022.11.001 · Cited by 17 · see citing works
Self-citationBurian, H., Böge, K., Burian, R., Burns, A., Nguyen, M. H., Ohse, L., Ta, T. M. T., Hahn, E., & Diefenbacher, A. (2021) · Journal of Psychosomatic Research · doi:10.1016/j.jpsychores.2021.110374 · Cited by 19 · see citing works
- Authors
- Herzog, P., Lauff, S., Rief, W., & Brakemeier, E.
- Affiliations
- Philipps University of Marburg (DE)
- Corresponding author
- Herzog, Philipp
- Venue
- Brain and Behavior
- Publisher
- Wiley
- Published
- 2019-10-24
- Volume
- 9
- Issue
- 12
- Pages
- e01447-e01447
- ISSN
- 2157-9032 (ISSN-L)
- 2162-3279
- PMC
- PMC6908878
- OpenAlex
- W2981659549
- Cited by
- 95
- References
- 53
- Citations by year
- 2026: 15 · 2025: 21 · 2024: 19 · 2023: 18 · 2022: 6 · 2021: 10 · 2020: 5 · 2019: 1
- FWCI
- 0.87
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- Brain and Behavior (publishedVersion, cc-by)
- Europe PMC (PubMed Central) (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Topics
- Social Sciences › Psychology › Clinical Psychology › Psychotherapy Techniques and Applications
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Anxiety, Depression, Psychometrics, Treatment, Cognitive Processes
- Social Sciences › Psychology › Clinical Psychology › Psychoanalysis and Psychopathology Research
- Keywords
- adverse effects, psychotherapy outcome assessment, assessment instruments, psychometric properties, psychiatric treatment
- MeSH terms
- Mental Disorders, Humans, Psychometrics, Psychotherapy
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
