The DOI matches its Crossref record; 93 of the 94 references it cites were found in a registry or an index.
Negative Effects of Psychological Treatments: An Exploratory Factor Analysis of the Negative Effects Questionnaire for Monitoring and Reporting Adverse and Unwanted Events
Open accessAbstract
Research conducted during the last decades has provided increasing evidence for the use of psychological treatments for a number of psychiatric disorders and somatic complaints. However, by focusing only on the positive outcomes, less attention has been given to the potential of negative effects. Despite indications of deterioration and other adverse and unwanted events during treatment, little is known about their occurrence and characteristics. Hence, in order to facilitate research of negative effects, a new instrument for monitoring and reporting their incidence and impact was developed using a consensus among researchers, self-reports by patients, and a literature review: the Negative Effects Questionnaire. Participants were recruited via a smartphone-delivered self-help treatment for social anxiety disorder and through the media (N = 653). An exploratory factor analysis was performed, resulting in a six-factor solution with 32 items, accounting for 57.64% of the variance. The derived factors were: symptoms, quality, dependency, stigma, hopelessness, and failure. Items related to unpleasant memories, stress, and anxiety were experienced by more than one-third of the participants. Further, increased or novel symptoms, as well as lack of quality in the treatment and therapeutic relationship rendered the highest self-reported negative impact. In addition, the findings were discussed in relation to prior research and other similar instruments of adverse and unwanted events, giving credence to the items that are included. The instrument is presently available in eleven different languages and can be freely downloaded and used from www.neqscale.com.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“On the one hand, this result fits literature that shows, that psychological interventions, online and offline, can initially evoke negative effects and symptom deteriorations (Ladwig et al., 2014; Rozental et al., 2016).”
From Feasibility of a Software agent providing a brief Intervention for Self-help to Uplift psychological wellbeing (“SISU”). A single-group pretest-posttest trial investigating the potential of SISU to act as therapeutic agent (2021) · cites this paper as Rozental et al., 2016
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Adverse events and harms during the RCT are measured with the Negative Experiences Questionnaire (NEQ), [61]”
From Transdiagnostic and tailored internet intervention to improve mental health among university students: Research protocol for a randomized controlled trial (2024) · cites this paper as [61]
The cited source says what the claim attributes to it.
In the citing paper
“The internal consistency for the full instrument is excellent (Cronbach's α = 0.95), and is scored on several dimensions; 1) “Did you experience this?” (yes/no), 2) “If yes – here is how negatively it affected me”, from 0 (“Not at all”) to 4 (“Extremely”), and 3) “Probably caused by” (“The treatment I received” or “Other circumstances”).”
From A study protocol of a randomized controlled study of internet-based cognitive behavioral therapy for adult attention deficit hyperactivity disorder (2023) · cites this paper as Rozental et al., 2016
The cited source says what the claim attributes to it.
In the citing paper
“Internal Consistency of the NEQ is reported to be α = 0.95 (90).”
The cited source says what the claim attributes to it.
In the citing paper
“Others have also implied that non-response, dropout, and interpersonal difficulties may be perceived as negative effects (Dimidjian 2010; Rozental 2016).”
From Psychotherapeutic treatments for depression in older adults (2024) · cites this paper as Rozental 2016
In the citing paper
“and trials have already begun (Miloff et al. 2019) to include and report results from broader measures of negative effects used elsewhere in psychotherapy research (Rozental et al. 2016).”
From Better, Virtually: the Past, Present, and Future of Virtual Reality Cognitive Behavior Therapy (2021) · cites this paper as Rozental et al. 2016
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Ryan, R. M., & Deci, E. L. (2000) · American Psychologist · doi:10.1037//0003-066x.55.1.68 · Cited by 54,599
An inventory for measuring depression
BECK, A. T. (1961) · Archives of General Psychiatry · doi:10.1001/archpsyc.1961.01710120031004 · Cited by 38,032
(1998) · The Journal of Clinical Psychiatry · Cited by 19,139
The Scree Test For The Number Of Factors
Cattell, R. B. (1966) · Multivariate Behavioral Research · doi:10.1207/s15327906mbr0102_10 · Cited by 13,554
The application of electronic computers to factor analysis
Kaiser, H. F. (1960) · Educational and Psychological Measurement · doi:10.1177/001316446002000116 · Cited by 12,459
Construct validity in psychological tests.
Cronbach, L. J., & Meehl, P. E. (1955) · Psychological Bulletin · doi:10.1037/h0040957 · Cited by 11,195
(1998) · PubMed · Cited by 10,216
Evaluating the use of exploratory factor analysis in psychological research.
Fabrigar, L. R., Wegener, D. T., MacCallum, R. C., & Strahan, E. J. (1999) · Psychological Methods · doi:10.1037/1082-989x.4.3.272 · Cited by 9,587
A Rationale and Test for the Number of Factors in Factor Analysis
Horn, J. L. (1965) · Psychometrika · doi:10.1007/bf02289447 · Cited by 8,965
Selective Cortical Undercutting As a Means of Modifying and Studying Frontal Lobe Function in Man
Scoville, W. B. (1949) · Journal of neurosurgery · doi:10.3171/jns.1949.6.1.0065 · Cited by 7,374
Flagged first, then by citations
1 of the 94 references couldn’t be matched to a known paper.
Research works printed but not linked: 2
- I Ladwig, 2014, What are the Risks and Side Effects to Psychotherapy?—Development of an Inventory for the Assessment of Negative Effects of Psychotherapy (INEP), Verhaltenstherapie, 24, 252
- S Moritz, 2015, No pain, no gain? Adverse effects of psychotherapy in obsessive-compulsive disorder and its relationship to treatment gains, J Obsess-Compuls Rel, 5, 61
Cited by
Sorted by citations
Items 201–223 of 223
Beiner, E., Baumeister, D., Buhai, D.-V., Löffler, M., Löffler, A., Schick, A., Ader, L., Eich, W., Sirazitdinov, A., Malone, C., et al. (2024) · Figshare · doi:10.6084/m9.figshare.26552533
Jensen, K. H. R., Dam, V. H., Ganz, M., Fisher, P. M., Ip, C.-T., Sankar, A., Marstrand-Joergensen, M. R., Ozenne, B., Osler, M., Penninx, B. W. J. H., et al. (2023) · Figshare · doi:10.6084/m9.figshare.22623808
Olsen, M. H., Hagstrøm, J., Lønfeldt, N. N., Uhre, C. F., Uhre, V., Pretzmann, L., Christensen, S. H., Thoustrup, C. L., Korsbjerg, N. L. J., Mora-Jensen, A.-R. C., et al. (2022) · Figshare · doi:10.6084/m9.figshare.21293625
Jensen, K. H. R., Dam, V. H., Ganz, M., Fisher, P. M., Ip, C.-T., Sankar, A., Marstrand-Joergensen, M. R., Ozenne, B., Osler, M., Penninx, B. W. J. H., et al. (2023) · Figshare · doi:10.6084/m9.figshare.22623805
Jensen, K. H. R., Dam, V. H., Ganz, M., Fisher, P. M., Ip, C.-T., Sankar, A., Marstrand-Joergensen, M. R., Ozenne, B., Osler, M., Penninx, B. W. J. H., et al. (2023) · Figshare · doi:10.6084/m9.figshare.22623802
Hybelius, J., Gustavsson, A., af Winklerfelt Hammarberg, S., Toth-Pal, E., Johansson, R., Ljótsson, B., & Axelsson, E. (2022) · Figshare · doi:10.6084/m9.figshare.20342406
Houben, S. T. L., Backus, A. C. P., Merckelbach, H. L. G. J., & Dandachi‐FitzGerald, B. (2026) · Clinical Psychology & Psychotherapy · doi:10.1002/cpp.70209
Madsen, M. S., Melau, M., Andersen, N. K., Nielsen, C. F., Larsen, N., Andersen, T. L., Nordentoft, M., Jepsen, J. R. M., Thorup, A. A., Jeppesen, P., et al. (2026) · Contemporary Clinical Trials · doi:10.1016/j.cct.2026.108253
Marcilla-Jordá, M., Montañés-Martí, S., Rubio‐Belmonte, C., & Ipola, M. M. (2026) · Research in Autism · doi:10.1016/j.reia.2026.202866
Paumen, T., Leidger, A., Ralfs, J., Modarressi, A., Ozga, A., Möhl, A., Daubmann, A., Zapf, A., Waldmann, T., von Gottberg, C., et al. (2026) · Frontiers in Psychiatry · doi:10.3389/fpsyt.2025.1735421
Bengtsson, D., Sahlin, H., Kaldo, V., & Falkenström, F. (2026) · Internet Interventions · doi:10.1016/j.invent.2026.100937
“The primary aim was to evaluate feasibility and acceptability, operationalized as adherence, engagement, and completion, client satisfaction, negative effects, recruitment source, and adherence to the assessment procedures (Attkisson and Zwick, 1982; Rozental et al., 2016).”
Janssen, W. C., Mérelle, S. Y. M., Gilissen, R., & Bockting, C. (2026) · Trials · doi:10.1186/s13063-026-09725-0
“The short version of the Negative Effects Questionnaire (NEQ; [37]) will be used to identify any adverse effects of either intervention.”
Skoko, A., Hlynsson, J. I., Čihařová, M., Berger, T., Carlbring, P., Krieger, T., Bergström, J., Daele, T. V., van Klaveren, C., Cornelisz, I., et al. (2026) · doi:10.2196/preprints.103262
Self-citationWilhelm, M., Basedow, L. A., Salzmann, S., Seewald, A., Bingel, U., & Rief, W. (2026) · Scientific Reports · doi:10.1038/s41598-026-60951-7
“These may include the emergence of new symptoms, interpersonal or emotional strain, or dependency on the therapist 21,22 .”
Jutzi, F. K., Witthöft, M., Baumeister, H., Denninger, M., Rief, W., Bleichhardt, G. M., Bailer, J., Kirsch, P., Erkic, M., Mier, D. U., et al. (2026) · Trials · doi:10.1186/s13063-026-09886-y
“…Systems Questionnaire [64]; TAI, Therapeutic Agency, therapeutic‑processing subscale [65]; NEQ, Negative Effects Questionnaire – 20 Item Version [66]; NoMAD, Normalization Measure Development questionnaire [67]; GEEE, Generic rating scale for previous treatment experiences, treatment…”
Beaudoin, M., Potvin, S., Giguère, S., Giguère, C., Aardema, F., Abdel‐Baki, A., Benedictis, L. D., Lipp, O., Lalonde, P., Stip, É., et al. (2026) · JMIR Mental Health · doi:10.2196/106791
“The Negative Effects Questionnaire (NEQ), an instrument designed to assess adverse and unwanted effects of psychological treatments, was administered [38].”
Infra-slow EEG neurofeedback for insomnia: a single-case experimental study in primary care
Edvardsson, A., Smith, M. G., Rolstad, S., Steingrímsson, S., Enriquez‐Geppert, S., Balt, K., Landahl, J., & Weineland, S. (2026) · Frontiers in Human Neuroscience · doi:10.3389/fnhum.2026.1832178
“The NEQ includes 32 questions about adverse events and effects of psychological treatment (Rozental et al., 2016).”
Řiháček, T., & Cviková, V. (2026) · Psychotherapy Research · doi:10.1080/10503307.2026.2704062
Hennemann, S., Fähnrich, J. M., Tietze, C., Witthöft, M., & Jungmann, S. M. (2026) · Digital Health · doi:10.1177/20552076261450336 · Cited by 1
“35 The NEQ consists of 20 items covering symptoms, quality, dependency, stigma, and hopelessness.”
Lindegaard, T., Berg, M., Käll, A., Lockman, J., Schönberg, E., Björklund, M., Frederick, R. J., & Andersson, G. (2026) · Psychodynamic Psychiatry · doi:10.1521/pdps.2026.54.3.511
Self-citationSkoko, A., Hlynsson, J. I., Čihařová, M., Berger, T., Carlbring, P., Krieger, T., Bergström, J., Daele, T. V., van Klaveren, C., Cornelisz, I., et al. (2026) · JMIR Research Protocols · doi:10.2196/103262
Self-citationMapping negative experiences in psychotherapy: A network analysis
Bobei, T., Flückiger, C., & Podina, I. R. (2026) · Psychotherapy Research · doi:10.1080/10503307.2026.2743645
Hlynsson, J. I., Andersson, G., & Carlbring, P. (2026) · Internet Interventions · doi:10.1016/j.invent.2026.101012
“Items can be summed to index the frequency of negative effects experienced, categorized by treatment or other circumstances (Rozental et al., 2016, 2019).”
Self-citation
Metadata
- Authors
- Rozental, A., Kottorp, A., Boettcher, J., Andersson, G., & Carlbring, P.
- Affiliations
- Stockholm University (SE)
- Karolinska Institutet (SE)
- Freie Universität Berlin (DE)
- Linköping University (SE)
- Corresponding author
- Rozental, Alexander
- Venue
- PLOS ONE
- Publisher
- Public Library of Science (PLoS)
- Published
- 2016-06-22
- Volume
- 11
- Issue
- 6
- Pages
- e0157503
- ISSN
- 1932-6203 (ISSN-L)
- PMC
- PMC4917117
- OpenAlex
- W2472395746
- Cited by
- 223
- References
- 94
- Citations by year
- 2026: 23 · 2025: 39 · 2024: 23 · 2023: 27 · 2022: 31 · 2021: 20 · 2020: 16 · 2019: 25 · 2018: 12 · 2017: 6 · 2016: 1
- FWCI
- 19.61
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- PLoS ONE (publishedVersion, cc-by)
- Europe PMC (PubMed Central) (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- DiVA (Linkoping University) (submittedVersion, other-oa)
- Zürcher Hochschule für Angewandte Wissenschaften digital collection (Zurich University of Applied Sciences) (submittedVersion, cc-by-nc-nd)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Refubium (Universitätsbibliothek der Freien Universität Berlin) (submittedVersion, cc-by)
- ZHAW Digital Collection (cc-by)
- Funders
- FORTE (FORTE 2013-1107)
- Topics
- Social Sciences › Psychology › Clinical Psychology › Psychotherapy Techniques and Applications
- Social Sciences › Psychology › Applied Psychology › Digital Mental Health Interventions
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Anxiety, Depression, Psychometrics, Treatment, Cognitive Processes
- Keywords
- adverse events, unwanted events, self-report instrument development, social anxiety disorder, symptom deterioration, treatment quality, therapeutic relationship, stigma, hopelessness, factor structure
- MeSH terms
- Adult, Mental Disorders, Demography, Factor Analysis, Statistical, Female, Humans, Male, Surveys and Questionnaires
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
