The Negative Effects Questionnaire: psychometric properties of an instrument for assessing negative effects in psychological treatments
Open accessAbstract
Background: Psychological treatments provide many benefits for patients with psychiatric disorders, but research also suggests that negative effects might occur from the interventions involved. The Negative Effects Questionnaire (NEQ) has previously been developed as a way of determining the occurrence and characteristics of such incidents, consisting of 32 items and six factors. However, the NEQ has yet to be examined using modern test theory, which could help to improve the understanding of how well the instrument works psychometrically. Aims: The current study investigated the reliability and validity of the NEQ from both a person and item perspective, establishing goodness-of-fit, item bias, and scale precision. Method: The NEQ was distributed to 564 patients in five clinical trials at post-treatment. Data were analysed using Rasch analysis, i.e. a modern test theory application. Results: (1) the NEQ exhibits fairness in testing across sociodemographics, (2) shows comparable validity for a final and condensed scale of 20 instead of 32 items, (3) uses a rating scale that advances monotonically in steps of 0 to 4, and (4) is suitable for monitoring negative effects on an item-level. Conclusions: The NEQ is proposed as a useful instrument for investigating negative effects in psychological treatments, and its newer shorter format could facilitate its use in clinical and research settings. However, further research is needed to explore the relationship between negative effects and treatment outcome, as well as to test it in more diverse patient populations.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The Negative Effects Questionnaire-20 has demonstrated satisfactory psychometric properties.
“The 20-item version has demonstrated satisfactory psychometric properties [52].”
From Internet-Delivered Therapy for Parents With Health Anxiety by Proxy: Protocol for a Single-Case Experimental Design Study (2023) · cites this paper as [52]
The NEQ has excellent internal consistency (Cronbach's α = 0.95).
“The internal consistency for the full instrument is excellent (Cronbach's α = 0.95), and is scored on several dimensions; 1) “Did you experience this?” (yes/no), 2) “If yes – here is how negatively it affected me”, from 0 (“Not at all”) to 4 (“Extremely”), and 3) “Probably caused by” (“The treatment I received” or “Other circumstances”).”
From A study protocol of a randomized controlled study of internet-based cognitive behavioral therapy for adult attention deficit hyperactivity disorder (2023) · cites this paper as Rozental et al., 2019
The 20-item NEQ shows comparable validity to the original 32-item version.
“The 20-item NEQ shows comparable validity to the original 32-item version.”
From Efficacy of an internet-based psychological intervention for problem gambling and gambling disorder: Study protocol for a randomized controlled trial (2021) · cites this paper as Rozental et al., 2019
Negative treatment effects are assessed post-treatment using the Negative Effects Questionnaire, a tool with reliable and valid psychometrics.
“Negative treatment effects are here assessed post-treatment using the negative effects questionnaire (NEQ), a tool with reliable and valid psychometrics.74”
From Development, evaluation and implementation of a digital behavioural health treatment for chronic pain: study protocol of the multiphase DAHLIA project (2022) · cites this paper as [74]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
A Brief Measure for Assessing Generalized Anxiety Disorder: The GAD-7
Spitzer, R. L., Kroenke, K., Williams, J. B. W., & Löwe, B. (2006) · Archives of Internal Medicine · doi:10.1001/archinte.166.10.1092 · Cited by 31,765 · see citing works
Lovibond, P. F., & Lovibond, S. H. (1995) · Behaviour Research and Therapy · doi:10.1016/0005-7967(94)00075-u · Cited by 14,523 · see citing works
Applying the Rasch Model: Fundamental Measurement in the Human Sciences
Bond, T. G., & Fox, C. M. (2003) · Journal of Educational Measurement · doi:10.1111/j.1745-3984.2003.tb01103.x · Cited by 4,503 · see citing works
The Efficacy of Cognitive Behavioral Therapy: A Review of Meta-analyses
Hofmann, S. G., Asnaani, A., Vonk, I. J. J., Sawyer, A. T., & Fang, A. (2012) · Cognitive Therapy and Research · doi:10.1007/s10608-012-9476-1 · Cited by 3,445 · cited 1× · see citing works
“Research on the efficacy of evidencebased approaches, such as cognitive behaviour therapy (CBT), suggest that a large number of patients improve thanks to the interventions they receive (Hofmann et al., 2012).”
BackgroundA Rating Formulation for Ordered Response Categories
Andrich, D. (1978) · Psychometrika · doi:10.1007/bf02293814 · Cited by 2,984 · see citing works
Chi-Square Tests with One Degree of Freedom; Extensions of the Mantel- Haenszel Procedure
Mantel, N. (1963) · Journal of the American Statistical Association · doi:10.1080/01621459.1963.10500879 · Cited by 2,899 · see citing works
Summated Rating Scale Construction
Spector, P. E. (1992) · Sage eBooks · doi:10.4135/9781412986038 · Cited by 2,173 · see citing works
Krosnick, J. A. (1999) · Annual Review of Psychology · doi:10.1146/annurev.psych.50.1.537 · Cited by 1,852 · see citing works
Löwe, B., Kroenke, K., Herzog, W., & Gräfe, K. (2004) · Journal of Affective Disorders · doi:10.1016/s0165-0327(03)00198-8 · Cited by 1,558 · cited 2× · see citing works
“Several secondary outcome measures were also administered; the Patient Health Questionnaire – 9 items (PHQ-9; Löwe et al., 2004), the Generalized Anxiety Disorder – 7 items (GAD-7; Spitzer et al., 2006), and the Brunnsviken Brief Quality of Life Scale (BBQ; Lindner et al., 2016).”
MethodThe psychotherapeutic process: A research handbook.
Greenberg, L. S., & Pinsof, W. M. (1986) · Cited by 1,386 · see citing works
Optimizing rating scale category effectiveness.
Linacre, J. M. (2002) · PubMed · Cited by 1,287 · see citing works
Chi-Square Tests with One Degree of Freedom; Extensions of the Mantel- Haenszel Procedure
Mantel, N. (1963) · Journal of the American Statistical Association · doi:10.2307/2282717 · Cited by 1,144 · see citing works
Rating Scale Analysis: Rasch Measurement
Wolins, L., Wright, B. D., & Masters, G. N. (1983) · Journal of the American Statistical Association · doi:10.2307/2288670 · Cited by 1,033 · see citing works
McHugh, R. K., & Barlow, D. H. (2010) · American Psychologist · doi:10.1037/a0018121 · Cited by 909 · see citing works
The psychotherapy dose-response effect and its implications for treatment delivery services.
Hansen, N. B., Lambert, M. J., & Forman, E. M. (2002) · Clinical Psychology: Science and Practice · doi:10.1093/clipsy.9.3.329 · Cited by 783 · see citing works
SOLVING MEASUREMENT PROBLEMS WITH THE RASCH MODEL
Wright, B. D. (1977) · Journal of Educational Measurement · doi:10.1111/j.1745-3984.1977.tb00031.x · Cited by 600 · see citing works
Internet-Delivered Psychological Treatments
Andersson, G. (2016) · Annual Review of Clinical Psychology · doi:10.1146/annurev-clinpsy-021815-093006 · Cited by 595 · see citing works
Self-citationResponse rates for CBT for anxiety disorders: need for standardized criteria
Loerinc, A. G., Meuret, A. E., Twohig, M. P., Rosenfield, D., Bluett, E. J., & Craske, M. G. (2015) · Clinical Psychology Review · doi:10.1016/j.cpr.2015.08.004 · Cited by 592 · see citing works
Implementing routine outcome monitoring in clinical practice: Benefits, challenges, and solutions
Boswell, J. F., Kraus, D. R., Miller, S. D., & Lambert, M. J. (2015) · Psychotherapy Research · doi:10.1080/10503307.2013.817696 · Cited by 494 · see citing works
Rasch fit statistics and sample size considerations for polytomous data
Smith, A. B., Rush, R., Fallowfield, L. J., Velikova, G., & Sharpe, M. (2008) · BMC Medical Research Methodology · doi:10.1186/1471-2288-8-33 · Cited by 387 · see citing works
Consensus statement on defining and measuring negative effects of internet interventions
Rozental, A., Andersson, G., Boettcher, J., Ebert, D. D., Cuijpers, P., Knaevelsrud, C., Ljótsson, B., Kaldo, V., Titov, N., & Carlbring, P. (2014) · Internet Interventions · doi:10.1016/j.invent.2014.02.001 · Cited by 371 · see citing works
Self-citationStrupp, H. H., & Hadley, S. W. (1977) · American Psychologist · doi:10.1037//0003-066x.32.3.187 · Cited by 336 · see citing works
Strupp, H. H., & Hadley, S. W. (1977) · American Psychologist · doi:10.1037/0003-066x.32.3.187 · Cited by 286 · see citing works
Bond, T. G., & Fox, C. M. (2013) · Psychology Press eBooks · doi:10.4324/9781410614575 · Cited by 270 · see citing works
Training implications of harmful effects of psychological treatments.
Castonguay, L. G., Boswell, J. F., Constantino, M. J., Goldfried, M. R., & Hill, C. E. (2010) · American Psychologist · doi:10.1037/a0017330 · Cited by 240 · see citing works
Features and functionality of the Iterapi platform for internet-based psychological treatment
Vlaescu, G., Alasjö, A., Miloff, A., Carlbring, P., & Andersson, G. (2016) · Internet Interventions · doi:10.1016/j.invent.2016.09.006 · Cited by 229 · see citing works
Self-citationLadwig, I., Rief, W., & Nestoriuc, Y. (2014) · Verhaltenstherapie · doi:10.1159/000367928 · Cited by 227 · see citing works
Patient experience of negative effects of psychological treatment: results of a national survey
Crawford, M. J., Thana, L., Farquharson, L., Palmer, L., Hancock, E., Bassett, P., Clarke, J., & Parry, G. D. (2016) · British Journal of Psychiatry · doi:10.1192/bjp.bp.114.162628 · Cited by 225 · see citing works
Rozental, A., Kottorp, A., Boettcher, J., Andersson, G., & Carlbring, P. (2016) · PLOS ONE · doi:10.1371/journal.pone.0157503 · Cited by 223 · see citing works
Self-citationComparing Rasch measurement and factor analysis
Wright, B. D. (1996) · Structural Equation Modeling A Multidisciplinary Journal · doi:10.1080/10705519609540026 · Cited by 215 · see citing works
A comparison of two spider fear questionnaires
Muris, P., & Merckelbach, H. (1996) · Journal of Behavior Therapy and Experimental Psychiatry · doi:10.1016/s0005-7916(96)00022-5 · Cited by 215 · see citing works
The Brunnsviken Brief Quality of Life Scale (BBQ): development and psychometric evaluation
Lindner, P., Frykheden, O., Forsström, D., Andersson, E., Ljótsson, B., Hedman, E., Andersson, G., & Carlbring, P. (2016) · Cognitive Behaviour Therapy · doi:10.1080/16506073.2016.1143526 · Cited by 212 · see citing works
Self-citationRozental, A., Boettcher, J., Andersson, G., Schmidt, B., & Carlbring, P. (2015) · Cognitive Behaviour Therapy · doi:10.1080/16506073.2015.1008033 · Cited by 189 · cited 1× · see citing works
“…statement among researchers (Rozental et al., 2014), a qualitative analysis of the responses to open-ended questions among patients in treatment (Rozental et al., 2015), and a literature review, items for the Negative Effects Questionnaire (NEQ) were generated and investigated using an…”
MethodSelf-citationRasch model analysis of the Depression, Anxiety and Stress Scales (DASS)
Shea, T. L., Tennant, A., & Pallant, J. (2009) · BMC Psychiatry · doi:10.1186/1471-244x-9-21 · Cited by 184 · see citing works
Rozental, A., Magnusson, K., Boettcher, J., Andersson, G., & Carlbring, P. (2017) · Journal of Consulting and Clinical Psychology · doi:10.1037/ccp0000158 · Cited by 137 · see citing works
Self-citationWaugh, R., & Chapman, E. S. (2005) · PubMed · Cited by 107 · see citing works
Suh, C. S., Strupp, H. H., & O’Malley, S. S. (1986) · Cited by 100 · see citing works
Rasch model estimation: further topics.
Linacre, J. M. (2004) · PubMed · Cited by 98 · see citing works
P-DRIVE: ASSESSMENT OF DRIVING PERFORMANCE AFTER STROKE
Patomella, A., Tham, K., & Kottorp, A. (2006) · Journal of Rehabilitation Medicine · doi:10.1080/16501970600632594 · Cited by 91 · see citing works
Self-citationSide effects in internet-based interventions for social anxiety disorder
Boettcher, J., Rozental, A., Andersson, G., & Carlbring, P. (2014) · Internet Interventions · doi:10.1016/j.invent.2014.02.002 · Cited by 88 · see citing works
Self-citationAbramovitch (2015) · doi:10.1016/j.jocrd.2015.02.002 · Cited by 72 · see citing works
Parker, G. B., Fletcher, K., Berk, M., & Paterson, A. (2013) · Psychiatry Research · doi:10.1016/j.psychres.2012.11.026 · Cited by 45 · see citing works
Rate and predictors of negative effects of psychotherapy in psychiatric and psychosomatic inpatients
Rheker, J., Beisel, S., Kräling, S., & Rief, W. (2017) · Psychiatry Research · doi:10.1016/j.psychres.2017.04.042 · Cited by 44 · see citing works
Lerdal, A., Kottorp, A., Gay, C., Aouizerat, B. E., Lee, K. A., & Miaskowski, C. A. (2016) · Journal of Pain and Symptom Management · doi:10.1016/j.jpainsymman.2015.12.331 · Cited by 32 · see citing works
Self-citationInternet-Based Cognitive Behavior Therapy
Andersson, G., Carlbring, P., & Hadjistavropoulos, H. (2017) · Elsevier eBooks · doi:10.1016/b978-0-12-803457-6.00021-0 · Cited by 24 · see citing works
Self-citationMechler, J., & Holmqvist, R. (2015) · Nordic Journal of Psychiatry · doi:10.3109/08039488.2015.1028438 · Cited by 17 · see citing works
Flagged first, then by citations
Research works printed but not linked: 4
- Fisher, W. (1992). Reliability, separation, strata statistics. Rasch Measurement Transactions, 6, 238. doi: https://doi.org/
- Linacre, J. M. (2005). Rasch dichotomous model vs. one-parameter logistic model. Rasch Measurement Transactions, 19, 1032.
- Raîche, G. (2005). Critical eigenvalue sizes in standardized residual principal components analysis. Rasch Measurement Transactions, 19, 1012.
- Spitzer, R. L., Kroenke, K., Williams, J. B. W. and Löwe, B. (2006). A brief measure for assessing generalized anxiety disor- der. Archives of Internal Medicine, 166, 1092. doi: https://doi.org/
Cited by
Sorted by citations
Defining and Measuring Meditation-Related Adverse Effects in Mindfulness-Based Programs
Britton, W. B., Lindahl, J. R., Cooper, D. J., Canby, N. K., & Palitsky, R. (2021) · Clinical Psychological Science · doi:10.1177/2167702621996340 · Cited by 278 · see citing works
Curran, J., Parry, G. D., Hardy, G. E., Darling, J., Mason, A.-M., & Chambers, E. (2019) · Frontiers in Psychology · doi:10.3389/fpsyg.2019.00347 · Cited by 84 · see citing works
Rozental, A., Andersson, G., & Carlbring, P. (2019) · Frontiers in Psychology · doi:10.3389/fpsyg.2019.00589 · Cited by 74 · see citing works
Riva, G., Bernardelli, L., Browning, M. H. E. M., Castelnuovo, G., Cavedoni, S., Chirico, A., Cipresso, P., de Paula, D. M. B., Di Lernia, D., Fernández-Álvarez, J., et al. (2020) · Frontiers in Psychiatry · doi:10.3389/fpsyt.2020.563319 · Cited by 72 · see citing works
The Safety of Digital Mental Health Interventions: Systematic Review and Recommendations
Taher, R., Hsu, C.-W., Hampshire, C., Fialho, C., Heaysman, C., Stahl, D., Shergill, S., & Yiend, J. (2023) · JMIR Mental Health · doi:10.2196/47433 · Cited by 68 · see citing works
Negative effects of psychotherapy: estimating the prevalence in a random national sample
Strauss, B. M., Gawlytta, R., Schleu, A., & Frenzl, D. (2021) · BJPsych Open · doi:10.1192/bjo.2021.1025 · Cited by 59 · see citing works
Fenlon, D., Maishman, T., Day, L., Nuttall, J., May, C. R., Ellis, M. R., Raftery, J. P., Turner, L., Fields, J., Griffiths, G. O., et al. (2020) · Psycho-Oncology · doi:10.1002/pon.5432 · Cited by 59 · see citing works
Hensler, I., Sveen, J., Cernvall, M., & Arnberg, F. K. (2022) · Journal of Medical Internet Research · doi:10.2196/31419 · Cited by 51 · see citing works
Pellas, J., Renner, F., Ji, J. L., & Damberg, M. (2021) · International Journal of Geriatric Psychiatry · doi:10.1002/gps.5646 · Cited by 45 · see citing works
Foulkes, L., Andrews, J. L., Reardon, T. C., & Stringaris, A. K. (2024) · Nature Mental Health · doi:10.1038/s44220-024-00208-2 · Cited by 44 · see citing works
Systematic review and meta-analysis of adverse events in clinical trials of mental health apps
Linardon, J., Fuller-Tyszkiewicz, M., Firth, J., Goldberg, S. B., Anderson, C., McClure, Z., & Torous, J. (2024) · npj Digital Medicine · doi:10.1038/s41746-024-01388-y · Cited by 44 · see citing works
Client perspectives on psychotherapy failure
Knox, S., Miller, C., Twidwell, R. E., & Knowlton, G. (2022) · Psychotherapy Research · doi:10.1080/10503307.2022.2110020 · Cited by 38 · see citing works
Lätth, J., Landgren, V., McMahan, A., Sparre, C., Eriksson, J., Malki, K., Söderquist, E., Öberg, K. G., Rozental, A., Andersson, G., et al. (2022) · Internet Interventions · doi:10.1016/j.invent.2022.100590 · Cited by 38 · see citing works
Self-citationSpanhel, K., Hovestadt, E., Lehr, D., Spiegelhalder, K., Baumeister, H., Bengel, J., & Sander, L. B. (2022) · Frontiers in Psychiatry · doi:10.3389/fpsyt.2022.832196 · Cited by 37 · see citing works
Effectiveness of a Web-Based Cognitive Behavioral Self-Help Intervention for Binge Eating Disorder
Pruessner, L., Timm, C., Barnow, S., Rubel, J. A., Lalk, C., & Hartmann, S. F. (2024) · JAMA Network Open · doi:10.1001/jamanetworkopen.2024.11127 · Cited by 39 · see citing works
Schaeuffele, C., Homeyer, S. L., Perea, L., Scharf, L., Schulz, A., Knaevelsrud, C., Renneberg, B., & Boettcher, J. (2022) · PLoS ONE · doi:10.1371/journal.pone.0270178 · Cited by 39 · see citing works
Self-citationMaroti, D., Ek, J., Widlund, R.-M., Schubiner, H. H., Lumley, M. A., Lilliengren, P., Bilevičiūtė-Ljungar, I., Ljótsson, B., & Johansson, R. (2021) · Frontiers in Psychiatry · doi:10.3389/fpsyt.2021.620359 · Cited by 32 · see citing works
Lundström, L., Flygare, O., Ivanova, E., Mataix‐Cols, D., Enander, J., Djurfeldt, D. R., Chen, L. L., Andersson, E., & Rück, C. (2023) · Internet Interventions · doi:10.1016/j.invent.2023.100608 · Cited by 32 · see citing works
Andersson, R., Åhlén, J., Mataix‐Cols, D., Lenhard, F., Henje, E., Månsson, C., Sahlin, H., Beckman, M., Serlachius, E., & Vigerland, S. (2022) · BMJ Open · doi:10.1136/bmjopen-2022-066357 · Cited by 31 · see citing works
Baumeister, H., Bauereiss, N., Zarski, A.-C., Braun, L., Buntrock, C., Hoherz, C., Idrees, A. R., Kraft, R., Meyer, P., Nguyen, T. B. D., et al. (2021) · Frontiers in Psychiatry · doi:10.3389/fpsyt.2021.660534 · Cited by 30 · see citing works
“Additionally, possible negative effects of the psychotherapies are assessed by means of the 20-item version of the Negative Effects Questionnaire [NEQ; 20 items; (90, 91)] which measures the frequency (score range: 0–20) and impact (score range: 0–80) of several different negative effects during the treatment period.”
Methodvan de Leur, J. C., Buhrman, M., Åhs, F., Rozental, A., & Jansen, G. B. (2020) · BMC Psychiatry · doi:10.1186/s12888-020-02907-3 · Cited by 30 · see citing works
Self-citationDülsen, P., & Baumeister, H. (2024) · European Child & Adolescent Psychiatry · doi:10.1007/s00787-024-02404-y · Cited by 28 · see citing works
Bartels, S. L., Johnsson, S. I., Boersma, K., Flink, I., McCracken, L. M., Petersson, S., Christie, H. L., Feldman, I., Simons, L. E., Onghena, P., et al. (2022) · BMJ Open · doi:10.1136/bmjopen-2021-059152 · Cited by 23 · see citing works
Gullickson, K. M., Hadjistavropoulos, H. D., Dear, B. F., & Titov, N. (2019) · Internet Interventions · doi:10.1016/j.invent.2019.100278 · Cited by 26 · see citing works
Wiklund, T., Molander, P., Lindner, P., Andersson, G., Gerdle, B. U. C., & Dragioti, E. (2022) · Journal of Medical Internet Research · doi:10.2196/29258 · Cited by 25 · see citing works
Self-citationHuang, S., Wang, Y., Li, G., Hall, B. J., & Nyman, T. J. (2024) · JMIR Mental Health · doi:10.2196/56650 · Cited by 25 · see citing works
Mutter, A., Küchler, A., Idrees, A. R., Kählke, F., Terhorst, Y., & Baumeister, H. (2023) · BMC Psychology · doi:10.1186/s40359-023-01312-1 · Cited by 22 · see citing works
Making the leap: From experimental psychopathology to clinical trials
Blackwell, S. E., & Woud, M. L. (2022) · Journal of Experimental Psychopathology · doi:10.1177/20438087221080075 · Cited by 18 · see citing works
Wiberg, A.-C., Ghaderi, A., Danielsson, H. B., Safarzadeh, K., Parling, T., Carlbring, P., Jansson, M., & Welch, E. (2022) · Internet Interventions · doi:10.1016/j.invent.2022.100570 · Cited by 22 · see citing works
Self-citationBlackwell, S. E., Schönbrodt, F. D., Woud, M. L., Wannemüller, A., Bektas, B., Rodrigues, M. B., Hirdes, J., Stumpp, M. T., & Margraf, J. (2022) · Psychological Medicine · doi:10.1017/s0033291722003294 · Cited by 20 · see citing works
Finazzi, E., & MacBeth, A. M. (2021) · Clinical Psychology & Psychotherapy · doi:10.1002/cpp.2650 · Cited by 19 · see citing works
Verkooyen, R., Broers, N. J., & Dandachi-FitzGerald, B. (2024) · Frontiers in Psychology · doi:10.3389/fpsyg.2024.1378456 · Cited by 18 · see citing works
Web-Based Cognitive Behavioral Treatment for Bulimia Nervosa
Hartmann, S. F., Timm, C., Barnow, S., Rubel, J. A., Lalk, C., & Pruessner, L. (2024) · JAMA Network Open · doi:10.1001/jamanetworkopen.2024.19019 · Cited by 17 · see citing works
Dietrichkeit, M., Hagemann-Goebel, M., Nestoriuc, Y., Moritz, S., & Jelinek, L. (2021) · Scientific Reports · doi:10.1038/s41598-021-87198-8 · Cited by 16 · see citing works
Spiegelhalder, K., Baumeister, H., Al-Kamaly, A., Bader, M., Bauereiß, N., Benz, F., Braun, L., Buntrock, C., Burkhardt, M., Cuijpers, P. J., et al. (2022) · BMJ Open · doi:10.1136/bmjopen-2021-058212 · Cited by 16 · see citing works
Dandachi‐FitzGerald, B., Houben, S. T. L., Broers, N. J., & Merckelbach, H. L. G. J. (2024) · Clinical Psychology & Psychotherapy · doi:10.1002/cpp.3014 · Cited by 15 · see citing works
Moriuchi, E., Berbary, C. M., & Easton, C. J. (2023) · Journal of Technology in Behavioral Science · doi:10.1007/s41347-022-00298-8 · Cited by 14 · see citing works
Jensen, K. H. R., Dam, V. H., Ganz, M., Fisher, P. M., Ip, C.-T., Sankar, A., Marstrand-Joergensen, M. R., Ozenne, B., Osler, M., Penninx, B. W. J. H., et al. (2023) · BMC Psychiatry · doi:10.1186/s12888-023-04618-x · Cited by 16 · see citing works
Mathiesen, A. S., Zoffmann, V., Skytte, T. B., Jakobsen, J. C., Gluud, C. N., Lindschou, J., Rasmussen, B., Marqvorsen, E. H. S., Thomsen, T., & Rothmann, M. J. (2021) · BMJ Open · doi:10.1136/bmjopen-2020-047037 · Cited by 13 · see citing works
Schaeuffele, C., Heinrich, M., Behr, S., Fenski, F., Hammelrath, L., Zagorscak, P., Jansen, A., Pohl, S., Boettcher, J., & Knaevelsrud, C. (2022) · Trials · doi:10.1186/s13063-022-06757-0 · Cited by 14 · see citing works
Self-citationTaher, R., Hall, C. L., Bergin, A. D. G., Gupta, N., Heaysman, C., Jacobsen, P., Kabir, T., Kalnad, N., Keppens, J., Hsu, C.-W., et al. (2024) · Trials · doi:10.1186/s13063-024-08421-1 · Cited by 8 · see citing works
Watson, P., Leroux, E. J., Chowdhury, M., Mehra, K., Henderson, J. L., Szatmári, P., & Hawke, L. D. (2022) · Journal of Child and Family Studies · doi:10.1007/s10826-022-02431-w · Cited by 12 · see citing works
Herzog, P., Häusler, S., Normann, C., & Brakemeier, E.-L. (2021) · Frontiers in Psychiatry · doi:10.3389/fpsyt.2021.575837 · Cited by 13 · see citing works
Internet-based self-help intervention for procrastination: randomized control group trial protocol
Ying, Z., & Wang, J. (2023) · Trials · doi:10.1186/s13063-023-07112-7 · Cited by 17 · see citing works
Wittmann, L., Blomert, E., & Linden, M. (2023) · Psychotherapy Research · doi:10.1080/10503307.2023.2290029 · Cited by 12 · see citing works
Austin, S. F., Frøsig, A. J., Buus, N., Lincoln, T. M., von Malachowski, A., Schlier, B., Frost, M. M., & Simonsen, E. H. (2021) · Qualitative Health Research · doi:10.1177/1049732320986556 · Cited by 13 · see citing works
Berman, A. H., Topooco, N., Lindfors, P., Bendtsen, M., Lindner, P., Molander, O., Kraepelien, M., Sundström, C., Talebizadeh, N., Engström, K., et al. (2024) · Trials · doi:10.1186/s13063-024-07986-1 · Cited by 9 · see citing works
Self-citationHooi, L. Y., Chen, P. L., Tan, K. W., de Vries, M., & Wong, H. K. (2025) · Scientific Reports · doi:10.1038/s41598-025-23727-z · Cited by 17 · see citing works
Adaptation and Feasibility of the Mandarin Version of PEERS® for Autistic Adolescents
Lao, U., Li, Y., Bai, W., Wang, Y., Li, Y., Xie, Y., Huang, X., Zhu, H., & Zou, X. (2023) · Journal of Autism and Developmental Disorders · doi:10.1007/s10803-023-06056-w · Cited by 11 · see citing works
Mide, M., Mattiasson, J., Norlin, D., Sehlin, H., Rasmusson, J., Ljung, S., Lindskog, A., Petersson, J., Saavedra, F., & Gordh, A. S. (2023) · Frontiers in Psychiatry · doi:10.3389/fpsyt.2023.1243826 · Cited by 12 · see citing works
Side Effects in Psychodynamic and Cognitive Behavior Therapy
Balder, T., Linden, M., & Rose, M. (2024) · Journal of Contemporary Psychotherapy · doi:10.1007/s10879-023-09615-5 · Cited by 12 · see citing works
The preliminary development and psychometric properties of the Psychotherapy Side Effects Scale
Chen, F., Liu, L., Xudong, Z., Qiang, F., Congcong, G., & Yunhan, Z. (2023) · Brain and Behavior · doi:10.1002/brb3.2885 · Cited by 10 · see citing works
School-based social skills group training (SKOLKONTAKT™): a pilot randomized controlled trial
Fridell, A., Coco, C., Borg, A., & Bölte, S. (2023) · Frontiers in Psychology · doi:10.3389/fpsyg.2023.1128288 · Cited by 10 · see citing works
Balci, S., Küchler, A., Ebert, D. D., & Baumeister, H. (2024) · BMC Digital Health · doi:10.1186/s44247-024-00074-z · Cited by 11 · see citing works
Recommendations on recording harms in randomised controlled trials of behaviour change interventions
Papaioannou, D. E., Hamer‐Kiwacz, S., Mooney, C., Sprange, K., Cooper, C. L., & O’Cathain, A. (2024) · BMJ · doi:10.1136/bmj-2023-077418 · Cited by 10 · see citing works
Lassen, N. F., Hougaard, E., Arendt, K. B., & Thastum, M. (2019) · Trials · doi:10.1186/s13063-019-3885-3 · Cited by 10 · see citing works
Ekbäck, E., Granåsen, G., Svärling, R., Blomqvist, I., & Henje, E. (2021) · Frontiers in Psychiatry · doi:10.3389/fpsyt.2021.674583 · Cited by 10 · see citing works
Thaysen‐Petersen, D., Hammerum, S. K., Vissing, A.-C., Arnfred, B. T., Nordahl, R., Adjorlu, A., Nordentoft, M., Oestrich, I. H., Düring, S. W., & Fink‐Jensen, A. (2023) · BMJ Open · doi:10.1136/bmjopen-2022-068658 · Cited by 10 · see citing works
Pedersen, S. S., Skovbakke, S. J., Skov, O., Carlbring, P., Burg, M. M., Habibović, M., & Ahm, R. (2023) · Current Cardiology Reports · doi:10.1007/s11886-023-01867-w · Cited by 11 · see citing works
Self-citationDiaz-Sanahuja, L., Campos, D., Mira, A., Castilla, D., García-Palacios, A., & Bretón-López, J. M. (2021) · Internet Interventions · doi:10.1016/j.invent.2021.100466 · Cited by 10 · see citing works
Lundin, J., Jansson‐Fröjmark, M., Gustafsson-Björverud, L., Grey, N., Santoft, F., Ehlers, A., Carlbring, P., Lundgren, T., Bragesjö, M., & Salomonsson, S. (2024) · Frontiers in Psychiatry · doi:10.3389/fpsyt.2024.1447651 · Cited by 9 · see citing works
Self-citationMolander, O., Berman, A. H., Jakobson, M., Gajecki, M., Hällström, H., Ramnerö, J., Bjureberg, J., Carlbring, P., & Lindner, P. (2024) · Journal of Medical Internet Research · doi:10.2196/54754 · Cited by 9 · see citing works
Self-citationAxelsson, E., & Hedman-Lagerlöf, E. (2023) · Expert review of pharmacoeconomics & outcomes research · doi:10.1080/14737167.2023.2250915 · Cited by 7 · see citing works
Benz, F., Grolig, L., Hannibal, S., Buntrock, C., Cuijpers, P. J., Domschke, K., Ebert, D. D., Ell, J., Janneck, M., Jenkner, C., et al. (2024) · Trials · doi:10.1186/s13063-024-08214-6 · Cited by 8 · see citing works
Stark, R. M., Markert, C., Golder, S., Psarros, R., Discher, J. P., Khatib, S., Metzger, J., Palmer, S., Rechmann, J., Storz, F., et al. (2024) · Journal of Behavioral Addictions · doi:10.1556/2006.2024.00046 · Cited by 10 · see citing works
Yiend, J., Taher, R., Fialho, C., Hampshire, C., Hsu, C.-W., Kabir, T., Keppens, J., McGuire, P. K., Mouchlianitis, E. D., PETERS, E. R., et al. (2024) · Trials · doi:10.1186/s13063-024-08570-3 · Cited by 7 · see citing works
Seitz, K. I., Schouler, N., Hundertmark, J., Wilhelm, M., Franz, S., Bauer, S., Taubner, S., Korn, C. W., Haun, M. W., Ditzen, B., et al. (2025) · BMJ Open · doi:10.1136/bmjopen-2024-090476 · Cited by 8 · see citing works
Hentati, A., Forsell, E., Ljótsson, B., Lindefors, N., & Kraepelien, M. (2022) · BJPsych Open · doi:10.1192/bjo.2022.14 · Cited by 8 · see citing works
Pruessner, L., Hartmann, S. F., Rubel, J. A., Lalk, C., Barnow, S., & Timm, C. (2022) · Internet Interventions · doi:10.1016/j.invent.2022.100514 · Cited by 8 · see citing works
Hartmann, S. F., Pruessner, L., Rubel, J. A., Lalk, C., Barnow, S., & Timm, C. (2022) · Internet Interventions · doi:10.1016/j.invent.2022.100512 · Cited by 8 · see citing works
Molander, O., Lindner, P., Ramnerö, J., Bjureberg, J., Carlbring, P., & Berman, A. H. (2020) · Pilot and Feasibility Studies · doi:10.1186/s40814-020-00647-5 · Cited by 8 · see citing works
Self-citationGholamrezaei, A., Magee, M. R., McNeilage, A. G., Dwyer, L., Sim, A., Ferreira, M. L., Darnall, B. D., Brake, T., Aggarwal, A., Craigie, M., et al. (2024) · PAIN Reports · doi:10.1097/pr9.0000000000001128 · Cited by 8 · see citing works
Sennerstam, V., Föyen, L. F., Kontio, E., Svärdman, F., Lekander, M., Lindsäter, E., & Hedman‐Lagerlöf, E. (2025) · Psychotherapy and Psychosomatics · doi:10.1159/000546221 · Cited by 7 · see citing works
Olsen, M. H., Hagstrøm, J., Lønfeldt, N. N., Uhre, C. F., Uhre, V., Pretzmann, L., Christensen, S. H., Thoustrup, C. L., Korsbjerg, N. L. J., Mora‐Jensen, A. C., et al. (2022) · Trials · doi:10.1186/s13063-022-06799-4 · Cited by 7 · see citing works
Hybelius, J., Gustavsson, A., Hammarberg, S. A. W., Toth-Pal, E., Johansson, R., Ljótsson, B., & Axelsson, E. (2022) · Pilot and Feasibility Studies · doi:10.1186/s40814-022-01105-0 · Cited by 7 · see citing works
Dülsen, P., Barck, K., Daubmann, A., Höller, A., Zeidler, J., Kilian, R., Wiegand-Grefe, S., & Baumeister, H. (2022) · Frontiers in Digital Health · doi:10.3389/fdgth.2022.816412 · Cited by 6 · see citing works
Riva, G. (2021) · http://isrctn.com/ · doi:10.1186/isrctn63887521 · Cited by 6 · see citing works
Forsström, D., Oscarsson, M., Buhrman, M., & Rozental, A. (2023) · Internet Interventions · doi:10.1016/j.invent.2023.100652 · Cited by 6 · see citing works
“Negative Effects Questionnaire (NEQ) The NEQ is a 32-item instrument that assesses unwanted and adverse events experienced by patients undergoing psychotherapy (Rozental et al., 2016; Rozental et al., 2019), e.g., “ I experienced more unpleasant feelings ” (Item 11).”
Self-citationArendt, I. T. P., Gondan, M., Juul, S., Hastrup, L. H., Hjorthøj, C. R., Bach, B., Videbech, P., Jørgensen, M. B., & Møeller, S. B. (2024) · Trials · doi:10.1186/s13063-024-08079-9 · Cited by 8 · see citing works
Digital Emotion Regulation Interventions for Patients With Congenital Heart Disease
Pruessner, L., Hartmann, S., Ehmann, A.-L., Barnow, S., Bauer, U. M. M., & Helm, P. (2025) · JAMA Network Open · doi:10.1001/jamanetworkopen.2025.38813 · Cited by 7 · see citing works
Lafortune, D., Lapointe, V. A., Dubé, S., & Bonneau, J. (2024) · The Journal of Sexual Medicine · doi:10.1093/jsxmed/qdae154 · Cited by 6 · see citing works
How to measure negative effects of psychological interventions
Rozental, A., & Řiháček, T. (2025) · Current Opinion in Psychology · doi:10.1016/j.copsyc.2025.102084 · Cited by 6 · see citing works
Self-citationStrauss, B. M. (2021) · International Journal of Group Psychotherapy · doi:10.1080/00207284.2021.1890089 · Cited by 6 · see citing works
Hensler, I., Sveen, J., Cernvall, M., & Arnberg, F. K. (2023) · Internet Interventions · doi:10.1016/j.invent.2023.100618 · Cited by 6 · see citing works
Cunningham, N. R., Miller, A. K., Ely, S. L., Reid, M. R., Danguecan, A. N., Mossad, S. I., Pereira, L. F., Abulaban, K. M., Kessler, E., Rosenwasser, N. L., et al. (2023) · Pediatric Rheumatology · doi:10.1186/s12969-023-00835-6 · Cited by 7 · see citing works
Eklund, M., Kiritsis, C., Livheim, F., & Ghaderi, A. (2023) · Journal of Contextual Behavioral Science · doi:10.1016/j.jcbs.2023.01.003 · Cited by 5 · see citing works
Balsom, A. A., Klest, B., Sander, B., & Gordon, J. L. (2024) · Reproductive Health · doi:10.1186/s12978-024-01766-5 · Cited by 6 · see citing works
Schaeuffele, C., Homeyer, S. L., Perea, L., Scharf, L., Schulz, A., Knaevelsrud, C., Renneberg, B., & Boettcher, J. (2020) · doi:10.31234/osf.io/528tw · Cited by 9 · see citing works
Self-citationPoulter, M. M. L., Wahl, T. D., Kiviharju, M. J., Campbell, T. S., & Gordon, J. L. (2025) · Human Reproduction · doi:10.1093/humrep/deaf237 · Cited by 5 · see citing works
Hlynsson, J. I., Mechler, J., Lindqvist, K., Andersson, G., & Carlbring, P. (2026) · Internet Interventions · doi:10.1016/j.invent.2026.100960 · Cited by 6 · see citing works
Self-citationGordon, J. L., Poulter, M. M. L., Balsom, A. A., & Campbell, T. S. (2023) · JMIR Research Protocols · doi:10.2196/52662 · Cited by 5 · see citing works
Ivanova, E. N., Fondberg, R., Flygare, O., Sannemalm, M., Asplund, S., Dahlén, S., Sampaio, F., Andersson, E., Mataix‐Cols, D., Ivanov, V. Z., et al. (2023) · BMJ Open · doi:10.1136/bmjopen-2023-076361 · Cited by 5 · see citing works
Seidler, K.-P., Grützmacher, S., Epner, A., & Schreiber‐Willnow, K. (2020) · Psychotherapeut · doi:10.1007/s00278-020-00432-y · Cited by 5 · see citing works
Remote blended treatment for individuals with suicidal ideation: A single‐arm proof‐of‐concept trial
Büscher, R., Teismann, T., Hartleitner, P., Klein, J. P., Baumeister, H., & Sander, L. (2023) · Clinical Psychology & Psychotherapy · doi:10.1002/cpp.2905 · Cited by 5 · see citing works
Gholamrezaei, A., Magee, M. R., McNeilage, A. G., Dwyer, L., Jafari, H., Sim, A., Ferreira, M. L., Darnall, B. D., Glare, P. A., & Ashton-James, C. E. (2023) · BMJ Open · doi:10.1136/bmjopen-2023-073297 · Cited by 5 · see citing works
von Malachowski, A., Schlier, B., Austin, S. F., Frost, M. M., Frøsig, A. J., Heinzle, M., Holzapfel, H., Lipps, A., Simonsen, E. H., & Lincoln, T. M. (2022) · Schizophrenia Research · doi:10.1016/j.schres.2021.12.011 · Cited by 5 · see citing works
Dandachi‐FitzGerald, B., Otgaar, H., & Merckelbach, H. L. G. J. (2022) · Springer eBooks · doi:10.1007/978-3-031-14332-8_16 · Cited by 5 · see citing works
Lindegaard, T., Persson, L., Thorängen, M., & Rozental, A. (2024) · Psychotherapy Research · doi:10.1080/10503307.2024.2420717 · Cited by 4 · see citing works
Self-citationBöge, K., Bergmann, N., Hahne, I., Emde, L., Cartner, S., Bell, I., Pillny, M., & Thomas, N. (2025) · Internet Interventions · doi:10.1016/j.invent.2025.100804 · Cited by 4 · see citing works
Loades, M. E., Perry, G., & Marshall, N. (2025) · JMIR Mental Health · doi:10.2196/72077 · Cited by 4 · see citing works
- Authors
- Rozental, A., Kottorp, A., Forsström, D., Månsson, K., Boettcher, J., Andersson, G., Furmark, T., & Carlbring, P.
- Affiliations
- Karolinska Institutet (SE)
- Malmö University (SE)
- Stockholm University (SE)
- Uppsala University (SE)
- Freie Universität Berlin (DE)
- Linköping University (SE)
- Corresponding author
- Rozental, Alexander
- ORCID
- Venue
- Behavioural and Cognitive Psychotherapy
- Publisher
- Cambridge University Press (CUP)
- Published
- 2019-03-15
- Volume
- 47
- Issue
- 5
- Pages
- 559-572
- ISSN
- 1352-4658 (ISSN-L)
- 1469-1833
- OpenAlex
- W2922529521
- Cited by
- 230
- References
- 46
- Citations by year
- 2026: 47 · 2025: 47 · 2024: 47 · 2023: 32 · 2022: 30 · 2021: 15 · 2020: 7 · 2019: 4 · 2013: 1
- FWCI
- 10.19
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Locations
- Behavioural and Cognitive Psychotherapy (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Uppsala University Publications (Uppsala University) (submittedVersion, other-oa)
- Malmö University Publications (Malmö University) (submittedVersion)
- University of Southern Denmark Research Portal (University of Southern Denmark) (submittedVersion, cc-by)
- Funders
- University of Illinois at Urbana-Champaign
- University of Illinois at Chicago
- Topics
- Social Sciences › Decision Sciences › Management Science and Operations Research › Psychometric Methodologies and Testing
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Anxiety, Depression, Psychometrics, Treatment, Cognitive Processes
- Social Sciences › Psychology › Applied Psychology › Psychological Testing and Assessment
- Keywords
- adverse effects, psychometric properties, Rasch analysis, differential item functioning, item response theory, short-form scale, treatment outcome monitoring, mental illness
- MeSH terms
- Negative Results, Adult, Mental Disorders, Female, Humans, Male, Psychology, Psychometrics, Surveys and Questionnaires, Reproducibility of Results, Treatment Failure
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
