The DOI matches its Crossref record; 64 of the 64 references it cites were found in a registry or an index.
Negative Effects of Internet Interventions: A Qualitative Content Analysis of Patients' Experiences with Treatments Delivered Online
Open accessAbstract
Internet interventions are defined as the delivery of health care-related treatments via an online or a smartphone interface, and have been shown to be a viable alternative to face-to-face treatments. However, not all patients benefit from such treatments, and it is possible that some may experience negative effects. Investigations of face-to-face treatments indicate that deterioration occurs in 5-10% of all patients. The nature and scope of other negative effects of Internet interventions is, however, largely unknown. Hence, the current study explored patients' reported negative experiences while undergoing treatments delivered via the Internet. Data from four large clinical trials (total N = 558) revealed that 9.3% of patients reported some type of negative effects. Qualitative content analysis was used to explore the patients' responses to open-ended questions regarding their negative experiences. Results yielded two broad categories and four subcategories of negative effects: patient-related negative effects (insight and symptom) and treatment-related negative effects (implementation and format). Results emphasize the importance of always considering negative effects in Internet-based interventions, and point to several ways of preventing such experiences, including regular assessment of negative events, increasing the flexibility of treatment schedules and therapist contact, as well as prolonging the treatment duration.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Rozental et al. (2015) in a review from four clinical trials with a total sample of 558 participants identified that 9.3% of the participants indicated a negative effect.”
From A Self-Applied Multi-Component Psychological Online Intervention Based on UX, for the Prevention of Complicated Grief Disorder in the Mexican Population During the COVID-19 Outbreak: Protocol of a Randomized Clinical Trial (2021) · cites this paper as Rozental et al. (2015)
In the citing paper
“Besides positive effects (eg, decrease of symptom severity), IMIs can also have negative side effects (eg, time pressure).118”
In the citing paper
“Rozental and colleagues shows that ICBT has the same rates of deterioration as other treatments, normally 5-10% of patients (Rozental et al., 2015)”
From Internet delivered cognitive behavior therapy for antenatal depression: A randomised controlled trial (2017) · cites this paper as Rozental et al., 2015
Citations
Cites
Three Approaches to Qualitative Content Analysis
Hsieh, H.-F., & Shannon, S. E. (2005) · Qualitative Health Research · doi:10.1177/1049732305276687 · Cited by 44,601
Content Analysis: An Introduction to its Methodology.
Shelley, M., & Krippendorff, K. (1984) · Journal of the American Statistical Association · doi:10.2307/2288384 · Cited by 24,351
The qualitative content analysis process
Elo, S., & Kyngäs, H. (2008) · Journal of Advanced Nursing · doi:10.1111/j.1365-2648.2007.04569.x · Cited by 22,617
Graneheim, U. H., & Lundman, B. (2004) · Nurse Education Today · doi:10.1016/j.nedt.2003.10.001 · Cited by 20,772
Constructing validity: Basic issues in objective scale development.
Clark, L. A., & Watson, D. (1995) · Psychological Assessment · doi:10.1037/1040-3590.7.3.309 · Cited by 6,567
Self-reports: How the questions shape the answers.
Schwarz, N. (1999) · American Psychologist · doi:10.1037/0003-066x.54.2.93 · Cited by 2,846
Content Analysis: A Flexible Methodology
White, M. D., & Marsh, E. (2006) · Library trends · doi:10.1353/lib.2006.0053 · Cited by 2,322
Krosnick, J. A. (1999) · Annual Review of Psychology · doi:10.1146/annurev.psych.50.1.537 · Cited by 1,852
Barak, A., Hen, L., Boniel-Nissim, M., & Shapira, N. (2008) · Journal of Technology in Human Services · doi:10.1080/15228830802094429 · Cited by 1,283
Andersson, G., Cuijpers, P., Carlbring, P., Riper, H., & Hedman, E. (2014) · World Psychiatry · doi:10.1002/wps.20151 · Cited by 1,077
Self-citationPsychological Treatments That Cause Harm
Lilienfeld, S. O. (2007) · Perspectives on Psychological Science · doi:10.1111/j.1745-6916.2007.00029.x · Cited by 959
A lab test and algorithms for identifying clients at risk for treatment failure.
Hannan, C. R., Lambert, M., Harmon, C., Nielsen, S. L., Smart, D., Shimokawa, K., & Sutton, S. (2005) · Journal of Clinical Psychology · doi:10.1002/jclp.20108 · Cited by 550
Internet-based psychological treatments for depression
Johansson, R., & Andersson, G. (2012) · Expert Review of Neurotherapeutics · doi:10.1586/ern.12.63 · Cited by 509
Self-citationLambert, M. (2007) · Psychotherapy Research · doi:10.1080/10503300601032506 · Cited by 461 · cited 3×
“deterioration and the emergence of novel symptoms—similar to the adverse events found in investigations of face-to-face treatments (Boisvert & Faust, 2002; Lambert, 2007)—as well as negative effects primarily associated with the online treatment format.”
BackgroundNegative effects from psychological treatments: a perspective
Barlow, D. H. (2010) · American Psychologist · doi:10.1037/a0015643 · Cited by 426 · cited 3×
“Furthermore, some treatments could result in unintended, negative effects, such as deterioration, novel symptoms, and decreased selfesteem, despite the fact that the patients might be receiving the best available care (Barlow, 2010; Berk & Parker, 2009).”
BackgroundLinden, M. (2013) · Clinical Psychology & Psychotherapy · doi:10.1002/cpp.1765 · Cited by 397 · cited 2×
“These include the Unwanted Event to Adverse Reaction Checklist, the Experience of Therapy Questionnaire and the Inventory for the Assessment of Negative Effects of Psychotherapy (Linden, 2013; Nestoriuc & Rief, 2012; Parker, Fletcher, Berk, & Paterson, 2013).”
MethodConsensus statement on defining and measuring negative effects of internet interventions
Rozental, A., Andersson, G., Boettcher, J., Ebert, D. D., Cuijpers, P., Knaevelsrud, C., Ljótsson, B., Kaldo, V., Titov, N., & Carlbring, P. (2014) · Internet Interventions · doi:10.1016/j.invent.2014.02.001 · Cited by 371
Self-citationHatfield, D., McCullough, L., Frantz, S. H. B., & Krieger, K. (2010) · Clinical Psychology & Psychotherapy · doi:10.1002/cpp.656 · Cited by 321
How would we know if psychotherapy were harmful?
Dimidjian, S., & Hollon, S. D. (2010) · American Psychologist · doi:10.1037/a0017299 · Cited by 314
Wagner, B., Horn, A. B., & Maercker, A. (2014) · Journal of Affective Disorders · doi:10.1016/j.jad.2013.06.032 · Cited by 311
Enhancing psychotherapy outcomes via providing feedback on client progress: a replication
Lambert, M. J., Whipple, J. L., Vermeersch, D. A., Smart, D. W., Hawkins, E. J., Nielsen, S. L., & Goates, M. (2002) · Clinical Psychology & Psychotherapy · doi:10.1002/cpp.324 · Cited by 306
Palmqvist, B., Carlbring, P., & Andersson, G. (2007) · Expert Review of Pharmacoeconomics & Outcomes Research · doi:10.1586/14737167.7.3.291 · Cited by 290
Self-citationCarlbring, P., Bohman, S., Brunt, S., Buhrman, M., Westling, B. E., Ekselius, L., & Andersson, G. (2006) · American Journal of Psychiatry · doi:10.1176/ajp.2006.163.12.2119 · Cited by 262
Self-citationHesser, H., Gustafsson, T., Lundén, C., Henrikson, O., Fattahi, K., Johnsson, E., Westin, V. Z., Carlbring, P., Mäki-Torkko, E., Kaldo, V., et al. (2012) · Journal of Consulting and Clinical Psychology · doi:10.1037/a0027021 · Cited by 257 · cited 1×
“…Dear, & Spence, 2013), irritable bowel syndrome (Ljótsson et al., 2011), social anxiety disorder (Andrews, Davies, & Titov, 2011), tinnitus (Hesser et al., 2012), insomnia (van Straten et al., 2014), generalized anxiety disorder (Titov et al., 2009), andpanic disorder (Carlbring et al.,…”
BackgroundSelf-citationSome implications of psychotherapy research for therapeutic practice.
Bergin, A. E. (1966) · Journal of Abnormal Psychology · doi:10.1037/h0023577 · Cited by 254
Definition, assessment and rate of psychotherapy side effects
Linden, M., & Schermuly-Haupt, M.-L. (2014) · World Psychiatry · doi:10.1002/wps.20153 · Cited by 220
Ljótsson, B., Hedman, E., Andersson, E., Hesser, H., Lindfors, P., Hursti, T., Rydh, S., Rück, C., Lindefors, N., & Andersson, G. (2011) · American Journal of Gastroenterology · doi:10.1038/ajg.2011.139 · Cited by 218 · cited 1×
“…2015 Swedish Association for Behaviour Therapy disorders (Johnston, Titov, Andrews, Dear, & Spence, 2013), irritable bowel syndrome (Ljótsson et al., 2011), social anxiety disorder (Andrews, Davies, & Titov, 2011), tinnitus (Hesser et al., 2012), insomnia (van Straten et al.,…”
BackgroundSelf-citationThe Elephant on the Couch: Side-Effects of Psychotherapy
Berk, M., & Parker, G. (2009) · Australian & New Zealand Journal of Psychiatry · doi:10.1080/00048670903107559 · Cited by 216
Establishing Guidelines for Executing and Reporting Internet Intervention Research
Proudfoot, J., Klein, B., Barak, A., Carlbring, P., Cuijpers, P., Lange, A., Ritterband, L., & Andersson, G. (2011) · Cognitive Behaviour Therapy · doi:10.1080/16506073.2011.573807 · Cited by 202
Self-citationChristensen, H., Griffiths, K., Groves, C., & Korten, A. (2006) · Australian & New Zealand Journal of Psychiatry · doi:10.1080/j.1440-1614.2006.01743.x · Cited by 201
Discontinuities and cognitive changes in an exposure-based cognitive therapy for depression
Hayes, A. M., Feldman, G. C., Beevers, C. G., Laurenceau, J.-P., Cardaciotto, L., & Lewis-Smith, J. (2007) · Journal of Consulting and Clinical Psychology · doi:10.1037/0022-006x.75.3.409 · Cited by 198 · cited 2×
“In other words, not all patient-related negative effects are necessarily problematic or enduring, even though they are perceived as distressing when they first occur, reflecting negative experiences that are temporary in nature and related to specific treatment interventions, but regarded as beneficial, or, at least,…”
BackgroundBendelin, N., Hesser, H., Dahl, J., Carlbring, P., Nelson, K. Z., & Andersson, G. (2011) · BMC Psychiatry · doi:10.1186/1471-244x-11-107 · Cited by 173 · cited 5×
“Even though the results are in line with prior qualitative analyses of patients’ experiences of undergoing Internet interventions (Bendelin et al., 2011; Olsson Halmetoja et al., 2014; Paxling et al., 2013), more research is warranted in order to replicate the findings of the current study concerning negative effects.”
BackgroundResultInfluentialSelf-citationPaxling, B., Lundgren, S., Norman, A., Almlöv, J., Carlbring, P., Cuijpers, P., & Andersson, G. (2013) · Behavioural and Cognitive Psychotherapy · doi:10.1017/s1352465812000240 · Cited by 162
Self-citationAssessing clinical significance: Application to the beck depression inventory
Seggar, L. B., Lambert, M. J., & Hansen, N. B. (2002) · Behavior Therapy · doi:10.1016/s0005-7894(02)80028-4 · Cited by 162
Mohr, D. C., Siddique, J., Ho, J., Duffecy, J., Jin, L., & Fokuo, J. K. (2010) · Annals of Behavioral Medicine · doi:10.1007/s12160-010-9203-7 · Cited by 154 · cited 1×
“For some individuals, direct contact with a health care provider, such as a general practitioner or a clinical psychologist, may still be favored (Mohr et al., 2010).”
BackgroundAndrews, G., Davies, M., & Titov, N. (2011) · Australian & New Zealand Journal of Psychiatry · doi:10.3109/00048674.2010.538840 · Cited by 152
Titov, N., Andrews, G., Robinson, E., Schwencke, G., Johnston, L., Solley, K., & Choi, I. (2009) · Australian & New Zealand Journal of Psychiatry · doi:10.1080/00048670903179269 · Cited by 147 · cited 2×
“…Dear, & Spence, 2013), irritable bowel syndrome (Ljótsson et al., 2011), social anxiety disorder (Andrews, Davies, & Titov, 2011), tinnitus (Hesser et al., 2012), insomnia (van Straten et al., 2014), generalized anxiety disorder (Titov et al., 2009), andpanic disorder (Carlbring et al., 2006).”
BackgroundGuided Internet-delivered cognitive behavioural treatment for insomnia: a randomized trial
van Straten, A., Emmelkamp, J., de Wit, J., Lancee, J., Andersson, G., Someren, E. J. W. V., & Cuijpers, P. J. (2013) · Psychological Medicine · doi:10.1017/s0033291713002249 · Cited by 131
Self-citationContemporary Views of Negative Effects in Psychotherapy: An Integrated Account
Hadley, S. W. (1976) · Archives of General Psychiatry · doi:10.1001/archpsyc.1976.01770110019001 · Cited by 126
Guided internet-based CBT for common mental disorders
Andersson, G., Carlbring, P., Ljótsson, B., & Hedman, E. (2013) · Journal of Contemporary Psychotherapy · doi:10.1007/s10879-013-9237-9 · Cited by 119
Self-citationThe Internet and CBT: A Clinical Guide
Andersson, G. (2014) · Cited by 110
Self-citationIatrogenic Symptoms in Psychotherapy
Boisvert, C. M., & Faust, D. (2002) · American Journal of Psychotherapy · doi:10.1176/appi.psychotherapy.2002.56.2.244 · Cited by 93
Kivi, M., Eriksson, M. C. M., Hange, D., Petersson, E.-L., Vernmark, K., Johansson, B., & Björkelund, C. (2014) · Cognitive Behaviour Therapy · doi:10.1080/16506073.2014.921834 · Cited by 92 · cited 1×
“Similarly, an effectiveness trial comparing ICBT to treatment as usual for mild to moderate depression in a primary health care setting found that 10% of the patients receiving treatment via the Internet deteriorated, compared to 3% in the treatment-as-usual condition (Kivi et al., 2014).”
BackgroundBergman Nordgren, L., Carlbring, P., Linna, E., & Andersson, G. (2013) · JMIR Research Protocols · doi:10.2196/resprot.2292 · Cited by 92
Self-citationSide effects in internet-based interventions for social anxiety disorder
Boettcher, J., Rozental, A., Andersson, G., & Carlbring, P. (2014) · Internet Interventions · doi:10.1016/j.invent.2014.02.002 · Cited by 88
Self-citationInternet-based treatment of pathological gambling with a three-year follow-up
Carlbring, P., Degerman, N., Jonsson, J., & Andersson, G. (2012) · Cognitive Behaviour Therapy · doi:10.1080/16506073.2012.689323 · Cited by 78
Self-citationClinicians' Perspectives on Negative Effects of Psychological Treatments
Bystedt, S., Rozental, A., Andersson, G., Boettcher, J., & Carlbring, P. (2014) · Cognitive Behaviour Therapy · doi:10.1080/16506073.2014.939593 · Cited by 78
Self-citationNegative outcome in psychotherapy and what to do about it
Mays, D. T., & Franks, C. M. (1985) · Springer eBooks · Cited by 75
Ström, M., Uckelstam, C.-J., Andersson, G., Hassmén, P., Umefjord, G., & Carlbring, P. (2013) · PeerJ · doi:10.7717/peerj.178 · Cited by 64
Self-citationThe Internet and CBT: A Clinical Guide
Andersson, G. (2014) · doi:10.1201/b13645 · Cited by 63
Self-citationO'Mahen, H. A., Grieve, H., Jones, J., McGinley, J., Woodford, J., & Wilkinson, E. L. (2015) · Internet Interventions · doi:10.1016/j.invent.2014.11.003 · Cited by 53
Parker, G. B., Fletcher, K., Berk, M., & Paterson, A. (2013) · Psychiatry Research · doi:10.1016/j.psychres.2012.11.026 · Cited by 50
Depression sudden gains and transient depression spikes during treatment for PTSD.
Keller, S. M., Feeny, N. C., & Zoellner, L. A. (2013) · Journal of Consulting and Clinical Psychology · doi:10.1037/a0035286 · Cited by 48
Rozental, A., & Carlbring, P. (2013) · JMIR Research Protocols · doi:10.2196/resprot.2801 · Cited by 47
Self-citationPossible Detrimental Effects of Cognitive Behaviour Therapy for Chronic Fatigue Syndrome
Heins, M. J., Knoop, H., Prins, J. B., Stulemeijer, M., van der Meer, J. W. M., & Bleijenberg, G. (2010) · Psychotherapy and Psychosomatics · doi:10.1159/000315130 · Cited by 47
Halmetoja, C. O., Malmquist, A., Carlbring, P., & Andersson, G. (2014) · Internet Interventions · doi:10.1016/j.invent.2014.08.001 · Cited by 46
Self-citationContraindications and Dangers of Psychotherapy
Crown, S. (1983) · British Journal of Psychiatry · doi:10.1192/bjp.143.5.436 · Cited by 46
Johnston, L., Titov, N., Andrews, G., Dear, B. F., & Spence, J. (2013) · Cognitive Behaviour Therapy · doi:10.1080/16506073.2012.753108 · Cited by 44
Dirkse, D., Hadjistavropoulos, H. D., Hesser, H., & Barak, A. (2014) · Cognitive Behaviour Therapy · doi:10.1080/16506073.2014.952773 · Cited by 43
Lindner, P., Ivanova, E., Ly, K. H., Andersson, G., & Carlbring, P. (2013) · Trials · doi:10.1186/1745-6215-14-437 · Cited by 39
Self-citationThe deterioration effect: A reply to Braucht.
Bergin, A. E. (1970) · Journal of Abnormal Psychology · doi:10.1037/h0029341 · Cited by 31
Getting worse: Psychotherapy or no treatment—the jury should still be out.
Mays, D. T., & Franks, C. M. (1980) · Professional Psychology · doi:10.1037/0735-7028.11.1.78 · Cited by 26
Boettcher, J., Andersson, G., & Carlbring, P. (2013) · Trials · doi:10.1186/1745-6215-14-68 · Cited by 18
Self-citationWhen Things Get Worse: The Problem of Negative Effects in Psychotherapy
Strupp, H. H., Hadley, S. W., & Gomes-Schwartz, B. (1977) · Cited by 11
Research works printed but not linked: 4
- 10.1176/appi.ajp.163.12.2119 · DOI · Paper page
- LambertM.J., BerginA.E. & CollinsJ.L. (1977). Therapist-induced deterioration in psychotherapy. In A.S.Gurman & A.M.Razin (Eds.), Effective psychotherapy: A handbook of research (pp. 452–481). New York: Pergamon Press.
- NestoriucY. & RiefW. (2012). Risiken und Nebenwirkungen von Verhaltenstherapie. In M.Linden & B.Strauss (Eds.), Risiken und Nebenwirkungen von Psychotherapie (pp. 59–74). Berlin: Medizinisch wissenschaftliche Verlagsgesellschaft.
- 10.1002/1098-240X(200008)23:43.0.CO;2-G · DOI
Cited by
Sorted by citations
Internet-Delivered Psychological Treatments
Andersson, G. (2016) · Annual Review of Clinical Psychology · doi:10.1146/annurev-clinpsy-021815-093006 · Cited by 595
Self-citationErbe, D., Eichert, H.-C., Riper, H., & Ebert, D. D. (2017) · Journal of Medical Internet Research · doi:10.2196/jmir.6588 · Cited by 532
“Also, negative effects such as frustration due to failure or time pressure might be associated with the Internet treatment format [10-13].”
Ebert, D. D., Van Daele, T., Nordgreen, T., Karekla, M., Compare, A., Zarbo, C., Brugnera, A., Øverland, S., Trebbi, G., Jensen, K. L., et al. (2018) · European Psychologist · doi:10.1027/1016-9040/a000318 · Cited by 437
Internet interventions: Past, present and future
Andersson, G. (2018) · Internet Interventions · doi:10.1016/j.invent.2018.03.008 · Cited by 315
Self-citationRozental, A., Kottorp, A., Forsström, D., Månsson, K., Boettcher, J., Andersson, G., Furmark, T., & Carlbring, P. (2019) · Behavioural and Cognitive Psychotherapy · doi:10.1017/s1352465819000018 · Cited by 230
“…statement among researchers (Rozental et al., 2014), a qualitative analysis of the responses to open-ended questions among patients in treatment (Rozental et al., 2015), and a literature review, items for the Negative Effects Questionnaire (NEQ) were generated and investigated using an…”
MethodSelf-citation
Metadata
- Authors
- Rozental, A., Boettcher, J., Andersson, G., Schmidt, B., & Carlbring, P.
- Affiliations
- Stockholm University (SE)
- Freie Universität Berlin (DE)
- Linköping University (SE)
- Florida State University (US)
- Corresponding author
- Rozental, Alexander
- Venue
- Cognitive Behaviour Therapy
- Publisher
- Informa UK Limited
- Published
- 2015-02-23
- Volume
- 44
- Issue
- 3
- Pages
- 223-236
- ISSN
- 1650-6073 (ISSN-L)
- 1651-2316
- OpenAlex
- W2061701598
- Cited by
- 189
- References
- 64
- Citations by year
- 2026: 10 · 2025: 19 · 2024: 15 · 2023: 19 · 2022: 13 · 2021: 23 · 2020: 23 · 2019: 16 · 2018: 20 · 2017: 11 · 2016: 12 · 2015: 8
- FWCI
- 27.55
- Type
- journal article
- Language
- en
- Open access
- green
- urn.kb.se
- Licence
- other-oa
- Locations
- Cognitive Behaviour Therapy (publishedVersion)
- PubMed (publishedVersion)
- DiVA (Stockholm University) (submittedVersion, other-oa)
- UCL Discovery (University College London) (submittedVersion)
- Topics
- Social Sciences › Psychology › Applied Psychology › Digital Mental Health Interventions
- Health Sciences › Medicine › Public Health, Environmental and Occupational Health › Telemedicine and Telehealth Implementation
- Health Sciences › Health Professions › General Health Professions › Health Literacy and Information Accessibility
- Keywords
- online intervention, online therapy, adverse outcomes, patient experience, treatment deterioration, treatment implementation, treatment format, therapist contact, treatment duration, treatment scheduling
- MeSH terms
- Mental Disorders, Clinical Trials as Topic, Humans, Therapy, Computer-Assisted, Patient Satisfaction, Internet, Qualitative Research
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
