The peril of self-reported adherence in digital interventions: A brief example
Open accessThe DOI matches its Crossref record; 23 of the 23 references it cites were found in a registry or an index.
Abstract
Adherence is an important predictor of intervention outcomes, but not all measures of adherence are created equally. Here, we analyzed whether there was a discrepancy between self-report adherence and objective adherence in a digital mindfulness meditation randomised, controlled trial. A sample of 174 young adult undergraduate university students trialled either an app-based or email-based mindfulness meditation program (or an app-based attention control). Participants' adherence (number of sessions completed) and mental health was self-reported. Objective adherence data were provided by the owners of the digital mindfulness programs. We found evidence of inflated self-reported adherence to the app-based intervention and argue that the inflation was not explained by social desirability biases because participants were aware we would have access to object data and no remuneration was tied to adherence. We also comment on the different conclusions we would have drawn about the effectiveness of the digital interventions on mental health, had we used the self-reported adherence data rather than the objective adherence data. We use this example to suggest that it may be perilous to rely on self-reported measures of adherence when assessing the effectiveness of digital interventions.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Effect size guidelines for individual differences researchers
Gignac, G. E., & Szodorai, E. T. (2016) · Personality and Individual Differences · doi:10.1016/j.paid.2016.06.069 · Cited by 2,860
Self-reports: How the questions shape the answers.
Schwarz, N. (1999) · American Psychologist · doi:10.1037/0003-066x.54.2.93 · Cited by 2,846 · cited 1×
“Although self-reported adherence data are easily they may be subject to biases that affect all self-report data (e.g., recall bias and response bias; Kimberlin and Winterstein, 2008; Schwarz, 1999), which may lead to inaccurate conclusions about the effectiveness of the intervention.”
BackgroundEysenbach, G. (2005) · Journal of Medical Internet Research · doi:10.2196/jmir.7.1.e11 · Cited by 2,524 · cited 1×
“One explanation for these modest effects might be that adherence to digital interventions is low (Cuijpers et al., 2011; Eysenbach, 2005).”
BackgroundPatient Adherence and Medical Treatment Outcomes: A Meta-Analysis
Robin DiMatteo, M., Giordani, P. J., Lepper, H. S., & Croghan, T. W. (2002) · Medical Care · doi:10.1097/00005650-200209000-00009 · Cited by 1,490
Internet-Based and Other Computerized Psychological Treatments for Adult Depression: A Meta-Analysis
Andersson, G., & Cuijpers, P. (2009) · Cognitive Behaviour Therapy · doi:10.1080/16506070903318960 · Cited by 1,368 · cited 2×
“…2009 and 2015, yearly publications on e-mental health interventions trebled (Firth et al., 2016), but meta-analytic reviews reveal that self-guided digital interventions often have only modest effects on mental health (Andersson and Cuijpers, 2009; Cuijpers et al., 2011; Spijkerman et al., 2016).”
BackgroundPersuasive System Design Does Matter: a Systematic Review of Adherence to Web-based Interventions
Kelders, S. M., Kok, R. N., Ossebaard, H. C., & Van Gemert-Pijnen, J. E. (2012) · Journal of Medical Internet Research · doi:10.2196/jmir.2104 · Cited by 1,308 · cited 2×
“…were skills acquired), nor do they distinguish between observed adherence (how much the individual experienced the content of the intervention) and prescribed adherence (how much the individual experienced the intervention as recommended or intended; Kelders et al., 2012; Sieverink et al., 2017).”
BackgroundAdherence in Internet Interventions for Anxiety and Depression
Christensen, H., Griffiths, K. M., & Farrer, L. (2009) · Journal of Medical Internet Research · doi:10.2196/jmir.1194 · Cited by 1,133 · cited 3×
“To date, the majority of research on adherence in digital interventions has focused on operationalizing adherence and identifying predictors of adherence (see: Christensen et al., 2009) but the complexity of adherence is often neglected (see: Sieverink et al., 2017 for a systematic review).”
BackgroundSpijkerman, M. P. J., Pots, W. T. M., & Bohlmeijer, E. T. (2016) · Clinical Psychology Review · doi:10.1016/j.cpr.2016.03.009 · Cited by 975
Self-report measures of medication adherence behavior: recommendations on optimal use
Stirratt, M. J., Dunbar-Jacob, J., Crane, H. M., Simoni, J. M., Czajkowski, S., Hilliard, M. E., Aikens, J. E., Hunter, C. M., Velligan, D. I., Huntley, K., et al. (2015) · Translational Behavioral Medicine · doi:10.1007/s13142-015-0315-2 · Cited by 824
Karyotaki, E., Riper, H., Twisk, J., Hoogendoorn, A., Kleiboer, A., Mira, A., Mackinnon, A., Meyer, B., Botella, C., Littlewood, E., et al. (2017) · JAMA Psychiatry · doi:10.1001/jamapsychiatry.2017.0044 · Cited by 691
Research works printed but not linked: 4
- Donkin, 2011, A systematic review of the impact of adherence on the effectiveness of e-therapies, J. Med. Internet Res., 13, 10.2196/jmir.1772 · DOI · Paper page
- Elefant, 2017, Microinterventions produce immediate but not lasting benefits in mood and distress, Internet Interv., 10, 17, 10.1016/j.invent.2017.08.004 · DOI
- Firth, 2016, Ecological momentary assessment and beyond: the rising interest in e-mental health research, J. Psychiatr. Res., 80, 3, 10.1016/j.jpsychires.2016.05.002 · DOI · Paper page
- Kimberlin, 2008, Validity and reliability of measurement instruments used in research, Am. J. Health Syst. Pharm., 65, 2276, 10.2146/ajhp070364 · DOI · Paper page
Cited by
Taylor, H., Strauss, C., & Cavanagh, K. (2021) · Clinical Psychology Review · doi:10.1016/j.cpr.2021.102078 · Cited by 102
Levin, M. E., Hicks, E. T., & Krafft, J. (2022) · Journal of American College Health · doi:10.1080/07448481.2020.1728281 · Cited by 65
Johnson, E., Corrick, S., Isley, S., Vandermeer, B., Dolgoy, N., Bates, J., Godfrey, E., Soltys, C., Muir, C., Vohra, S., et al. (2024) · PLOS Digital Health · doi:10.1371/journal.pdig.0000435 · Cited by 24
“Survey completion and self-reports were the most common markers, despite their limitations like recall bias, overestimation, and the possibility that participants may feel clinical improvement and stop using the intervention [140].”
BackgroundEconomides, M., Bolton, H., Male, R., & Cavanagh, K. (2022) · JMIR Formative Research · doi:10.2196/34032 · Cited by 19
“This is important, given the recent evidence of substantial discrepancies between self-reported and objective adherence in digital interventions [68].”
Taylor, R. W., Male, R., Economides, M., Bolton, H., & Cavanagh, K. (2023) · JMIR Formative Research · doi:10.2196/41590 · Cited by 15
“Intervention engagement was measured objectively, which is important given the evidence of inflated rates of self-reported digital intervention use in a study setting [72].”
Atefi, G. L., van Knippenberg, R. J. M., Bartels, S. L., Losada-Baltar, A., Márquez-González, M., Verhey, F. R. J., & de Vugt, M. E. (2024) · JMIR Aging · doi:10.2196/53489 · Cited by 15
Isbăşoiu, A. B., Tulbure, B. T., Rusu, A., & Sava, F. A. (2021) · Frontiers in Psychology · doi:10.3389/fpsyg.2021.752249 · Cited by 14
“As already pointed out in the literature on internet-interventions, the high selfregulatory effort needed from behalf of the participants is one key hindering factor when it comes to their adherence to such treatments (Donkin and Glozier, 2012; Beatty and Binnion, 2016; Zarski et al., 2018; Flett et al., 2019; Arndt e…”
BackgroundLaMontagne, L. G., Doty, J. L., Diehl, D. C., Nesbit, T. S., Gage, N. A., Kumbkarni, N., & Leon, S. P. (2024) · Journal of Affective Disorders · doi:10.1016/j.jad.2024.09.014 · Cited by 13
Metadata
- Authors
- Flett, J. A. M., Fletcher, B. D., Riordan, B. C., Patterson, T., Hayne, H., & Conner, T. S.
- Venue
- Internet Interventions
- Publisher
- Elsevier BV
- Published
- 2019-12
- Volume
- 18
- Pages
- 100267
- ISSN
- 2214-7829 (ISSN-L)
- OpenAlex
- W2969414414
- Cited by
- 37
- References
- 23
- FWCI
- 3.22
- Type
- journal article
- Language
- en
- Open access
- gold
- Licence
- www.elsevier.com/tdm/userlicense/1.0/ (tdm)
- www.elsevier.com/legal/tdmrep-license (tdm)
- creativecommons.org/licenses/by-nc-nd/4.0/ (vor)
- Funders
- University of Otago
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.