Seeing the “Big” Picture: Big Data Methods for Exploring Relationships Between Usage, Language, and Outcome in Internet Intervention Data
Open accessThe DOI matches its Crossref record; 19 of the 19 references it cites were found in a registry or an index.
Abstract
BACKGROUND: Assessing the efficacy of Internet interventions that are already in the market introduces both challenges and opportunities. While vast, often unprecedented amounts of data may be available (hundreds of thousands, and sometimes millions of participants with high dimensions of assessed variables), the data are observational in nature, are partly unstructured (eg, free text, images, sensor data), do not include a natural control group to be used for comparison, and typically exhibit high attrition rates. New approaches are therefore needed to use these existing data and derive new insights that can augment traditional smaller-group randomized controlled trials. OBJECTIVE: Our objective was to demonstrate how emerging big data approaches can help explore questions about the effectiveness and process of an Internet well-being intervention. METHODS: We drew data from the user base of a well-being website and app called Happify. To explore effectiveness, multilevel models focusing on within-person variation explored whether greater usage predicted higher well-being in a sample of 152,747 users. In addition, to explore the underlying processes that accompany improvement, we analyzed language for 10,818 users who had a sufficient volume of free-text response and timespan of platform usage. A topic model constructed from this free text provided language-based correlates of individual user improvement in outcome measures, providing insights into the beneficial underlying processes experienced by users. RESULTS: On a measure of positive emotion, the average user improved 1.38 points per week (SE 0.01, t122,455=113.60, P<.001, 95% CI 1.36-1.41), about an 11% increase over 8 weeks. Within a given individual user, more usage predicted more positive emotion and less usage predicted less positive emotion (estimate 0.09, SE 0.01, t6047=9.15, P=.001, 95% CI .07-.12). This estimate predicted that a given user would report positive emotion 1.26 points (or 1.26%) higher after a 2-week period when they used Happify daily than during a week when they didn't use it at all. Among highly engaged users, 200 automatically clustered topics showed a significant (corrected P<.001) effect on change in well-being over time, illustrating which topics may be more beneficial than others when engaging with the interventions. In particular, topics that are related to addressing negative thoughts and feelings were correlated with improvement over time. CONCLUSIONS: Using observational analyses on naturalistic big data, we can explore the relationship between usage and well-being among people using an Internet well-being intervention and provide new insights into the underlying mechanisms that accompany it. By leveraging big data to power these new types of analyses, we can explore the workings of an intervention from new angles, and harness the insights that surface to feed back into the intervention and improve it further in the future.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Positive Psychology Progress: Empirical Validation of Interventions.
Seligman, M. E. P., Steen, T. A., Park, N., & Peterson, C. (2005) · American Psychologist · doi:10.1037/0003-066x.60.5.410 · Cited by 5,947
Blei, D. M. (2012) · Communications of the ACM · doi:10.1145/2133806.2133826 · Cited by 5,516 · cited 2×
“To use topics as features, we calculated the probability of a user’s topic usage, p(topic | user), using LDA outputs and user word probabilities (see [41] for details).”
MethodCognitive Therapy of Depression
HOLLON, S. D., & BECK, A. T. (1979) · Cognitive-Behavioral Interventions · doi:10.1016/b978-0-12-404480-7.50012-5 · Cited by 4,685
Emmons, R. A., & McCullough, M. E. (2003) · Journal of Personality and Social Psychology · doi:10.1037//0022-3514.84.2.377 · Cited by 3,826 · cited 1×
“…organized activities into the following 5 categories using the acronym STAGE: savor (mindfulness activities) [20,21], thank (gratitude activities) [22,23], aspire (optimism, best possible selves, goal setting, and meaning or purpose activities) [24,25], give (kindness, prosocial spending, and…”
MethodCrawford, J. R., & Henry, J. D. (2004) · British Journal of Clinical Psychology · doi:10.1348/0144665031752934 · Cited by 2,957 · cited 1×
“We chose to focus our analysis on the positive emotion subscale of the Happify Scale, as previous research has shown that life satisfaction is relatively stable in the short term, while positive emotion is more susceptible to change [30,31].”
MethodSin, N. L., & Lyubomirsky, S. (2009) · Journal of Clinical Psychology · doi:10.1002/jclp.20593 · Cited by 2,955 · cited 4×
“Furthermore, consistent with moderator analyses from meta-analyses of behavioral well-being interventions, we expected that this effect would be magnified for users who began with lower well-being [17,18].”
BackgroundResultInfluentialEysenbach, G. (2005) · Journal of Medical Internet Research · doi:10.2196/jmir.7.1.e11 · Cited by 2,524 · cited 3×
“In a consumer environment, where users are customers and it is a buyer’s market, the frequency of dropout becomes even greater [51].”
BackgroundInfluentialThoits, P. A., & Hewitt, L. N. (2001) · Journal of Health and Social Behavior · doi:10.2307/3090173 · Cited by 1,712
Personality, Gender, and Age in the Language of Social Media: The Open-Vocabulary Approach
Schwartz, H. A., Eichstaedt, J. C., Kern, M. L., Dziurzynski, L., Ramones, S. M., Agrawal, M., Shah, A., Kosinski, M., Stillwell, D., Seligman, M. E. P., et al. (2013) · PLoS ONE · doi:10.1371/journal.pone.0073791 · Cited by 1,700
Self-citationSelf‐Compassion, Stress, and Coping
Allen, A. B., & Leary, M. R. (2010) · Social and Personality Psychology Compass · doi:10.1111/j.1751-9004.2009.00246.x · Cited by 757
Lyubomirsky, S., Dickerhoof, R., Boehm, J. K., & Sheldon, K. M. (2011) · Emotion · doi:10.1037/a0022575 · Cited by 720
Tutorial in biostatistics: the self‐controlled case series method
Whitaker, H. J., Paddy Farrington, C., Spiessens, B., & Musonda, P. (2006) · Statistics in Medicine · doi:10.1002/sim.2302 · Cited by 719
Mohr, D. C., Burns, M. N., Schueller, S. M., Clarke, G., & Klinkman, M. (2013) · General Hospital Psychiatry · doi:10.1016/j.genhosppsych.2013.03.008 · Cited by 693 · cited 1×
“In this way, our work is an example of an existing literature that uses passive forms of data collection, such as search behavior and phone sensors [11], to measure and characterize psychological constructs and to develop opportunities for targeted intervention [12].”
BackgroundLimitations of the Randomized Controlled Trial in Evaluating Population-Based Health Interventions
Sanson-Fisher, R. W., Bonevski, B., Green, L. W., & D’Este, C. (2007) · American Journal of Preventive Medicine · doi:10.1016/j.amepre.2007.04.007 · Cited by 514 · cited 2×
“It is easy, therefore, to relegate any data analysis that springs from a real-world Internet intervention to a lower status than that from a randomized controlled trial (RCT) [3].”
BackgroundMindfulness Goes to Work: Impact of an Online Workplace Intervention
Aikens, K. A., Astin, J., Pelletier, K. R., Levanovich, K., Baase, C. M., Park, Y. Y., & Bodnar, C. M. (2014) · Journal of Occupational & Environmental Medicine · doi:10.1097/jom.0000000000000209 · Cited by 338
Disseminating Self-Help: Positive Psychology Exercises in an Online Trial
Schueller, S. M., & Parks, A. C. (2012) · Journal of Medical Internet Research · doi:10.2196/jmir.1850 · Cited by 246 · cited 1×
“In line with previous work finding that Internet interventions are able to improve well-being effectively, we expected that users would exhibit higher well-being during periods in which they used the intervention more frequently [14,15].”
BackgroundSelf-citationMohr, D. C., Schueller, S. M., Riley, W. T., Brown, C. H., Cuijpers, P., Duan, N., Kwasny, M. J., Stiles-Shields, C., & Cheung, K. (2015) · Journal of Medical Internet Research · doi:10.2196/jmir.4391 · Cited by 216 · cited 1×
“Real-world interventions are constantly changing by releasing new content, revamping existing features, and providing interactions with other users via discussion forums, and that presents a challenge to researchers who are trained to keep as many factors as constant as possible [50].”
BackgroundAnguera, J. A., Jordan, J. T., Castaneda, D., Gazzaley, A., & Areán, P. A. (2016) · BMJ Innovations · doi:10.1136/bmjinnov-2015-000098 · Cited by 172 · cited 1×
“Whereas many RCTs aim to standardize or maximize engagement (ideally, every participant would show 100% engagement) [8], multilevel models can harness that variation, creating a dose-response relationship between behavior and outcome.”
BackgroundRealizing the Potential of Behavioral Intervention Technologies
Schueller, S. M., Muñoz, R. F., & Mohr, D. C. (2013) · Current Directions in Psychological Science · doi:10.1177/0963721413495872 · Cited by 152 · cited 3×
“…outside of the restrictions of a traditional randomized, controlled design (including unstructured data such as text produced by users), researchers are opened up to a new variety of potential research questions [2] and can directly examine the relationship between spontaneous usage and outcome.”
Background
Cited by
Objective user engagement with mental health apps: systematic search and panel-based usage analysis
Baumel, A., Muench, F., Edan, S., & Kane, J. M. (2019) · Journal of Medical Internet Research · doi:10.2196/14567 · Cited by 965
“These findings reflect the lower ranges of real-world retention rates reported in developer-led studies [17-20,22].”
BackgroundFleming, T., Bavin, L., Lucassen, M., Stasiak, K., Hopkins, S., & Merry, S. (2018) · Journal of Medical Internet Research · doi:10.2196/jmir.9275 · Cited by 670
“…gaming elements and games 58 core activities Mixed (includes CBT & positive psychology) Anxiety and mood/ depression Happify (app & online program) Carpenter et al (2016) [23] aCBT: cognitive behavioral therapy. bRCT: randomized controlled trial. cOn the basis of the study’s description of the…”
BackgroundMethodInfluentialMorris, R. R., Kouddous, K., Kshirsagar, R., & Schueller, S. M. (2018) · Journal of Medical Internet Research · doi:10.2196/10148 · Cited by 273
Bakker, D., & Rickard, N. (2018) · Journal of Affective Disorders · doi:10.1016/j.jad.2017.11.016 · Cited by 231
A randomized controlled trial of three smartphone apps for enhancing public mental health
Bakker, D., Kazantzis, N., Rickwood, D., & Rickard, N. (2018) · Behaviour Research and Therapy · doi:10.1016/j.brat.2018.08.003 · Cited by 176
Metadata
- Authors
- Carpenter, J., Crutchley, P., Zilca, R. D., Schwartz, H. A., Smith, L. K., Cobb, A. M., & Parks, A. C.
- Venue
- Journal of Medical Internet Research
- Publisher
- JMIR Publications Inc.
- Published
- 2016-08-31
- Volume
- 18
- Issue
- 8
- Pages
- e241
- ISSN
- 1438-8871 (ISSN-L)
- OpenAlex
- W2517026589
- Cited by
- 60
- References
- 19
- FWCI
- 10.43
- Type
- journal article
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.