The DOI matches its Crossref record; 147 of the 147 references it cites were found in a registry or an index.
Digital Mental Health Interventions for Depression, Anxiety, and Enhancement of Psychological Well-Being Among College Students: Systematic Review
Open accessPublished version of a preprint
Abstract
BACKGROUND: College students are increasingly reporting common mental health problems, such as depression and anxiety, and they frequently encounter barriers to seeking traditional mental health treatments. Digital mental health interventions, such as those delivered via the Web and apps, offer the potential to improve access to mental health treatment. OBJECTIVE: This study aimed to review the literature on digital mental health interventions focused on depression, anxiety, and enhancement of psychological well-being among samples of college students to identify the effectiveness, usability, acceptability, uptake, and adoption of such programs. METHODS: We conducted a systematic review using the Preferred Reporting Items for Systematic Reviews and Meta-Analyses guidelines (registration number CRD42018092800), and the search strategy was conducted by a medical research librarian in the following databases: MEDLINE (Ovid), EMBASE (Elsevier), PsycINFO (EbscoHost), the Cochrane Library (Wiley), and Web of Science (Thomson Reuters) from the date of inception to April 2019. Data were synthesized using a systematic narrative synthesis framework, and formal quality assessments were conducted to address the risk of bias. RESULTS: A total of 89 studies met the inclusion criteria. The majority of interventions (71/89, 80%) were delivered via a website, and the most common intervention was internet-based cognitive behavioral therapy (28, 31%). Many programs (33, 37%) featured human support in the form of coaching. The majority of programs were either effective (42, 47%) or partially effective (30, 34%) in producing beneficial changes in the main psychological outcome variables. Approximately half of the studies (45, 51%) did not present any usability or acceptability outcomes, and few studies (4, 4%) examined a broad implementation of digital mental health interventions on college campuses. Quality assessments revealed a moderate-to-severe risk of bias in many of the studies. CONCLUSIONS: Results suggest that digital mental health interventions can be effective for improving depression, anxiety, and psychological well-being among college students, but more rigorous studies are needed to ascertain the effective elements of these interventions. Continued research on improving the user experience of, and thus user engagement with, these programs appears vital for the sustainable implementation of digital mental health interventions on college campuses.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Additionally, our results support the importance of involving students in the process of developing new services, whether it is a digital mental health tool or another form of support.25,34”
From Aligning students and counseling centers on student mental health needs and treatment resources. (2022) · cites this paper as [25]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Few studies have described the results and benefits of joining students and counseling center clinicians together to address mental health service delivery, and this paper adds to this emerging literature.25,26”
From Aligning students and counseling centers on student mental health needs and treatment resources. (2022) · cites this paper as [25]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“which is consistent with studies in the same age group.4,33”
From Mental Health, Suicidality, Health, and Social Indicators Among College Students Across Nine States in India (2025) · cites this paper as [33]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Increasing evidence focusing on the effectiveness of these interventions for MDD and GAD exists in the literature [31,32].”
From A Digital Mental Health Support Program for Depression and Anxiety in Populations With Attention-Deficit/Hyperactivity Disorder: Feasibility and Usability Study (2023) · cites this paper as [31]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Digital tools supporting MH have been proven to be good solutions for existing barriers to care, such as long waiting lines and MH-related stigma, owing to their accessibility and ability to maintain students’ privacy [14-16]”
From Optimization of a Transdiagnostic Mobile Emotion Regulation Intervention for University Students: Protocol for a Microrandomized Trial (2023) · cites this paper as [14]
Experimental — accuracy not yet measured
10 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Preferred Reporting Items for Systematic Reviews and Meta-Analyses: the PRISMA statement
Moher, D., Liberati, A., Tetzlaff, J., Altman, D. G., & for the PRISMA Group (2009) · BMJ · doi:10.1136/bmj.b2535 · Cited by 83,823
Preferred Reporting Items for Systematic Reviews and Meta-Analyses: The PRISMA Statement
Moher, D., Liberati, A., Tetzlaff, J., Altman, D. G., & The PRISMA Group (2009) · PLoS Medicine · doi:10.1371/journal.pmed.1000097 · Cited by 64,042
The PHQ-9: Validity of a brief depression severity measure
Kroenke, K., Spitzer, R. L., & Williams, J. B. W. (2001) · Journal of General Internal Medicine · doi:10.1046/j.1525-1497.2001.016009606.x · Cited by 44,910
A Global Measure of Perceived Stress
Cohen, S., Kamarck, T., & Mermelstein, R. (1983) · Journal of Health and Social Behavior · doi:10.2307/2136404 · Cited by 32,807 · cited 1×
“…Inventory-II [28] and Patient Health Questionnaire [29]), measures of anxious symptomatology (eg, Beck Anxiety Inventory [30] and Anxiety Sensitivity Inventory [31]), and measures of psychological distress and well-being (eg, Perceived Stress Scale [32] and Scales of Psychological Well-being [33]).”
Rayyan-a web and mobile app for systematic reviews.
Ouzzani, M., Hammady, H., Fedorowicz, Z., & Elmagarmid, A. (2016) · Systematic Reviews · doi:10.1186/s13643-016-0384-4 · Cited by 26,092 · cited 1×
“Search results were uploaded into Rayyan, a Web-based software program that allows for reviewers to collaborate during the study selection process [27].”
ROBINS-I: a tool for assessing risk of bias in non-randomised studies of interventions
Sterne, J. A., Hernán, M. A., Reeves, B. C., Savović, J., Berkman, N. D., Viswanathan, M., Henry, D., Altman, D. G., Ansari, M. T., Boutron, I., et al. (2016) · BMJ · doi:10.1136/bmj.i4919 · Cited by 20,329 · cited 1×
“…both randomized trials and nonrandomized trials, the risk of bias was assessed using 2 separate tools: the Cochrane Collaboration’s tool for assessing risk of bias in randomized trials [36] and the Cochrane Collaboration’s tool for assessing risk in nonrandomized studies of interventions [37].”
Wang, G., Ma, R., Qiao, G., Wada, K., Aizawa, Y., & Satoh, T. (2015) · Integrative Medicine International · doi:10.1159/000368408 · Cited by 18,657
An inventory for measuring clinical anxiety: Psychometric properties.
Beck, A. T., Epstein, N., Brown, G., & Steer, R. A. (1988) · Journal of Consulting and Clinical Psychology · doi:10.1037/0022-006x.56.6.893 · Cited by 14,407 · cited 1×
“…(eg, Beck Depression Inventory-II [28] and Patient Health Questionnaire [29]), measures of anxious symptomatology (eg, Beck Anxiety Inventory [30] and Anxiety Sensitivity Inventory [31]), and measures of psychological distress and well-being (eg, Perceived Stress Scale [32] and Scales of…”
The structure of psychological well-being revisited.
Ryff, C. D., & Keyes, C. L. M. (1995) · Journal of Personality and Social Psychology · doi:10.1037//0022-3514.69.4.719 · Cited by 10,285 · cited 1×
“…Inventory-II [28] and Patient Health Questionnaire [29]), measures of anxious symptomatology (eg, Beck Anxiety Inventory [30] and Anxiety Sensitivity Inventory [31]), and measures of psychological distress and well-being (eg, Perceived Stress Scale [32] and Scales of Psychological Well-being [33]).”
Proctor, E., Silmere, H., Raghavan, R., Hovmand, P., Aarons, G., Bunger, A., Griffey, R., & Hensley, M. (2011) · Administration and Policy in Mental Health and Mental Health Services Research · doi:10.1007/s10488-010-0319-7 · Cited by 8,721 · cited 3×
“Thus, uptake and adoption outcomes were primarily metrics on the number of downloads and uses and were intended to be described alongside the number of users relative to the population of potential users when available (to determine the rates of service penetration [35]).”
Curran, G. M., Bauer, M., Mittman, B., Pyne, J. M., & Stetler, C. (2012) · Medical Care · doi:10.1097/mlr.0b013e3182408812 · Cited by 4,058 · cited 1×
“…for college students, digital health researchers need to embrace methods and models from implementation science, such as hybrid trial designs [153], and add to the body of knowledge on how to create and support a campus mental health system that actively uses digital mental health programs.”
Greenhalgh, T., Wherton, J., Papoutsi, C., Lynch, J., Hughes, G., A'Court, C., Hinder, S., Fahy, N., Procter, R., & Shaw, S. (2017) · Journal of Medical Internet Research · doi:10.2196/jmir.8775 · Cited by 2,887 · cited 1×
“The research-to-practice gap for digital mental health interventions is increasingly being recognized, and leaders in the field have proposed strategies to routinely incorporate implementation science methods into the study of digital mental health interventions [148,149].”
Fitzpatrick, K. K., Darcy, A., & Vierhile, M. (2017) · JMIR Mental Health · doi:10.2196/mental.7785 · Cited by 2,730 · cited 10×
“Some concerns were noted in this domain for roughly one-third of the studies (n=27 [42,46,49,50,52,53,55,56,58,59,61-63, 65,67-70,72-74,76,81,82,85,87,89]) because outcomes were self-reported in nature and the participants were aware of the intervention they received.”
InfluentialSiddaway, A. P., Wood, A. M., & Hedges, L. V. (2019) · Annual Review of Psychology · doi:10.1146/annurev-psych-010418-102803 · Cited by 2,550 · cited 1×
“Following the systematic narrative framework for literature reviews [39], the results of included studies were synthesized and presented without reference to the statistical significance of the findings.”
Auerbach, R. P., Collaborators, W. W.-I., Mortier, P., Bruffaerts, R., Alonso, J., Benjet, C., Cuijpers, P., Demyttenaere, K., Ebert, D. D., Green, J. G., et al. (2018) · Journal of Abnormal Psychology · doi:10.1037/abn0000362 · Cited by 2,522 · cited 1×
“Globally, approximately 31% of college students screened positive for a mental health disorder over the course of the last year [4].”
The client satisfaction questionnaire
Attkisson, C. C., & Zwick, R. (1982) · Evaluation and Program Planning · doi:10.1016/0149-7189(82)90074-x · Cited by 1,743 · cited 1×
“A minority of studies used validated measures such as the System Usability Scale [65-67,120,132] to assess usability and the Client Satisfaction Questionnaire [61,71,86,133] to assess acceptability.”
Writing Narrative Literature Reviews
Baumeister, R. F., & Leary, M. R. (1997) · Review of General Psychology · doi:10.1037/1089-2680.1.3.311 · Cited by 1,482 · cited 1×
“A systematic narrative framework was used to synthesize the data [38-40].”
Andrews, G., Cuijpers, P., Craske, M. G., McEvoy, P., & Titov, N. (2010) · PLoS ONE · doi:10.1371/journal.pone.0013196 · Cited by 1,311 · cited 1×
“The evidence base for digital mental health interventions for general adult populations is vast [12-15], and the evidence base for college and university student populations is rapidly accumulating.”
Help-Seeking and Access to Mental Health Care in a University Student Population
Eisenberg, D., Golberstein, E., & Gollust, S. E. (2007) · Medical Care · doi:10.1097/mlr.0b013e31803bb4c1 · Cited by 1,131 · cited 1×
“Many students have low mental health literacy and do not recognize a need for treatment but rather believe that these depression and anxiety symptoms are typical college stress and, therefore, do not warrant treatment [5].”
Firth, J., Torous, J., Nicholas, J., Carney, R., Pratap, A., Rosenbaum, S., & Sarris, J. (2017) · World Psychiatry · doi:10.1002/wps.20472 · Cited by 1,097 · cited 1×
“The evidence base for digital mental health interventions for general adult populations is vast [12-15], and the evidence base for college and university student populations is rapidly accumulating.”
Lipson, S. K., Lattie, E. G., & Eisenberg, D. (2019) · Psychiatric Services · doi:10.1176/appi.ps.201800332 · Cited by 947 · cited 1×
“In the last decade, rising rates of college students experiencing symptoms of depression and anxiety have been observed [1-3].”
Self-citationAndrews, G., Basu, A., Cuijpers, P., Craske, M. G., McEvoy, P., English, C. L., & Newby, J. M. (2018) · Journal of Anxiety Disorders · doi:10.1016/j.janxdis.2018.01.001 · Cited by 922 · cited 2×
“Similar to what has been observed in digital mental health intervention programs for general adult populations [22,134], there were notable rates of participant attrition and early program discontinuation in many of the studies.”
Torous, J., Nicholas, J., Larsen, M. E., Firth, J., & Christensen, H. (2018) · Evidence Based Mental Health · doi:10.1136/eb-2018-102891 · Cited by 790 · cited 2×
“Although this review focused on mental health apps [140], these themes appear to be translatable to reasons for poor engagement with other types of digital mental health interventions.”
Fulmer, R., Joerin, A., Gentile, B., Lakerink, L., & Rauws, M. (2018) · doi:10.2196/preprints.9782 · Cited by 715 · cited 10×
“Most studies included were focused on either universal prevention programs (n=36 [43,48,53,54,63-67,70-73,75-78,90,91, 94,95,97,99,100,102,106-108,110-112,114,118,119,125,126]), or on treatment intervention programs (n=22 [41,49,50,52, 58,69,74,79,80,83-86,88,89,92,98,113,117,120,122,127]).”
InfluentialKaryotaki, E., Riper, H., Twisk, J., Hoogendoorn, A., Kleiboer, A., Mira, A., Mackinnon, A., Meyer, B., Botella, C., Littlewood, E., et al. (2017) · JAMA Psychiatry · doi:10.1001/jamapsychiatry.2017.0044 · Cited by 691
Firth, J., Torous, J., Nicholas, J., Carney, R., Rosenbaum, S., & Sarris, J. (2017) · Journal of Affective Disorders · doi:10.1016/j.jad.2017.04.046 · Cited by 675 · cited 1×
“The evidence base for digital mental health interventions for general adult populations is vast [12-15], and the evidence base for college and university student populations is rapidly accumulating.”
Fleming, T., Bavin, L., Lucassen, M., Stasiak, K., Hopkins, S., & Merry, S. (2018) · Journal of Medical Internet Research · doi:10.2196/jmir.9275 · Cited by 670 · cited 2×
“Similar to what has been observed in digital mental health intervention programs for general adult populations [22,134], there were notable rates of participant attrition and early program discontinuation in many of the studies.”
Mental Health Service Utilization Among College Students in the United States
Eisenberg, D., Hunt, J., Speer, N., & Zivin, K. (2011) · Journal of Nervous & Mental Disease · doi:10.1097/nmd.0b013e3182175123 · Cited by 633 · cited 1×
“Students who do recognize a need for mental health services often face multiple barriers to accessing care, perceive the care available to them as inconvenient, and are skeptical about the efficacy of care [6,7].”
Snilstveit, B., Oliver, S., & Vojtkova, M. (2012) · Journal of Development Effectiveness · doi:10.1080/19439342.2012.710641 · Cited by 589 · cited 1×
“A systematic narrative framework was used to synthesize the data [38-40].”
Empirical Studies on Usability of mHealth Apps: A Systematic Literature Review
Zapata, B. C., Fernández-Alemán, J. L., Idri, A., & Toval, A. (2015) · Journal of Medical Systems · doi:10.1007/s10916-014-0182-2 · Cited by 585 · cited 1×
“Furthermore, in recent years, increased focus has been on assessing the user experience (including the usability and acceptability) of such interventions to identify and rectify user experience failings that could limit one’s ability and desire to continue to use a program [23-25].”
Ebert, D. D., Zarski, A.-C., Christensen, H., Stikkelbroek, Y., Cuijpers, P., Berking, M., & Riper, H. (2015) · PLOS ONE · doi:10.1371/journal.pone.0119895 · Cited by 533 · cited 1×
“This is consistent with past meta-analyses on digital mental health programs for college students [17,18] and is consistent with the broader literature on digital mental health interventions [134,135].”
Do Student Samples Provide an Accurate Estimate of the General Public?
Hanel, P. H. P., & Vione, K. C. (2016) · PLOS ONE · doi:10.1371/journal.pone.0168354 · Cited by 510 · cited 1×
“College students are often used as a convenience sample for psychological research [139].”
Davies, E. B., Morriss, R., & Glazebrook, C. (2014) · Journal of Medical Internet Research · doi:10.2196/jmir.3142 · Cited by 499 · cited 2×
“A 2014 systematic review and meta-analysis of computer-delivered and Web-based interventions for university students found that these types of interventions can be effective in improving depression, anxiety, and stress among students [17].”
mHealth: A Mechanism to Deliver More Accessible, More Effective Mental Health Care
Price, M., Yuen, E. K., Goetter, E. M., Herbert, J. D., Forman, E. M., Acierno, R., & Ruggiero, K. J. (2014) · Clinical Psychology & Psychotherapy · doi:10.1002/cpp.1855 · Cited by 488 · cited 1×
“Indeed, these themes suggest usability problems, which decrease the likelihood of user engagement because of a mismatch of design with user needs [141,142].”
mHealth Consumer Apps : The Case for User-Centered Design
McCurdie, T., Taneva, S., Casselman, M., Yeung, M., McDaniel, C., Ho, W., & Cafazzo, J. (2012) · Biomedical Instrumentation & Technology · doi:10.2345/0899-8205-46.s2.49 · Cited by 478 · cited 1×
“Furthermore, in recent years, increased focus has been on assessing the user experience (including the usability and acceptability) of such interventions to identify and rectify user experience failings that could limit one’s ability and desire to continue to use a program [23-25].”
Are we in crisis? National mental health and treatment trends in college counseling centers.
Xiao, H., Carney, D. M., Youn, S. J., Janis, R. A., Castonguay, L. G., Hayes, J. A., & Locke, B. D. (2017) · Psychological Services · doi:10.1037/ser0000130 · Cited by 447 · cited 1×
“However, many counseling centers across the country are underresourced, have difficulty reaching students in need, and operate at full capacity during much of the year [8].”
Harrer, M., Adam, S. H., Baumeister, H., Cuijpers, P., Karyotaki, E., Auerbach, R. P., Kessler, R. C., Bruffaerts, R., Berking, M., & Ebert, D. D. (2019) · International Journal of Methods in Psychiatric Research · doi:10.1002/mpr.1759 · Cited by 445 · cited 2×
“More recently, a 2018 systematic review and meta-analysis found that internet interventions can have small-to-moderate effects on a range of mental health conditions [18].”
Campus mental health services: Recommendations for change.
Mowbray, C. T., Megivern, D., Mandiberg, J. M., Strauss, S., Stein, C. H., Collins, K., Kopels, S., Curlin, C., & Lett, R. (2006) · American Journal of Orthopsychiatry · doi:10.1037/0002-9432.76.2.226 · Cited by 435 · cited 1×
“Students who do recognize a need for mental health services often face multiple barriers to accessing care, perceive the care available to them as inconvenient, and are skeptical about the efficacy of care [6,7].”
Eisenberg, D., Hunt, J., & Speer, N. (2012) · Harvard Review of Psychiatry · doi:10.3109/10673229.2012.712839 · Cited by 365 · cited 1×
“In the last decade, rising rates of college students experiencing symptoms of depression and anxiety have been observed [1-3].”
Flett, J. A. M., Hayne, H., Riordan, B. C., Thompson, L. M., & Conner, T. S. (2019) · Mindfulness · doi:10.1007/s12671-018-1050-9 · Cited by 338 · cited 2×
“]); Therapist-Assisted Online, n=1 [113]; Headspace and Smiling Mind, n=1 [95]; Family eJournal, n=1 [90]; Tess chatbot, n=1 [97]; DeStressify, n=1 [102]; Overcome Social Anxiety, n=1 [103]); SilverCloud Health’s Space from Depression, Space from Anxiety, and Space from Stress, n=1 [128]; and…”
Brief Virtual Reality Therapy for Public Speaking Anxiety
Harris, S. R., Kemmerling, R. L., & North, M. M. (2002) · CyberPsychology & Behavior · doi:10.1089/109493102321018187 · Cited by 314 · cited 10×
“Most studies included were focused on either universal prevention programs (n=36 [43,48,53,54,63-67,70-73,75-78,90,91, 94,95,97,99,100,102,106-108,110-112,114,118,119,125,126]), or on treatment intervention programs (n=22 [41,49,50,52, 58,69,74,79,80,83-86,88,89,92,98,113,117,120,122,127]).”
InfluentialUsability of Commercially Available Mobile Applications for Diverse Patients
Sarkar, U., Gourley, G. I., Lyles, C. R., Tieu, L., Clarity, C., Newmark, L., Singh, K., & Bates, D. W. (2016) · Journal of General Internal Medicine · doi:10.1007/s11606-016-3771-6 · Cited by 257 · cited 1×
“Utilization of user-centered design and usability testing is a growing practice in digital health interventions for depression [143], smoking cessation [144,145], and diverse patient groups [146].”
Räsänen, P., Lappalainen, P., Muotka, J., Tolvanen, A., & Lappalainen, R. (2016) · Behaviour Research and Therapy · doi:10.1016/j.brat.2016.01.001 · Cited by 221 · cited 1×
“As seen in Concerns were often noted regarding potential bias arising from the randomization process, and potential bias because of missing outcome data that were not analyzed in a manner to minimize risk [80-82].”
Harrer, M., Adam, S. H., Fleischmann, R. J., Baumeister, H., Auerbach, R., Bruffaerts, R., Cuijpers, P., Kessler, R. C., Berking, M., Lehr, D., et al. (2018) · Journal of Medical Internet Research · doi:10.2196/jmir.9293 · Cited by 216 · cited 1×
“…utilized a control condition; although the quality of control conditions varied broadly and ranged from treatment as usual or waitlists to other digital programs or face-to-face t r e a t m e n t ([41,42,45,47,48,50,51,54,56,57,59,60,71,78,79,84,86,88,89,92,93,100,101,103,105,107,109,112,113,125]).”
Levin, M. E., Haeger, J. A., Pierce, B. G., & Twohig, M. P. (2017) · Behavior Modification · doi:10.1177/0145445516659645 · Cited by 207
Georgsson, M., & Staggers, N. (2016) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocv099 · Cited by 200 · cited 1×
“Furthermore, in recent years, increased focus has been on assessing the user experience (including the usability and acceptability) of such interventions to identify and rectify user experience failings that could limit one’s ability and desire to continue to use a program [23-25].”
Technology-based interventions for mental health in tertiary students: Systematic review
Farrer, L., Gulliver, A., Chan, J. K., Batterham, P. J., Reynolds, J., Calear, A., Tait, R., Bennett, K., & Griffiths, K. M. (2013) · Journal of Medical Internet Research · doi:10.2196/jmir.2639 · Cited by 188 · cited 1×
“In 2013, a systematic review of technology-based interventions for mental health in tertiary students found that these types of interventions offer promise for improving symptoms of certain mental health problems, but it concluded that more research was needed [16].”
Group prevention of depression and anxiety symptoms
Seligman, M. E. P., Schulman, P., & Tryon, A. M. (2007) · Behaviour Research and Therapy · doi:10.1016/j.brat.2006.09.010 · Cited by 187
Vis, C., Mol, M., Kleiboer, A., Bührmann, L., Finch, T., Smit, J., & Riper, H. (2018) · JMIR Mental Health · doi:10.2196/mental.9769 · Cited by 183 · cited 1×
“Increasingly, calls are being made to collect implementation-relevant data while testing new digital mental health programs or testing existing programs on new populations [150-152].”
Levin, M. E., Pistorello, J., Seeley, J. R., & Hayes, S. C. (2014) · Journal of American College Health · doi:10.1080/07448481.2013.843533 · Cited by 179 · cited 1×
“A minority of studies used validated measures such as the System Usability Scale [65-67, 120”
Henriques, G. R., Keffer, S. L., Abrahamson, C., & Horst, S. J. (2011) · Applied Psychophysiology and Biofeedback · doi:10.1007/s10484-011-9151-4 · Cited by 179 · cited 2×
“The majority of these studies (n=46 [43-48,51-55,57-59, 62,65-67,69,72,76-78,86,87,113,116,119,120,122,124-127,131]) took place in the United States.”
Implementation science approaches for integrating eHealth research into practice and policy
Glasgow, R. E., Phillips, S. M., & Sanchez, M. A. (2013) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2013.07.002 · Cited by 174 · cited 1×
“…digital health tools under more pragmatic conditions to maximize the transfer of knowledge from research trials to real-world implementation [19-21], and studies examining the real-world uptake and engagement with digital mental health tools have generally found low engagement and…”
Continuous Evaluation of Evolving Behavioral Intervention Technologies
Mohr, D. C., Cheung, K., Schueller, S. M., Hendricks Brown, C., & Duan, N. (2013) · American Journal of Preventive Medicine · doi:10.1016/j.amepre.2013.06.006 · Cited by 168 · cited 1×
“Many have called for testing digital health tools under more pragmatic conditions to maximize the transfer of knowledge from research trials to real-world implementation [19-21], and studies examining the real-world uptake and engagement with digital mental health tools have generally found low engagement and completi…”
An Interactive Web-Based Method of Outreach to College Students at Risk for Suicide
Haas, A., Koestner, B., Rosenberg, J., Moore, D., Garlow, S. J., Sedway, J., Nicholas, L., Hendin, H., Mann, J. J., & Nemeroff, C. B. (2008) · Journal of American College Health · doi:10.3200/jach.57.1.15-22 · Cited by 163 · cited 4×
“…the importance of the systems in which interventions are to be introduced, and although all studies included in this review focused on college students as a population of interest, very few examined college campus systems and tested the implementation of programs onto campuses [86,109,116,119].”
InfluentialBeyond the Randomized Controlled Trial: A Review of Alternatives in mHealth Clinical Trial Methods
Pham, Q., Wiljer, D., & Cafazzo, J. A. (2016) · JMIR mHealth and uHealth · doi:10.2196/mhealth.5720 · Cited by 162 · cited 1×
“Many have called for testing digital health tools under more pragmatic conditions to maximize the transfer of knowledge from research trials to real-world implementation [19-21], and studies examining the real-world uptake and engagement with digital mental health tools have generally found low engagement and completi…”
Guille, C., Zhao, Z., Krystal, J., Nichols, B., Brady, K., & Sen, S. (2015) · JAMA Psychiatry · doi:10.1001/jamapsychiatry.2015.1880 · Cited by 160
Lintvedt, O. K., Griffiths, K. M., Sørensen, K., Østvik, A. R., Wang, C. E. A., Eisemann, M., & Waterloo, K. (2013) · Clinical Psychology & Psychotherapy · doi:10.1002/cpp.770 · Cited by 157 · cited 2×
“…appeared to use college students as a convenience sample, and for an additional 9 studies, it was unclear if the program was being tested specifically for college student mental health or if college students were a convenience sample [47,50,51,61,64,68,84,105,122].”
Rose, R. D., Buckey, J. C., Zbozinek, T. D., Motivala, S. J., Glenn, D. E., Cartreine, J. A., & Craske, M. G. (2013) · Behaviour Research and Therapy · doi:10.1016/j.brat.2012.11.003 · Cited by 153 · cited 2×
“…appeared to use college students as a convenience sample, and for an additional 9 studies, it was unclear if the program was being tested specifically for college student mental health or if college students were a convenience sample [47,50,51,61,64,68,84,105,122].”
Chiu, T., & Eysenbach, G. (2010) · BMC Medical Informatics and Decision Making · doi:10.1186/1472-6947-10-73 · Cited by 152 · cited 1×
“Indeed, these themes suggest usability problems, which decrease the likelihood of user engagement because of a mismatch of design with user needs [141,142].”
King, C. A., Eisenberg, D., Zheng, K., Czyz, E., Kramer, A., Horwitz, A., & Chermack, S. (2015) · Journal of Consulting and Clinical Psychology · doi:10.1037/a0038805 · Cited by 147 · cited 1×
“Approximately half of the studies included in this review (n=45 [41-44,48,54-57,59,60,62,63,70,73,76-78,81,84,85,87-93,96,100,102,104,106,110,111-113,116,118,121-125,127]) did not present any usability or acceptability outcomes.”
Day, V., McGrath, P. J., & Wojtowicz, M. (2013) · Behaviour Research and Therapy · doi:10.1016/j.brat.2013.03.003 · Cited by 145 · cited 10×
“Most studies included were focused on either universal prevention programs (n=36 [43,48,53,54,63-67,70-73,75-78,90,91, 94,95,97,99,100,102,106-108,110-112,114,118,119,125,126]), or on treatment intervention programs (n=22 [41,49,50,52, 58,69,74,79,80,83-86,88,89,92,98,113,117,120,122,127]).”
InfluentialYang, W., Ding, Z., Dai, T., Fang, P., & Zhang, J. X. (2014) · Journal of Behavior Therapy and Experimental Psychiatry · doi:10.1016/j.jbtep.2014.08.005 · Cited by 145 · cited 2×
“…utilized a control condition; although the quality of control conditions varied broadly and ranged from treatment as usual or waitlists to other digital programs or face-to-face t r e a t m e n t ([41,42,45,47,48,50,51,54,56,57,59,60,71,78,79,84,86,88,89,92,93,100,101,103,105,107,109,112,113,125]).”
Finlay‐Jones, A., Kane, R., & Rees, C. (2017) · Journal of Clinical Psychology · doi:10.1002/jclp.22375 · Cited by 139 · cited 1×
“Of the 16 nonrandomized studies screened for risk of bias, the majority of studies (n=10 [113-115, 117 , 120, 122-125 122-125,”
Evaluation of an mHealth App (DeStressify) on University Students’ Mental Health: Pilot Trial
Lee, R. A., & Jung, M. E. (2018) · JMIR Mental Health · doi:10.2196/mental.8324 · Cited by 138 · cited 2×
“Approximately half of the studies included in this review (n=45 [41-44,48,54-57,59,60,62,63,70,73,76-78,81,84,85,87-93,96,100,102,104,106,110,111-113,116,118,121-125,127]) did not present any usability or acceptability outcomes.”
Tillfors, M., Carlbring, P., Furmark, T., Lewenhaupt, S., Spak, M., Eriksson, A., Westling, B. E., & Andersson, G. (2008) · Depression and Anxiety · doi:10.1002/da.20416 · Cited by 132 · cited 10×
“Most studies included were focused on either universal prevention programs (n=36 [43,48,53,54,63-67,70-73,75-78,90,91, 94,95,97,99,100,102,106-108,110-112,114,118,119,125,126]), or on treatment intervention programs (n=22 [41,49,50,52, 58,69,74,79,80,83-86,88,89,92,98,113,117,120,122,127]).”
InfluentialMailey, E. L., Wójcicki, T. R., Motl, R. W., Hu, L., Strauser, D. R., Collins, K. D., & McAuley, E. (2010) · Psychology Health & Medicine · doi:10.1080/13548506.2010.498894 · Cited by 132 · cited 1×
“However, response rates were often low (which was specifically noted in the studies by Lintvedt et al [68] and Mailey et al [69]), so it is difficult to ascertain the true acceptability of these interventions as those who continued to engage with the study procedures may have found the…”
Spadaro, K. C., & Hunker, D. F. (2016) · Nurse Education Today · doi:10.1016/j.nedt.2016.02.006 · Cited by 132 · cited 1×
“103,109-111,114,118,119,121,124,125]) in which the participants only had contact with the study staff for research assessments; few studies focused on technology as an adjunct to therapy (n=5 [47,86,100,120,126]), 2 studies provided peer support [96,117], and 1 study had an unclear presence of…”
Evaluating an online stress management intervention for college students.
Hintz, S., Frazier, P. A., & Meredith, L. (2015) · Journal of Counseling Psychology · doi:10.1037/cou0000014 · Cited by 128 · cited 2×
“Approximately half of the studies included in this review (n=45 [41-44,48,54-57,59,60,62,63,70,73,76-78,81,84,85,87-93,96,100,102,104,106,110,111-113,116,118,121-125,127]) did not present any usability or acceptability outcomes.”
An evaluation of an online peer support forum for university students with depressive symptoms
Horgan, A., McCarthy, G., & Sweeney, J. (2013) · Archives of Psychiatric Nursing · doi:10.1016/j.apnu.2012.12.005 · Cited by 127 · cited 3×
“Of the 16 nonrandomized studies screened for risk of bias, the majority of studies (n=10 [113-115, 117 , 120, 122-125 122-125,”
ePREP: Computer Based Prevention of Relationship Dysfunction, Depression and Anxiety
Braithwaite, S. R., & Fincham, F. D. (2007) · Journal of Social and Clinical Psychology · doi:10.1521/jscp.2007.26.5.609 · Cited by 127 · cited 4×
“Approximately half of the studies included in this review (n=45 [41-44,48,54-57,59,60,62,63,70,73,76-78,81,84,85,87-93,96,100,102,104,106,110,111-113,116,118,121-125,127]) did not present any usability or acceptability outcomes.”
InfluentialMental Health Mobile Apps: From Infusion to Diffusion in the Mental Health Social System
East, M. L., & Havard, B. (2015) · JMIR Mental Health · doi:10.2196/mental.3954 · Cited by 127
Internet-Delivered Indicated Prevention For Anxiety Disorders: A Randomized Controlled Trial
Kenardy, J., McCafferty, K., & Rosa, V. (2003) · Behavioural and Cognitive Psychotherapy · doi:10.1017/s1352465803003047 · Cited by 121 · cited 1×
“…appeared to use college students as a convenience sample, and for an additional 9 studies, it was unclear if the program was being tested specifically for college student mental health or if college students were a convenience sample [47,50,51,61,64,68,84,105,122].”
MyStudentBody-stress: an online stress management intervention for college students
Chiauzzi, E., Brevard, J., Thurn, C., Decembrele, S., & Lord, S. (2008) · Journal of Health Communication · doi:10.1080/10810730802281668 · Cited by 119 · cited 10×
“Some concerns were noted in this domain for roughly one-third of the studies (n=27 [42,46,49,50,52,53,55,56,58,59,61-63, 65,67-70,72-74,76,81,82,85,87,89]) because outcomes were self-reported in nature and the participants were aware of the intervention they received.”
InfluentialFortney, J. C., Curran, G. M., Hunt, J. B., Cheney, A. M., Lu, L., Valenstein, M., & Eisenberg, D. (2016) · General Hospital Psychiatry · doi:10.1016/j.genhosppsych.2015.09.007 · Cited by 119 · cited 1×
“It was notable that only 1 study comprised a community college sample, as it is widely recognized that community college students have higher rates of unmet mental health needs compared with students in traditional 4-year colleges and universities [136-138].”
Web-Based Intervention Programs for Depression: A Scoping Review and Evaluation
Renton, T., Tang, H., Ennis, N., Cusimano, M. D., Bhalerao, S., Schweizer, T. A., & Topolovec-Vranic, J. (2014) · Journal of Medical Internet Research · doi:10.2196/jmir.3147 · Cited by 116 · cited 1×
“…health interventions, such as those delivered via mobile-and Web-based platforms, offer the possibility of treatment to college students with common mental health problems while circumventing many existing barriers to receiving traditional mental health services, including stigma and time [9-11].”
Braithwaite, S. R., & Fincham, F. D. (2009) · Journal of Family Psychology · doi:10.1037/a0014061 · Cited by 112 · cited 2×
“Approximately half of the studies included in this review (n=45 [41-44,48,54-57,59,60,62,63,70,73,76-78,81,84,85,87-93,96,100,102,104,106,110,111-113,116,118,121-125,127]) did not present any usability or acceptability outcomes.”
Cohen, G. E., & KERR, B. A. (1999) · Journal of Technology in Human Services · doi:10.1300/j407v15n04_02 · Cited by 111 · cited 3×
“…in which the participants only had contact with the study staff for research assessments; few studies focused on technology as an adjunct to therapy (n=5 [47,86,100,120,126]), 2 studies provided peer support [96,117], and 1 study had an unclear presence of support (n=1 [127]).”
Mak, W. W., Chan, A. T., Cheung, E. Y., Lin, C. L., & Ngai, K. C. (2015) · Journal of Medical Internet Research · doi:10.2196/jmir.3746 · Cited by 110 · cited 1×
“Approximately half of the studies included in this review (n=45 [41-44,48,54-57,59,60,62,63,70,73,76-78,81,84,85,87-93,96,100,102,104,106,110,111-113,116,118,121-125,127]) did not present any usability or acceptability outcomes.”
Trockel, M., Manber, R., Chang, V., Thurston, A., & Tailor, C. B. (2011) · Journal of Clinical Sleep Medicine · doi:10.5664/jcsm.1072 · Cited by 108 · cited 4×
“Approximately half of the studies included in this review (n=45 [41-44,48,54-57,59,60,62,63,70,73,76-78,81,84,85,87-93,96,100,102,104,106,110,111-113,116,118,121-125,127]) did not present any usability or acceptability outcomes.”
InfluentialSontag-Padilla, L. M., Woodbridge, M. W., Mendelsohn, J., D’Amico, E. J., Osilla, K. C., Jaycox, L. H., Eberhart, N. K., Burnam, A., & Stein, B. D. (2016) · Psychiatric Services · doi:10.1176/appi.ps.201500307 · Cited by 105 · cited 2×
“Furthermore, community college students are likely to face additional barriers to accessing care as they are more likely to attend school part-time while balancing other responsibilities and commitments, and many community college campuses do not provide mental health services [138].”
Levin, M. E., Hayes, S. C., Pistorello, J., & Seeley, J. R. (2016) · Journal of Clinical Psychology · doi:10.1002/jclp.22254 · Cited by 103 · cited 1×
“As seen in Multimedia Appendix 55,64,66,98,114,117]) or did not report on psychological outcomes and focused on program usage or usability (n=6 [116,118,119,121,126,129]).”
Musiat, P., Conrod, P., Treasure, J. L., Tylee, A. T., Williams, C., & Schmidt, U. H. (2014) · PLoS ONE · doi:10.1371/journal.pone.0093621 · Cited by 98 · cited 10×
“Most studies included were focused on either universal prevention programs (n=36 [43,48,53,54,63-67,70-73,75-78,90,91, 94,95,97,99,100,102,106-108,110-112,114,118,119,125,126]), or on treatment intervention programs (n=22 [41,49,50,52, 58,69,74,79,80,83-86,88,89,92,98,113,117,120,122,127]).”
InfluentialRadhu, N., Daskalakis, Z. J., Arpin-Cribbie, C. A., Irvine, J., & Ritvo, P. (2012) · Journal of American College Health · doi:10.1080/07448481.2011.630703 · Cited by 94 · cited 1×
“Approximately half of the studies included in this review (n=45 [41-44,48,54-57,59,60,62,63,70,73,76-78,81,84,85,87-93,96,100,102,104,106,110,111-113,116,118,121-125,127]) did not present any usability or acceptability outcomes.”
Mak, W. W., Chio, F. H., Chan, A. T., Lui, W. W., & Wu, E. K. (2017) · Journal of Medical Internet Research · doi:10.2196/jmir.6737 · Cited by 94 · cited 2×
“…utilized a control condition; although the quality of control conditions varied broadly and ranged from treatment as usual or waitlists to other digital programs or face-to-face t r e a t m e n t ([41,42,45,47,48,50,51,54,56,57,59,60,71,78,79,84,86,88,89,92,93,100,101,103,105,107,109,112,113,125]).”
Vilardaga, R., Rizo, J., Zeng, E., Kientz, J. A., Ries, R., Otis, C., & Hernandez, K. (2018) · JMIR Serious Games · doi:10.2196/games.8881 · Cited by 90 · cited 1×
“Utilization of user-centered design and usability testing is a growing practice in digital health interventions for depression [143], smoking cessation [144,145], and diverse patient groups [146].”
A randomised placebo-controlled trial of a self-help Internet-based intervention for test anxiety
Orbach, G., Lindsay, S., & Grey, S. J. (2006) · Behaviour Research and Therapy · doi:10.1016/j.brat.2006.04.002 · Cited by 77 · cited 1×
“As seen in Concerns were often noted regarding potential bias arising from the randomization process, and potential bias because of missing outcome data that were not analyzed in a manner to minimize risk [80-82].”
Mullin, A., Dear, B. F., Karin, E., Wootton, B. M., Staples, L., Johnston, L., Gandy, M., Fogliati, V. J., & Titov, N. (2015) · Internet Interventions · doi:10.1016/j.invent.2015.02.002 · Cited by 76 · cited 10×
“Most studies included were focused on either universal prevention programs (n=36 [43,48,53,54,63-67,70-73,75-78,90,91, 94,95,97,99,100,102,106-108,110-112,114,118,119,125,126]), or on treatment intervention programs (n=22 [41,49,50,52, 58,69,74,79,80,83-86,88,89,92,98,113,117,120,122,127]).”
InfluentialNguyen-Feng, V. N., Greer, C. S., & Frazier, P. (2017) · Psychological Services · doi:10.1037/ser0000154 · Cited by 73 · cited 2×
“Approximately half of the studies included in this review (n=45 [41-44,48,54-57,59,60,62,63,70,73,76-78,81,84,85,87-93,96,100,102,104,106,110,111-113,116,118,121-125,127]) did not present any usability or acceptability outcomes.”
Internet-delivered cognitive behavior therapy for anxiety and insomnia in a higher education context
Morris, J., Firkins, A., Millings, A., Möhr, C., Redford, P., & Rowe, A. (2015) · Anxiety Stress & Coping · doi:10.1080/10615806.2015.1058924 · Cited by 72 · cited 1×
“Approximately half of the studies included in this review (n=45 [41-44,48,54-57,59,60,62,63,70,73,76-78,81,84,85,87-93,96,100,102,104,106,110,111-113,116,118,121-125,127]) did not present any usability or acceptability outcomes.”
Web-based cognitive-behavioral therapy for perfectionism: A randomized controlled trial
Arpin-Cribbie, C., Irvine, J., & Ritvo, P. (2012) · Psychotherapy Research · doi:10.1080/10503307.2011.637242 · Cited by 72 · cited 1×
“…utilized a control condition; although the quality of control conditions varied broadly and ranged from treatment as usual or waitlists to other digital programs or face-to-face t r e a t m e n t ([41,42,45,47,48,50,51,54,56,57,59,60,71,78,79,84,86,88,89,92,93,100,101,103,105,107,109,112,113,125]).”
McCall, H. C., Richardson, C. G., Helgadottir, F. D., & Chen, F. S. (2018) · Journal of Medical Internet Research · doi:10.2196/jmir.8630 · Cited by 71 · cited 3×
“…Headspace and Smiling Mind, n=1 [95]; Family eJournal, n=1 [90]; Tess chatbot, n=1 [97]; DeStressify, n=1 [102]; Overcome Social Anxiety, n=1 [103]); SilverCloud Health’s Space from Depression, Space from Anxiety, and Space from Stress, n=1 [128]; and CareCollaborateConnect: Student Success…”
Computer-Based Intervention for Anxious and Depressive Symptoms in a Non-Clinical Population
Cukrowicz, K. C., & Joiner, T. E. (2007) · Cognitive Therapy and Research · doi:10.1007/s10608-006-9094-x · Cited by 68 · cited 2×
“Approximately half of the studies included in this review (n=45 [41-44,48,54-57,59,60,62,63,70,73,76-78,81,84,85,87-93,96,100,102,104,106,110,111-113,116,118,121-125,127]) did not present any usability or acceptability outcomes.”
Geisner, I. M., Varvil‐Weld, L., Mittmann, A. J., Mallett, K. A., & Turrisi, R. (2014) · Addictive Behaviors · doi:10.1016/j.addbeh.2014.10.030 · Cited by 67 · cited 10×
“Some concerns were noted in this domain for roughly one-third of the studies (n=27 [42,46,49,50,52,53,55,56,58,59,61-63, 65,67-70,72-74,76,81,82,85,87,89]) because outcomes were self-reported in nature and the participants were aware of the intervention they received.”
InfluentialMelnyk, B. M., Amaya, M., Szalacha, L. A., Hoying, J., Taylor, T., & Bowersox, K. (2015) · Journal of Child and Adolescent Psychiatric Nursing · doi:10.1111/jcap.12119 · Cited by 66 · cited 10×
“Most studies included were focused on either universal prevention programs (n=36 [43,48,53,54,63-67,70-73,75-78,90,91, 94,95,97,99,100,102,106-108,110-112,114,118,119,125,126]), or on treatment intervention programs (n=22 [41,49,50,52, 58,69,74,79,80,83-86,88,89,92,98,113,117,120,122,127]).”
InfluentialKaczmarek, L. D., Kashdan, T. B., Kleiman, E. M., Bączkowski, B. M., Enko, J., Siebers, A., Szäefer, A., Król, M., & Baran, B. (2013) · Personality and Individual Differences · doi:10.1016/j.paid.2013.06.013 · Cited by 66 · cited 4×
“Approximately half of the studies included in this review (n=45 [41-44,48,54-57,59,60,62,63,70,73,76-78,81,84,85,87-93,96,100,102,104,106,110,111-113,116,118,121-125,127]) did not present any usability or acceptability outcomes.”
InfluentialKim, E.-H., Coumar, A., Lober, W. B., & Kim, Y. (2011) · IEEE Transactions on Information Technology in Biomedicine · doi:10.1109/titb.2011.2107561 · Cited by 65 · cited 4×
“…the importance of the systems in which interventions are to be introduced, and although all studies included in this review focused on college students as a population of interest, very few examined college campus systems and tested the implementation of programs onto campuses [86,109,116,119].”
InfluentialHoppitt, L., Illingworth, J. L., MacLeod, C., Hampshire, A., Dunn, B. D., & Mackintosh, B. (2013) · Behaviour Research and Therapy · doi:10.1016/j.brat.2013.10.008 · Cited by 63 · cited 2×
“Approximately half of the studies included in this review (n=45 [41-44,48,54-57,59,60,62,63,70,73,76-78,81,84,85,87-93,96,100,102,104,106,110,111-113,116,118,121-125,127]) did not present any usability or acceptability outcomes.”
Koydemir, S., & Selişik, Z. E. S. (2015) · British Journal of Guidance and Counselling · doi:10.1080/03069885.2015.1110562 · Cited by 60 · cited 1×
“Approximately half of the studies included in this review (n=45 [41-44,48,54-57,59,60,62,63,70,73,76-78,81,84,85,87-93,96,100,102,104,106,110,111-113,116,118,121-125,127]) did not present any usability or acceptability outcomes.”
Kass, A. E., Balantekin, K. N., Fitzsimmons‐Craft, E. E., Jacobi, C., Wilfley, D. E., & Taylor, C. B. (2017) · International Journal of Eating Disorders · doi:10.1002/eat.22680 · Cited by 60 · cited 1×
“…health interventions, such as those delivered via mobile-and Web-based platforms, offer the possibility of treatment to college students with common mental health problems while circumventing many existing barriers to receiving traditional mental health services, including stigma and time [9-11].”
Palacios, J. E., Richards, D., Palmer, R., Coudray, C., Hofmann, S. G., Palmieri, P. A., & Frazier, P. (2018) · JMIR Mental Health · doi:10.2196/11467 · Cited by 58 · cited 4×
“…that were implemented on college campuses [86,116,119,128], the Beating the Blues implementation by Santucci et al [86] and the SilverCloud implementation by Palacios et al [128] were the only studies in which the feasibility of implementing a digital mental health program was explicitly discussed.”
InfluentialEustis, E. H., Hayes-Skelton, S. A., Orsillo, S. M., & Roemer, L. (2018) · Behavior Therapy · doi:10.1016/j.beth.2018.05.009 · Cited by 57 · cited 9×
“Most studies included were focused on either universal prevention programs (n=36 [43,48,53,54,63-67,70-73,75-78,90,91, 94,95,97,99,100,102,106-108,110-112,114,118,119,125,126]), or on treatment intervention programs (n=22 [41,49,50,52, 58,69,74,79,80,83-86,88,89,92,98,113,117,120,122,127]).”
InfluentialFrazier, P. A., Meredith, L. N., Greer, C. S., Paulsen, J. A., Howard, K. G., Dietz, L. R., & Qin, K. (2014) · Anxiety Stress & Coping · doi:10.1080/10615806.2014.987666 · Cited by 56 · cited 10×
“Most studies included were focused on either universal prevention programs (n=36 [43,48,53,54,63-67,70-73,75-78,90,91, 94,95,97,99,100,102,106-108,110-112,114,118,119,125,126]), or on treatment intervention programs (n=22 [41,49,50,52, 58,69,74,79,80,83-86,88,89,92,98,113,117,120,122,127]).”
InfluentialKvillemo, P., Brandberg, Y., & Bränström, R. (2016) · JMIR Mental Health · doi:10.2196/mental.5457 · Cited by 56
Hall, B. J., Xiong, P., Guo, X., Sou, E. K. L., Chou, U. I., & Shen, Z. (2018) · Psychiatry Research · doi:10.1016/j.psychres.2018.09.060 · Cited by 56 · cited 3×
“Approximately half of the studies included in this review (n=45 [41-44,48,54-57,59,60,62,63,70,73,76-78,81,84,85,87-93,96,100,102,104,106,110,111-113,116,118,121-125,127]) did not present any usability or acceptability outcomes.”
Auyeung, L., & Mo, P. K. H. (2018) · Journal of Happiness Studies · doi:10.1007/s10902-018-0054-4 · Cited by 56 · cited 10×
“Most studies included were focused on either universal prevention programs (n=36 [43,48,53,54,63-67,70-73,75-78,90,91, 94,95,97,99,100,102,106-108,110-112,114,118,119,125,126]), or on treatment intervention programs (n=22 [41,49,50,52, 58,69,74,79,80,83-86,88,89,92,98,113,117,120,122,127]).”
InfluentialSharry, J., Davidson, R., McLoughlin, O., & Doherty, G. (2013) · Journal of Medical Internet Research · doi:10.2196/jmir.2248 · Cited by 55 · cited 1×
“As seen in Multimedia Appendix 2, a m i n o r i t y o f t h e s t u d i e s ( n = 1 2 [44,71,89,90,93,95,104,110,111,114,118,123]) appeared to use college students as a convenience sample, and for an additional 9 studies, it was unclear if the program was being tested specifically for college…”
Richards, D., Timulak, L., & Hevey, D. (2013) · Counselling and Psychotherapy Research · doi:10.1080/14733145.2012.733715 · Cited by 55 · cited 3×
“Approximately half of the studies included in this review (n=45 [41-44,48,54-57,59,60,62,63,70,73,76-78,81,84,85,87-93,96,100,102,104,106,110,111-113,116,118,121-125,127]) did not present any usability or acceptability outcomes.”
Álvarez, L. M., Sotres, J. F. C., León, S. O., Estrella, J., & Sosa, J. J. S. (2007) · Computers in Human Behavior · doi:10.1016/j.chb.2007.02.013 · Cited by 54 · cited 3×
“Of the 89 studies included in this review, 72 employed randomized study designs [ 41-112 41-112 ], whereas the remaining 17 We examined if each study appeared to be designed specifically for students (eg, the purpose of the study was focused on college student mental health) or if college students…”
Williams, A., LaRocca, R. A., Chang, T. E., Trinh, N., D, M. F. M., Kvedar, J. C., & Yeung, A. (2014) · International Journal of Telemedicine and Applications · doi:10.1155/2014/580786 · Cited by 53 · cited 2×
“…in which the participants only had contact with the study staff for research assessments; few studies focused on technology as an adjunct to therapy (n=5 [47,86,100,120,126]), 2 studies provided peer support [96,117], and 1 study had an unclear presence of support (n=1 [127]).”
Outcome of an Online Mutual Support Group for College Students with Psychological Problems
Freeman, E., Barker, C., & Pistrang, N. (2008) · CyberPsychology & Behavior · doi:10.1089/cpb.2007.0133 · Cited by 51 · cited 2×
“Approximately half of the studies included in this review (n=45 [41-44,48,54-57,59,60,62,63,70,73,76-78,81,84,85,87-93,96,100,102,104,106,110,111-113,116,118,121-125,127]) did not present any usability or acceptability outcomes.”
Richards, D., Timulák, L., Rashleigh, C., McLoughlin, O., Colla, A., Joyce, C., Doherty, G. J., Sharry, J., Duffy, D. J., & Anderson-Gibbons, M. (2016) · Internet Interventions · doi:10.1016/j.invent.2016.10.003 · Cited by 50 · cited 10×
“Most studies included were focused on either universal prevention programs (n=36 [43,48,53,54,63-67,70-73,75-78,90,91, 94,95,97,99,100,102,106-108,110-112,114,118,119,125,126]), or on treatment intervention programs (n=22 [41,49,50,52, 58,69,74,79,80,83-86,88,89,92,98,113,117,120,122,127]).”
InfluentialWozney, L., McGrath, P. J., Gehring, N. D., Bennett, K., Huguet, A., Hartling, L., Dyson, M. P., Soleimani, A., & Newton, A. S. (2018) · JMIR Mental Health · doi:10.2196/mental.9655 · Cited by 50 · cited 1×
“Increasingly, calls are being made to collect implementation-relevant data while testing new digital mental health programs or testing existing programs on new populations [150-152].”
Can we learn to manage stress? A randomized controlled trial carried out on university students
Saleh, D., Camart, N., Sbeira, F., & Romo, L. (2018) · PLoS ONE · doi:10.1371/journal.pone.0200997 · Cited by 50 · cited 1×
“…utilized a control condition; although the quality of control conditions varied broadly and ranged from treatment as usual or waitlists to other digital programs or face-to-face t r e a t m e n t ([41,42,45,47,48,50,51,54,56,57,59,60,71,78,79,84,86,88,89,92,93,100,101,103,105,107,109,112,113,125]).”
Levin, M. E., Pistorello, J., Hayes, S. C., Seeley, J. R., & Levin, C. (2015) · Journal of Counseling Psychology · doi:10.1037/cou0000083 · Cited by 47 · cited 3×
“A minority of studies used validated measures such as the System Usability Scale [65-67, 120”
Benton, S., Heesacker, M., Snowden, S., & Lee, G. (2016) · Professional Psychology Research and Practice · doi:10.1037/pro0000097 · Cited by 47 · cited 4×
“Approximately half of the studies included in this review (n=45 [41-44,48,54-57,59,60,62,63,70,73,76-78,81,84,85,87-93,96,100,102,104,106,110,111-113,116,118,121-125,127]) did not present any usability or acceptability outcomes.”
InfluentialFleischmann, R. J., Harrer, M., Zarski, A.-C., Baumeister, H., Lehr, D., & Ebert, D. D. (2018) · Internet Interventions · doi:10.1016/j.invent.2017.12.001 · Cited by 47 · cited 9×
“Although the interventions examined were variable in content and in length, the most common type of intervention examined was internet-based cognitive behavioral therapy (n=28 [42,49,50,57,61,68,72-75,79-81,83-86,88,98,101, 103,107,109,112,113,123,128,129]).”
InfluentialUser Experience Evaluation of a Smoking Cessation App in People With Serious Mental Illness
Vilardaga, R., Rizo, J. L., Kientz, J. A., McDonell, M. G., Ries, R. K., & Sobel, K. (2015) · Nicotine & Tobacco Research · doi:10.1093/ntr/ntv256 · Cited by 46 · cited 1×
“Utilization of user-centered design and usability testing is a growing practice in digital health interventions for depression [143], smoking cessation [144,145], and diverse patient groups [146].”
Hill, C., Creswell, C., Vigerland, S., Nauta, M. H., March, S., Donovan, C., Wolters, L., Spence, S. H., Martin, J. L., Wozney, L., et al. (2018) · Internet Interventions · doi:10.1016/j.invent.2018.02.002 · Cited by 44 · cited 1×
“Increasingly, calls are being made to collect implementation-relevant data while testing new digital mental health programs or testing existing programs on new populations [150-152].”
Mogoaşe, C., Brăilean, A., & David, D. (2013) · Cognitive Therapy and Research · doi:10.1007/s10608-012-9514-z · Cited by 43 · cited 10×
“Furthermore, many studies did not present the metrics on the usage of the digital mental health interventions (n=29 [44,45,48,50,53,56,69,70,73,74,76-78,81, 89,90,93,98,99,103-105,107,109-111,113,114,124].”
InfluentialHowell, A. N., Rheingold, A. A., Uhde, T. W., & Guille, C. (2018) · Cognitive Behaviour Therapy · doi:10.1080/16506073.2018.1533575 · Cited by 41 · cited 2×
“Of the 89 studies included in this review, 72 employed randomized study designs [ 41-112 41-112 ], whereas the remaining 17 We examined if each study appeared to be designed specifically for students (eg, the purpose of the study was focused on college student mental health) or if college students…”
Richards, D., & Timulak, L. (2013) · British Journal of Guidance & Counselling · doi:10.1080/03069885.2012.726347 · Cited by 40 · cited 10×
“Most studies included were focused on either universal prevention programs (n=36 [43,48,53,54,63-67,70-73,75-78,90,91, 94,95,97,99,100,102,106-108,110-112,114,118,119,125,126]), or on treatment intervention programs (n=22 [41,49,50,52, 58,69,74,79,80,83-86,88,89,92,98,113,117,120,122,127]).”
InfluentialRai, M., Vigod, S. N., & Hensel, J. M. (2016) · JMIR Mental Health · doi:10.2196/mental.6068 · Cited by 38 · cited 1×
“Digital mental health interventions, such as those delivered via mobile- and Web-based platforms, offer the possibility of treatment to college students with common mental health problems while circumventing many existing barriers to receiving traditional mental health services, including stigma and time [9-11].”
Nguyen‐Feng, V. N., Frazier, P. A., Greer, C. S., Howard, K. G., Paulsen, J. A., Meredith, L. N., & Kim, S. (2015) · Psychology of Violence · doi:10.1037/a0039596 · Cited by 37 · cited 1×
“Approximately half of the studies included in this review (n=45 [41-44,48,54-57,59,60,62,63,70,73,76-78,81,84,85,87-93,96,100,102,104,106,110,111-113,116,118,121-125,127]) did not present any usability or acceptability outcomes.”
Booker, J. A., & Dunsmore, J. C. (2017) · Journal of Social and Clinical Psychology · doi:10.1521/jscp.2017.36.7.580 · Cited by 34 · cited 10×
“Most studies included were focused on either universal prevention programs (n=36 [43,48,53,54,63-67,70-73,75-78,90,91, 94,95,97,99,100,102,106-108,110-112,114,118,119,125,126]), or on treatment intervention programs (n=22 [41,49,50,52, 58,69,74,79,80,83-86,88,89,92,98,113,117,120,122,127]).”
InfluentialReducing distress in university students: A randomised control trial of two online interventions
Stallman, H. M., Ohan, J. L., & Chiera, B. (2019) · Australian Psychologist · doi:10.1111/ap.12375 · Cited by 32 · cited 4×
“…the importance of the systems in which interventions are to be introduced, and although all studies included in this review focused on college students as a population of interest, very few examined college campus systems and tested the implementation of programs onto campuses [86,109,116,119].”
InfluentialA Bioecological Systems Approach for Navigating the College Mental Health Crisis
Pinder-Amaker, S., & Bell, C. (2012) · Harvard Review of Psychiatry · doi:10.3109/10673229.2012.712842 · Cited by 31 · cited 1×
“In the last decade, rising rates of college students experiencing symptoms of depression and anxiety have been observed [1-3].”
Traditional group therapy and computer-administered treatment for test anxiety
Buglione, S. A., DeVito, A. J., & Mulloy, J. M. (1990) · Anxiety Research · doi:10.1080/08917779008248739 · Cited by 31 · cited 1×
“…utilized a control condition; although the quality of control conditions varied broadly and ranged from treatment as usual or waitlists to other digital programs or face-to-face t r e a t m e n t ([41,42,45,47,48,50,51,54,56,57,59,60,71,78,79,84,86,88,89,92,93,100,101,103,105,107,109,112,113,125]).”
Lattie, E. G., Ho, J., Sargent, E., Tomasino, K. N., Smith, J. D., Brown, C. H., & Mohr, D. C. (2017) · Internet Interventions · doi:10.1016/j.invent.2017.02.003 · Cited by 31 · cited 1×
“The research-to-practice gap for digital mental health interventions is increasingly being recognized, and leaders in the field have proposed strategies to routinely incorporate implementation science methods into the study of digital mental health interventions [148,149].”
Self-citationNordmo, M., Sinding, A. I., Carlbring, P., Andersson, G., Havik, O. E., & Nordgreen, T. (2015) · Internet Interventions · doi:10.1016/j.invent.2015.10.003 · Cited by 30 · cited 1×
“…utilized a control condition; although the quality of control conditions varied broadly and ranged from treatment as usual or waitlists to other digital programs or face-to-face t r e a t m e n t ([41,42,45,47,48,50,51,54,56,57,59,60,71,78,79,84,86,88,89,92,93,100,101,103,105,107,109,112,113,125]).”
Crookston, B. T., West, J. H., Hall, C. P., Dahle, K. M., Heaton, T. L., Beck, R. N., & Muralidharan, C. (2017) · JMIR Mental Health · doi:10.2196/mental.7262 · Cited by 30
New Technologies to Manage Exam Anxiety
Grassi, A., Gaggioli, A., & Giuseppe, R. (2011) · Studies in health technology and informatics · doi:10.3233/978-1-60750-766-6-57 · Cited by 29 · cited 10×
“Furthermore, many studies did not present the metrics on the usage of the digital mental health interventions (n=29 [44,45,48,50,53,56,69,70,73,74,76-78,81, 89,90,93,98,99,103-105,107,109-111,113,114,124].”
InfluentialCarey, T. A., Haviland, J., Tai, S. J., Vanags, T., & Mansell, W. (2016) · BMC Psychiatry · doi:10.1186/s12888-016-1168-z · Cited by 29 · cited 3×
“As seen in Multimedia Appendix 55,64,66,98,114,117]) or did not report on psychological outcomes and focused on program usage or usability (n=6 [116,118,119,121,126,129]).”
Exploring user learnability and learning performance in an app for depression: usability study
Stiles-Shields, C., Montague, E., Lattie, E. G., Schueller, S. M., Kwasny, M. J., & Mohr, D. C. (2017) · JMIR Human Factors · doi:10.2196/humanfactors.7951 · Cited by 29
Learning Lucid Dreaming and its Effect on Depression in Undergraduates
Taitz, I. Y. (2011) · International journal of dream research · doi:10.11588/ijodr.2011.2.9123 · Cited by 27
Pilot Implementation of Computerized Cognitive Behavioral Therapy in a University Health Setting
Santucci, L. C., McHugh, R. K., Elkins, R. M., Schechter, B., Ross, M. S., Landa, C. E., Eisen, S. V., & Barlow, D. H. (2013) · Administration and Policy in Mental Health and Mental Health Services Research · doi:10.1007/s10488-013-0488-2 · Cited by 27 · cited 6×
“Santucci et al set out to assess the feasibility, acceptability, and effectiveness of Beating the Blues for university students (as benchmarked to published trials) and outlined their process of conducting a needs assessment and engaging stakeholders before commencing the trial [86].”
InfluentialNguyen‐Feng, V. N., Frazier, P. A., Greer, C. S., Meredith, L. N., Howard, K. G., & Paulsen, J. A. (2016) · Translational Issues in Psychological Science · doi:10.1037/tps0000099 · Cited by 21 · cited 10×
“Most studies included were focused on either universal prevention programs (n=36 [43,48,53,54,63-67,70-73,75-78,90,91, 94,95,97,99,100,102,106-108,110-112,114,118,119,125,126]), or on treatment intervention programs (n=22 [41,49,50,52, 58,69,74,79,80,83-86,88,89,92,98,113,117,120,122,127]).”
InfluentialBedford, L. A., Dietch, J. R., Taylor, D. J., Boals, A., & Zayfert, C. (2017) · Behavior Therapy · doi:10.1016/j.beth.2017.11.010 · Cited by 18 · cited 1×
“…utilized a control condition; although the quality of control conditions varied broadly and ranged from treatment as usual or waitlists to other digital programs or face-to-face t r e a t m e n t ([41,42,45,47,48,50,51,54,56,57,59,60,71,78,79,84,86,88,89,92,93,100,101,103,105,107,109,112,113,125]).”
Mental health treatment seeking among veteran and civilian community college students
Fortney, J. C., Curran, G. M., Hunt, J. B., Lu, L., Eisenberg, D., & Valenstein, M. (2017) · Psychiatric Services · doi:10.1176/appi.ps.201600240 · Cited by 17 · cited 1×
“It was notable that only 1 study comprised a community college sample, as it is widely recognized that community college students have higher rates of unmet mental health needs compared with students in traditional 4-year colleges and universities [136-138].”
Sagon, A. L., Danitz, S. B., Suvak, M. K., & Orsillo, S. M. (2018) · Journal of Contextual Behavioral Science · doi:10.1016/j.jcbs.2018.06.004 · Cited by 16 · cited 1×
“Approximately half of the studies included in this review (n=45 [41-44,48,54-57,59,60,62,63,70,73,76-78,81,84,85,87-93,96,100,102,104,106,110,111-113,116,118,121-125,127]) did not present any usability or acceptability outcomes.”
Frith, E., & Loprinzi, P. D. (2017) · JMIR Mental Health · doi:10.2196/mental.8086 · Cited by 14 · cited 2×
“Approximately half of the studies included in this review (n=45 [41-44,48,54-57,59,60,62,63,70,73,76-78,81,84,85,87-93,96,100,102,104,106,110,111-113,116,118,121-125,127]) did not present any usability or acceptability outcomes.”
Virtual Reality Therapy: Case Study of Fear of Public Speaking
North, M. M., Schoeneman, C. M., & Mathis, J. R. (2002) · Studies in health technology and informatics · doi:10.3233/978-1-60750-929-5-318 · Cited by 13
FAMAP: A Framework for Developing m-Health Apps
García‐Magariño, I., Bedía, M. G., & Palacios-Navarro, G. (2018) · Advances in intelligent systems and computing · doi:10.1007/978-3-319-77703-0_83 · Cited by 13 · cited 1×
“The vast majority of the included studies (n=81 [41-44,47-52,54-56,58-111,113,114,116-120,122,123,125-129]) took place at universities, with far fewer taking place at 4-year colleges (n=2 [45,46]), at health professional schools (eg, medical school and nursing school; n=5 [57,115,121,124,130]), and at community colleg…”
Family eJournal: benefits of online guided group journaling for women
Asbury, E. T., Casey, J., & Desai, K. (2018) · Journal of Public Mental Health · doi:10.1108/jpmh-01-2018-0008 · Cited by 9 · cited 2×
“]); Therapist-Assisted Online, n=1 [113]; Headspace and Smiling Mind, n=1 [95]; Family eJournal, n=1 [90]; Tess chatbot, n=1 [97]; DeStressify, n=1 [102]; Overcome Social Anxiety, n=1 [103]); SilverCloud Health’s Space from Depression, Space from Anxiety, and Space from Stress, n=1 [128]; and…”
Coping with test stress: Microcomputer software for treatment of test anxiety
Wilson, F. R., Omeltschenko, L., & Yager, G. G. (1991) · Journal of Behavior Therapy and Experimental Psychiatry · doi:10.1016/0005-7916(91)90008-s · Cited by 8 · cited 2×
“Approximately half of the studies included in this review (n=45 [41-44,48,54-57,59,60,62,63,70,73,76-78,81,84,85,87-93,96,100,102,104,106,110,111-113,116,118,121-125,127]) did not present any usability or acceptability outcomes.”
Moir, F., Fernando, A. T., Kumar, S., Henning, M. A., Moyes, S. A., & Elley, C. R. (2015) · PubMed · Cited by 7
Stiles‐Shields, C., & Holmbeck, G. N. (2018) · Journal of Pediatric Psychology · doi:10.1093/jpepsy/jsy018 · Cited by 2 · cited 1×
“Therefore, assessing for usability is a critical component in establishing the feasibility, efficacy, and generalizability of a digital health intervention, particularly for specialty populations [147].”
Self-citation
Research works printed but not linked: 14
- Beck, AT, 1996, BDI-II, Beck Depression Inventory: Manual
- 10.1037//0022-006X.56.6.893 · doi:10.1037//0022-006x.56.6.893 · Paper page
- Peterson, RA, 1992, Anxiety Sensitivity Index Manual
- 10.1037/0022-3514.69.4.719 · doi:10.1037/0022-3514.69.4.719 · Paper page
- Sauro, J, 2011, A Practical Guide to the System Usability Scale: Background, Benchmarks & Best Practices
- Higgins, JP, 2016, Cochrane Methods 2016, 29
- 10.1037//1089-2680.1.3.311 · DOI · Paper page
- Ellis, LA, 2011, J Cyber Ther Rehabil, 4, 461
- Fernandez, E, 1986, J Coll Stud Dev, 27, 224
- Grassi, A, 2011, Stud Health Technol Inform, 167, 57
- 10.2196/mental.9782 · doi:10.2196/mental.9782 · Paper page
- Villani, D, 2008, PsychNol J, 6, 7
- Moir, F, 2015, N Z Med J, 128, 51
- Brooke, J, 1996, J Usability Stud, 189, 4
Cited by
Holmes, E. A., O'Connor, R. C., Perry, V. H., Tracey, I., Wessely, S., Arseneault, L., Ballard, C., Christensen, H., Cohen Silver, R., Everall, I., et al. (2020) · The Lancet Psychiatry · doi:10.1016/s2215-0366(20)30168-1 · Cited by 5,715
“The digital response is crucial, 37–39 not only because of social isolation measures but also because less than a third of people who die by suicide have been in contact with mental health services in the 12 months before death.”
BackgroundBorghouts, J., Eikey, E., Mark, G., De Leon, C., Schueller, S. M., Schneider, M., Stadnick, N., Zheng, K., Mukamel, D., & Sorkin, D. H. (2021) · Journal of Medical Internet Research · doi:10.2196/24387 · Cited by 1,069
“As seen in Multimedia Appendix 2, the 208 articles included in this review [27-237] reported on 2 types of user studies: (1) 69 studies were needs assessments that aimed to understand user needs and attitudes toward DMHIs without or before engaging with a specific intervention as part of a study and (2) 135 studies we…”
BackgroundArtificially intelligent chatbots in digital mental health interventions: a review
Boucher, E. M., Harake, N. R., Ward, H. E., Stoeckl, S. E., Vargas, J., Minkel, J., Parks, A. C., & Zilca, R. (2021) · Expert Review of Medical Devices · doi:10.1080/17434440.2021.2013200 · Cited by 440
“%) were at least partially effective in improving depressive symptoms, anxiety, psychological distress, or well-being [33].”
BackgroundDigital Mental Health for Young People: A Scoping Review of Ethical Promises and Challenges
Wies, B., Landers, C., & Ienca, M. (2021) · Frontiers in Digital Health · doi:10.3389/fdgth.2021.697072 · Cited by 242
He, Y., Yang, L., Qian, C., Li, T., Su, Z., Zhang, Q., & Hou, X. (2023) · Journal of Medical Internet Research · doi:10.2196/43862 · Cited by 213
“This may be because digital interventions such as internet-based CBT largely consisted of linear, structured psychotherapy modules [81,82], whereas more freedom for participants was offered in the context of CAIs [12], where interference and repetition would instead have an adverse effect on…”
Metadata
- Authors
- Lattie, E. G., Adkins, E. C., Winquist, N., Stiles-Shields, C., Wafford, Q. E., & Graham, A. K.
- Affiliations
- Northwestern University (US)
- Corresponding author
- Lattie, Emily G
- ORCID
- Venue
- Journal of Medical Internet Research
- Publisher
- JMIR Publications Inc.
- Published
- 2019-07-22
- Volume
- 21
- Issue
- 7
- Pages
- e12869
- ISSN
- 1438-8871 (ISSN-L)
- PMC
- PMC6681642
- OpenAlex
- W2950229699
- Cited by
- 843
- References
- 147
- Citations by year
- 2026: 175 · 2025: 145 · 2024: 142 · 2023: 142 · 2022: 134 · 2021: 70 · 2020: 32 · 2019: 3
- FWCI
- 8.35
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- Locations
- Journal of Medical Internet Research (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Funders
- National Institute of Mental Health (K08MH112878)
- National Institute of Diabetes and Digestive and Kidney Diseases (K01DK116925)
- Topics
- Social Sciences › Psychology › Applied Psychology › Digital Mental Health Interventions
- Health Sciences › Medicine › Psychiatry and Mental health › Mental Health Treatment and Access
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Anxiety, Depression, Psychometrics, Treatment, Cognitive Processes
- Keywords
- digital mental health interventions, depression, anxiety, psychological well-being, university students, internet-based cognitive behavioral therapy, mental health apps, usability, acceptability, university mental health services
- MeSH terms
- Adult, Anxiety, Depression, Humans, Mental Health, Psychotherapy, Students, Universities, Telemedicine, Young Adult
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.
