CoChat
Sign Up
Cited by 5Mobile Health and mHealth ApplicationsUsability and User Interface DesignCancer survivorship and care

Usability and Usefulness of a Symptom Management Coaching System for Patients With Cancer Treated With Immune Checkpoint Inhibitors: Comparative Mixed Methods Study

Open access
Glaser, Savannah Lucia Caterina, Fraterman, Itske, van Brummelen, Noah, Tibollo, Valentina, Del Campo, Laura Maria, Mallo, Henk, Wilgenhof, Sofie, Wilk, Szymon +5 more·2025·JMIR Formative Research
DOI 10.2196/57659PMC PMC11803325

Published version of a preprint

Open this paper in CoChat — free accountOpen PDF

Abstract

Background: The prognosis for patients with several types of cancer has substantially improved following the introduction of immune checkpoint inhibitors, a novel type of immunotherapy. However, patients may experience symptoms both from the cancer itself and from the medication. A prototype of the eHealth tool Cancer Patients Better Life Experience (CAPABLE) was developed to facilitate symptom management, aimed at patients with melanoma and renal cell carcinoma treated with immunotherapy. Better usability of such eHealth tools can lead to improved user well-being and reduced risk of harm. It is unknown for usability evaluations whether certain usability problems would only be evident to patients whose condition closely resembles the target population, or if a broader group of patients would lead to the identification of a broader range of potential usability issues. Objective: This study aims to evaluate the CAPABLE prototype by conducting tests to assess usability, user experience, and perceived acceptability among end users, and to assess any agreements or differences in the results of our wide range of participants. Methods: This usability study was executed by interviewing participants with a melanoma or renal cell carcinoma diagnosis who have received immunotherapy and participants without direct experience with the targeted cancer types who have not received immunotherapy. Participants were asked to review the concept of the tool, perform think-aloud tasks, and complete the System Usability Scale and a Perceived Usefulness questionnaire. Usability problems were extracted from the interview data by independent coding and mapped to an eHealth Usability Problem Framework. Results: We included 21 participants in the study, aged 29 to 73 years; 13 participants who had received immunotherapy and 8 participants who had not received immunotherapy. In total, 76 usability problems were identified. A total of 22 usability problems were in the task-technology fit category of the usability framework, mostly regarding the coaching and symptom functionality of the prototype. Critical problems regarding the symptom monitoring functionality were mainly found by participants who had received immunotherapy. For 8 out of 10 statements in the Perceived Usefulness questionnaire, more than 75% of participants agreed or strongly agreed. The overall mean System Usability Scale score was 80 out of 100 (SD 11.3). Conclusions: Despite identified usability issues, participants responded positively to the Perceived Usefulness questionnaire regarding the evaluated tool. Further analysis of the usability problems indicates that it was essential to include participants who matched the target end users. Participants treated with immunotherapy, specifically with previous experience in immune-related adverse events, encountered critical problems with symptom reporting that would not have been identified if these participants were not included. For other tasks and functionalities, it seems likely that loosening the inclusion criteria would have resulted in sufficient feedback without critical missing usability issues.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.2196/57659

Cites

6 of 6 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Glaser et al [38] evaluated the CAPABLE prototype for immunotherapy patients, showing that involving target users improved perceived usefulness and revealed critical symptom-reporting issues that would have gone unnoticed without their participation.

    “Glaser et al [38] evaluated the CAPABLE prototype for immunotherapy patients, showing that involving target users improved perceived usefulness and revealed critical symptom–reporting issues that would have gone unnoticed without their participation.”

    From User-Centered Design for Digital Patient-Navigation Tools in Oncology: Scoping Review (2026) · cites this paper as [38]

    Split into 2 atoms — 0 confirmed, 2 partially supported, 0 disputed, 0 misattributed, 0 unverifiable. Expand the claim to see per-atom verdicts.

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 32

  1. [1]Larkin J Chiarion-Sileni V Gonzalez R Grob JJ Rutkowski P Lao CD Cowey CL Schadendorf D Wagstaff J Dummer R Ferrucci PF Smylie M Hogg D Hill A Márquez-Rodas I Haanen J Guidoboni M Maio M Schöffski P Carlino MS Lebbé C McArthur G Ascierto PA Daniels GA Long GV Bastholt L Rizzo JI Balogh A Moshyk A Hodi FS Wolchok JD Five-year survival with combined nivolumab and ipilimumab in advanced melanoma N Engl J Med 2019 381 16 1535 1546 10.1056/NEJMoa1910836 31562797 31562797 · DOI · Paper page
  2. [2]Robert C A decade of immune-checkpoint inhibitors in cancer therapy Nat Commun 2020 11 1 3801 10.1038/s41467-020-17670-y 32732879 10.1038/s41467-020-17670-y 32732879 PMC7393098 · DOI · Paper page
  3. [3]Hargadon KM Johnson CE Williams CJ Immune checkpoint blockade therapy for cancer: an overview of FDA-approved immune checkpoint inhibitors Int Immunopharmacol 2018 62 29 39 10.1016/j.intimp.2018.06.001 29990692 S1567-5769(18)30252-2 29990692 · DOI · Paper page
  4. [4]Shiravand Y Khodadadi F Kashani SMA Hosseini-Fard SR Hosseini S Sadeghirad H Ladwa R O'Byrne K Kulasinghe A Immune checkpoint inhibitors in cancer therapy Curr Oncol 2022 29 5 3044 3060 10.3390/curroncol29050247 35621637 curroncol29050247 35621637 PMC9139602 · DOI · Paper page
  5. [5]Rogiers A Boekhout A Schwarze J Awada G Blank C Neyns B Long-term survival, quality of life, and psychosocial outcomes in advanced melanoma patients treated with immune checkpoint inhibitors J Oncol 2019 2019 5269062 10.1155/2019/5269062 31182961 31182961 PMC6512024 · DOI
  6. [6]Slev VN Mistiaen P Pasman HRW Verdonck-de LIM van UCF Francke AL Effects of eHealth for patients and informal caregivers confronted with cancer: a meta-review Int J Med Inform 2016 87 54 67 10.1016/j.ijmedinf.2015.12.013 26806712 S1386-5056(15)30080-0 26806712 · DOI
  7. [7]Fraterman I Glaser S Wilgenhof S Medlock S Mallo H Cornet R van de Poll-Franse LV Boekhout AH Exploring supportive care and information needs through a proposed eHealth application among melanoma patients undergoing systemic therapy: a qualitative study Support Care Cancer 2022 30 9 7249 7260 10.1007/s00520-022-07133-z 35589878 10.1007/s00520-022-07133-z 35589878 PMC9119379 · DOI
  8. [8]Kotronoulas G Kearney N Maguire R Harrow A Di Domenico D Croy S MacGillivray S What is the value of the routine use of patient-reported outcome measures toward improvement of patient outcomes, processes of care, and health service outcomes in cancer care? A systematic review of controlled trials J Clin Oncol 2014 32 14 1480 501 10.1200/JCO.2013.53.5948 24711559 JCO.2013.53.5948 24711559 · DOI
  9. [9]Lubberding S van Uden-Kraan CF Te Velde EA Cuijpers P Leemans CR Verdonck-de Leeuw IM Improving access to supportive cancer care through an eHealth application: a qualitative needs assessment among cancer survivors J Clin Nurs 2015 24 9-10 1367 1379 10.1111/jocn.12753 25677218 25677218 · DOI
  10. [10]Iivanainen S Alanko T Vihinen P Konkola T Ekstrom J Virtanen H Koivunen J Follow-Up of cancer patients receiving anti-PD-(L)1 therapy using an electronic patient-reported outcomes tool (KISS): prospective feasibility cohort study JMIR Form Res 2020 4 10 e17898 10.2196/17898 33112242 v4i10e17898 33112242 PMC7657724 · DOI
  11. [11]Hægermark EA Kongshaug N Raj SX Hofsli E Faxvaag A Design, development, and evaluation of an mHealth app for reporting of side effects during cytostatic treatment: usability test and interview study JMIR Form Res 2023 7 e47374 10.2196/47374 37856183 v7i1e47374 37856183 PMC10623228 · DOI
  12. [13]Chen J Ou L Hollis SJ A systematic review of the impact of routine collection of patient reported outcome measures on patients, providers and health organisations in an oncologic setting BMC Health Serv Res 2013 13 211 10.1186/1472-6963-13-211 23758898 1472-6963-13-211 23758898 PMC3700832 · DOI
  13. [14]van Deursen L Versluis A van der Vaart R Standaar L Struijs J Chavannes N Aardoom JJ eHealth interventions for Dutch cancer care: systematic review using the triple aim lens JMIR Cancer 2022 8 2 e37093 10.2196/37093 35699991 v8i2e37093 35699991 PMC9240931 · DOI
  14. [16]Ardito V Golubev G Ciani O Tarricone R Evaluating barriers and facilitators to the uptake of mHealth apps in cancer care using the consolidated framework for implementation research: scoping literature review JMIR Cancer 2023 9 e42092 10.2196/42092 36995750 v9i1e42092 36995750 PMC10131717 · DOI
  15. [17]De Vito Dabbs A Myers BA Mc Curry KR Dunbar-Jacob J Hawkins RP Begey A Dew MA User-centered design and interactive health technologies for patients Comput Inform Nurs 2009 27 3 175 183 10.1097/NCN.0b013e31819f7c7c 19411947 00024665-200905000-00011 19411947 PMC2818536 · DOI · Paper page
  16. [18]Saparamadu AADNS Fernando P Zeng P Teo H Goh A Lee JMY Lam CWL User-centered design process of an mHealth app for health professionals: case study JMIR Mhealth Uhealth 2021 9 3 e18079 10.2196/18079 33769297 v9i3e18079 33769297 PMC8088861 · DOI
  17. [20]Jaspers MWM A comparison of usability methods for testing interactive health technologies: methodological aspects and empirical evidence Int J Med Inform 2009 78 5 340 353 10.1016/j.ijmedinf.2008.10.002 19046928 S1386-5056(08)00180-9 19046928 · DOI · Paper page
  18. [22]Neal D Engelsma T Tan J Craven MP Marcilly R Peute L Dening T Jaspers M Dröes RM Limitations of the new ISO standard for health and wellness apps Lancet Digital Health 2022 4 2 e80 e82 10.1016/S2589-7500(21)00273-9 35090677 S2589-7500(21)00273-9 35090677 · DOI
  19. [24]Fraterman I Wollersheim BM Tibollo V Glaser SLC Medlock S Cornet R Gabetta M Gisko V Barkan E di Flora N Glasspool D Kogan A Lanzola G Leizer R Mallo H Ottaviano M Peleg M van de Poll-Franse LV Veggiotti N Śniatała K Wilk S Parimbelli E Quaglini S Rizzo M Locati LD Boekhout A Sacchi L Wilgenhof S An eHealth app (CAPABLE) providing symptom monitoring, well-being interventions, and educational material for patients with melanoma treated with immune checkpoint inhibitors: protocol for an exploratory intervention trial JMIR Res Protoc 2023 12 e49252 10.2196/49252 37819691 v12i1e49252 37819691 PMC10600650 · DOI
  20. [25]Jaspers MWM Steen T van den Bos Cor Geenen M The think aloud method: a guide to user interface design Int J Med Inform 2004 73 11 781 795 10.1016/j.ijmedinf.2004.08.003 15491929 S1386-5056(04)00182-0 15491929 · DOI · Paper page
  21. [27]Lisowska A Wilk S Peleg M SATO (IDEAS expAnded wiTh BCIO): workflow for designers of patient-centered mobile health behaviour change intervention applications J Biomed Inform 2023 138 104276 10.1016/j.jbi.2022.104276 36586499 S1532-0464(22)00281-7 36586499 PMC9975785 · DOI
  22. [29]Nielsen J Estimating the number of subjects needed for a thinking aloud test Int J Hum-Comput Stud 1994 41 3 385 397 10.1006/ijhc.1994.1065 · DOI · Paper page
  23. [30]Bangor A Kortum PT Miller JT An empirical evaluation of the system usability scale Int J Hum-Comput Interact 2008 24 6 574 594 10.1080/10447310802205776 · DOI · Paper page
  24. [31]Davis FD Sloan School of Management A technology acceptance model for empirically testing new end-user information systems: theory and results [Thesis] 1985 2023-07-18 https://dspace.mit.edu/handle/1721.1/15192
  25. [32]Broekhuis M van Velsen L Peute L Halim M Hermens H Conceptualizing usability for the eHealth context: content analysis of usability problems of eHealth applications JMIR Form Res 2021 5 7 e18198 10.2196/18198 34313594 v5i7e18198 34313594 PMC8367108 · DOI
  26. [33]Peute LW Driest KF Marcilly R Bras Da Costa S Beuscart-Zephir M Jaspers MWM A framework for reporting on human factor/usability studies of health information technologies Stud Health Technol Inform 2013 194 54 60 23941930 23941930
  27. [35]Mirkovic J Kaufman DR Ruland CM Supporting cancer patients in illness management: usability evaluation of a mobile app JMIR Mhealth Uhealth 2014 2 3 e33 10.2196/mhealth.3359 25119490 v2i3e33 25119490 PMC4147703 · DOI
  28. [36]O'Malley DM Davis SN Devine KA Sullivan B Bator A Clemow L Ferrante JM Findley PA Miller SM Hudson SV Development and usability testing of the e-EXCELS tool to guide cancer survivorship follow-up care Psychooncology 2020 29 1 123 131 10.1002/pon.5222 31626397 31626397 PMC7279518 · DOI · Paper page
  29. [37]Duman-Lubberding S van Uden-Kraan CF Peek N Cuijpers P Leemans CR Verdonck-de Leeuw IM An eHealth application in head and neck cancer survivorship care: health care professionals' perspectives J Med Internet Res 2015 17 10 e235 10.2196/jmir.4870 26489918 v17i10e235 26489918 PMC4642398 · DOI
  30. [39]Ju B Does domain knowledge matter: mapping users' expertise to their information interactions J Am Soc Inf Sci 2007 58 13 2007 2020 10.1002/asi.20680 · DOI
  31. [42]Liu Y Osvalder A Karlsson M Holzinger A User's expertise differences when interacting with simple medical user interfaces HCI and Usability for Medicine and Health Care 2007 Berlin, Heidelberg Springer 441
  32. [44]Hertzum M Jacobsen NE The evaluator effect: a chilling fact about usability evaluation methods Int J Hum-Comput Interact 2003 15 1 183 204 10.1207/s15327590ijhc1501_14 · DOI

Cited by

Sorted by citations

  • User-Centered Design for Digital Patient-Navigation Tools in Oncology: Scoping Review

    Kheirinejad, S., White, B. M., Kheirkhah Rahimabad, P., Zink, J. A., Hashtarkhani, S., Kumsa, F. A., Rashid, R., Chinthala, L., Brett, C. L., Davis, R. L., et al. (2026) · JMIR Human Factors · doi:10.2196/87686

    “In line with these findings, Glaser et al [38] evaluated the CAPABLE prototype for immunotherapy patients, showing that involving target users improved perceived usefulness and revealed critical symptom–reporting issues that would have gone unnoticed without their participation.”

Authors
Glaser, S. L. C., Fraterman, I., van Brummelen, N., Tibollo, V., Del Campo, L. M., Mallo, H., Wilgenhof, S., Wilk, S., Gisko, V., Khadakou, V., Cornet, R., Ottaviano, M., & Medlock, S.
Affiliations
  • The Netherlands Cancer Institute (NL)
  • Istituti Clinici Scientifici Maugeri (IT)
  • Universidad Politécnica de Madrid (ES)
  • Amsterdam Public Health (NL)
Corresponding author
Medlock, Stephanie
ORCID
Venue
JMIR Formative Research
Publisher
JMIR Publications Inc.
Published
2025-01-23
Volume
9
Pages
e57659
ISSN
2561-326X (ISSN-L)
DOI
10.2196/57659
PMC
PMC11803325
OpenAlex
W4404333693
Cited by
5
References
6
Citations by year
2026: 1 · 2025: 2 · 2024: 2
FWCI
2.92
Type
journal article
Language
en
Open access
Licence
cc-by
Locations
  • JMIR Formative Research (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • Pure Amsterdam UMC (submittedVersion, cc-by)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
Funders
European Commission (HORIZON2020, 875052)
Topics
  • Health Sciences › Health Professions › General Health Professions › Mobile Health and mHealth Applications
  • Physical Sciences › Computer Science › Human-Computer Interaction › Usability and User Interface Design
  • Health Sciences › Medicine › Oncology › Cancer survivorship and care
Keywords
symptom management, immune checkpoint inhibitors, immunotherapy, melanoma, renal cell carcinoma, immune-related adverse events, symptom monitoring, user experience, perceived usefulness, usability problems, think-aloud, System Usability Scale, patient-reported outcomes
MeSH terms
Mentoring, Immune Checkpoint Inhibitors, Adult, Aged, Carcinoma, Renal Cell, Female, Humans, Kidney Neoplasms, Male, Melanoma, Middle Aged, Neoplasms, Surveys and Questionnaires, Telemedicine

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review