Development and Evaluation of the Usefulness, Usability, and Feasibility of iNNOV Breast Cancer: Mixed Methods Study
Open accessPublished version of a preprint
The DOI matches its Crossref record; 73 of the 74 references it cites were found in a registry or an index; 49 of the 81 sources its claims cite were checked for support.
Abstract
Background: Despite the efficacy of psychosocial interventions in minimizing psychosocial morbidity in breast cancer survivors (BCSs), intervention delivery across survivorship is limited by physical, organizational, and attitudinal barriers, which contribute to a mental health care treatment gap in cancer settings. Objective: The aim of this study is to develop iNNOV Breast Cancer (iNNOVBC), a guided, internet-delivered, individually tailored, acceptance and commitment therapy–influenced cognitive behavioral intervention program aiming to treat mild to moderate anxiety and depression in BCSs as well as to improve fatigue, insomnia, sexual dysfunction, and health-related quality of life in this group. This study also aims to evaluate the usefulness, usability, and preliminary feasibility of iNNOVBC. Methods: iNNOVBC was developed using a user-centered design approach involving its primary and secondary end users, that is, BCSs (11/24, 46%) and mental health professionals (13/24, 54%). We used mixed methods, namely in-depth semistructured interviews, laboratory-based usability tests, short-term field trials, and surveys, to assess iNNOVBC’s usefulness, usability, and preliminary feasibility among these target users. Descriptive statistics were used to characterize the study sample, evaluate performance data, and assess survey responses. Qualitative data were recorded, transcribed verbatim, and thematically analyzed. Results: Overall, participants considered iNNOVBC highly useful, with most participants reporting on the pertinence of its scope, the digital format, the relevant content, and the appropriate features. However, various usability issues were identified, and participants suggested that the program should be refined by simplifying navigation paths, using a more dynamic color scheme, including more icons and images, displaying information in different formats and versions, and developing smartphone and tablet versions. In addition, participants suggested that tables should be converted into plain textboxes and data visualization dashboards should be included to facilitate the tracking of progress. The possibility of using iNNOVBC in a flexible manner, tailoring it according to BCSs’ changing needs and along the cancer care continuum, was another suggestion that was identified. Conclusions: The study results suggest that iNNOVBC is considered useful by both BCSs and mental health professionals, configuring a promising point-of-need solution to bridge the psychological supportive care gap experienced by BCSs across the survivorship trajectory. We believe that our results may be applicable to other similar programs. However, to fulfill their full supportive role, such programs should be comprehensive, highly usable, and tailorable and must adopt a flexible yet integrated structure capable of evolving in accordance with survivors’ changing needs and the cancer continuum. © Cristina Mendes-Santos, Francisco Nunes, Elisabete Weiderpass, Rui Santana, Gerhard Andersson. Originally published in JMIR Cancer (https://cancer.jmir.org), 15.02.2022. This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Cancer, is properly cited. The complete bibliographic information, a link to the original publication on https://cancer.jmir.org/, as well as this copyright and license information must be included.
Overview
CoChat Summary
TL;DR
iNNOV Breast Cancer (iNNOVBC) is a guided, internet-delivered, individually tailored ACT-influenced CBT program for breast cancer survivors, developed and assessed with mixed methods. Participants found it highly useful, but usability problems and practical barriers suggest it needs refinement before broader implementation.
Key findings
- The study involved 24 participants: 11 breast cancer survivors (BCSs, 46%) and 13 mental health professionals (MHPs, 54%); among BCSs, the median age was 48 years, and among MHPs, the median age was 35 years.
- Usefulness was rated highly: in the field trial, modules were rated 5/5 stars by BCSs, and all survivors who rated modules (8/8, 100%) said they would recommend them to a friend or family member in the same situation.
- Usability scores were below average: the median System Usability Scale score was 65 (IQR 35) for BCSs and 47.5 (IQR 25) for MHPs; usability test issues were mainly inefficient interaction (12/43, 28%), ineffective information presentation (9/43, 21%), and cognitive overload (7/43, 16%).
- Feasibility was viewed as promising but challenging: participants said the program could fit into care, but BCSs worried about “finding the time to fit the program into the day-to-day life,” and MHPs raised interoperability, practical, and attitudinal concerns.
Methods
- Mixed-methods user-centered design study combining in-depth semistructured interviews, laboratory-based usability tests, short-term field trials, and surveys to evaluate usefulness, usability, and preliminary feasibility.
- Nonprobabilistic purposive sample recruited in Porto, Portugal: 24 participants total (11 BCSs and 13 MHPs); BCSs had completed primary adjuvant treatment except hormonal therapy, and MHPs were registered psychologists or psychiatrists.
- Quantitative data were summarized with descriptive statistics; qualitative interview and written-comment data were transcribed verbatim and analyzed thematically.
Limitations
- The paper states limitations of a small sample size and minimal diversity in educational level and experience using digital mental health programs.
- The authors also note that BCS participants were younger than the general Portuguese BCS population and were not screened for mild to moderate anxiety or depression, which may have limited generalizability; this is stated in the paper.
- Evident from the text: data collection ended after 13 of 15 MHPs and 11 of 16 BCSs completed the protocol, so not all who agreed participated through completion.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures



Verification
Claims the paper makes
28 Valid · 1 Not valid · 7 Not checked · 81 Own results and interpretations
Every source the paper cites for this claim was checked and supports it.
In the paper
“Another cause for the low uptake of internet interventions in clinical settings is the peripheral position end users are often referred to during development [38].”
§ Development and Evaluation of the Usefulness, Usability, and Feasibility of iNNOV Breast Cancer: Mixed Methods Study > Introduction > Background
What the sources say
- Supportedcites [38] source
The Introduction says eHealth technologies are often developed with only marginal engagement from end users. It links that lack of involvement to high attrition and says people stop using technologies that do not fit their daily lives, supporting the claim about low uptake. (LLM, conf=0.88)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Recent meta-analyses demonstrated the efficacy of such interventions in improving a range of psychosocial outcomes [13,14].”
§ Development and Evaluation of the Usefulness, Usability, and Feasibility of iNNOV Breast Cancer: Mixed Methods Study > Introduction > Background
What the sources say
- Supportedcites [13] source
The Abstract (p. 1) reports that this meta-analysis found improvements across several psychosocial outcomes following breast cancer surgery, including anxiety, depression, quality of life, and sleep disturbance. The Discussion (p. 11) likewise describes effects across a range of outcomes. This supports the claim as attributed to this source; the plural “meta-analyses” refers to the citing sentence’s two citations, not to this paper alone. (LLM, conf=0.94)
- Partly supportedcites [14] sourceExperimental — accuracy not yet measured
The Discussion reports improvements in quality of life and mood and reductions in anxiety and depression from cognitive behavioural and psycho-educational therapies. However, the Eligibility Criteria include women diagnosed with breast cancer at any stage and under any treatment; the analysis is not specific to breast cancer survivors as the claim frames it. (LLM, conf=0.82)
boostedstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“To guarantee optimal use of the program, the study intervention was developed according to the following persuasive system principles categorized by Oinas-Kukkonen and Harjumaa [53]: responsiveness, tunneling, tailoring, personalization, reminders, and professional support.”
§ Development and Evaluation of the Usefulness, Usability, and Feasibility of iNNOV Breast Cancer: Mixed Methods Study > Introduction > iNNOVBC Overview
What the sources say
- Partly supportedcites [53] sourceExperimental — accuracy not yet measured
Section IV lists tunneling, tailoring, and personalization as primary-task design principles and describes reminders as a dialogue-support principle. Section II discusses responsiveness, but calls it a general software quality rather than a persuasive-system principle; the passages found do not identify professional support as a principle. The source therefore supports much, but not all, of the attributed list. (LLM, conf=0.88)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“iNNOVBC was developed using a user-centered design approach involving its primary and secondary end users, that is, BCSs (11/24, 46%) and mental health professionals (13/24, 54%).”
- Also stated in: § iNNOVBC Overview · plain
“iNNOVBC (Figures 1 and 2) is a guided, internet-delivered, individually tailored, ACT-influenced CBT program and was developed with a user-centered design approach [37].”
§ Development and Evaluation of the Usefulness, Usability, and Feasibility of iNNOV Breast Cancer: Mixed Methods Study > Abstract > Methods
What the sources say
- Partly supportedcites [37] sourceExperimental — accuracy not yet measured
The “Identifying Users and User Needs” section describes user-centered design as identifying primary and secondary users and incorporating their perspectives during intervention development (page unavailable). It does not discuss iNNOVBC, breast cancer survivors, or the reported participant counts, so it supports the design approach but not its application to this intervention. (LLM, conf=0.85)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § iNNOVBC Overview · plain
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“The intervention structure and content build up on prior CBT- and ACT-inspired interventions developed by the Department of Behavioural Sciences and Learning at Linköping University, targeting other populations [41-45].”
§ Development and Evaluation of the Usefulness, Usability, and Feasibility of iNNOV Breast Cancer: Mixed Methods Study > Introduction > iNNOVBC Overview
What the sources say
- Not checked — no full text of the sourcecites [41] source
- Not checked — no full text of the sourcecites [42] source
- Not checked — no full text of the sourcecites [43] source
- Not checked — no full text of the sourcecites [44] source
- Partly supportedcites [45] sourceExperimental — accuracy not yet measured
The Methods (Intervention Procedure; page unavailable) describe a guided CBT intervention for patients with heart failure and depressive symptoms, so this paper can serve as an example of a prior CBT intervention targeting another population. Our copy does not establish that the citing paper’s intervention built on this program, that it was ACT-inspired, or that it was developed by the named department. (LLM, conf=0.79)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
27 more claims · 5 with their cited sources checked
Create a free account to see all 117 claims and to verify the 4 references still open.
- Probably not a substantial claim. 41
- It may state two separate claims as one. 65
- The quote only partly states this claim. 1
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“interventions such as iNNOVBC presented by Mendes-Santos et al [20] highlighted the underexplored potential of navigation tools to provide psychological support, positioning digital navigation to address care gaps in survivorship.”
From User-Centered Design for Digital Patient-Navigation Tools in Oncology: Scoping Review (2026) · cites this paper as [20]
Experimental — accuracy not yet measured
Citations
Cites
Using thematic analysis in psychology
Braun, V., & Clarke, V. (2006) · Qualitative Research in Psychology · doi:10.1191/1478088706qp063oa · Cited by 194,414
Sung, H., Ferlay, J., Siegel, R. L., Laversanne, M., Soerjomataram, I., Jemal, A., & Bray, F. (2021) · CA: A Cancer Journal for Clinicians · doi:10.3322/caac.21660 · Cited by 108,699 · cited 2×
“In Portugal, the 5-year prevalence of breast cancer was estimated at 27,051 in 2020, making breast cancer survivors (BCSs) the largest group of cancer survivors in the country [1,2].”
BackgroundCode Saturation Versus Meaning Saturation: How Many Interviews Are Enough?
Hennink, M. M., Kaiser, B. N., & Marconi, V. C. (2017) · Qualitative Health Research · doi:10.1177/1049732316665344 · Cited by 4,597 · cited 1×
“Meaning saturation was used as a stopping criterion, which meant that new participants would not be enrolled once novel fieldwork insights stopped changing analysis significantly [55].”
A mathematical model of the finding of usability problems
Nielsen, J., & Landauer, T. K. (1993) · Proceedings of the SIGCHI conference on Human factors in computing systems - CHI '93 · doi:10.1145/169059.169166 · Cited by 1,961 · cited 1×
“Nevertheless, there is some consensus that usability tests may include a minimum of 5 participants per iteration and that approximately 80% of usability issues are discovered with as few as 4-6 participants [80].”
BackgroundPersuasive Systems Design: Key Issues, Process Model, and System Features
Oinas-Kukkonen, H., & Harjumaa, M. (2009) · Communications of the Association for Information Systems · doi:10.17705/1cais.02428 · Cited by 1,663 · cited 1×
“To guarantee optimal use of the program, the study intervention was developed according to the following persuasive system principles categorized by Oinas-Kukkonen and Harjumaa [53]: responsiveness, tunneling, tailoring, personalization, reminders, and professional support.”
MethodAdvantages and limitations of Internet-based interventions for common mental disorders
Andersson, G., & Titov, N. (2014) · World Psychiatry · doi:10.1002/wps.20083 · Cited by 1,160 · cited 1×
“Internet interventions—self-help technology-enabled interventions that provide synchronous or asynchronous health-related and mental health–related assistance based on established psychotherapy models [20]—provide an opportunity to fulfill the mental health care gap within oncology and offer BCSs with patient-centered…”
BackgroundSelf-citationA Holistic Framework to Improve the Uptake and Impact of eHealth Technologies
van Gemert-Pijnen, J. E., Nijland, N., van Limburg, M., Ossebaard, H. C., Kelders, S. M., Eysenbach, G., & Seydel, E. R. (2011) · Journal of Medical Internet Research · doi:10.2196/jmir.1672 · Cited by 1,126 · cited 2×
“Intervention programs are frequently planned by neglecting end users’ perspective (eg, individual’s goals, needs, skills, and contexts) and researchers often fail to involve end users in the development process [37,38].”
BackgroundRunowicz, C. D., Leach, C. R., Henry, N. L., Henry, K. S., Mackey, H. T., Cowens-Alvarado, R. L., Cannady, R. S., Pratt-Chapman, M. L., Edge, S. B., Jacobs, L. A., et al. (2016) · Journal of Clinical Oncology · doi:10.1200/jco.2015.64.3809 · Cited by 1,071
Seasons of Survival: Reflections of a Physician with Cancer
Mullan, F. (1985) · New England Journal of Medicine · doi:10.1056/nejm198507253130421 · Cited by 810 · cited 1×
“Considering the survivorship status [61], the sample was heterogeneous, including participants at acute (2/11, 18%), extended (3/11, 27%), and permanent (6/11, 55%) survivorship stages.”
BackgroundAcceptance and Commitment Therapy as a Unified Model of Behavior Change
Hayes, S. C., Pistorello, J., & Levin, M. E. (2012) · The Counseling Psychologist · doi:10.1177/0011000012460836 · Cited by 718 · cited 1×
“Additional content was developed based on peer-reviewed sources [46-50], as well as mixed methods research conducted by the research team and involving iNNOVBC’s primary and secondary end users [51,52].”
Method
Research works printed but not linked: 2
- Kim, AR, 2015, Stud Health Technol Inform, 216, 142
- Short, CE, 2015, Eur Health Psychol, 17, 32
Cited by
Examining Challenges to Co-Design Digital Health Interventions With End Users: Systematic Review
Duffy, A., Boroumandzad, N., Sherman, A. L., Christie, G., Riadi, I., & Moreno, S. (2024) · Journal of Medical Internet Research · doi:10.2196/50178 · Cited by 82
“…and behavioral objectives), leading toward more transdisciplinary approaches in DHI design Theme: Pragmatic hybridized frameworks Subthemes 24 (40) Despite the advantages that mixed methods approaches bring to DHIs (ie, blending emotional and functional feedback for better design decisions),…”
Wu, W., Graziano, T. A., Salner, A., Chen, M., Judge, M. P., Cong, X. S., & Xu, W. (2024) · JMIR mhealth and uhealth · doi:10.2196/53652 · Cited by 38
“For instance, a 2-way text message strategy has been shown to be effective in improving engagement and adherence in the survivorship management program, although it may require the research team to commit more effort to respond to text messages personally [74,75].”
BackgroundAnders, C., Moorthy, P., Svensson, L., Müller, J., Heinze, O., Knaup, P., Wallwiener, M., Deutsch, T. M., Le, T.-V., & Weinert, L. (2024) · JMIR Human Factors · doi:10.2196/50926 · Cited by 17
“Relevant usability themes of similar apps were, for example, streamlining of the navigation paths, a clearer information architecture, or the desire for personalization [15,16].”
BackgroundBäuerle, A., Martus, P., Erim, Y., Schug, C., Heinen, J., Krakowczyk, J. B., Steinbach, J., Damerau, M., Bethge, W., Dinkel, A., et al. (2022) · BMJ Open · doi:10.1136/bmjopen-2021-056973 · Cited by 12
Weirauch, V., Soehnchen, C., Burmann, A., & Meister, S. (2024) · Journal of Medical Internet Research · doi:10.2196/55714 · Cited by 11
Metadata
- Authors
- Mendes-Santos, C., Nunes, F., Weiderpass, E., Santana, R., & Andersson, G.
- Affiliations
- Linköping University (SE)
- Fraunhofer Portugal Research (PT)
- Centre international de recherche sur le cancer (FR)
- Comprehensive Health Research Centre (PT)
- Corresponding author
- Mendes-Santos, Cristina
- Venue
- JMIR Cancer
- Publisher
- JMIR Publications Inc.
- Published
- 2022-02-15
- Volume
- 8
- Issue
- 1
- Pages
- e33550
- ISSN
- 2369-1999 (ISSN-L)
- PMC
- PMC8889471
- OpenAlex
- W4205179018
- Cited by
- 19
- References
- 74
- Citations by year
- 2026: 4 · 2025: 6 · 2024: 6 · 2023: 2 · 2022: 1
- FWCI
- 1.57
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- Locations
- JMIR Cancer (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Portuguese National Funding Agency for Science, Research and Technology (RCAAP Project by FCT) (submittedVersion, cc-by)
- DiVA (Linkoping University) (submittedVersion, other-oa)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
- Fraunhofer-Publica (Fraunhofer-Gesellschaft) (submittedVersion)
- Funders
- World Health Organization
- Centre International de Recherche sur le Cancer
- European Commission (2020 ‘)
- Linköpings Universitet
- Karolinska Institutet
- Erasmus+
- Topics
- Health Sciences › Medicine › Oncology › Cancer survivorship and care
- Social Sciences › Psychology › Applied Psychology › Digital Mental Health Interventions
- Health Sciences › Health Professions › General Health Professions › Mobile Health and mHealth Applications
- Keywords
- breast cancer survivors, online intervention, acceptance and commitment therapy, cognitive behavioral therapy, anxiety, depression, fatigue, insomnia, sexual dysfunction, health-related quality of life, usability evaluation, user-centered design, mental health treatment gap, psychosocial interventions, intervention development, feasibility, mental health professionals
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-11, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-11 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.