Methods for Human-Centered eHealth Development: Narrative Scoping Review
Open accessPublished version of a preprint
The DOI matches its Crossref record; 187 of the 187 references it cites were found in a registry or an index; 45 of the 145 sources its claims cite were checked for support.
Abstract
Background: Thorough holistic development of eHealth can contribute to a good fit among the technology, its users, and the context. However, despite the availability of frameworks, not much is known about specific research activities for different aims, phases, and settings. This results in researchers having to reinvent the wheel. Consequently, there is a need to synthesize existing knowledge on research activities for participatory eHealth development processes. Objective: The 3 main goals of this review are to create an overview of the development strategies used in studies based on the CeHRes (Center for eHealth Research) Roadmap, create an overview of the goals for which these methods can be used, and provide insight into the lessons learned about these methods. Methods: We included eHealth development studies that were based on the phases and/or principles of the CeHRes Roadmap. This framework was selected because of its focus on participatory, iterative eHealth design in context and to limit the scope of this review. Data were extracted about the type of strategy used, rationale for using the strategy, research questions, and reported information on lessons learned. The most frequently mentioned lessons learned were summarized using a narrative, inductive approach. Results: In the included 160 papers, a distinction was made between overarching development methods (n=10) and products (n=7). Methods are used to gather new data, whereas products can be used to synthesize previously collected data and support the collection of new data. The identified methods were focus groups, interviews, questionnaires, usability tests, literature studies, desk research, log data analyses, card sorting, Delphi studies, and experience sampling. The identified products were prototypes, requirements, stakeholder maps, values, behavior change strategies, personas, and business models. Examples of how these methods and products were applied in the development process and information about lessons learned were provided. Conclusions: This study shows that there is a plethora of methods and products that can be used at different points in the development process and in different settings. To do justice to the complexity of eHealth development, it seems that multiple strategies should be combined. In addition, we found no evidence for an optimal single step-by-step approach to develop eHealth. Rather, researchers need to select the most suitable research methods for their research objectives, the context in which data are collected, and the characteristics of the participants. This study serves as a first step toward creating a toolkit to support researchers in applying the CeHRes Roadmap to practice. In this way, they can shape the most suitable and efficient eHealth development process.
Overview
CoChat Summary
TL;DR
This narrative scoping review synthesized 160 eHealth development studies based on the CeHRes Roadmap, identifying 10 overarching methods and 7 products used across contextual inquiry, value specification, and design phases. It concludes there is no single optimal step-by-step approach; researchers should combine multiple strategies selected for the research objective, context, and participant characteristics.
Key findings
- In 160 included papers, the review identified 10 methods (focus groups, interviews, questionnaires, usability tests, literature studies, desk research, log data analyses, card sorting, Delphi studies, experience sampling) and 7 products (prototypes, requirements, stakeholder maps, values, behavior change strategies, personas, business models).
- Most development studies used a multi-method approach (90/160, 56.3%); web-based interventions were the most common technology (74/160, 46.3%), followed by mobile apps (34/160, 21.3%), VR (3/160, 1.9%), and no clear technology description (27/160, 16.9%).
- Most frequently reported methods were focus groups (71/160, 44.4%), interviews (70/160, 43.8%), questionnaires (51/160, 31.9%), usability tests (51/160, 31.9%), and literature studies (43/160, 26.9%); most frequently reported products were prototypes (32/160, 20%), requirements (11/160, 6.9%), stakeholder maps (10/160, 6.3%), and values (7/160, 4.4%).
- Lessons learned emphasized that methods and products should fit the context, participant characteristics and skills, and earlier findings; multiple methods should be combined; and there was no evidence for an optimal single step-by-step approach. Experience sampling (1/160, 0.6%), Delphi studies (3/160, 1.9%), personas (5/160, 3.1%), and behavior change strategies (5/160, 3.1%) were underrepresented.
Methods
- Narrative scoping review of eHealth development studies based on the CeHRes Roadmap phases and/or principles; inclusion required reference to the original 2011 CeHRes paper in the introduction or methods and use of ≥1 phase or principle, while implementation/summative-evaluation-only studies, abstract guideline papers, non-peer-reviewed records, and records not in English, Dutch, German, or Portuguese were excluded.
- Searched Scopus, Google Scholar, and Web of Science up to June 2021, plus snowball sampling for records coauthored by the CeHRes Roadmap founder; 1713 unique records were screened, 377 remained after title/abstract screening, and 160 were included after full-text screening.
- Data were extracted by 3 researchers on strategy type, rationale, research questions, and lessons learned; methods and products were categorized by consensus, and lessons learned were summarized using a narrative, inductive approach. Phases were deduced by the authors when not explicitly reported.
Limitations
- Stated: focusing only on the CeHRes Roadmap means other suitable methods or products may have been overlooked, so the overview is neither exhaustive nor unconditionally generalizable; lessons learned are not exhaustive; no quality assessment of included studies was performed; and the qualitative approach is prone to researcher bias, though screening and summarizing were systematic and conducted by 3 researchers with coauthor discussion.
- Stated: not all relevant development information may have been published, and reporting clarity and extensiveness varied across studies; suboptimal or incomplete reflection by authors resulted in some lessons learned remaining general.
- Evident from the text: CeHRes phases were explicitly mentioned in only 30% (48/160) of studies, so phase categorization was often deduced by the authors, which may introduce misclassification; products also appeared less clearly reported than methods.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures
Verification
Claims the paper makes
22 Valid · 0 Not valid · 55 Not checked · 111 Own results and interpretations
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“First, multiple authors indicated that usability tests should be conducted with a broad range of stakeholders: not only end users such as patients, but also caregivers, managers, technology developers, and experts on content and design”
§ Methods for Human-Centered eHealth Development: Narrative Scoping Review > Results > Definition, Applicability, and Lessons Learned Per Method
What the sources say
- Not checked yetcites [21] Van Gemert-Pijnen, L, 2016, Handbook of Smart Homes, Health Care and Well-Being
- Partly supportedcites [43] sourceExperimental — accuracy not yet measured
Our copy supports only the descriptive half of the claim: the paper's own usability testing (Results, Part 4) involved people beyond patients — a testicular cancer survivor, a nurse, a cancer care assistant, a spouse of a cancer survivor with fatigue, and 2 health-psychology PhD students — and its development consulted content/design specialists (Results, Part 3: 'Specialists in the development and evaluation of Web-based behavior change interventions... expertise in the design and implementation of interventions built using LifeGuide'; 'the views of stakeholders, such as health care staff'). The Discussion (Principal Findings) endorses 'cocreation of the intervention with the experts and service users'. What the paper does NOT say is the normative, cross-study assertion attributed to it: it is a single intervention-development paper, not a review of 'included studies', and no passage states that usability tests SHOULD be conducted with a broad range of stakeholders including managers and technology developers (searched: stakeholders, caregivers, managers, technology developers, recommendations for usability testing). (LLM, conf=0.68)
- Not checked yetcites [54] source
- Partly supportedcites [64] sourceExperimental — accuracy not yet measured
The Discussion (p. n/a, 'This mixed method approach…') states the paper's lesson that they 'focused on diverse stakeholders to catch different perspectives' and 'involved end-users and stakeholders in the whole analysis through multiple and small tests', and the Methods/Results (Characteristics of participants, Table 1; Implementation) show those stakeholders were healthcare managers, clinicians/nurses, plus software developers, biomedical engineers and graphic designers acting as HCI/content-design experts — so the managers/technology-developers/design-experts part of the claim is supported. The patients-and-caregivers part is not: the paper's end users were healthcare professionals only, and the Results (Implementation) state the patient-facing use case UC1.3 'directly involves patients, and it is not included in this work'; caregivers are never mentioned. Searched: 'usability tests stakeholders patients/caregivers', 'patients and caregivers involvement', 'human-centered design stakeholders'. (LLM, conf=0.72)
- Not checked yetcites [77] Verhoeven, F, 2010, HCI in Work and Learning, Life and Leisure
- Partly supportedcites [112] sourceExperimental — accuracy not yet measured
Our copy supports the multi-stakeholder part of the claim at the level of intervention development, not usability testing: the Development team section (p. 6) states that 'different perspectives were considered important' and lists researchers, technicians with software development and design expertise, ten health professionals and five patients, and the Discussion (p. 11) reflects on collaboration of researchers, health professionals, patients and technicians. Phase 3 (p. 7) adds that usability findings were discussed with the development team and translated with two software engineers. However, the paper's actual usability tests were run with three outpatients only (p. 7, p. 9), it makes no recommendation that usability tests include caregivers or managers (neither group appears), and it reports its own co-creative practice rather than stating a normative point credited to 'multiple authors'. (LLM, conf=0.72)
- Partly supportedcites [127] sourceExperimental — accuracy not yet measured
Our copy of Derks et al. (2019) supports the core idea that usability testing should extend beyond end users: the Discussion Conclusions state that 'the involvement of multiple user groups proved to be of added value during design and realization, as evidenced by the complementary requirements and perspectives,' and the Abstract Methods/Results describe three cycles with patients, therapists (professional caregivers) and user-centered design experts, with the Limitations and Strengths section noting 'the use of the multiple user groups is of added value during design and realization.' However, the paper's groups are only patients, caregivers/therapists and UCD (content/design) experts; it nowhere recommends including managers or technology developers in usability tests, and it frames multi-group testing as its own demonstrated practice rather than a general normative statement by 'multiple authors.' The stakeholder list in the claim (managers, technology developers) is therefore not supported by this source. (LLM, conf=0.72)
- Partly supportedcites [131] sourceExperimental — accuracy not yet measured
Split into 2 atoms — 0 confirmed, 2 partially supported, 0 disputed, 0 misattributed, 0 unverifiable. Expand the claim to see per-atom verdicts.
- Not checked yetcites [152] source
- Not checked yetcites [155] source
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Every source the paper cites for this claim was checked and supports it.
In the paper
“First, the included studies showed that there is not 1 single way to create a prototype.”
§ Methods for Human-Centered eHealth Development: Narrative Scoping Review > Results > Definition, Applicability, and Lessons Learned Per Product
What the sources say
- Supportedcites [40] source
The Methods (Overview; page unavailable) describe prototypes ranging from rough paper sketches to interactive tablet mock-ups. The Methods (Prototype and testing of PRO dashboard; page unavailable) also describe six iterations of a paper prototype shaped by feedback. This study supports the citing authors’ point that prototyping need not follow one fixed form. (LLM, conf=0.86)
- Supportedcites [55] source
The Results > Design section (page unavailable) describes four preliminary sketch concepts, two of which became distinct digital prototypes: a list-based version and a pie-chart version. This study therefore provides an example of more than one approach to creating a prototype, consistent with the citing sentence’s synthesis of the included studies. (LLM, conf=0.88)
- Supportedcites [121] source
The Background (page unavailable) lists several frameworks for developing web-based interventions and says this study used scrum. The Discussion, ‘Lessons learned & study limitations’ (page unavailable), considers IDEAS and traditional methods as alternatives. Together, these passages support the point that there is no single way to develop a prototype. (LLM, conf=0.88)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“Second, it can be challenging to recruit participants for a Delphi study because, for example, there are not many experts in a specific or new field, which can be especially relevant when studying novel applications of an eHealth technology”
§ Methods for Human-Centered eHealth Development: Narrative Scoping Review > Results > Definition, Applicability, and Lessons Learned Per Method
What the sources say
- Supportedcites [101] source
The Discussion (p. of passage [1]) states directly that 'It proved to be difficult to recruit participants for such an explorative study, since not many therapists have experience with online treatment, and therapists do not receive extra time to take part in research from their management,' which is exactly the recruitment-difficulty-because-few-experts point the claim attributes to this Delphi study. The surrounding Discussion (passage [2]) reinforces the novelty qualifier: 'online therapy is still scarce in clinical practice,' i.e. a novel eHealth application. The citing sentence adds a second reason (management time) only implicitly, but the core attribution is present in our copy. (LLM, conf=0.85)
plainstates a possible cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“First, in the included papers it was shown that stakeholder maps are not created from scratch but are based on data.”
§ Methods for Human-Centered eHealth Development: Narrative Scoping Review > Results > Definition, Applicability, and Lessons Learned Per Product
What the sources say
- Supportedcites [27] source
In Methods (Stakeholder Identification Method Number 3: Snowball Sampling With Stakeholders; page unavailable), the authors say they developed a stakeholder map from an initial list, then used questionnaire responses to revise and expand it. Their account supports the claim that the map was grounded in gathered information rather than created from scratch. (LLM, conf=0.90)
- Supportedcites [129] source
The Methods section (Key stakeholder identification; page not provided) says the initial stakeholder list was built from a literature scan rather than from scratch. It also says an online survey was used to identify key stakeholders from that list, supporting the claim that the stakeholder identification was based on data. (LLM, conf=0.87)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Second, it might not be possible to include all requirements in an eHealth technology because of practical, technical, or financial limitations.”
§ Methods for Human-Centered eHealth Development: Narrative Scoping Review > Results > Definition, Applicability, and Lessons Learned Per Product
What the sources say
- Supportedcites [38] source
The Methods section says that selecting requirements for bWell depended on technical feasibility, context of use, and available finance and time. The Discussion reports that financial limitations prevented the team from including all identified user needs and requirements. Our copy does not provide page numbers for these passages. (LLM, conf=0.91)
- Could not verifycites [39] sourceExperimental — accuracy not yet measured
The Results (Phase 2: Findings; page unavailable) say the team evaluated system requirements against usability feedback and addressed identified changes. Neither that passage nor the searched passages state that practical, technical, or financial limitations can prevent an eHealth technology from including all requirements (searched for those limitations, feasibility, resources, and unimplemented requirements). (LLM, conf=0.72)
hedgedstates a possible cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
81 more claims · 3 with their cited sources checked
Create a free account to see all 188 claims and to verify the 100 references still open.
- It may state two separate claims as one. 40
- Probably not a substantial claim. 79
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Methods for Human-Centered eHealth Development: Narrative Scoping Review > Results > Overview of Methods and Products Per Phase of the CeHRes Roadmap
“| | Usability test (n=51) | 0 (0) | 0 (0) | 1 (2) | 1 (2) | 67 (131) | 67 (131) |”
§ Methods for Human-Centered eHealth Development: Narrative Scoping Review > Results > Definition, Applicability, and Lessons Learned Per Method
“Of the 160 included studies, 51 (31.9%) reported on usability tests.”
Citations
Cites
Preferred Reporting Items for Systematic Reviews and Meta-Analyses: the PRISMA statement
Moher, D., Liberati, A., Tetzlaff, J., Altman, D. G., & for the PRISMA Group (2009) · BMJ · doi:10.1136/bmj.b2535 · Cited by 83,823
Preferred Reporting Items for Systematic Reviews and Meta-Analyses: The PRISMA Statement
Moher, D., Liberati, A., Tetzlaff, J., Altman, D. G., & The PRISMA Group (2009) · PLoS Medicine · doi:10.1371/journal.pmed.1000097 · Cited by 64,042
Wang, G., Ma, R., Qiao, G., Wada, K., Aizawa, Y., & Satoh, T. (2015) · Integrative Medicine International · doi:10.1159/000368408 · Cited by 18,657
A typology of reviews: an analysis of 14 review types and associated methodologies
Grant, M. J., & Booth, A. (2009) · Health Information & Libraries Journal · doi:10.1111/j.1471-1842.2009.00848.x · Cited by 10,817
Michie, S., Richardson, M., Johnston, M., Abraham, C., Francis, J., Hardeman, W., Eccles, M. P., Cane, J., & Wood, C. E. (2013) · Annals of Behavioral Medicine · doi:10.1007/s12160-013-9486-6 · Cited by 7,629
Webb, T. L., Joseph, J., Yardley, L., & Michie, S. (2010) · Journal of Medical Internet Research · doi:10.2196/jmir.1376 · Cited by 2,517 · cited 1×
“This is quite surprising, especially considering the relationship between behavior change theory and effectiveness of (eHealth) interventions [184,185].”
Eysenbach, G. (2001) · Journal of Medical Internet Research · doi:10.2196/jmir.3.2.e20 · Cited by 2,449
CONSORT 2010 Statement: updated guidelines for reporting parallel group randomised trials
the CONSORT Group, Schulz, K. F., Altman, D. G., & Moher, D. (2010) · Trials · doi:10.1186/1745-6215-11-32 · Cited by 1,871
Planning Health Promotion Programs: An Intervention Mapping Approach
Hicking-Woodison, L. (2017) · Nursing Standard · doi:10.7748/ns.32.6.33.s38 · Cited by 1,841
Persuasive Systems Design: Key Issues, Process Model, and System Features
Oinas-Kukkonen, H., & Harjumaa, M. (2009) · Communications of the Association for Information Systems · doi:10.17705/1cais.02428 · Cited by 1,663
Research works printed but not linked: 21
- Van DykLWentzelMLimburg, vanMGemert-Pijnen, vanJEWCSchutteCSBusiness models for sustained ehealth implementation: Lessons from two continentsProceedings of International Conference on Computers and Industrial Engineering2012Proceedings of International Conference on Computers and Industrial EngineeringJul 16-18, 2012Cape Town, South Africa
- KulykOVan Gemert-PijnenLDaas DenCDavidSHow persuasive are serious games, social media and mHealth technologies for vulnerable young adults? Design factors for health behavior and lifestyle change support: sexual health caseProceedings of the 3rd International Workshop on Behavior Change Support Systems, BCSS 2015 - co-located with the 10th International Conference on Persuasive Technology, PERSUASIVE 201520153rd International Workshop on Behavior Change Support Systems, BCSS 2015 - co-located with the 10th International Conference on Persuasive Technology, PERSUASIVE 2015Jun 3, 2015Chicago, United States
- Kulyk, O, 2014, Advances Life Sci, 6, 300
- Beerlage-de Jong, N, 2014, Int J Advances Life Sci, 6, 262
- LimburgMOssebaardHWentzelMvan Gemert-PijnenJCo-creating with stakeholders: ideating eHealth applications to support antibiotic stewardship in hospitalsProceedings of the eTELEMED 2016: The Eighth International Conference on eHealth, Telemedicine, and Social Medicine 20162016eTELEMED 2016: The Eighth International Conference on eHealth, Telemedicine, and Social Medicine 2016Apr 24 - 28, 2016Venice, Italy
- McGrath, SP, 2017, AMIA Annu Symp Proc, 2017, 1282
- SpanMHettingaMSmitsCGroen-van de VenLJukemaJVernooij-DassenMEefstingJDeveloping a supportive tool to facilitate shared decision making in dementiaProceedings of the The Sixth International Conference on eHealth, Telemedicine, and Social Medicine2014The Sixth International Conference on eHealth, Telemedicine, and Social MedicineMar 23-27, 2014Barcelona, Spain
- Span, M, 2014, Int J Adv Life Sci, 6, 338
- Imhemed, S, 2017, J Eng Applied Sci, 12, 559
- Verhoeven, F, 2008, Hygiene Medizin, 33, 412
- Cruz-MartínezRSieverinkFWesselinkRvan Gemert-PijnenJTowards data-driven persuasive coaching in a heart failure telemonitoring technologyProceedings of the Sixth International Workshop on Behavior Change Support Systems (BCSS 2018). In conjunction with the 13th International Conference on Persuasive Technology2018Sixth International Workshop on Behavior Change Support Systems (BCSS 2018). In conjunction with the 13th International Conference on Persuasive TechnologyApr 16-19, 2018University of Waterloo, Canada
- Nobakht, Z, 2018, Iran J Child Neurol, 12, 65
- Span, M, 2014, Int J Adv Life Sci, 6, 107
- Ihmhemed, SM, 2017, J Advanced Sci Eng Res, 7, 27
- Gude, WT, 2012, Stud Health Technol Inform, 180, 1209
- Walters, LE, 2016, Stud Health Technol Inform, 231, 143
- OrganDDickSHurleyCHeavinCLinehanCDockraySDavorenMBryneMA systematic review of user-centred design practices in illicit substance use interventions for higher education studentsProceedings of the European Conference on Information Systems 2018: Beyond Digitization-Facets of Socio-Technical Change2018European Conference on Information Systems 2018: Beyond Digitization-Facets of Socio-Technical ChangeJun 23 - 28, 2018Portsmouth, UK
- WentzelMKarremanJvanGJvan VanGemert-PijneJEOssebaardHHamalainenPTowards an internet-based infectious disease management platform to increase patient safetyProceedings of the Third International Conference on eHealth, Telemedicine, and Social Medicine2011Third International Conference on eHealth, Telemedicine, and Social MedicineFeb 23-28, 2011Guadeloupe, France
- TschimmelKDesign thinking as an effective toolkit for innovationProceedings of the XXIII ISPIM Conference: Action for Innovation: Innovating from Experience2012XXIII ISPIM Conference: Action for Innovation: Innovating from ExperienceJun 17-20, 2012Barcelona, Spain
- Kip, H, 2018, eHealth Research, Theory and Development: A Multi-Disciplinary Approach
- 2014, Guidelines for Reporting Health Research: A User's Manual
Cited by
An umbrella review on how digital health intervention co-design is conducted and described
Kilfoy, A., Hsu, T. C., Stockton-Powdrell, C., Whelan, P., Chu, C. H., & Jibb, L. (2024) · npj Digital Medicine · doi:10.1038/s41746-024-01385-1 · Cited by 128
“Nine reviews 16,18,19,22,25,28,30,33,20 reported on challenges to co-design of digital health interventions.”
InfluentialBienefeld, N., Keller, E., & Grote, G. (2024) · Journal of Medical Internet Research · doi:10.2196/50852 · Cited by 88
Kip, H., Jong, N. B.-D., van Gemert‐Pijnen, J. E. W. C., & Kelders, S. M. (2024) · Journal of Medical Internet Research · doi:10.2196/59601 · Cited by 62
“Over the past years, the original CeHRes Roadmap has been applied to a broad range of contexts and has proven to be a useful tool for researchers and practitioners in the field of eHealth (refer to the study by Kip et al [13] for a partial overview).”
InfluentialSelf-citationRossetto, F., Borgnis, F., Isernia, S., Foglia, E., Garagiola, E., Realdon, O., & Baglio, F. (2023) · Frontiers in Public Health · doi:10.3389/fpubh.2023.1154481 · Cited by 39
Asbjørnsen, R. A., Hjelmesaeth, J. S., Smedsrød, M. L., Wentzel, J., Ollivier, M., Clark, M. M., van Gemert‐Pijnen, J. E. W. C., & Nes, L. S. (2022) · JMIR Human Factors · doi:10.2196/37372 · Cited by 33
Austin, J., Drossaert, C. H. C., van Dijk, J., Sanderman, R., Børøsund, E., Mirković, J., Schotanus-Dijkstra, M., Peeters, N. J., Klooster, J. W. J. R. V. '., Schroevers, M. J., et al. (2022) · JMIR Cancer · doi:10.2196/37502 · Cited by 25
“Indeed, in a recent scoping review of methods used in eHealth development, Kip et al [61] found that very few studies reported”
MethodLangerak, A. J., Regterschot, G. R. H., Selles, R. W., Meskers, C. G. M., Evers, M., Ribbers, G. M., van Beijnum, B.-J. F., & Bussmann, J. B. J. (2023) · Disability and Rehabilitation Assistive Technology · doi:10.1080/17483107.2023.2183993 · Cited by 20
“Ideally, multiple data sources should be combined, such as scientific literature and qualitative data collected from various stakeholders [30].”
BackgroundHouwen, T., Vugts, M. A. P., Lansink, K. W. W., Theeuwes, H. P., Neequaye, N., Beerekamp, M. S. H., Joosen, M. C. W., & de Jongh, M. A. C. (2022) · JMIR Human Factors · doi:10.2196/35342 · Cited by 17
“Although the CeHRes road map has been used in this study, numerous other frameworks are available to develop eHealth or mHealth initiatives [41].”
BackgroundMethodInfluentialWeimar, S. N., Martjan, R. S., & Terzidis, O. (2024) · Management Review Quarterly · doi:10.1007/s11301-024-00417-0 · Cited by 31
“Various toolkits and methods have been created to assist with the practical application of these methods, providing more detailed instructions for the different design phases of digital health products (Hughes et al. 2021; Kayser et al. 2022; Kip et al. 2022; Marvel et al. 2018).”
Haneveld, L. K., Kip, H., Bouman, Y. H. A., Weerdmeester, J., Scholten, H., & Kelders, S. M. (2023) · Frontiers in Psychology · doi:10.3389/fpsyg.2023.1201485 · Cited by 18
Self-citationKang, B., Poddar, M., Luitel, A., Rimal, R. N., Ayalew, B. M., & Black, D. P. (2025) · Global Health Science and Practice · doi:10.9745/ghsp-d-24-00164 · Cited by 26
“30 The gap also stems from health programs often relying on certain elements of HCD rather than adopting a comprehensive process.”
Tran, T. A., Mattap, S. M., Warren, N., Teng, J. H. J., Đức, D. M., Minh, H. V., Yasin, S., & Mohan, D. (2025) · Aging & Mental Health · doi:10.1080/13607863.2025.2462110 · Cited by 20
“In terms of health-related factors , the presence of behavior and psychological symptoms in dementia (BPSD) (Adana et al., 2022; Kamalzadeh et al., 2022; Kip et al., 2022; S. Liu et al., 2017, 2018 were associated with higher levels of caregiver burden.”
Arigo, D., Lassiter, J. M., Baga, K., Jackson, D. A., Lobo, A. F., & Guetterman, T. C. (2023) · Digital Health · doi:10.1177/20552076231210654 · Cited by 17
“44 Thus, multimodal or mixed methods research designs are most popular for understanding the usability and acceptability of web and other digital tools 45 ; these approaches can capture behavioral data on user navigation of an application and ratings of subjective experience (quantitative) as well…”
BackgroundMethodvan Gurp, J. L. P., van Lent, L. G. G., Stoel, N., van der Rijt, C. C. D., van Weert, J. C. M., & Hasselaar, J. G. J. (2023) · Patient Education and Counseling · doi:10.1016/j.pec.2023.108075 · Cited by 11
André, M., Enez, J., Charras, K., Besançon, M., & Delouvée, S. (2024) · Disability and Rehabilitation Assistive Technology · doi:10.1080/17483107.2024.2359472 · Cited by 15
“Participatory creation processes are needed [63] can help to ensure a holist process of development and encourage acceptance.”
Brandl, L., Cabrita, M., Brodbeck, J., Heylen, D., & van Velsen, L. (2022) · Internet Interventions · doi:10.1016/j.invent.2022.100534 · Cited by 13
Lukka, L., Karhulahti, V., Bergman, V.-R., & Palva, J. M. (2024) · Internet Interventions · doi:10.1016/j.invent.2023.100706 · Cited by 14
“Capturing these changes requires long-term data gathering, but earlier reviews have not found EMAs being used to study the digital intervention user experience over time (Doherty et al., 2020; Kip et al., 2022; Short et al., 2018; Smith et al., 2023).”
BackgroundMethodInfluentialBadaghi, N., van Kruijsbergen, M., Prins, J., Kelders, S., Cillessen, L., Compen, F., Donders, R., Kwakkenbos, L., & Speckens, A. (2023) · BMC Psychology · doi:10.1186/s40359-023-01052-2 · Cited by 8
“Although it is still under debate on how optimal support should be given [17], it has been suggested that good quality online interventions consider the heterogeneity in patients characteristics and skills, the context, and evidence from previous development activities [17].”
BackgroundSelf-citationAbbasi, R., Alavi, N., Farzandipour, M., Gong, Y., & Nabovati, E. (2023) · Informatics in Medicine Unlocked · doi:10.1016/j.imu.2023.101198 · Cited by 7
User journey method: a case study for improving digital intervention use measurement
Lukka, L., Vesterinen, M., Salonen, A., Bergman, V.-R., Torkki, P. M., Palva, S., & Palva, J. M. (2025) · BMC Health Services Research · doi:10.1186/s12913-025-12641-9 · Cited by 9
“Also known as analytics [16] and log data [17], intervention use measurement evaluates how and how much the patient interacts with the intervention.”
InfluentialDeveloping a Guided Web App for Postpartum Depression Symptoms: User-Centered Design Approach
Franco, P., Olhaberry, M., Muzard, A., Harismendy, Á., & Kelders, S. M. (2024) · JMIR Formative Research · doi:10.2196/56319 · Cited by 11
“Early and integrative inclusion of user involvement in the development process has been suggested as a strategy to address these challenges [1,4,5].”
BackgroundMethodSelf-citationLaure, T., Boffo, M., Engels, R. C. M. E., & Remmerswaal, D. (2024) · Internet Interventions · doi:10.1016/j.invent.2024.100750 · Cited by 12
Badaghi, N., van Kruijsbergen, M., Speckens, A. E. M., Vilé, J., Prins, J. B., Kelders, S. M., & Kwakkenbos, L. (2024) · JMIR Formative Research · doi:10.2196/52338 · Cited by 9
“Although eHealth interventions are complex and relatively new [10], online MBIs offer multiple advantages over face-to-face interventions.”
BackgroundInfluentialSelf-citationEffects of technology-assisted rehabilitation for patients with hip arthroplasty: A meta-analysis
Wu, L., Li, X., Hua, L., & Sun, Q. (2023) · Medicine · doi:10.1097/md.0000000000035921 · Cited by 13
“[15] Despite systematic reviews of technology-assisted rehabilitation, there is insufficient evidence to support the effectiveness of this method in the rehabilitation of THA patients.”
MethodGhaben, S. J., Ludin, A. F. M., Ali, N. M., Gan, K. B., & Singh, D. K. A. (2023) · Digital Health · doi:10.1177/20552076231191014 · Cited by 16
Zoellner, J., Reid, A., Porter, K., Frederick, C., Hilgart, M. M., & Ritterband, L. M. (2022) · JMIR Human Factors · doi:10.2196/41262 · Cited by 10
“A scoping review of 160 papers regarding research activities for participatory eHealth development processes identified a variety of methods and products [14].”
BackgroundMethodResultInfluentialÅsberg, K., Eldh, A. C., Löf, M., & Bendtsen, M. (2024) · Digital Health · doi:10.1177/20552076241245905 · Cited by 9
“47 Another example is derived from a qualitative study by Burgermaster et al. 48 Utilizing a phenomenological and social construct-ivist approach, the study identi fi ed four psychosocial phenotypes of behavior change in response to a school-based childhood obesity prevention intervention.”
MethodCardwell, E. T., Walsh, O., Chang, S., Coombe, J., Fairley, C. K., Hocking, J. S., Kong, F. Y. S., & Ludwick, T. (2024) · Sexually Transmitted Infections · doi:10.1136/sextrans-2024-056292 · Cited by 6
Asbjørnsen, R. A., Børøsund, E., Hjelmesaeth, J. S., Smedsrød, M. L., Ollivier, M., Wentzel, J., Clark, M. M., van Gemert‐Pijnen, J. E. W. C., & Nes, L. S. (2024) · Behaviour and Information Technology · doi:10.1080/0144929x.2024.2399299 · Cited by 7
“There is also a lack of guidelines and standardised reporting on how behaviour change techniques can be combined with persuasive design features and translated into human-centred and effective digital interventions (Asbjørnsen et al. 2022; Encantado et al. 2022; Kip et al. 2022).”
Laure, T., Engels, R. C. M. E., Remmerswaal, D., Spruijt-Metz, D., Konigorski, S., & Boffo, M. (2023) · JMIR Research Protocols · doi:10.2196/46603 · Cited by 4
“The development of the mobile app was guided by the Center for eHealth Research (CeHRes) road map, an established framework for the development of eHealth tools [19,34] in which evidence and theoretical insights are intertwined with design methods.”
MethodCOHERENCE, CREATIVITY AND AUDACITY IN METHODOLOGICAL DECISIONS
Zanchetta, M. S., & Metersky, K. (2023) · Texto & Contexto - Enfermagem · doi:10.1590/1980-265x-tce-2023-e001en · Cited by 6
Exploring Older Adults’ Needs for a Healthy Life and eHealth: Qualitative Interview Study
Valkonen, P., Kujala, S., Savolainen, K., & Helminen, R.-R. (2025) · JMIR Human Factors · doi:10.2196/50329 · Cited by 6
“In eHealth, technology, tasks, individuals, and contexts are holistically combined, which should all be considered in developments [36-39]. eHealth services should genuinely support people pursuing healthier lifestyles and be accessible to all [10], although older adults may perceive the technology…”
Breaking Down Barriers to a Suicide Prevention Helpline: Web-Based Randomized Controlled Trial
van der Burgt, M. C. A., Mérelle, S. Y. M., Brinkman, W., Beekman, A. T. F., & Gilissen, R. (2024) · JMIR Mental Health · doi:10.2196/56396 · Cited by 5
“In their review of methods for human-centered eHealth development, Kip et al [30] describe many varieties of methods and products that can be used throughout the development process to prioritize user perspectives to enhance the effectiveness and usability of eHealth solutions.”
MethodHartmann, E. M., Küper, A., Swoboda, J., Lodde, G. C., Livingstone, E., Beckmann, C. L., Schadendorf, D., & Sachweh, S. (2024) · Scientific Reports · doi:10.1038/s41598-024-67857-2 · Cited by 4
“Conducting a CI is relevant for a holistic view of the context 15,34,35 .”
BackgroundKeizer, J., Bente, B. E., al Naiemi, N., van Gemert‐Pijnen, J. E. W. C., & Jong, N. B.-D. (2021) · Journal of Medical Internet Research · doi:10.2196/33531 · Cited by 6
Self-citationIbrahim, R. H., Yaas, M. H., Hamarash, M. Q., Mukhtar, S. H. A., Abdulghani, M. F., & Almushhadany, O. I. (2024) · JMIR Pediatrics and Parenting · doi:10.2196/67137 · Cited by 6
“CBT focuses on helping individuals identify and modify negative thought patterns and behaviors, making it well suited for delivery via mobile platforms that offer interactive and self-guided modules [25-28].”
Organizational and economic aspects of standardization of processes in the healthcare system
Русских, С. В., Al, L., Moskvicheva, L. I., Vasilieva, T. P., Мануковна, С. С., & Gorbacheva, N. А. (2023) · Health Care of the Russian Federation · doi:10.47470/0044-197x-2023-67-5-367-373 · Cited by 3
Branquinho, M., Canavarro, M. C., & Fonseca, A. (2023) · International Journal of Cognitive Therapy · doi:10.1007/s41811-023-00182-0 · Cited by 5
“…intervention involved the target population at different stages (development and evaluation) and using different methods (e.g., questionnaires and interviews) following a patient-centered approach, which is an established practice in the development of e-health interventions (Kip et al., 2022).”
Kadir, M. N. N. A., Hairon, S. M., Yaacob, N. M., Yusof, S. N., Musa, K. I., Yahya, M. M., Isa, S. A. M., Azlan, M. H. M., & Hadi, I. S. A. (2023) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph20042985 · Cited by 4
“In each stage, various methods such as literature reviews, questionnaires, interviews, or focus groups could be used according to the developer’s feasibility and the product’s needs [22,24].”
MethodBranquinho, M., Monteiro, F., & Fonseca, A. (2025) · International Journal of Cognitive Behavioral Therapy · doi:10.1007/s41811-025-00266-z · Cited by 2
“Considering the recommendations of including relevant stakeholders and the target population in the early stages of development of e-health interventions (Kip et al., 2022), this study aimed to gather perspectives of health professionals and women in (peri)menopause about such an intervention.”
Zonneveld, M., Pol, M., Achterberg, W. P., & van Isselt, E. F. V. D. (2025) · INQUIRY The Journal of Health Care Organization Provision and Financing · doi:10.1177/00469580251347139 · Cited by 4
“19,23 The information obtained from the interviews and the focus groups was shared with the project team before multiple individual brainstorming sessions were held to generate ideas for what eHealth technology fits the context.”
Alexithymia in borderline personality pathology: From theory to a biosensor application
Derks, Y. P. M. J. (2022) · doi:10.3990/1.9789464217896 · Cited by 2
Jonsson, M., Gustavsson, C., Gulliksen, J., & Johansson, S. (2025) · International Journal of Human-Computer Interaction · doi:10.1080/10447318.2025.2601277 · Cited by 1
“Prototypes are common in the development of eHealth (Kip et al., 2022).”
Implementation of digital health interventions in thyroid cancer care: A scoping review
Zhu, M., Wang, X., & Mei, L. (2025) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2025.106221 · Cited by 2
Al-Dhahir, I., Breeman, L. D., Faber, J. S., Wentzel, J., van den Berg-Emons, R. J. G., Kraal, J. J., Janssen, V. R., Kraaijenhagen, R. A., Visch, V., Chavannes, N. H., et al. (2025) · Frontiers in Digital Health · doi:10.3389/fdgth.2025.1528860 · Cited by 3
Enhancing the Visibility of Handicraft-Based MSMEs through Strategic Web Design
Nisa, A. R., & Zulaikha, E. (2025) · Journal of Management Studies and Development · doi:10.56741/jmsd.v4i03.1223 · Cited by 1
“This method is particularly suitable for website development, as it emphasizes empathy with users and continuous refinement based on feedback [14].”
Consumer Involvement in the Co-Design of Diabetes Self-Management Smartphone Apps: A Scoping Review
Martin, C. L., Bakker, C., Sarkar, S., Kang, R. M., Reid, N., Chih, M., Sittig, S., Gao, G., Smith, C., Morse, B., et al. (2025) · Applied Clinical Informatics · doi:10.1055/a-2564-7682 · Cited by 1
Valkonen, P., Hölsä, S., Viitanen, J., Leinonen, S., Karisalmi, N., & Rauta, V. (2025) · JMIR Human Factors · doi:10.2196/48221 · Cited by 3
“Findings from user research can also be presented as user profile illustrations, such as personas [15,34], stakeholder maps [15], experience maps [34], and visualizations of stakeholders [34].”
InfluentialNye, K. E., Tryon, V. L., Pierce, K. M., Tully, L. M., Ereshefsky, S., Savill, M., Smith, L., Wilcox, A. B., Hakusui, C. K., Padilla, V. E., et al. (2025) · JMIR Human Factors · doi:10.2196/65889 · Cited by 4
“This iterative approach in response to user feedback is consistent with the development process of other eHealth apps [47].”
Borgnis, F., Isernia, S., Rossetto, F., Pagliari, C., Tavanelli, M., Brambilla, L., & Baglio, F. (2024) · Heliyon · doi:10.1016/j.heliyon.2024.e40436 · Cited by 5
“In this regard, co-designing the technological platform with end-users, both patients and clinical staff, may help to avoid technological constraints [34].”
Kip, H., Jong, N. B.-D., van Gemert‐Pijnen, J. E. W. C., et al. (2024) · doi:10.2196/preprints.59601 · Cited by 1
Self-citationLiebe, J.-D., & Esdar, M. (2024) · Springer eBooks · doi:10.1007/978-3-658-43236-2_57 · Cited by 2
Eriksson‐Backa, K., Enwald, H., Hirvonen, N., & Huvila, I. (2023) · Proceedings of the Association for Information Science and Technology · doi:10.1002/pra2.818 · Cited by 1
“In this sense, collecting data through different methods might be the most useful solution, as also Kip et al. (2022) suggest.”
BackgroundMethodInfluentialApplying the Nominal Group Technique for the Conceptual Validation of ehealth Solutions
Rosa, A. F., Martins, A. I., Cerqueira, M., Santos, M., Silva, A. G., & Rocha, N. P. (2023) · Procedia Computer Science · doi:10.1016/j.procs.2023.01.407 · Cited by 6
Gómez, L., Ríos, V. L., González, J. C. R., & Abascal-Mena, R. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-48041-6_34 · Cited by 1
Lindayani, L., Taryudi, T., Mutiar, A., Darmawati, I., & Calisanie, N. N. P. (2026) · Progress in Disaster Science · doi:10.1016/j.pdisas.2026.100707
Moura, A. B. F., Motter, F. R., Silva, L. M., Silva, I. C., Silva, W. A. S., Leão, B. D. F., Col, S. V. C. D., & Gadenz, S. D. (2026) · Frontiers in Digital Health · doi:10.3389/fdgth.2026.1713349
“Desk research is a question-driven review of existing materials—often including non-scientific sources such as intervention descriptions, management reports, and stakeholder artifacts—to build a broad understanding of the topic (24).”
Hartmann, E. M., Küper, A., Swoboda, J., Beckmann, C. L., Lodde, G. C., Livingstone, E., Schadendorf, D., & Sachweh, S. (2026) · JMIR Cancer · doi:10.2196/79294 · Cited by 1
“The TA method [26] is a widespread methodology [27-29] for assessing the suitability of implementations or prototypes within their intended context [27].”
Oti, O., Foley, S., & Pitt, I. J. (2025) · Frontiers in Digital Health · doi:10.3389/fdgth.2025.1633987 · Cited by 1
“Researchers suggest that usability tests should be comprehensive, including a range of stakeholders and not only end-users [33].”
Hartmann, E. M., Küper, A., Swoboda, J., et al. (2025) · doi:10.2196/preprints.79294
Mitra, A., & Pal, A. (2025) · e-Prime - Advances in Electrical Engineering Electronics and Energy · doi:10.1016/j.prime.2025.100983 · Cited by 5
Kelly, M., Fullen, B. M., Martin, D., Bradley, C., Riain, E. Ó., & McVeigh, J. G. (2024) · Clinical Rehabilitation · doi:10.1177/02692155241289097 · Cited by 2
“12 This approach aims to enhance self-ef fi cacy, improve self-management skills and increase overall intervention effectiveness.”
BackgroundMethodInfluentialIbrahim, R. H., Yaas, M. H., Hamarash, M. Q., Mukhtar, S. H. A., & Abdulghani, M. F. (2024) · doi:10.2196/preprints.67137
Hoang, V. L., Nguyen, T. T. H., Tran, N. T., Nguyen, T. H. H., Bui, K. L., Anderson, D. J., Baird, K. M., Akbar, H. G., & Hoang, A. P. (2024) · JMIR Research Protocols · doi:10.2196/66060 · Cited by 1
Franco, P., Olhaberry, M., Muzard, A., et al. (2024) · doi:10.2196/preprints.56319
Self-citation(2023) · doi:10.2196/preprints.52338
Self-citationExploring Older Adults’ Needs for a Healthy Life and eHealth: Qualitative Interview Study (Preprint)Preprint
Valkonen, P., Kujala, S., Savolainen, K., et al. (2023) · doi:10.2196/preprints.50329
Valkonen, P., Hölsä, S., Viitanen, J., et al. (2023) · doi:10.2196/preprints.48221
COERÊNCIA, CRIATIVIDADE E AUDÁCIA NAS DECISÕES METODOLÓGICAS
Zanchetta, M. S., & Metersky, K. (2023) · Texto & Contexto - Enfermagem · doi:10.1590/1980-265x-tce-2023-e001pt
COHERENCIA, CREATIVIDAD Y AUDACIA EN DECISIONES METODOLÓGICAS
Zanchetta, M. S., & Metersky, K. (2023) · Texto & Contexto - Enfermagem · doi:10.1590/1980-265x-tce-2023-e001es
Hong, S. J., Chung, M., Gu, Y., & Kim, K. (2022) · CIN Computers Informatics Nursing · doi:10.1097/cin.0000000000000975 · Cited by 3
Keizer, J. (2021) · doi:10.3990/1.9789036553230 · Cited by 2
Self-citation
Metadata
- Authors
- Kip, H., Keizer, J., da Silva, M. C., Beerlage-de Jong, N., Köhle, N., & Kelders, S. M.
- Affiliations
- University of Twente (NL)
- North-West University (ZA)
- Corresponding author
- Kip, Hanneke
- ORCID
- Venue
- Journal of Medical Internet Research
- Publisher
- JMIR Publications Inc.
- Published
- 2022-01-27
- Volume
- 24
- Issue
- 1
- Pages
- e31858
- ISSN
- 1438-8871 (ISSN-L)
- PMC
- PMC8832261
- OpenAlex
- W3213720648
- Cited by
- 67
- References
- 183
- Citations by year
- 2026: 4 · 2025: 15 · 2024: 20 · 2023: 19 · 2022: 7
- FWCI
- 2.48
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- Locations
- Journal of Medical Internet Research (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- University of Twente Research Information (publishedVersion, cc-by)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
- University of Twente Research Information (submittedVersion, cc-by)
- Topics
- Health Sciences › Health Professions › General Health Professions › Mobile Health and mHealth Applications
- Physical Sciences › Computer Science › Human-Computer Interaction › Usability and User Interface Design
- Physical Sciences › Computer Science › Human-Computer Interaction › Innovative Human-Technology Interaction
- Keywords
- participatory design, human-centered design, iterative design, development methods, lessons learned, stakeholder engagement, personas, behavior change strategies, business models, prototype development, usability testing, stakeholder mapping
- MeSH terms
- Behavior Therapy, Humans, Surveys and Questionnaires, Research Design, Focus Groups, Telemedicine
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-04, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-04 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (qwen3.8-omni-flash and gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.
![Figure 1 The Center for eHealth Research Roadmap [2,11,18].](https://figures.cochat.ai/paper-figures/98136266f8df4aaa942d75b242ef32bd/methods-for-human-centered-ehealth-development-narrative-scoping-review-figure-1.jpg)
