The Importance of Systematically Reporting and Reflecting on eHealth Development: Participatory Development Process of a Virtual Reality Application for Forensic Mental Health Care
Open accessAbstract
Background: The use of electronic health (eHealth) technologies in practice often is lower than expected, mostly because there is no optimal fit among a technology, the characteristics of prospective users, and their context. To improve this fit, a thorough systematic development process is recommended. However, more knowledge about suitable development methods is necessary to create a tool kit that guides researchers in choosing development methods that are appropriate for their context and users. In addition, there is a need for reflection on the existing frameworks for eHealth development to be able to constantly improve them. Objective: The two main objectives of this case study were to present and reflect on the (1) methods used in the development process of a virtual reality application for forensic mental health care and (2) development model that was used: the CeHRes Roadmap (the Centre for eHealth Research Roadmap). Methods: In the development process, multiple methods were used to operationalize the first 2 phases of the CeHRes Roadmap: the contextual inquiry and value specification. To summarize the most relevant information for the goals of this study, the following information was extracted per method: (1) research goal, (2) explanation of the method used, (3) main results, (4) main conclusions, and (5) lessons learned about the method. Results: Information on 10 methods used is presented in a structured manner. These 10 methods were stakeholder identification, project team composition, focus groups, literature study, semistructured interviews, idea generation with scenarios, Web-based questionnaire, value specification, idea generation with prototyping, and a second round of interviews. The lessons learned showed that although each method added new insights to the development process, not every method appeared to be the most appropriate for each research goal. Conclusions: Reflection on the methods used pointed out that brief methods with concrete examples or scenarios fit the forensic psychiatric patients the best, among other things, because of difficulties with abstract reasoning and low motivation to invest much time in participating in research. Formulating clear research questions based on a model’s underlying principles and composing a multidisciplinary project team with prospective end users appeared to be important in this study. The research questions supported the project team in keeping the complex development processes structured and prevented tunnel vision. With regard to the CeHRes Roadmap, continuous stakeholder involvement and formative evaluations were evaluated as strong points. A suggestion to further improve the Roadmap is to explicitly integrate the use of domain-specific theories and models. To create a tool kit with a broad range of methods for eHealth development and further improve development models, studies that report and reflect on development processes in a consistent and structured manner are needed.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The CeHRes Roadmap has been successfully used to develop effective digital interventions.
“The framework has been successfully used to develop effective digital interventions (Kelders et al., 2013; Kip et al., 2019).”
From Strength Back – A qualitative study on the co-creation of a positive psychology digital health intervention for spinal surgery patients (2023) · cites this paper as Kip et al., 2019
The Methods section, “The CeHRes Roadmap” (page unavailable), says the framework has proved useful for eHealth development in multiple settings. The Discussion section, “Operationalization of the Development Model” (page unavailable), says it guided development of a VR application concept but explicitly does not conclude that it produced better outcomes. The paper therefore supports successful use in development, not the claim that the resulting digital interventions were effective. (LLM, conf=0.88)
Experimental — accuracy not yet measuredIt was suggested that desk research is a suitable method to look outside of a project's scope to learn from the use of an eHealth technology in other domains.
“Third, it was suggested that desk research is a suitable method to look outside of a project’s scope to learn from the use of an eHealth technology in other domains.”
From Methods for Human-Centered eHealth Development: Narrative Scoping Review (2022) · cites this paper as [15]
Desk research should be used as a method that requires a clear research question and, if possible, a systematic search strategy that is clearly connected to the goal of the eHealth development process.
“Desk research should be used as a method that requires a clear research question and, if possible, a systematic search strategy that is clearly connected to the goal of the eHealth development process”
From Methods for Human-Centered eHealth Development: Narrative Scoping Review (2022) · cites this paper as [15]
Authors stated that values should be formulated in such a way that they are neither too specific to prevent overlap with requirements nor too broad and vague.
“First, authors stated that values should be formulated in such a way that they are neither too specific to prevent overlap with requirements nor too broad and vague.”
From Methods for Human-Centered eHealth Development: Narrative Scoping Review (2022) · cites this paper as [15]
It was considered to be important to include a wide variety of stakeholders, where researchers should look beyond their own setting.
“Third, it was considered to be important to include a wide variety of stakeholders, where researchers should look beyond their own setting.”
From Methods for Human-Centered eHealth Development: Narrative Scoping Review (2022) · cites this paper as [15]
Methods can be used to validate findings; for example, interviews can provide more context to the results of a literature review.
“methods can be used to validate findings; for example, interviews can be used to provide more context to the results of a literature review [15].”
From Methods for Human-Centered eHealth Development: Narrative Scoping Review (2022) · cites this paper as [15]
The distinction between methods and products is in line with design-thinking approaches such as the double diamond model, which pays attention to diverging, for example, by means of collecting data, and converging, for example, by means of integrating the findings in a set of requirements or a prototype.
“This is in line with design-thinking approaches such as the double diamond model, which pays attention to diverging, for example, by means of collecting data, and converging, for example, by means of integrating the findings in a set of requirements or a prototype”
From Methods for Human-Centered eHealth Development: Narrative Scoping Review (2022) · cites this paper as [15]
Multiple included studies showed that a visualization of the development process might also be of added value.
“multiple included studies showed that a visualization of the development process might also be of added value [15].”
From Methods for Human-Centered eHealth Development: Narrative Scoping Review (2022) · cites this paper as [15]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
1 retracted
Renaud, P., Chartier, S., Rouleau, J. L., Proulx, J., Goyette, M., Trottier, D., Fedoroff, P., Bradford, J.-P., Dassylva, B., & Bouchard, S. (2011) · Journal of Sexual Aggression · doi:10.1080/13552600.2011.617014 · Cited by 42 · see citing works
RetractedA Theoretical Extension of the Technology Acceptance Model: Four Longitudinal Field Studies
Venkatesh, V., & Davis, F. D. (2000) · Management Science · doi:10.1287/mnsc.46.2.186.11926 · Cited by 23,066 · see citing works
Why do people use information technology? A critical review of the technology acceptance model
Legris, P., Ingham, J., & Collerette, P. (2003) · Information & Management · doi:10.1016/s0378-7206(01)00143-4 · Cited by 4,250 · see citing works
Greenhalgh, T., Wherton, J., Papoutsi, C., Lynch, J., Hughes, G., A'Court, C., Hinder, S., Fahy, N., Procter, R., & Shaw, S. (2017) · Journal of Medical Internet Research · doi:10.2196/jmir.8775 · Cited by 2,887 · cited 2× · see citing works
“• eHealth development and implementation should be intertwined; implementation is a very complex activity that should be accounted for from the start of the development process [24,25].”
Webb, T. L., Joseph, J., Yardley, L., & Michie, S. (2010) · Journal of Medical Internet Research · doi:10.2196/jmir.1376 · Cited by 2,517 · see citing works
The Personal Involvement Inventory: Reduction, Revision, and Application to Advertising
Zaichkowsky, J. L. (1994) · Journal of Advertising · doi:10.1080/00913367.1943.10673459 · Cited by 1,737 · see citing works
Persuasive Systems Design: Key Issues, Process Model, and System Features
Oinas-Kukkonen, H., & Harjumaa, M. (2009) · Communications of the Association for Information Systems · doi:10.17705/1cais.02428 · Cited by 1,663 · see citing works
Yardley, L., Morrison, L., Bradbury, K., & Muller, I. (2015) · Journal of Medical Internet Research · doi:10.2196/jmir.4055 · Cited by 1,442 · see citing works
A Holistic Framework to Improve the Uptake and Impact of eHealth Technologies
van Gemert-Pijnen, J. E., Nijland, N., van Limburg, M., Ossebaard, H. C., Kelders, S. M., Eysenbach, G., & Seydel, E. R. (2011) · Journal of Medical Internet Research · doi:10.2196/jmir.1672 · Cited by 1,126 · see citing works
Self-citationRoss, J., Stevenson, F., Lau, R., & Murray, E. (2016) · Implementation Science · doi:10.1186/s13012-016-0510-7 · Cited by 1,011 · see citing works
Michie, S., Yardley, L., West, R., Patrick, K., & Greaves, F. (2017) · Journal of Medical Internet Research · doi:10.2196/jmir.7126 · Cited by 865 · see citing works
Ten principles of good interdisciplinary team work
Nancarrow, S., Booth, A., Ariss, S., Smith, T., Enderby, P., & Roots, A. (2013) · Human Resources for Health · doi:10.1186/1478-4491-11-19 · Cited by 683 · see citing works
Conceptualizing vulnerable populations health-related research.
Flaskerud, J., & Winslow, B. J. (1998) · Nursing Research · doi:10.1097/00006199-199803000-00005 · Cited by 563 · see citing works
Mohr, D. C., Schueller, S. M., Montague, E., Burns, M. N., & Rashidi, P. (2014) · Journal of Medical Internet Research · doi:10.2196/jmir.3077 · Cited by 510 · see citing works
Mohr, D. C., Lyon, A. R., Lattie, E. G., Reddy, M., & Schueller, S. M. (2017) · Journal of Medical Internet Research · doi:10.2196/jmir.7725 · Cited by 432 · see citing works
Designing for or designing with? Informant design for interactive learning environments
Scaife, M., Rogers, Y., Aldrich, F. K., & Davies, M. (1997) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/258549.258789 · Cited by 388 · see citing works
van der Vaart, R., Witting, M., Riper, H., Kooistra, L., Bohlmeijer, E. T., & van Gemert-Pijnen, L. J. (2014) · BMC Psychiatry · doi:10.1186/s12888-014-0355-z · Cited by 273 · cited 1× · see citing works
“First, although involving patients proved to be very valuable, the development process also benefited from the perspectives of other types of stakeholders, such as therapists, managers, researchers, and technology developers, as they might have different needs or a more overarching view [28,48].”
Designing eHealth that Matters via a Multidisciplinary Requirements Development Approach
Van Velsen, L., Wentzel, J., & Van Gemert-Pijnen, J. E. (2013) · JMIR Research Protocols · doi:10.2196/resprot.2547 · Cited by 262 · see citing works
Agile science: creating useful products for behavior change in the real world
Hekler, E. B., Klasnja, P., Riley, W. T., Buman, M. P., Huberty, J., Rivera, D. E., & Martin, C. A. (2016) · Translational Behavioral Medicine · doi:10.1007/s13142-016-0395-7 · Cited by 208 · see citing works
The Pace of Technologic Change
Patrick, K., Hekler, E. B., Estrin, D., Mohr, D. C., Riper, H., Crane, D., Godino, J., & Riley, W. T. (2016) · American Journal of Preventive Medicine · doi:10.1016/j.amepre.2016.05.001 · Cited by 169 · see citing works
eHealth Research, Theory and Development: A Multidisciplinary Approach
(2018) · doi:10.4324/9781315385907 · Cited by 169 · see citing works
Self-citationMoller, A. C., Merchant, G., Conroy, D. E., West, R., Hekler, E., Kugler, K. C., & Michie, S. (2017) · Journal of Behavioral Medicine · doi:10.1007/s10865-016-9818-7 · Cited by 151 · see citing works
DeSmet, A., Thompson, D., Baranowski, T., Palmeira, A., Verloigne, M., & De Bourdeaudhuij, I. (2016) · Journal of Medical Internet Research · doi:10.2196/jmir.4444 · Cited by 138 · see citing works
van der Weegen, S., Verwey, R., Spreeuwenberg, M., Tange, H., van der Weijden, T., & de Witte, L. (2013) · JMIR mhealth and uhealth · doi:10.2196/mhealth.2526 · Cited by 128 · see citing works
eHealth in Treatment of Offenders in Forensic Mental Health: A Review of the Current State
Kip, H., Bouman, Y. H. A., Kelders, S. M., & van Gemert-Pijnen, L. J. E. W. C. (2018) · Frontiers in Psychiatry · doi:10.3389/fpsyt.2018.00042 · Cited by 89 · see citing works
Self-citationDevelopment of a web-based intervention for the indicated prevention of depression
Kelders, S. M., Pots, W. T., Oskam, M. J., Bohlmeijer, E. T., & van Gemert-Pijnen, J. E. (2013) · BMC Medical Informatics and Decision Making · doi:10.1186/1472-6947-13-26 · Cited by 80 · see citing works
Self-citationParticipatory eHealth development to support nurses in antimicrobial stewardship
Wentzel, J., van Velsen, L., van Limburg, M., de Jong, N., Karreman, J., Hendrix, R., & van Gemert-Pijnen, J. J. E. W. C. (2014) · BMC Medical Informatics and Decision Making · doi:10.1186/1472-6947-14-45 · Cited by 73 · see citing works
Renaud, P., Trottier, D., Rouleau, J. L., Goyette, M., Saumur, C., Boukhalfi, T., & Bouchard, S. (2013) · Virtual Reality · doi:10.1007/s10055-013-0235-8 · Cited by 65 · see citing works
van Woezik, A. F. G., Braakman-Jansen, L. M. A., Kulyk, O., Siemons, L., & van Gemert-Pijnen, J. E. W. C. (2016) · Antimicrobial Resistance & Infection Control · doi:10.1186/s13756-016-0119-2 · Cited by 57 · see citing works
Kip, H., Kelders, S. M., Weerink, K. M. A., Kuiper, A., Brüninghoff, I., Bouman, Y. H. A., Dijkslag, D., & van Gemert-Pijnen, L. J. E. W. C. (2019) · Frontiers in Psychology · doi:10.3389/fpsyg.2019.00406 · Cited by 52 · see citing works
Self-citationDugas, M., Trottier, M.-È., Dansokho, S. C., Vaisson, G., Provencher, T., Colquhoun, H. L., Dogba, M. J., Dupéré, S., Fagerlin, A., Giguere, A. M. C., et al. (2017) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-016-0399-8 · Cited by 50 · see citing works
Geissbühler, A. (2013) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2012.11.002 · Cited by 45 · see citing works
Feldman, S. S., Schooley, B. L., & Bhavsar, G. P. (2014) · JMIR Medical Informatics · doi:10.2196/medinform.3455 · Cited by 44 · see citing works
van Velsen, L., Evers, M., Bara, C.-D., den Akker, H. O., Boerema, S., & Hermens, H. J. (2018) · JMIR Formative Research · doi:10.2196/10474 · Cited by 37 · see citing works
Wolpin, S. E., Halpenny, B., Whitman, G. J., McReynolds, J., Stewart, M., Lober, W. B., & Berry, D. (2014) · Health Informatics Journal · doi:10.1177/1460458213495744 · Cited by 36 · see citing works
Virtual reality applications in forensic psychiatry
Benbouriche, M., Nolet, K., Trottier, D., & Renaud, P. (2014) · Virtual Reality International Conference · doi:10.1145/2617841.2620692 · Cited by 34 · see citing works
Fromberger, P., Meyer, S., Kempf, C., Jordan, K., & Müller, J. L. (2015) · PLoS ONE · doi:10.1371/journal.pone.0127156 · Cited by 32 · see citing works
Beerlage-de Jong, N., Van Gemert-Pijnen, L., Wentzel, J., Hendrix, R., & Siemons, L. (2017) · Infectious Disease Reports · doi:10.4081/idr.2017.6829 · Cited by 28 · see citing works
Self-citationInterpretive Flexibility in Mobile Health: Lessons From a Government-Sponsored Home Care Program
Nielsen, J. A., & Mathiassen, L. (2013) · Journal of Medical Internet Research · doi:10.2196/jmir.2816 · Cited by 25 · see citing works
Involvement as a Working Mechanism for Persuasive Technology
Kelders, S. M. (2015) · Lecture notes in computer science · doi:10.1007/978-3-319-20306-5_1 · Cited by 25 · see citing works
Self-citationBeerlage-de Jong, N., Wentzel, J., Hendrix, R., & van Gemert-Pijnen, L. (2017) · American Journal of Infection Control · doi:10.1016/j.ajic.2016.12.001 · Cited by 24 · see citing works
Self-citationKip, H., Kelders, S. M., & Van Gemert-Pijnen, L. J. E. W. C. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300365 · Cited by 11 · see citing works
Self-citation
Flagged first, then by citations
Research works printed but not linked: 12
- [5]Pieterse ME Kip H Cruz-Martínez RR van Gemert-Pijnen L Kelders SM Kip H Sanderman R The complexity of ehealth implementation: a theoretical and practical perspective eHealth Research, Theory and Development: A Multi-Disciplinary Approach 2018 London Routledge 247 70
- [6]Kip H van Gemert-Pijnen L van Gemert-Pijnen L Kelders SM Kip H Sanderman R Holistic development of ehealth technology eHealth Research, Theory and Development: A Multi-Disciplinary Approach 2018 London Routledge 151 86
- [12]Eldredge LK Markham CM Ruiter RA Fernández ME Kok G Parcel GS Planning Health Promotion Programs: An Intervention Mapping Approach 2016 Hoboken, New Jersey John Wiley & Sons
- [15]van Gemert-Pijnen L Kelders SM Kip H Sanderman R eHealth Research, Theory and Development: A Multi-Disciplinary Approach 2018 London Routledge
- [23]Covvey HD van Gemert-Pijnen L Kelders SM Kip H Sanderman R Healthcare as a complex adaptive system eHealth Research, Theory and Development: A Multi-Disciplinary Approach 2018 London Routledge 89 110
- [26]Sieverink F Köhle N Cheung K Roefs A Trompetter H Keizer J Braakman-Jansen A Kelders SM van Gemert-Pijnen L Kelders SM Kip H Sanderman R Evaluating ehealth eHealth Research, Theory and Development: A Multi-Disciplinary Approach 2018 London Routledge
- [35]Renaud P Rouleau JL Proulx J Trottier D Goyette M Bradford JP Fedoroff P Dufresne MH Dassylva B Côté G Bouchard S Virtual characters designed for forensic assessment and rehabilitation of sex offenders: standardized and made-to-measure J Virtual Real Broadcast 2010 7 5 1 9 10.20385/1860-2037/7.2010.5 · DOI
- [36]Fromberger P Jordan K Müller JL [Use of virtual reality in forensic psychiatry. A new paradigm?] Nervenarzt 2014 3 85 3 298 303 10.1007/s00115-013-3904-7 24549691 24549691 · DOI
- [46]Burns C. van Gemert-Pijnen L Kelders SM Kip H Sanderman R Human-centred design eHealth Research, Theory and Development: A Multi-Disciplinary Approach 2018 London Routledge
- [53]Yip J Clegg T Bonsignore E Gelderblom H Rhodes E Druin A Brownies or Bags-of-Stuff?: Domain Expertise in Cooperative Inquiry With Children Proceedings of the 12th International Conference on Interaction Design and Children 2013 IDC'13 June 24-27, 2013 New York, USA 201 10 10.1145/2485760.2485763 · DOI · Paper page
- [54]Kip H Beerlage-de Jong N Wentzel J van Gemert-Pijnen JE Kelders SM Kip H Sanderman R The contextual inquiry eHealth Research, Theory and Development: A Multi-Disciplinary Approach 2018 London Routledge 187 206
- [58]Gottfredson MR A General Theory Of Crime 1990 Stanford Stanford University Press
Cited by
Sorted by citations
Dincelli, E., & Yayla, A. (2022) · The Journal of Strategic Information Systems · doi:10.1016/j.jsis.2022.101717 · Cited by 490 · see citing works
Methods for Human-Centered eHealth Development: Narrative Scoping Review
Kip, H., Keizer, J., da Silva, M. C., Beerlage-de Jong, N., Köhle, N., & Kelders, S. M. (2022) · Journal of Medical Internet Research · doi:10.2196/31858 · Cited by 67 · see citing works
Self-citationMethods for Human-Centered eHealth Development: Narrative Scoping Review (Preprint)
Kip, H., Keizer, J., da Silva, M. C., Beerlage-de Jong, N., Köhle, N., & Kelders, S. M. (2021) · doi:10.2196/preprints.31858 · Cited by 67 · see citing works
Self-citationSchouten, S. E., Kip, H., Dekkers, T., Deenik, J., Beerlage-de Jong, N., Ludden, G. D. S., & Kelders, S. M. (2022) · Design for Health · doi:10.1080/24735132.2022.2145814 · Cited by 37 · see citing works
Van Der Horst, A. Y., Bohlmeijer, E. T., Schreurs, K. M. G., & Kelders, S. M. (2023) · Frontiers in Psychology · doi:10.3389/fpsyg.2023.1117357 · Cited by 4 · see citing works
“The framework has been successfully used to develop effective digital interventions (Kelders et al., 2013; Kip et al., 2019).”
MethodSelf-citation
- Authors
- Kip, H., Kelders, S. M., Bouman, Y. H. A., & van Gemert-Pijnen, L. J. E. W. C.
- Affiliations
- University of Twente (NL)
- North-West University (ZA)
- Corresponding author
- Kip, Hanneke
- Venue
- Journal of Medical Internet Research
- Publisher
- JMIR Publications Inc.
- Published
- 2019-08-19
- Volume
- 21
- Issue
- 8
- Pages
- e12972
- ISSN
- 1438-8871 (ISSN-L)
- PMC
- PMC6718085
- OpenAlex
- W2969448992
- Cited by
- 40
- References
- 42
- Citations by year
- 2026: 2 · 2025: 11 · 2024: 3 · 2023: 6 · 2022: 9 · 2021: 8 · 2020: 1
- FWCI
- 5.08
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- Locations
- Journal of Medical Internet Research (publishedVersion, cc-by)
- Europe PMC (PubMed Central) (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- University of Groningen research database (University of Groningen / Centre for Information Technology) (publishedVersion, cc-by)
- University of Twente Research Information (publishedVersion, cc-by)
- University of Groningen research database (University of Groningen / Centre for Information Technology) (submittedVersion, cc-by)
- University of Twente Research Information (submittedVersion, cc-by)
- Topics
- Social Sciences › Psychology › Applied Psychology › Digital Mental Health Interventions
- Physical Sciences › Computer Science › Human-Computer Interaction › Usability and User Interface Design
- Health Sciences › Health Professions › General Health Professions › Mobile Health and mHealth Applications
- Keywords
- virtual reality applications, forensic mental health services, forensic psychiatric patients, participatory development, contextual inquiry, stakeholder engagement, development methods, lessons learned, multidisciplinary project teams, user-centered design, formative assessment, prototype development
- MeSH terms
- Virtual Reality, Humans, Mental Health, Prospective Studies, Research Design, Telemedicine
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
