Putting the Value in VR: How to Systematically and Iteratively Develop a Value-Based VR Application with a Complex Target Group
Open accessAbstract
In development, implementation and evaluation of eHealth it is essential to account for stakeholders' perspectives, opinions and values, which are statements that specify what stakeholders want to achieve or improve via a technology. The use of values enables developers to systematically include stakeholders' perspectives and the context of use in an eHealth development process. However, there are relatively few papers that explain how to use values in technology development. Consequently, in this paper we show how we formulated values during the multi-method, interdisciplinary and iterative development process of a VR application for a complex setting: forensic mental healthcare. We report the main foundations for these values: the outcomes of an online questionnaire with patients, therapists and other stakeholders (n=146) and interviews with patients and therapists (n=18). We show how a multidisciplinary project team used these qualitative results to formulate and adapt values and create lo-fi prototypes of a VR application. We discuss the importance of a systematic development process with multiple formative evaluations for eHealth and reflect on the role of values within this.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Conflicting values might arise related to, for example, costs or the focus of an eHealth technology.
“Third, conflicting values might arise related to, for example, costs or the focus of an eHealth technology.”
From Methods for Human-Centered eHealth Development: Narrative Scoping Review (2022) · cites this paper as [97]
Authors stated that values should be formulated in such a way that they are neither too specific to prevent overlap with requirements nor too broad and vague.
“First, authors stated that values should be formulated in such a way that they are neither too specific to prevent overlap with requirements nor too broad and vague.”
From Methods for Human-Centered eHealth Development: Narrative Scoping Review (2022) · cites this paper as [97]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
A Theoretical Extension of the Technology Acceptance Model: Four Longitudinal Field Studies
Venkatesh, V., & Davis, F. D. (2000) · Management Science · doi:10.1287/mnsc.46.2.186.11926 · Cited by 23,066 · see citing works
Why do people use information technology? A critical review of the technology acceptance model
Legris, P., Ingham, J., & Collerette, P. (2003) · Information & Management · doi:10.1016/s0378-7206(01)00143-4 · Cited by 4,250 · see citing works
A Purposeful Approach to the Constant Comparative Method in the Analysis of Qualitative Interviews
Boeije, H. (2002) · Quality and Quantity · doi:10.1023/a:1020909529486 · Cited by 3,073 · see citing works
Greenhalgh, T., Wherton, J., Papoutsi, C., Lynch, J., Hughes, G., A'Court, C., Hinder, S., Fahy, N., Procter, R., & Shaw, S. (2017) · Journal of Medical Internet Research · doi:10.2196/jmir.8775 · Cited by 2,887 · see citing works
The Personal Involvement Inventory: Reduction, Revision, and Application to Advertising
Zaichkowsky, J. L. (1994) · Journal of Advertising · doi:10.1080/00913367.1943.10673459 · Cited by 1,737 · see citing works
Yardley, L., Morrison, L., Bradbury, K., & Muller, I. (2015) · Journal of Medical Internet Research · doi:10.2196/jmir.4055 · Cited by 1,442 · see citing works
Virtual Reality in the Assessment, Understanding, and Treatment of Mental Health Disorders
Freeman, D., Reeve, S., Robinson, A., Ehlers, A., Clark, D., Spanlang, B., & Slater, M. (2017) · Psychological Medicine · doi:10.1017/s003329171700040x · Cited by 1,428 · see citing works
A Holistic Framework to Improve the Uptake and Impact of eHealth Technologies
van Gemert-Pijnen, J. E., Nijland, N., van Limburg, M., Ossebaard, H. C., Kelders, S. M., Eysenbach, G., & Seydel, E. R. (2011) · Journal of Medical Internet Research · doi:10.2196/jmir.1672 · Cited by 1,126 · see citing works
Self-citationDiemer, J., Alpers, G. W., Peperkorn, H. M., Shiban, Y., & Mühlberger, A. (2015) · Frontiers in Psychology · doi:10.3389/fpsyg.2015.00026 · Cited by 959 · see citing works
Michie, S., Yardley, L., West, R., Patrick, K., & Greaves, F. (2017) · Journal of Medical Internet Research · doi:10.2196/jmir.7126 · Cited by 865 · see citing works
Pan, X., & Hamilton, A. F. D. C. (2018) · British Journal of Psychology · doi:10.1111/bjop.12290 · Cited by 738 · see citing works
BMC Medical Informatics and Decision Making
McConnell, P. I., Dash, R., Chilukuri, R., Pietrobon, R. S., Johnson, K. S., Annechiarico, R. P., & Cuticchia, A. J. (2008) · Cited by 678 · see citing works
A Theoretical Extension of the Technology Acceptance Model
VenkateshViswanath, & DavisFred, D. (2000) · Management Science · doi:10.5555/2786232.2786234 · Cited by 601 · see citing works
Conceptualizing vulnerable populations health-related research.
Flaskerud, J., & Winslow, B. J. (1998) · Nursing Research · doi:10.1097/00006199-199803000-00005 · Cited by 563 · see citing works
Design and Evaluation in eHealth: Challenges and Implications for an Interdisciplinary Field
Pagliari, C. (2007) · Journal of Medical Internet Research · doi:10.2196/jmir.9.2.e15 · Cited by 275 · see citing works
Designing eHealth that Matters via a Multidisciplinary Requirements Development Approach
Van Velsen, L., Wentzel, J., & Van Gemert-Pijnen, J. E. (2013) · JMIR Research Protocols · doi:10.2196/resprot.2547 · Cited by 262 · see citing works
Agile science: creating useful products for behavior change in the real world
Hekler, E. B., Klasnja, P., Riley, W. T., Buman, M. P., Huberty, J., Rivera, D. E., & Martin, C. A. (2016) · Translational Behavioral Medicine · doi:10.1007/s13142-016-0395-7 · Cited by 208 · see citing works
Requirements engineering: making the connection between the software developer and customer
Saiedian, H., & Dale, R. (2000) · Information and Software Technology · doi:10.1016/s0950-5849(99)00101-9 · Cited by 183 · see citing works
The Pace of Technologic Change
Patrick, K., Hekler, E. B., Estrin, D., Mohr, D. C., Riper, H., Crane, D., Godino, J., & Riley, W. T. (2016) · American Journal of Preventive Medicine · doi:10.1016/j.amepre.2016.05.001 · Cited by 169 · see citing works
Mullen, P. E. (2000) · The British Journal of Psychiatry · doi:10.1192/bjp.176.4.307 · Cited by 159 · see citing works
Implementation science approaches for integrating eHealth research into practice and policy
Glasgow, R. E., Phillips, S. M., & Sanchez, M. A. (2013) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2013.07.002 · Cited by 155 · see citing works
Greenberg, E., Dunleavy, E. M., & Kutner, M. A. (2007) · National Center for Education Statistics · Cited by 146 · see citing works
Outcomes associated with virtual reality in psychological interventions: where are we now?
Turner, W. A., & Casey, L. M. (2014) · Clinical Psychology Review · doi:10.1016/j.cpr.2014.10.003 · Cited by 116 · see citing works
eHealth in Treatment of Offenders in Forensic Mental Health: A Review of the Current State
Kip, H., Bouman, Y. H. A., Kelders, S. M., & van Gemert-Pijnen, L. J. E. W. C. (2018) · Frontiers in Psychiatry · doi:10.3389/fpsyt.2018.00042 · Cited by 89 · see citing works
Self-citationParticipatory eHealth development to support nurses in antimicrobial stewardship
Wentzel, J., van Velsen, L., van Limburg, M., de Jong, N., Karreman, J., Hendrix, R., & van Gemert-Pijnen, J. J. E. W. C. (2014) · BMC Medical Informatics and Decision Making · doi:10.1186/1472-6947-14-45 · Cited by 73 · see citing works
Renaud, P., Trottier, D., Rouleau, J. L., Goyette, M., Saumur, C., Boukhalfi, T., & Bouchard, S. (2013) · Virtual Reality · doi:10.1007/s10055-013-0235-8 · Cited by 65 · see citing works
The Treatment Motivation Scales for Forensic Outpatient Treatment (TMS-F)
Drieschner, K. H., & Boomsma, A. (2008) · Assessment · doi:10.1177/1073191107311650 · Cited by 55 · see citing works
An Investigation of Presence Response across Variations in Visual Realism
Vinayagamoorthy, Brogni, A., Gillies, M. F. P., Slater, M., & Steed, A. J. (2004) · Cited by 52 · see citing works
Kip, H., Kelders, S. M., Weerink, K. M. A., Kuiper, A., Brüninghoff, I., Bouman, Y. H. A., Dijkslag, D., & van Gemert-Pijnen, L. J. E. W. C. (2019) · Frontiers in Psychology · doi:10.3389/fpsyg.2019.00406 · Cited by 52 · see citing works
Self-citationDugas, M., Trottier, M.-È., Dansokho, S. C., Vaisson, G., Provencher, T., Colquhoun, H. L., Dogba, M. J., Dupéré, S., Fagerlin, A., Giguere, A. M. C., et al. (2017) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-016-0399-8 · Cited by 50 · see citing works
Feldman, S. S., Schooley, B. L., & Bhavsar, G. P. (2014) · JMIR Medical Informatics · doi:10.2196/medinform.3455 · Cited by 44 · see citing works
van Velsen, L., Evers, M., Bara, C.-D., den Akker, H. O., Boerema, S., & Hermens, H. J. (2018) · JMIR Formative Research · doi:10.2196/10474 · Cited by 37 · see citing works
Holistic development of eHealth technology
Kip, H., & van Gemert-Pijnen, L. (. E. W. C. ). (2018) · Routledge eBooks · doi:10.4324/9781315385907-7 · Cited by 36 · see citing works
Self-citationVirtual reality applications in forensic psychiatry
Benbouriche, M., Nolet, K., Trottier, D., & Renaud, P. (2014) · Virtual Reality International Conference · doi:10.1145/2617841.2620692 · Cited by 34 · see citing works
Anwendung virtueller Realitäten in der forensischen Psychiatrie
Fromberger, P., Jordan, K., & Müller, J. L. (2014) · Der Nervenarzt · doi:10.1007/s00115-013-3904-7 · Cited by 34 · see citing works
Diagnostic Comorbidity in Psychotic Offenders and Their Criminal History: A Review of the Literature
Goethals, K., Vorstenbosch, E., & Marle, H. J. C. V. (2008) · International Journal of Forensic Mental Health · doi:10.1080/14999013.2008.9914411 · Cited by 31 · see citing works
Involvement as a Working Mechanism for Persuasive Technology
Kelders, S. M. (2015) · Lecture notes in computer science · doi:10.1007/978-3-319-20306-5_1 · Cited by 25 · see citing works
Self-citationInterpretive Flexibility in Mobile Health: Lessons From a Government-Sponsored Home Care Program
Nielsen, J. A., & Mathiassen, L. (2013) · Journal of Medical Internet Research · doi:10.2196/jmir.2816 · Cited by 25 · see citing works
Beerlage-de Jong, N., Wentzel, J., Hendrix, R., & van Gemert-Pijnen, L. (2017) · American Journal of Infection Control · doi:10.1016/j.ajic.2016.12.001 · Cited by 24 · see citing works
User Expressions Translated into Requirements
Bergvall‐Kåreborn, B., & Ståhlbröst, A. (2010) · Human Technology · doi:10.17011/ht/urn.201011173092 · Cited by 15 · see citing works
Virtual Reality Behavioral Therapy
Kim, B., Schwartz, W., Catacora, D., & Vaughn-Cooke, M. (2016) · Proceedings of the Human Factors and Ergonomics Society Annual Meeting · doi:10.1177/1541931213601081 · Cited by 13 · see citing works
Flagged first, then by citations
Research works printed but not linked: 1
- Catherine Burns. Human-centred design, book section 10. Routledge, Abingdon, 2018.
Cited by
Sorted by citations
Methods for Human-Centered eHealth Development: Narrative Scoping Review
Kip, H., Keizer, J., da Silva, M. C., Beerlage-de Jong, N., Köhle, N., & Kelders, S. M. (2022) · Journal of Medical Internet Research · doi:10.2196/31858 · Cited by 67 · see citing works
Self-citationMethods for Human-Centered eHealth Development: Narrative Scoping Review (Preprint)
Kip, H., Keizer, J., da Silva, M. C., Beerlage-de Jong, N., Köhle, N., & Kelders, S. M. (2021) · doi:10.2196/preprints.31858 · Cited by 67 · see citing works
Self-citationKip, H., Kelders, S. M., Bouman, Y. H. A., & van Gemert-Pijnen, L. J. E. W. C. (2019) · Journal of Medical Internet Research · doi:10.2196/12972 · Cited by 40 · see citing works
Self-citationKip, H., Jong, N. B.-D., van Gemert‐Pijnen, J. E. W. C., & Kelders, S. M. (2024) · Journal of Medical Internet Research · doi:10.2196/59601 · Cited by 41 · see citing works
Self-citationKip, H., & Bouman, Y. H. A. (2021) · Frontiers in Psychiatry · doi:10.3389/fpsyt.2021.703043 · Cited by 13 · see citing works
Self-citationMerino, M., del Barrio, J., Nuño-Solínis, R., & Rodríguez, M. E. (2024) · Digital Health · doi:10.1177/20552076241277438 · Cited by 16 · see citing works
Keizer, J., Jong, N. B.-D., al Naiemi, N., & van Gemert‐Pijnen, J. E. W. C. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-45712-9_9 · Cited by 8 · see citing works
Digital service delivery applications in corrections: A scoping review
Ross, S., Wood, M., Baird, R., & Lundberg, K. (2023) · doi:10.52922/crg77109 · Cited by 7 · see citing works
Gamble, C. J., van Isselt, E. F. V. D., Zwakhalen, S. M. G., Adams, N., Roekx, D. P. H. G., & van Haastregt, J. C. M. (2025) · International Journal of Telerehabilitation · doi:10.5195/ijt.2024.6681 · Cited by 1 · see citing works
Zonneveld, M., Pol, M., Achterberg, W. P., & van Isselt, E. F. V. D. (2025) · INQUIRY The Journal of Health Care Organization Provision and Financing · doi:10.1177/00469580251347139 · Cited by 3 · see citing works
Murray, S., Langener, S., Kip, H., Klaassen, R., Kelders, S. M., Heylen, D., & VanDerNagel, J. (2025) · doi:10.2196/preprints.82601
Self-citationMurray, S., Langener, S., Kip, H., Klaassen, R., Kelders, S. M., Heylen, D., & VanDerNagel, J. (2026) · JMIR XR and spatial computing. · doi:10.2196/82601
Self-citationConsideration of Human Values in Extended Reality: A Systematic Review
Li, M., Zhang, Y., Dwyer, T., Alshehri, T., Evans, J., & Goodwin, S. (2026) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2026.3680698
- Authors
- Kip, H., Kelders, S. M., & Van Gemert-Pijnen, L. J. E. W. C.
- Affiliations
- University of Twente (NL)
- University of Twente, Enschede, Netherlands
- Venue
- Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems
- Publisher
- ACM
- Published
- 2019-05-02
- Pages
- 1-12
- OpenAlex
- W2942068202
- Cited by
- 11
- References
- 41
- Citations by year
- 2026: 2 · 2025: 3 · 2024: 2 · 2021: 2 · 2020: 1 · 2019: 1
- FWCI
- 2.59
- Type
- conference paper
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- CHI Conference on Human Factors in Computing Systems (publishedVersion)
- University of Twente Research Information (submittedVersion, other-oa)
- Topics
- Social Sciences › Psychology › Applied Psychology › Digital Mental Health Interventions
- Physical Sciences › Computer Science › Human-Computer Interaction › Virtual Reality Applications and Impacts
- Physical Sciences › Computer Science › Human-Computer Interaction › Innovative Human-Technology Interaction
- Keywords
- stakeholder perspectives, virtual reality applications, stakeholder engagement, formative assessment
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
