Breaking The Experience: Effects of Questionnaires in VR User Studies
Open accessThe DOI matches its Crossref record; 100 of the 100 references it cites were found in a registry or an index.
Abstract
Questionnaires are among the most common research tools in virtual reality (VR) evaluations and user studies. However, transitioning from virtual worlds to the physical world to respond to VR experience questionnaires can potentially lead to systematic biases. Administering questionnaires in VR (inVRQs) is becoming more common in contemporary research. This is based on the intuitive notion that inVRQs may ease participation, reduce the Break in Presence (BIP) and avoid biases. In this paper, we perform a systematic investigation into the effects of interrupting the VR experience through questionnaires using physiological data as a continuous and objective measure of presence. In a user study (n=50), we evaluated question-asking procedures using a VR shooter with two different levels of immersion. The users rated their player experience with a questionnaire either inside or outside of VR. Our results indicate a reduced BIP for the employed inVRQ without affecting the self-reported player experience.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“The other brain-signal sensing technology in the database is in Shi et al. [84], where a functional near-infrared spectroscopy (fNIRS) device (a NIRSportTM worn on the head) is used along with an HMD integrated eye tracker (Tobii) to measure stress levels among users who were being trained on industrial shutdown maintenance.”
From A systematic review of immersive virtual reality for industrial skills training (2021) · cites this paper as [84]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Measuring Presence in Virtual Environments: A Presence Questionnaire
Witmer, B. G., & Singer, M. J. (1998) · Presence: Teleoperators and Virtual Environments · doi:10.1162/105474698565686 · Cited by 6,371 · cited 12×
“[38, 55, 82, 102]), there is a consensus that presence is a multidimensional construct that is driven by media characteristics, such as technological factors [99, 56, 68, 81], and personal characteristics [4].”
BackgroundMethodInfluential(2013) · Encyclopedia of Creativity, Invention, Innovation and Entrepreneurship · doi:10.1007/978-1-4614-3858-8_100431 · Cited by 4,799
Flow: The Psychology of Optimal Experience
Mirvis, P. H., & Csikszentmihalyi, M. (1991) · Academy of Management Review · doi:10.5465/amr.1991.4279513 · Cited by 3,782
At the Heart of It All: The Concept of Presence
Lombard, M., & Ditton, T. (2006) · Journal of Computer-Mediated Communication · doi:10.1111/j.1083-6101.1997.tb00072.x · Cited by 3,706
Flow: The Psychology of Optimal Experience
Mirvis, P. H., & Csikszentmihalyi, M. (1991) · The Academy of Management Review · doi:10.2307/258925 · Cited by 3,611
The motivational pull of video games: a self-determination theory approach
Ryan, R. M., Rigby, C. S., & Przybylski, A. (2006) · Motivation and Emotion · doi:10.1007/s11031-006-9051-8 · Cited by 3,332 · cited 13×
“With 21 items PENS is similar in length to other questionnaires (e.g. PANAS, IPQ, NASA-TLX) used in previous user studies with IN VRQ S [45, 63, 76].”
MethodInfluentialSlater, M., & Wilbur, S. (1997) · Presence: Teleoperators and Virtual Environments · doi:10.1162/pres.1997.6.6.603 · Cited by 3,034 · cited 1×
“In contrast to presence, immersion is recurrently described as the objective properties of the VE and the applied technology that induce the sense of presence [9, 88, 92].”
BackgroundThe Experience of Presence: Factor Analytic Insights
Schubert, T., Friedmann, F., & Regenbrecht, H. (2001) · Presence: Teleoperators and Virtual Environments · doi:10.1162/105474601300343603 · Cited by 2,303 · cited 14×
“Recent research started taking interest in administering questionnaires in VR. Schwind et al. investigated the effect of filling out questionnaires on presence in VR. 36 participants played a VR shooter with two degrees of realism and filled out 3 presence questionnaires (IPQ [70], SUS [96], and PQ [102]) on a desktop P…”
BackgroundMethodInfluentialFrom presence to consciousness through virtual reality.
Sanchez-Vives, M. V., & Slater, M. (2005) · Nature reviews. Neuroscience · doi:10.1038/nrn1651 · Cited by 1,831 · cited 6×
“extend this definition and describe presence as “how well a person’s behaviour in the VE matches their behaviour in similar circumstances in real life” [88] and argue that how data is displayed – and how the participants are able to interact in VR – is more important for presence than the level of realism [67, 84].”
BackgroundMethodInfluentialHow Immersive Is Enough? A Meta-Analysis of the Effect of Immersive Technology on User Presence
Cummings, J. J., & Bailenson, J. N. (2016) · Media Psychology · doi:10.1080/15213269.2015.1015740 · Cited by 1,749 · cited 2×
“Stereoscopic rendering, resolution, frame rate, field of view, levels of user tracking and fidelity of sensory input are considered as driving factors that enable immersion and thus, facilitate the sense of presence [9, 19, 36, 39, 58, 91, 99].”
Background
Research works printed but not linked: 10
- Cath Dillon, Edmund Keogh, and Jonathan Freeman. 2002. It's Been Emotional: Affect, Physiology, and Presence. In Proceedings of the Fifth Annual International Workshop on Presence, Porto, Portugal.
- Cath Dillon, Edmund Keogh, Jonathan Freeman, and Jules Davidoff. 2000. Aroused and Immersed: The Psychophysiology of Presence. In Proceedings of 3rd International Workshop on Presence, Delft University of Technology, Delft, the Netherlands. 27--28.
- 10.1016/b978-0-08-051574--8.50096-0 · doi:10.1016/b978-0-08-051574--8.50096-0
- Larry F. Hodges, Barbara O. Rothbaum, Rob Kooper, Dan Opdyke, Thomas Meyer, Johannes J. de Graaff, James S. Williford, and Max M. North. 1994. Presence as The Defining Factor in a VR Application: Virtual Reality Graded Exposure in the Treatment of Acrophobia. (1994).
- Wijnand A. IJsselsteijn, Huib de Ridder, Jonathan Freeman, and Steve E. Avons. 2000. Presence: Concept, Determinants, and Measurement. In Electronic Imaging, Bernice E. Rogowitz and Thrasyvoulos N. Pappas (Eds.). San Jose, CA, 520--529. DOI: http://dx.doi.org/10/fgk32n
- Alexander Kunst. 2019. Kinds of Video Games for VR Headsets U.S. 2017. https://www.statista.com/forecasts/790465/kinds-ofvideo-games-us-consumers-would-like-to-use-a-vrheadset-for. (2019).
- Jonathan Lazar, Jinjuan Heidi Feng, and Harry Hochheiser. 2017. Research Methods in Human-Computer Interaction (2 ed.). Morgan Kaufmann, Cambridge, MA.
- 10.5555/2870721.2870732 · doi:10.5555/2870721.2870732
- Mel Slater. 2003. A Note on Presence Terminology. Presence connect 3, 3 (2003), 1--5.
- Mel Slater, Martin Usoh, and Yiorgos Chrysanthou. 1995. The Influence of Dynamic Shadows on Presence in Immersive Virtual Environments. In Virtual Environments '95 (Eurographics), Martin Göbel (Ed.). Springer Vienna, 8--21.
Cited by
A Separate Reality: An Update on Place Illusion and Plausibility in Virtual Reality
Slater, M., Banakou, D., Beacco, A., Gallego, J., Macia-Varela, F., & Oliva, R. (2022) · Frontiers in Virtual Reality · doi:10.3389/frvir.2022.914392 · Cited by 286
“Putze et al. (2020) investigated the relationship between physiological responses and questionnaire administration during and after the VR experience, and the results suggested less disruption to presence for questionnaires administered during the experience, and the result of Graf and Schwind…”
ResultA systematic review of immersive virtual reality for industrial skills training
Radhakrishnan, U., Koumaditis, K., & Chinello, F. (2021) · Behaviour & Information Technology · doi:10.1080/0144929x.2021.1954693 · Cited by 200
“This refers to any factor that can make the user lose their perceived feeling of presence in the virtual environment, for example when they are asked to remove the HMD in between the IVR experience to fi ll in questionnaires (Putze et al. 2020). o Negative.”
BackgroundThe Virtual Reality Questionnaire Toolkit
Feick, M., Kleer, N., Tang, A., & Krüger, A. (2020) · ACM Symposium on User Interface Software and Technology (UIST) · doi:10.1145/3379350.3416188 · Cited by 117
Alamirah, H., Schweiker, M., & Azar, E. (2021) · Building and Environment · doi:10.1016/j.buildenv.2021.108396 · Cited by 109
A Scoping Survey on Cross-reality Systems
Auda, J., Gruenefeld, U., Faltaous, S., Mayer, S., & Schneegaß, S. (2023) · ACM Computing Surveys · doi:10.1145/3616536 · Cited by 107
Examining Design Choices of Questionnaires in VR User Studies
Alexandrovsky, D., Putze, S., Bonfert, M., Höffner, S., Michelmann, P., Wenig, D., Malaka, R., & Smeddinck, J. D. (2020) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376260 · Cited by 101
“This supports prior findings that presence questionnaires are inadequate to assess BIPs [171].”
BackgroundInfluentialSelf-citationUnderstanding and Designing Avatar Biosignal Visualizations for Social Virtual Reality Entertainment
Lee, S., Ali, A. E., Wijntjes, M., & Cesar, P. S. (2022) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3491102.3517451 · Cited by 96
“This allows users to stay immersed in the VR experience [76].”
BackgroundXue, T., Ali, A. E., Zhang, T., Ding, G., & Cesar, P. S. (2021) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2021.3124080 · Cited by 92
“rating [17] panel was embedded in VR to visualize the 9-point scales of valence and arousal, which allows users to stay closer to the context of an ongoing exposure than outside of the VR [43].”
A Critique of Electrodermal Activity Practices at CHI
Babaei, E., Tag, B., Dingler, T., & Velloso, E. (2021) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445370 · Cited by 81
“As a good example of reporting pre-treatment on the skin we refer the reader to Putze et al. [78]: “before placement of the electrodes, the participants cleaned their non-dominant hand with a wet wipe.”
MethodA Survey on Cross‐Virtuality Analytics
Fröhler, B., Anthes, C., Pointecker, F., Friedl-Knirsch, J., Schwajda, D., Riegler, A., Tripathi, S., Holzmann, C., Brunner, M., Jodlbauer, H., et al. (2022) · Computer Graphics Forum · doi:10.1111/cgf.14447 · Cited by 80
“Can I Touch This?”: Survey of Virtual Reality Interactions via Haptic Solutions
Bouzbib, E., Bailly, G., Haliyo, S. D., & Frey, P. (2021) · Conference on l'Interaction Homme-Machine (IHM) · doi:10.1145/3450522.3451323 · Cited by 76
“Surveys for instance compare the results whenever the questionnaires are asked in VR or in the real world [9, 116].”
ResultLuong, T., Martin, N., Raison, A., Argelaguet, F., Diverrez, J.-M., & Lécuyer, A. (2020) · Proceedings - International Symposium on Mixed and Augmented Reality, ISMAR · doi:10.1109/ismar50242.2020.00068 · Cited by 65
“, physiological responses and/or task performance) without requesting any evaluation from the users, which could disrupt their experience [61].”
Mathis, F., Vaniea, K. E., & Khamis, M. (2021) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445478 · Cited by 62
“After entering 16 PINs per condition, participants flled in in-VR questionnaires [4, 81].”
BackgroundMethodRealism and Field of View Affect Presence in VR but Not the Way You Think
Jicol, C., Clarke, C., Tor, E., Dakin, R. M., Lancaster, T. C., Chang, S. T., Petrini, K., O'Neill, E. J., Proulx, M. J., & Lutteroth, C. (2023) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581448 · Cited by 54
“While within-VR measures of presence have their merits, assessing presence with a traditional questionnaire after the VR experience has been shown to not affect obtained scores [74].”
BackgroundFacial Electromyography-based Adaptive Virtual Reality Gaming for Cognitive Training
Reidy, L., Chan, D., Nduka, C. C., & Güneş, H. (2020) · INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION · doi:10.1145/3382507.3418845 · Cited by 54
“performed a systematic investigation into the effects of interrupting the VR experience through a questionnaire either inside or outside of VR [49].”
BackgroundThe Effect of Multisensory Pseudo-Haptic Feedback on Perception of Virtual Weight
Kim, J., Kim, S., & Lee, J. (2022) · IEEE Access · doi:10.1109/access.2022.3140438 · Cited by 54
“After the weight perception task, a survey based on the Witmer-Singer presence questionnaire [36], [37], [50] was conducted.”
MethodDöllinger, N., Wolf, E., Mal, D., Wenninger, S., Botsch, M., Latoschik, M. E., & Wienrich, C. (2022) · Frontiers in Virtual Reality · doi:10.3389/frvir.2022.935449 · Cited by 51
“Although interaction between the experimenter and the user may cause small breaks in presence (Putze et al., 2020), we considered this approach as part of the evaluation, since interaction between patient and therapist would also likely occur in clinical settings and advanced in virtuo interaction to answer questionna…”
BackgroundThe Influence of in-VR Questionnaire Design on the User Experience
Safikhani, S., Holly, M., Kainz, A., & Pirker, J. (2021) · ACM Symposium on Virtual Reality Software and Technology (VRST) · doi:10.1145/3489849.3489884 · Cited by 50
“As found by Putze et al. [14] in-VR questionnaire provides more reliable self-reports in comparison to the out-VR questionnaire.”
BackgroundResultIntelligent autonomous agents and trust in virtual reality
Sun, N., & Botev, J. (2021) · Computers in Human Behavior Reports · doi:10.1016/j.chbr.2021.100146 · Cited by 44
Ubiq-exp: A toolkit to build and run remote and distributed mixed reality experiments
Steed, A. J., Izzouzi, L., Brandstätter, K., Friston, S., Congdon, B. J., Olkkonen, O., Giunchi, D., Numan, N., & Swapp, D. (2022) · Frontiers in Virtual Reality · doi:10.3389/frvir.2022.912078 · Cited by 44
Xue, T., Ali, A. E., Zhang, T., Ding, G., & Cesar, P. S. (2021) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445487 · Cited by 41
“’s [80] fndings that administering questionnaires in VR can reduce breaks in presence and avoid biases.”
BackgroundMethodGiuseppe, E. D., Marcelli, L., Latini, A., & D’Orazio, M. (2024) · Journal of Building Engineering · doi:10.1016/j.jobe.2024.111127 · Cited by 39
ShareYourReality: Investigating Haptic Feedback and Agency in Virtual Avatar Co-embodiment
Venkatraj, K. P., Meijer, W., Perusquía-Hernández, M., Huisman, G., & Ali, A. E. (2024) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642425 · Cited by 38
“These questions were embedded as panels in VR, allowing participants to stay immersed in the VR experience [47].”
Wölfel, M., Hepperle, D., Purps, C. F., Deuchler, J., & Hettmann, W. (2021) · International Conference on Cyberworlds (CW) · doi:10.1109/cw52790.2021.00038 · Cited by 38
Mathis, F., O’Hagan, J., Vaniea, K. E., & Khamis, M. (2022) · International Conference on Advanced Visual Interfaces (AVI) · doi:10.1145/3531073.3531087 · Cited by 35
“, for NASA-TLX [19], SUS [4]) to guide participants through the study and to not break their VR experience [43].”
BackgroundInconsistencies of Presence Questionnaires in Virtual Reality
Graf, S., & Schwind, V. (2020) · ACM Symposium on Virtual Reality Software and Technology (VRST) · doi:10.1145/3385956.3422105 · Cited by 34
“…using an head-mounted display (HMD) requires removing the headset and causes a state of disorientation and confusion while “returning” to the real-world [1, 3, 8], the procedure increases the probability that confounds interfere with the subjective ratings on items of presence questionnaires [4].”
BackgroundTatzgern, M., Domhardt, M., Wolf, M., Cenger, M., Emsenhuber, G., Dinic, R., Gerner, N., & Hartl, A. J. (2022) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3491102.3502090 · Cited by 33
“Studies have also shown that administering questionnaires in VR has the advantage that it avoids breaks in presence [2, 51].”
BackgroundWagener, N., Ackermann, A., Savino, G.-L., Dänekas, B., Niess, J., & Schöning, J. (2022) · INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION · doi:10.1145/3536221.3556622 · Cited by 32
“Although criticised, among others due to its length [72] and not diferentiating enough in their question setting [65], it is the most often used presence measurement in literature [2, 55].”
BackgroundMethodFrommel, J., Phillips, C. J., & Mandryk, R. L. (2021) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445411 · Cited by 31
“In a related context, research has investigated the use of questionnaires in virtual reality (VR) environments, in particular their validity in measuring experience [2, 59, 63] and how to design them [57, 59].”
BackgroundWang, P., Miller, M. R., Queiroz, A. C. M., & Bailenson, J. N. (2024) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642244 · Cited by 30
“…this procedure resembled real-world scenarios of users joining and leaving asynchronous VR social interactions, switching between VR and desktop PC when completing questionnaires can yield disorientation, disturbances, and undesirable effects on the variance of subjective responses [28, 64, 71].”
MethodEvaluating User Experiences in Mixed Reality
Alexandrovsky, D., Putze, S., Schwind, V., Mekler, E. D., Smeddinck, J. D., Kahl, D., Krüger, A., & Malaka, R. (2021) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3411763.3441337 · Cited by 30
“3441337 assessment methods of user experience, like objective measurements using biosignals [26, 27] or behavioural measures [32, 36].”
BackgroundMethodSelf-citationAn Immersive Self-Report Tool for the Affective Appraisal of 360° VR Videos
Toet, A., Heijn, F., Brouwer, A.-M., Mioch, T., & van Erp, J. B. F. (2020) · Frontiers in Virtual Reality · doi:10.3389/frvir.2020.552587 · Cited by 28
“It has also been shown that self-report tools are less invasive and yield more reliable results when they used inside a VR, compared to their application outside the VR (Schwind et al., 2019; Putze et al., 2020).”
BackgroundInvestigating the Correlation Between Presence and Reaction Time in Mixed Reality
Chandio, Y., Bashir, N., Interrante, V. L., & Anwar, F. M. (2023) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2023.3319563 · Cited by 26
“This helps us set a starting point for our study [13,58,67].”
BackgroundAli, A. E., Ney, R., van Berlo, Z. M. C., & Cesar, P. S. (2023) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2023.3320228 · Cited by 26
“After the trials, participants filled in the SUS and IPQ questionnaires within-VR, which allows users to stay closer to the context of an ongoing exposure than outside of VR [62].”
MethodSafikhani, S., Gattringer, V., Schmied, M., Wriessnegger, S. C., & Pirker, J. (2024) · Multimodal Technologies and Interaction · doi:10.3390/mti8110104 · Cited by 25
RemoteLab: A VR Remote Study Toolkit
Lee, J., Natarrajan, R., Rodríguez, S., Panda, P., & Ofek, E. (2022) · ACM Symposium on User Interface Software and Technology (UIST) · doi:10.1145/3526113.3545679 · Cited by 25
“However, research indicates that there is value in including questionnaires as part of the VR experience to reduce break-in-presence, and increase the ecological validity of the participant’s evaluation of a VR task [6, 20].”
BackgroundInfluentialBANS: Evaluation of Bystander Awareness Notification Systems for Productivity in VR
Mansour, S., Knierim, P., O’Hagan, J., Alt, F., & Mathis, F. (2023) · Workshop on Usable Security (USEC) · doi:10.14722/usec.2023.234566 · Cited by 25
“We kept participants in the nested realities (i.e., in VR) to not break their VR experience and to not require them to take off the headset several times during the study [72].”
MethodPhysiological Data for User Experience and Quality of Experience: A Systematic Review (2018–2022)
da Silveira, A. C., de Souza, M. L., Ghinea, G., & Santos, C. A. S. (2024) · International Journal of Human-Computer Interaction · doi:10.1080/10447318.2024.2311972 · Cited by 25
“Moreover, when dealing with immersive experiences like VR, questionnaires and surveys can disrupt user immersion, leading to systematic biases as users transition between the virtual and physical worlds to respond to experience-related inquiries (Putze et al., 2020; Safikhani et al., 2021).”
Lau, K. H. C., Sen, S., Stark, P., Bozkir, E., & Kasneci, E. (2025) · INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION · doi:10.1145/3716553.3750760 · Cited by 25
“Our multimodal evaluation framework, integrating eye-tracking, head movements, and verbal interactions, provides deeper insights than traditional single-modality or self-report approaches common in HCI research [24, 57].”
Virtual Reality Observations: Using Virtual Reality to Augment Lab-Based Shoulder Surfing Research
Mathis, F., O’Hagan, J., Khamis, M., & Vaniea, K. E. (2022) · IEEE Conference on Virtual Reality and 3D User Interfaces (VR) · doi:10.1109/vr51125.2022.00048 · Cited by 25
“We did this to ensure a consistent VR experience and not break participants’ focus [55].”
BackgroundKicking in Virtual Reality: The Influence of Foot Visibility on the Shooting Experience and Accuracy
Bonfert, M., Lemke, S., Porzel, R., & Malaka, R. (2022) · IEEE Conference on Virtual Reality and 3D User Interfaces (VR) · doi:10.1109/vr51125.2022.00092 · Cited by 23
Self-citationPiumsomboon, T., Ong, G., Urban, C., Ens, B. M., Topliss, J., Bai, X., & Hoermann, S. (2022) · Frontiers in Virtual Reality · doi:10.3389/frvir.2022.943696 · Cited by 23
“For example, experience is worsened in an IVE by lowering the fi delity of the environment or, in other words introducing BIP effects (considering presence as an appetitive stimulus).”
MethodInfluentialPlayful User-Generated Treatment: A Novel Game Design Approach for VR Exposure Therapy
Alexandrovsky, D., Volkmar, G., Spliethöver, M., Finke, S., Herrlich, M., Döring, T., Smeddinck, J. D., & Malaka, R. (2020) · Annual Symposium on Computer-Human Interaction in Play · doi:10.1145/3410404.3414222 · Cited by 21
“as world-anchored in-VR questionnaires (inVRQs) [3, 87].”
Self-citationReducing the Human Factor in Virtual Reality Research to Increase Reproducibility and Replicability
Hepperle, D., Dienlin, T., & Wölfel, M. (2021) · IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct) · doi:10.1109/ismar-adjunct54149.2021.00030 · Cited by 21
Tadayyoni, H., Campos, M. S. R., Uribe-Quevedo, Á., & Murphy, B. A. (2024) · Brain Sciences · doi:10.3390/brainsci14050470 · Cited by 19
“Additionally, asking about immersion while the user is inside the virtual environment breaks the immersion, as it distracts the user from their subjective experience [20], and asking about it afterwards makes the results highly dependent on the recollection of the user’s experience [21].”
BackgroundWang, Q., Kang, B. R., & Kristensson, P. O. (2022) · IEEE Conference on Virtual Reality and 3D User Interfaces (VR) · doi:10.1109/vr51125.2022.00070 · Cited by 18
“Future work may want to consider embedding virtual questionnaire forms or using avatars to represent researchers in VR, which may be beneficial to help maintain the impression of a neutral experimental environment [14, 45].”
BackgroundDifferent Types of Questionnaires as a Tool in Placemaking Research
Kimic, K., Polko, P., Hansen, P., & Mehan, A. (2024) · BRILL eBooks · doi:10.1163/9789004691919_007 · Cited by 17
Show, don’t tell: Using Go-along Interviews in Immersive Virtual Reality
Vindenes, J., & Wasson, B. (2021) · Designing Interactive Systems Conference · doi:10.1145/3461778.3462014 · Cited by 17
“For instance, researchers had participants fill out presence questionnaires while still inside the VE [1, 17, 63, 68, 69].”
BackgroundAffective Landscapes: Navigating the Emotional Impact of Multisensory Stimuli in Virtual Reality
Magalhães, M., Melo, M., Coelho, A. F., & Bessa, M. (2024) · IEEE Access · doi:10.1109/access.2024.3499858 · Cited by 16
“…methods of administering questionnaires in VR, such as paper-based or computer-based forms [109], [110], known as out-VR [111], often result in a significant loss of presence [109], [112], [113] due to the shift from virtual to real environments, introducing BiP [114] which can skew results.”
Effects of Image Realism on the Stress Response in Virtual Reality
Weiß, S., Klassen, N., & Heuten, W. (2021) · ACM Symposium on Virtual Reality Software and Technology (VRST) · doi:10.1145/3489849.3489885 · Cited by 16
“In retrospect, we believe that completing the questionnaires in VR will yield better results, as it will avoid a break in presence and the memory of the situation will still be current [26].”
BackgroundCognitive load Classification with a Stroop task in Virtual Reality based on Physiological data
Souchet, A. D., Diallo, M. L., & Lourdeaux, D. (2022) · Proceedings - International Symposium on Mixed and Augmented Reality, ISMAR · doi:10.1109/ismar55827.2022.00083 · Cited by 15
“We adapted the NASA-TLX questionnaire in VR to reduce experiment time and disorientation (by going back and forth between VR and the real world) [62].”
MethodReaction Time as a Proxy for Presence in Mixed Reality with Distraction
Chandio, Y., Interrante, V. L., & Anwar, F. M. (2025) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2025.3549575 · Cited by 15
“(c) Cognitive overload. occurs when users are overwhelmed by task complexity or by processing real and virtual stimuli simultaneously [26], or discontinuation for secondary task [46].”
InfluentialImmersive Emotion Analysis in VR Environments: A Sensor-Based Approach to Prevent Distortion
Joo, J.-H., Han, S., Park, I., & Chung, T. (2024) · Electronics · doi:10.3390/electronics13081494 · Cited by 14
Ongoing Challenges of Evaluating Mulsemedia QoE
da Silveira, A. C., & Santos, C. A. S. (2022) · doi:10.5753/sensoryx.2022.20005 · Cited by 14
“…cheaper, are also a source of criticism: questionnaires and surveys can break user immersion when dealing with immersive experiences as in VR because the transition from the virtual world to the physical world to respond to VR experience questionnaires can lead to systematic biases [23, 24].”
BackgroundAltaf, B., Tavakoli, A., Ruth, P. S., Green, A., Xu, J., Jain, S., Chiu, E., Bencharit, L. Z., Murnane, E. L., Landay, J. A., et al. (2025) · Building and Environment · doi:10.1016/j.buildenv.2025.113147 · Cited by 13
RetroSketch: A Retrospective Method for Measuring Emotions and Presence in Virtual Reality
Potts, D., Gada, M., Gupta, A., Goel, K., Krzok, K. P., Pate, G., Hartley, J. P., Weston-Arnold, M., Aylott, J., Clarke, C., et al. (2025) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713957 · Cited by 12
“In VEs designed to elicit emotions, presence can sometimes be estimated through physiological measures [47] such as ECG and EDA [136, 144, 174, 237], eye movements/pupillometry [116], fEMG [182] or EEG signals [14, 58, 110, 166] because of its association with emotional response [225].”
Mirror Dwellers in Social VR: Investigating Reasons and Perception of Mirror Watching
Chen, Q., Bellucci, A., & Jacucci, G. (2025) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3711033 · Cited by 12
“Studies have shown that extensive text-based materials (e.g., questionnaires) in VR can cause a ‘break in presence’ [61], putting an excessive cognitive load on users.”
Kahl, D., Ruble, M., & Krüger, A. (2022) · IEEE Conference on Virtual Reality and 3D User Interfaces (VR) · doi:10.1109/vr51125.2022.00030 · Cited by 12
“However, advantages of virtual questionnaires are known from VR research [21,22] and we try to minimize the break between the AR experience and the questionnaires that refer to it.”
BackgroundA Virtual Reality Whiteboard System for Remote Collaboration Using Natural Handwriting
Wang, J., & Jing, L. (2022) · Electronics · doi:10.3390/electronics11244152 · Cited by 12
Deuchler, J., Hettmann, W., Hepperle, D., & Wölfel, M. (2023) · IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW) · doi:10.1109/vrw58643.2023.00105 · Cited by 12
“Research has shown that integrating a questionnaire directly within VR instead of presenting it afterwards in the real-world offers several advantages such as less spatial disorientation [7] and reduced systematic bias due to eventual presence-loss [9].”
BackgroundChandio, Y., Interrante, V. L., & Anwar, F. M. (2024) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2024.3372120 · Cited by 12
“Traditional methods of measuring this immersion have relied on post-experience questionnaires [75, 94, 66, 65, 29, 58, 63, 22], capturing users’ retrospective feelings about the VE to assess elements like sensory fidelity and emotional engagement.”
MethodROVER: A Standalone Overlay Tool for Questionnaires in Virtual Reality
Küntzer, L., Schwab, S. U., Spaderna, H., & Rock, G. (2024) · ACM SIGCHI Symposium on Engineering Interactive Computing Systems (EICS) · doi:10.1145/3660515.3661328 · Cited by 11
“The flat 2D canvases are surrounded by 3D frames resembling monitor frames for a more cohesively embedded intradiegetic UI [37, 48], resulting in a more immersive user experience, which has been shown to mitigate BIPs [3, 34, 48].”
BackgroundMethodResultInfluentialShoot Down Drones with Your Answer, an Integration of a Questionnaire into a VR Experience
Tamaki, R., & Nakajima, T. (2021) · Symposium on Spatial User Interaction · doi:10.1145/3485279.3488282 · Cited by 11
“However, Putze et al. argued that the loss of presence still occurs at the transition from the VR experience to the questionnaire [5].”
BackgroundMethodAnswering With Bow and Arrow: Questionnaires and VR Blend Without Distorting the Outcome
Gründling, J. P., Zeiler, D., & Weyers, B. (2022) · IEEE Conference on Virtual Reality and 3D User Interfaces (VR) · doi:10.1109/vr51125.2022.00089 · Cited by 11
“This aligns with findings of Putze et al. [35] who report that the average electrodermal activity (EDA) varies between inVRQ and outVRQ modalities, without having an effect on the answers.”
BackgroundMethodResultInfluentialWagener, N., Stamer, M., Schöning, J., & Tümler, J. (2020) · ACM Symposium on Virtual Reality Software and Technology (VRST) · doi:10.1145/3385956.3418972 · Cited by 11
“In some experimental setups, the VR-scene (mostly a game) was replaced entirely by a new VE, either by showing the questionnaire in front of a plain background [9, 23] or in a virtual replica of the laboratory [19, 30, 35].”
BackgroundMethodInfluentialAkhoroz, M., & Yildirim, C. (2024) · INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION · doi:10.1145/3678957.3685734 · Cited by 10
Haptic Biosignals Affect Proxemics Toward Virtual Reality Agents
Ooms, S., Lee, M., Stepanova, E. R., Cesar, P. S., & Ali, A. E. (2025) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713231 · Cited by 10
“All measures include: (a) Trial duration (b) Objective distance to agent (through position logging) (c) Subjective distance ratings (trial-wise) (d) Arousal ratings (trial-wise) (e) Stimulation comfort ratings (trial-wise) (f) IPQ Presence questionnaire [90] (g) [83].”
Du, H., Chhatre, K., Peters, C., Keegan, B., McDonnell, R., & Ennis, C. (2025) · ACM International Conference on Intelligent Virtual Agents (IVA) · doi:10.1145/3717511.3747074 · Cited by 10
“After each scenario, participants completed a questionnaire within the same immersive setting to maintain their sense of immersion [61].”
Kocur, M., Henze, N., & Schwind, V. (2021) · University of Regensburg Publication Server (University of Regensburg) · doi:10.5283/epub.45543 · Cited by 10
Effects of PCG on Creativity in Playful City-Building Environments in VR
Volkmar, G., Alexandrovsky, D., Eilks, A. E., Queck, D., Herrlich, M., & Malaka, R. (2022) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3549493 · Cited by 11
“To avoid breaks in presence [97], those were presented as world-anchored in-VR questionnaires (inVRQs) [1, 83].”
MethodSelf-citationUnderstanding Software Quality Metrics for Virtual Reality Products - A Mapping Study
Kuri, M., Karre, S. A., & Reddy, Y. R. (2021) · Innovations in Software Engineering Conference (ISEC) · doi:10.1145/3452383.3452391 · Cited by 10
“Susanne et al. conducted the effectiveness of Questionnaires in VR User Studies as a quality aspect and found that questionnaires reduces Break in Presence (BIP) and theoretical bias [59].”
BackgroundMeasuring User Experience of Older Adults during Virtual Reality Exergaming
Küntzer, L., Schwab, S. U., Spaderna, H., & Rock, G. (2024) · International Conference on Quality of Multimedia Experience (QoMEX) · doi:10.1109/qomex61742.2024.10598263 · Cited by 9
“The act of self-reporting in VR likely leads to BIPs independent of the specific method [12].”
BackgroundMethodInfluentialChandio, Y., Interrante, V. L., & Anwar, F. M. (2025) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2025.3549580 · Cited by 8
“Traditional post-experience questionnaires [18,22,50,54,56,57,64,77], often used to measure presence, capture retrospective impressions of sensory fidelity and engagement but can lack real-time insights and may suffer from memory biases [58].”
"I Shot the Interviewer!": The Effects of In-VR Interviews on Participant Feedback and Rapport
Young, J., Ferreira, J., & Pantidi, N. (2024) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642707 · Cited by 8
“…having the participant keep their VR headset on during the interview, as in the Immersed and Avatar conditions, would reduce or remove the break in presence associated with leaving virtual reality [21, 54] while increasing self-reported presence [16] and reducing the variability of results [61].”
InfluentialAkhoroz, M., & Yildirim, C. (2024) · International Conference on Mobile and Ubiquitous Multimedia (MUM) · doi:10.1145/3701571.3701596 · Cited by 8
“We presented the questionnaires in the VE to avoid causing a break in participants’ sense of presence [20] and inducing recollection biases [25].”
MR-RIEW: An MR Toolkit for Designing Remote Immersive Experiment Workflows
Bovo, R., Giunchi, D., Steed, A. J., & Heinis, T. (2022) · IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW) · doi:10.1109/vrw55335.2022.00234 · Cited by 8
Dongas, R., Grace, K. S., Gillespie, S. D., Hoggenmüeller, M., Tomitsch, M., & Worrall, S. (2023) · Multimodal Technologies and Interaction · doi:10.3390/mti7080082 · Cited by 8
“They also “embody self-reports and therefore gather the participants’ subjective experiences” [52].”
Background“Can You Move It?”: The Design and Evaluation of Moving VR Shots in Sport Broadcast
Zhu, X., Wang, C., Guo, Z., Zhao, Y., & Jiao, Y. (2023) · Proceedings - International Symposium on Mixed and Augmented Reality, ISMAR · doi:10.1109/ismar59233.2023.00099 · Cited by 8
“A 5-minute break separated each clip viewing to decrease fatigue [50].”
BackgroundTheory of Mind and Delegation to Robotic Virtual Agents
Sun, N., Botev, J., Khaluf, Y., & Simoens, P. (2022) · IEEE RO-MAN · doi:10.1109/ro-man53752.2022.9900789 · Cited by 8
“The entire experiment, including the CT game and the post-assessment questionnaire, is conducted within the same virtual environment to reduce the break in presence [32] and is distributed as an online browser game for better accessibility.”
MethodQuantifying Social Connection With Verbal and Non-Verbal Behaviors in Virtual Reality Conversations
Kim, H., & Lee, J. (2025) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713674 · Cited by 7
“First, user responses inter-rupt their experience, making it unsuitable for capturing dynamic user experiences in real-time communication [87].”
Why Do We Delegate to Intelligent Virtual Agents? Influencing Factors on Delegation Decisions
Sun, N., & Botev, J. (2021) · International Conference on Human-Agent Interaction (HAI) · doi:10.1145/3472307.3484680 · Cited by 7
“The experiment, including the interaction and the survey, is realized as a Unity WebGL program and entirely conducted in a browserbased virtual environment to avoid breaks in presence [17].”
MethodFoodChoices(Q): Exploring the Design of a Serious Game Proxy for Likert-style Survey Questionnaires
Rogoda, K., Daniszewski, P., Florowski, K., Mathur, R., Amouzgar, K., Mackenzie, J., Sauvé, K., & Karnik, A. (2022) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3549499 · Cited by 7
“HCI researchers have explored mechanisms to investigate and mitigate the disruptive effects of administration of a questionnaire [2, 29] in VR.”
BackgroundTowards Understanding Sense of Inclusion in Social VR Onboarding
Fang, S., & Yatani, K. (2023) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3544549.3585751 · Cited by 6
“To avoid the possible systematic bias in participants’ subjective responses due to break in presence [45], we used VR questionnaires for more reliable self-reports.”
An experimental study of gaze-controlled telepresence robots
Zhang, G., Hansen, J. P., & Minakata, K. (2025) · Behaviour and Information Technology · doi:10.1080/0144929x.2025.2470382 · Cited by 5
“In doing so, the person had to re-orientate in the real world, causing a BIP problem (Jerald 2015), which could distort the subjective experience that the questionnaires are meant to measure (Putze et al. 2020; Schwind et al. 2019).”
The Untapped Potential of Escape Rooms as Gamified Research Environments
Kleinman, E., & Harteveld, C. (2024) · Annual Symposium on Computer-Human Interaction in Play · doi:10.1145/3665463.3678865 · Cited by 5
“Much prior work has sought to find ways to integrate such moments seamlessly into the gamified research experience so as not to interrupt the flow of the gameplay [6, 13, 19, 20].”
A Literature Review and Taxonomy of In-VR Questionnaire User Interfaces
Safikhani, S., Nacke, L. E., & Pirker, J. (2025) · Communications in computer and information science · doi:10.1007/978-3-031-80475-5_7 · Cited by 5
Magalhães, M., Melo, M., Coelho, A. F., & Bessa, M. (2025) · Computers & Graphics · doi:10.1016/j.cag.2025.104443 · Cited by 5
MEEC: First Workshop on Momentary Emotion Elicitation and Capture
Ali, A. E., Perusquía-Hernández, M., Denman, P., Abdelrahman, Y., Hassib, M. E., Meschtscherjakov, A., Ferreira, D., & Henze, N. (2020) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3334480.3375175 · Cited by 5
Use of Virtual Reality Technology for CANDU 6 Reactor Fuel Channel Operation Training
Fan, Z., Brown, K. M., Nistor, S., Seepaul, K., Wood, K., Uribe-Quevedo, Á., Perera, S., Waller, E. J., & Lowe, S. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-63464-3_9 · Cited by 5
Hu, Y.-H., Yokoi, S., Hatada, Y., Hiroi, Y., Narumi, T., & Hiraki, T. (2026) · Frontiers in Virtual Reality · doi:10.3389/frvir.2026.1847904 · Cited by 4
Ratcliffe, J., & Tokarchuk, L. N. (2022) · Frontiers in Computer Science · doi:10.3389/fcomp.2022.952996 · Cited by 4
“Research into how surveying participants directly in VR is already been conducted (Schwind et al., 2019; Alexandrovsky et al., 2020; Putze et al., 2020) without adverse impacts on outcomes, and with the additional benefits of reducing a study’s duration and reducing participant disorientation (especially if, after a s…”
BackgroundSouchet, A. D., Diallo, M. L., & Lourdeaux, D. (2023) · IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct) · doi:10.1109/ismar-adjunct60411.2023.00115 · Cited by 4
“We adapted the STAI-6 questionnaire in VR to reduce experiment time and disorientation [23].”
MethodZhang, L., Cummings, J. J., Liu, X., Mu, D., & Sun, B. (2024) · Media Psychology · doi:10.1080/15213269.2023.2294282 · Cited by 4
“With respect to physiological measures of arousal, Putze et al. (2020) provide perhaps the only direct investigation to date.”
Effects of Different Tracker-driven Direction Sources on Continuous Artificial Locomotion in VR
Lougiakis, C., Mandilaras, T., Katifori, A., Ganias, G., Ioannidis, I. P., & Roussou, M. (2024) · ACM Symposium on Virtual Reality Software and Technology (VRST) · doi:10.1145/3641825.3687735 · Cited by 4
ScryVR: A Systematic Framework for Accelerating Experimental Research in VR
Scully, L., Cerna, J. T., Chu, P., Harris, F. C., & Dascalu, S. M. (2025) · Electronic Imaging · doi:10.2352/ei.2025.37.13.ervr-168 · Cited by 4
Perceived environmental safety and alertness in virtual urban environments
Jedon, R., Haans, A., & de Kort, Y. A. W. (2025) · Journal of Transport & Health · doi:10.1016/j.jth.2025.102138 · Cited by 4
Michailidis, L. (2021) · Bournemouth University Research Online (Bournemouth University) · Cited by 4
Exploring the Design Space for Security Warnings in Immersive Environments
Mengascini, A., Weil, R., Walle, A., Steimle, J., & Pellegrino, G. (2025) · IEEE European Symposium on Security and Privacy · doi:10.1109/eurosp63326.2025.00022 · Cited by 3
“It was administered within the VR environment, following established methodology [5], to avoid potential bias from exiting the virtual environment before responding [113], [121].”
Design and Evaluation of a Continuous Interface for Real-time Self-reporting of VR Sickness
Plaza, M., Real, C., Serrano, A., & Gutiérrez, D. (2026) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3772363.3798515 · Cited by 3
“Recent research has proposed embedding questionnaires or scales directly inside the VR environment [29, 36, 40], developing tools that facilitate less intrusive and more frequent data collection [26, 31, 41, 49].”
Tulaskar, R., Opel, A., Turunen, M., Erdal, I. J., & Gulbrandsen, A. K. (2025) · Norsk medietidsskrift · doi:10.18261/nmt.31.2-3.7 · Cited by 3
Metadata
- Authors
- Putze, S., Alexandrovsky, D., Putze, F., Höffner, S., Smeddinck, J. D., & Malaka, R.
- Affiliations
- University of Bremen (DE)
- Newcastle University (GB)
- ORCID
- Venue
- Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems
- Publisher
- ACM
- Published
- 2020-04-21
- Pages
- 1-15
- OpenAlex
- W3020477709
- Cited by
- 138
- References
- 99
- Citations by year
- 2026: 12 · 2025: 27 · 2024: 22 · 2023: 14 · 2022: 28 · 2021: 24 · 2020: 11
- FWCI
- 14.06
- Type
- conference paper
- Language
- en
- Open access
- green
- doi.org
- Locations
- CHI Conference on Human Factors in Computing Systems (publishedVersion)
- OSF Preprints (OSF Preprints) (submittedVersion)
- Newcastle University ePrints (Newcastle Univesity)
- Funders
- Horizon 2020 (688244)
- DFG (SFB 1320, 402779631)
- Topics
- Physical Sciences › Computer Science › Human-Computer Interaction › Virtual Reality Applications and Impacts
- Physical Sciences › Computer Science › Human-Computer Interaction › Interactive and Immersive Displays
- Physical Sciences › Computer Science › Human-Computer Interaction › Innovative Human-Technology Interaction
- Keywords
- presence, questionnaire administration, immersion level
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.