CoChat
Sign Up
Cited by 29Information Retrieval and Search BehaviorUsability and User Interface DesignInnovative Human-Technology Interaction

A Mixed-Method Analysis of Text and Audio Search Interfaces with Varying Task Complexity

Open access
Vtyurina, Alexandra, Clarke, Charles L. A., Law, Edith, Trippas, Johanne R., Bota, Horatiu·2020·Proceedings of the 2020 ACM SIGIR on International Conference on Theory of Information Retrieval
DOI 10.1145/3409256.3409822
Open this paper in CoChat — free accountOpen PDF

Abstract

Voice-based assistants have become a popular tool for conducting web search, particularly for factoid question answering. However, for more complex web searches, their functionality remains limited, as does our understanding of the ways in which users can best interact with audio-based search results. In this paper, we compare and contrast user behaviour through the representation of search results over two mediums: text and audio. We begin by conducting a crowdsourced study exposing the differences in user selection of search results when those are presented in text and audio formats. We further confirm these differences and investigate the reasons behind them through a mixed-methods laboratory study. Through a qualitative analysis of the collected data, we produce a list of guidelines for an audio-based presentation of search results.

Integrity signals

Registry

Crossref · 10.1145/3409256.3409822

Cites

36 of 36 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

3 of 9 cited sources checked · 1 supported · 0 flagged · accuracy not yet measured

  • Premisecitation

    Current state-of-the-art voice search systems perform well for factoid or simple questions, where an exact answer can be read out loud and easily digested by the listener.

    “Current state-of-the-art voice search systems perform well for factoid or simple questions, where an exact answer can be read out loud and easily digested by the listener [26].”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites [26] 26. Gary Ren, Xiaochuan Ni, Manish Malik, and Qifa Ke. 2018. Conversational Query Understanding Using Sequence to Sequence Modeling (WWW '18). 1715-1724. source

      The Introduction (page unavailable) says search and question-answering systems are designed for standalone queries and that reformulating conversational queries can make them understandable to those systems. It does not establish that voice search systems perform well on factoid or simple questions, or that their answers can be read aloud and easily digested. (LLM, conf=0.73)

      Experimental — accuracy not yet measured
  • Findingown data

    There were significant differences between all five NASA-TLX scales between Text and Audio conditions, with medium effect size.

    “We found that there were significant differences between all five scales and medium effect size (calculated using Cohen's d) [20], as shown in Table 2.”
    • Also stated in: § 5 QUALITATIVE FINDINGS · plain
      “Wilcoxon Signed Rank (W) test showed that for all scales the differences in scores between Audio and Text conditions are unlikely due to chance.”
    • Also stated in: § 5 QUALITATIVE FINDINGS · plain
      “Cohen's d (d) values correspond to medium effect size [20]”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites [20] 20. Daniël Lakens. 2013. Calculating and reporting effect sizes to facilitate cumulative science: a practical primer for t-tests and ANOVAs. Frontiers in psychology 4 (2013), 863. source
  • Findingown data

    Participants tended to refer back to selected results using a word, a phrase, or the result's source.

    “Participants tended to refer back to the results using a word, a phrase, or the result's source.”
    Quote in the paperNot a substantial claim
  • Findingown data

    There is no significant interaction between the complexity of a search task and the medium, both in terms of search result preference and perceived workload.

    “Regarding RQ2, our analysis does not reveal a significant interaction between the complexity of a search task and the medium, both in terms of search result preference and perceived workload.”
    Quote in the paperSubstantial claim
  • Findingown data

    Some participants had difficulty distinguishing between the title, URL, and snippet when listening to audio results.

    “In particular, some participants had difficulty distinguishing between the title, URL, and the snippet when listening to the audio results.”
    Quote in the paperSubstantial claim
  • Classified as probably not a substantial claim. 18
  • May combine two claims. 12
  • The quote only partly states this claim. 1

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    The level of mental effort is likely dependent on users' working memory; prior work found an effect between working memory level and search process outcome.

    “It is not unlikely that the level of mental effort is dependant on the users' working memory: prior work found an effect between the level of working memory and the outcome of a search process [7].”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites [7] 7. Bogeum Choi, Robert Capra, and Jaime Arguello. 2019. The Effects of Working Memory during Search Tasks of Varying Complexity. In CHIIR. ACM, 261-265. source

      The Abstract and Discussion report that participants with higher working memory exerted more search effort and had better outcomes, particularly on complex tasks. But the Discussion says working memory had no significant effect on post-task workload measures, so the proposed dependence of mental effort on working memory is not established. (LLM, conf=0.84)

      Experimental — accuracy not yet measured

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

    Findingnone given

    Aspects such as pre-determined snippet length, fully formed sentences, and lack of redundancy across caption parts are highly important for user perception of audio-based search results.

    “In this work, we confirm that aspects such as pre-determined snippet length, fully formed sentences, and lack of redundancy across caption parts are highly important for user perception of audio-based search results.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    Another source of uncertainty was the unknown length of the audio captions.

    “Another source of uncertainty was the unknown length of the audio captions.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Task complexity has a main effect on the probability of selecting the top-ranked result as most useful, with the Understand complexity level leading to fewest most useful choices consistent with true result ranking.

    “We note that, although complexity has a main effect on the probability of selecting the top-ranked result as most useful (with the Understand complexity level leading to fewest most useful choices consistent with true result ranking), our analysis did not reveal an interaction effect between task complexity and the medium.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    Twelve participants talked about a specific search result referring to a genre of the result.

    “Additionally, twelve people talked about a specific search result referring to a genre of the result”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    In both studies, differences between Text and Audio conditions in users' ability to identify the top-ranked search result as most useful are present but not statistically significant.

    “In both our studies, differences between Text and Audio conditions related to users' ability to identify the top-ranked search result as most useful are present, but not significant.”
    • Also stated in: § 6 CONCLUSION · plain
      “However, the differences in picking only the most useful result were not significant.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Skimming through results was impossible in the Audio interface, as noted by sixteen participants who said reading through results felt faster than listening.

    “Skimming through results was impossible in the Audio interface, which was noted by sixteen people, who said that reading through results felt faster than listening to them.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    Twenty participants used a multi-word phrase to refer to a result.

    “Twenty people used a multi-word phrase for the same purpose.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Findingown data

    Qualitative analysis of rich interview data with LAB study participants revealed a number of aspects that should be considered by designers of future end-to-end voice-based search systems.

    “Notably, our qualitative analysis of rich interview data with the LAB study participants revealed a number of aspects that should be considered by designers of future end-to-end voice-based search systems.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    No such comments about repetitions were made for the text condition, though the content was identical.

    “Interestingly, no such comments were made for text condition, though the content was identical, which leads us to assume that audio is a more sensitive medium in this respect.”
    Quote in the paperSubstantial claim

  • The quote only partly states this claim.

    Findingown data

    Some of the same aspects are important for sighted users when processing audio search captions as for blind users.

    “In this work, we find some of the same aspects being of importance for sighted users when processing audio search captions.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Voice-based interaction systems, including voice-activated personal assistants, are steadily increasing in popularity.

    “Voice-based interaction systems, and voice-activated personal assistants in particular (e.g., Amazon Alexa), are steadily increasing in popularity [12].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [12] 12. Forbes. 2018. Okay, Google, Will Voice Be The Future Of Search? Retrieved October, 2019 from https://www.forbes.com/sites/nicolemartin1/2018/11/06/ok- google-will-voice-be-the-future-of-search/

      Not checked in the free run.

  • May combine two claims.

    Study design and conductnone given

    The queries used came from a standardised dataset and were not reflective of the users' information needs.

    “Notably, the queries we used came from a standardised dataset and were not reflective of the users' information needs, in addition, the lack of the full search process (query reformulation and access to full documents) could have produced such effect.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductown data

    For the LAB study, 37 participants were recruited from the local university, and data for one person who did not fully understand the instructions were excluded.

    “For LAB study, we recruited 37 participants from the local university, of which the data for one person, who did not fully understand the instructions, were excluded.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductown data

    After removing submissions that did not pass the quality check, data from 69 crowdworkers were collected for the AMT study.

    “After removing submissions that did not pass our quality check, we collected data from 69 crowdworkers.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Interpretationown data

    In an end-to-end voice-based web search system, users will ultimately select a result to hear fully, the voice equivalent of clicking on a result.

    “In an end-to-end voice-based web search system, the users will ultimately select a result to hear fully, the voice equivalent of clicking on a result.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    The order of search tasks and interfaces was counterbalanced using a Latin square design, such that each task occurred with every interface.

    “rotating them in a Latin square design, such that each task occurred with every interface.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Study design and conductnone given

    A golden task was included at the end of the AMT study, always in Audio condition, with two relevant and three non-relevant results; submissions were considered acceptable if the two relevant results were selected as the two most useful.

    “Additionally, we included an”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Study design and conductreasoning

    To ensure a high quality of submissions, the participant pool was restricted to workers with approval rating 95% or higher, who have completed more than 1,000 HITs, and lived in the US.

    “To ensure a high quality of submissions, we restricted the participant pool to workers with approval rating 95% or higher, who have completed more than 1,000 HITs, and lived in the US.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationown data

    The mean average difference and its associated 95% confidence interval, in both studies, is entirely positive, and the null hypothesis of no differences between Text and Audio conditions with respect to the average number of consistent choices is rejected at the 95% confidence level.

    “Furthermore, the mean average difference and its associated 95% confidence interval, in both studies, is entirely positive -under the null hypothesis this difference is expected to be 0 -and therefore we reject the null hypothesis of no differences between Text and Audio conditions with respect to the average number of consistent choices at the 95% confidence level.”
    Quote in the paperSubstantial claim
    • The authors state this as an assumption: The 1st, 5th, and 10th Google search results will be more relevant than the 50th and 100th results.
    • Rests on an assumption: Google's SERP ranking is a valid ground truth for the true ranking of search results.

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

    Premisecitation

    One of the most attractive features of a voice assistant is its ability to support hands-free interaction and multitasking.

    “One of the most attractive features of a voice assistant is its ability to support hands-free interaction and multitasking [22].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [22] 22. Ewa Luger and Abigail Sellen. 2016. Like having a really bad PA: the gulf between user expectation and experience of conversational agents. In CHI. ACM, 5286- 5297. source

  • Classified as probably not a substantial claim.

    Study design and conductnone given

    Judgements of workers who did not listen to all clips in the Audio condition were discarded.

    “Finally, we discarded judgements of the workers who did not listen to all clips in the Audio condition.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Study design and conductreasoning

    The lack of the full search process (query reformulation and access to full documents) could have produced the lack of interaction effect.

    “Notably, the queries we used came from a standardised dataset and were not reflective of the users' information needs, in addition, the lack of the full search process (query reformulation and access to full documents) could have produced such effect.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Premisecitation

    More complex tasks lead to greater levels of search interactivity, for example, through increased clicks, queries, and time on task.

    “Prior research in visual text search has shown that more complex tasks lead to greater levels of search interactivity, for example, through increased clicks, queries, and time on task [18].”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [18] 18. Diane Kelly, Jaime Arguello, Ashlee Edwards, and Wan-Ching Wu. 2015. Devel- opment and Evaluation of Search Tasks for IIR Experiments using a Cognitive Complexity Framework. In ICTIR. 101-110. source

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductcitation

    Three researchers analysed the transcribed interviews and jointly developed a codebook using the method of affinity diagramming.

    “Three researchers then analysed the transcribed interviews and jointly developed a codebook using the method of affinity diagramming [21].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [21] 21. Jonathan Lazar, Jinjuan Heidi Feng, and Harry Hochheiser. 2017. Research methods in human-computer interaction. Morgan Kaufmann.

      Not checked in the free run.

25 of 25

  • The paper relies on this without support.

    Little is known about how people perceive voice-only search captions.

    “Despite the popularity of voice interfaces, little is known about how people perceive voice-only search captions.”

  • The paper relies on this without support.

    For more complex questions, a voice assistant may redirect its user to a companion app (usually phone-based), where search results will be displayed on the screen.

    “For more complex questions, a voice assistant may redirect its user to a companion app (usually phone-based), where search results will be displayed on the screen.”

  • The authors state this as an assumption.

    The 1st, 5th, and 10th Google search results will be more relevant than the 50th and 100th results.

    “with the assumption that the 1st, 5th, and 10th results will be more relevant than 50th and 100th.”

  • The paper relies on this without support.

    Google's SERP ranking is a valid ground truth for the true ranking of search results.

    “we assert that participant choices are consistent with the true ranking of the results (i.e., the ranked result position on Google's SERP)”

  • The paper relies on this without support.

    For more complex web searches, voice-based assistants' functionality remains limited.

    “However, for more complex web searches, their functionality remains limited, as does our understanding of the ways in which users can best interact with audio-based search results.”

  • The authors state this as an assumption.

    Further improvements in audio representation of document surrogates can result in the emergence of successful voice-based search interfaces.

    “We therefore assume that further improvements in audio representation of document surrogates can result in the emergence of successful voice-based search interfaces.”

  • The authors state this as an assumption.

    Audio is a more sensitive medium than text with respect to repetitions.

    “Interestingly, no such comments were made for text condition, though the content was identical, which leads us to assume that audio is a more sensitive medium in this respect.”

  • The paper relies on this without support.

    The top-ranked Google result represents the true ranking of search results.

    “user relevance judgements in Text condition are significantly more consistent with the results' true ranking than those in Audio condition.”
8 of 8

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

  • Conclusion denies a reported task-complexity effect

    Major

    The results report a main effect of task complexity on selecting the top-ranked result, which is a measure of search-result preference. The conclusion instead says task complexity had no effect on search-result preferences.

  • Seven task IDs are listed for a six-task study

    Minor

    The methods state that six search tasks were selected, but the subsequent list contains seven task IDs.

2 of 2

Cites

Flagged first, then by citations

Research works printed but not linked: 1

  1. Jonathan Lazar , Jinjuan Heidi Feng, and Harry Hochheiser . 2017 . Research methods in human-computer interaction .Morgan Kaufmann . Jonathan Lazar, Jinjuan Heidi Feng, and Harry Hochheiser. 2017. Research methods in human-computer interaction .Morgan Kaufmann.

Cited by

Sorted by citations

Related

Authors
Vtyurina, A., Clarke, C. L. A., Law, E., Trippas, J. R., & Bota, H.
Affiliations
  • University of Waterloo (CA)
  • The University of Melbourne (AU)
  • Independent Researcher, Cluj-Napoca, Romania
ORCID
Venue
Proceedings of the 2020 ACM SIGIR on International Conference on Theory of Information Retrieval
Publisher
ACM
Published
2020-09-14
Pages
61-68
DOI
10.1145/3409256.3409822
OpenAlex
W3048731759
Semantic Scholar
fcc0269773793e450232878845410d65e3a662d5
Cited by
29
References
36
Citations by year
2026: 1 · 2025: 3 · 2024: 21 · 2023: 2 · 2022: 1 · 2021: 1
FWCI
0.76
Type
conference paper
Language
en
Open access
Licence
www.acm.org/publications/policies/copyright_policy#Background (vor)
Locations
  • ACM SIGIR International Conference on Theory of Information Retrieval (ICTIR) (publishedVersion)
  • Minerva Access (University of Melbourne) (submittedVersion)
Topics
  • Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
  • Physical Sciences › Computer Science › Human-Computer Interaction › Usability and User Interface Design
  • Physical Sciences › Computer Science › Human-Computer Interaction › Innovative Human-Technology Interaction
Keywords
task complexity, factoid question answering, voice assistants, user behavior

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-11.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-11.

Cited sources are checked by gpt-6-sol.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review