CoChat
Sign Up
PreprintCited by 3

ChatGPT vs. Google: A Comparative Study of Search Performance and User Experience

Xu, Ruiyun, Feng, Yue, Chen, Hailiang·2023·arXiv
DOI 10.48550/arxiv.2307.01135arXiv 2307.01135
Open this paper in CoChat — free account

Abstract

The advent of ChatGPT, a large language model-powered chatbot, has prompted questions about its potential implications for traditional search engines. In this study, we investigate the differences in user behavior when employing search engines and chatbot tools for information-seeking tasks. We carry out a randomized online experiment, dividing participants into two groups: one using a ChatGPT-like tool and the other using a Google Search-like tool. Our findings reveal that the ChatGPT group consistently spends less time on all tasks, with no significant difference in overall task performance between the groups. Notably, ChatGPT levels user search performance across different education levels and excels in answering straightforward questions and providing general solutions but falls short in fact-checking tasks. Users perceive ChatGPT's responses as having higher information quality compared to Google Search, despite displaying a similar level of trust in both tools. Furthermore, participants using ChatGPT report significantly better user experiences in terms of usefulness, enjoyment, and satisfaction, while perceived ease of use remains comparable between the two tools. However, ChatGPT may also lead to overreliance and generate or replicate misinformation, yielding inconsistent results. Our study offers valuable insights for search engine management and highlights opportunities for integrating chatbot technologies into search engine designs.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2307.01135

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • The ChatGPT group spent less time on all tasks, and the difference between the two groups is not significant.

    “The results show that the ChatGPT group spent less time on all tasks and the difference between these two groups is not significant.”

    From A Survey on Evaluation of Large Language Models (2024) · cites this paper as [232]

    Section 4.2 (Search Efficiency) supports the claim that the ChatGPT group spent less time on each task. It directly contradicts the claim that the time differences were not significant: the paper says they were statistically significant at the 1% level across all three tasks. (LLM, conf=0.95)

    Checked by a reasoning model — may be wrong.

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

No references could be resolved yet.

Research works printed but not linked: 21

  1. Adipat, B., Zhang, D., and Zhou, L. 2011. "The Effects of Tree-view based Presentation Adaptation on Mobile Web Browsing," MIS Quarterly (35:1), pp. 99-121.
  2. Brin, S., Page, L. 1998. "The Anatomy of a Large-Scale Hypertextual Web Search Engine," Computer Networks and ISDN Systems (30), pp. 107-117.
  3. Croft, W. B., Metzler, D., & Strohman, T. (2010). Search engines: Information retrieval in practice (Vol. 520, pp. 131-141). Reading: Addison-Wesley.
  4. Dowling, M., & Lucey, B. (2023). ChatGPT for (finance) research: The Bananarama conjecture. Finance Research Letters, 53, 1-6.
  5. Eloundou, T., Manning, S., Mishkin, P., & Rock, D. (2023). GPTs are GPTs: An Early Look at the Labor Market Impact Potential of Large Language Models. arXiv:2303.10130. Working Paper. · arXiv
  6. Goodwin (2021). A Complete Guide to the Google Panda Update: 2011-21. Search Engine Journal. Accessed on June 25, 2023.
  7. Hopkins, A. M., Logan, J. M., Kichenadasse, G., & Sorich, M. J. (2023). Artificial intelligence chatbots will revolutionize how cancer patients access information: ChatGPT represents a paradigm-shift. JNCI Cancer Spectrum, 7(2), 1-3.
  8. Hutson, M. (2022). Could AI help you to write your next paper?. Nature, 611(7934), 192-193.
  9. Jo, A. (2023). The promise and peril of generative AI. Nature, 614(1), 214-216.
  10. Kleinberg, J. M. 1999. "Authoritative Sources in a Hyperlinked Environment," Journal of the ACM (46:5), pp. 604-632.
  11. Liu, J. (2021). Deconstructing search tasks in interactive information retrieval: A systematic review of task dimensions and predictors. Information Processing & Management, 58(3), 1- 17.
  12. Liu, J., Sarkar, S., & Shah, C. (2020). Identifying and predicting the states of complex search tasks. In Proceedings of the 2020 conference on human information interaction and retrieval (pp. 193-202).
  13. Lopez-Lira, A., & Tang, Y. (2023). Can ChatGPT forecast stock price movements? Return predictability and large language models. arXiv preprint arXiv:2304.07619. Working Paper. · arXiv
  14. Montti (2022). Google's Hummingbird Update: How It Changed Search. Search Engine Journal. Accessed on June 25, 2023.
  15. Pokorný, J. 2004. "Web Searching and Information Retrieval," Computing in Science & Engineering (6:4), pp. 43-48.
  16. Schwartz (2016). Google updates Penguin, says it now runs in real time within the core search algorithm. Search Engine Land. Accessed on June 25, 2023.
  17. Seach Engine Journal (2023). History of Google Algorithm Updates. Accessed on June 25, 2023.
  18. Storey, V. C., Burton-Jones, A., Sugumaran, V., and Purao, S. 2008. "CONQUER: Methodology for Context-Aware Query Processing on the World Wide Web," Information Systems Research (19:1), pp. 3-25.
  19. Van Dis, E. A., Bollen, J., Zuidema, W., van Rooij, R., & Bockting, C. L. (2023). ChatGPT: Five priorities for research. Nature, 614(7947), 224-226.
  20. Wu, S., Irsoy, O., Lu, S., Dabravolski, V., Dredze, M., Gehrmann, S., Kambadur, P., Rosenberg, D., & Mann, G. (2023). BloombergGPT: A Large Language Model for Finance. arXiv:2303.17564. Working Paper. · arXiv
  21. Van Bulck, L., & Moons, P. (2023). What if your patient switches from Dr. Google to Dr. ChatGPT? A vignette-based survey of the trustworthiness, value, and danger of ChatGPT-generated responses to health questions. European Journal of Cardiovascular Nursing, 00, 1-4.

Cited by

Sorted by citations

Authors
Xu, R., Feng, Y., & Chen, H.
Affiliations
  • Rayna
  • Katherine
Venue
arXiv
Publisher
arXiv
Published
2023
DOI
10.48550/arxiv.2307.01135
arXiv
2307.01135
References
0
Type
Preprint
Licence
creativecommons.org/licenses/by/4.0/legalcode

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review