Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat
Cited by 6

The DOI matches its Crossref record; 2 of the 2 references it cites were found in a registry or an index.

Effectiveness of Google keyword suggestion on users’ relevance judgment: A mixed method approach to query expansion

Fattahi, Rahmatollah, Parirokh, Mehri, Dayyani, Mohammd Hosien, Khosravi, Abdolrasoul, Zareivenovel, Mojgan · 2016 · The Electronic Library

Abstract

Purpose: – One of the most effective ways information retrieval (IR) systems including Web search engines can improve relevance performance is to provide their users with tools for facilitating query expansion. Search engines such as Google provide users with keyword suggest tools. This paper aims to investigate users’ criteria in relevance judgment regarding Google’s keywords suggest tool and to see how such keywords would lead to more relevant results from the viewpoint of users. Design/methodology/approach: – Through a mixed method approach, quantitative and qualitative data were collected from 60 postgraduate students at Ferdowsi University of Mashhad, Iran, using four different instruments (questionnaire, thinking aloud technique, query logs and interviews). Findings: – Among other criteria, the “relation between suggested keywords and the information need” (with the mean rate of 3.53 of four) was considered the most important by searchers in selecting suggested keywords for query expansion. Also, the “relation between suggested Keywords and the retrieved items” (with the mean rate of 3.62) was considered the second most important criterion in judging the relevance of the retrieved results. The participants agreed that the suggested keywords by Google improved the retrieval relevance. The content analysis of the participants’ aloud-thinking sessions and the interviews approved such findings. Originality/value: – This research makes a contribution to the need of designers of IR systems regarding the use of add words for query expansion. It also helps librarians how to instruct searchers with expanding their queries to retrieve more relevant results. Another contribution of the study is the identification of a number of new relevance judgment criteria for Web-based environments.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1108/el-03-2015-0035

Cites

2 of 2 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 2 references

Research works printed but not linked: 38

  1. Abam, Z. (2010), “The study of users’ browsing behavior and relevance judgment in web-based meta-link environment”, PhD Dissertation, Library and Information Science, Ferdowsi University of Mashhsad, Mashhsad
  2. Ahn, M. (2003), “Exploring factors affecting users’ link-following decision and evaluation behavior during web browsing”, PhD Thesis, School of Information Studies, University of Pittsburgh, Pittsburgh, PA.
  3. Andreou, A. (2005), “Ontologies and query expansion”, Master’s thesis, School of Informatics, University of Edinburgh, Edinburgh.
  4. Anick, P. (2003), “Using terminological feedback for web search refinement: a log-based study”, Proceedings of the 26th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, ACM, Bethesda, MD, pp. 88-95. · DOI · Paper page
  5. Baeza-Yates, R. , Hurtado, C.A. and Mendoza, M. (2004), “Query recommendation using query logs in search engines”, in Lindner, W. , Mesiti, M. , Türker, C. , Tzitzikas, Y. and Vakali, A. (Eds), EDBT Workshops , Springer, New York, NY, pp. 588-596. · DOI · Paper page
  6. Barry, C.L. (1994), “User-defined relevance criteria: an exploratory study”, Journal of the American Society for Information Science , Vol. 45 No. 3, pp. 149-159. · DOI · Paper page
  7. Barry, C.L. and Schamber, L. (1998), “User’s criteria for relevance evaluation: a cross-situational comparison”, Information Processing and Management , Vol. 34 Nos 2/3, pp. 219-236. · DOI · Paper page
  8. Borland, P. (2003), “The concept of relevance”, Journal of the American Society for Information Science & Technology , Vol. 54 No. 10, pp. 913-925. · DOI · Paper page
  9. Chen, Z. and Xu, Y. (2005), “User-oriented relevance judgment: a conceptual model”, paper presented at Proceedings of the 38th Hawaii International Conference on System Sciences, Hawaii, HI.
  10. Choi, Y. and Rasmussen, E.M. (2002), “User’s relevance criteria in image retrieval in American history”, Information Processing and Management , Vol. 38 No. 50, pp. 695-726. · doi:10.1016/s0306-4573(01)00059-0
  11. Chowdhury, A. and Soboroff, I. (2002), “Automatic evaluation of World Wide Web search services”, Proceedings of the 25th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, ACM, Bethesda, MD, pp. 421-422. · doi:10.1145/564376.564474
  12. Chu, H. (2010), “Factor affecting relevance judgment: a report from TREC legal track”, Journal of Documentation , Vol. 67 No. 2, pp. 264-278. · doi:10.1108/00220411111109467
  13. Efthimiadis, E.N. (2000), “Interactive query expansion: a user-based evaluation in a relevance feedback environment”, Journal of the American Society for Information Science , Vol. 51 No. 11, pp. 989-1003. · DOI
  14. Ellis, D. , Ford, N. and Furner, J. (1998), “In search of the unknown user: indexing and hypertext and the world wide web”, Journal of Documentation , Vol. 54 No. 1, pp. 28-47. · DOI
  15. Fattahi, R. , Wilson, C.S. and Cole, F. (2008), “An alternative approach to natural language query expansion in search engines: text analysis of non-topical terms in web documents”, Information Processing and Management , Vol. 44 No. 4, pp. 1503-1516. · DOI
  16. Ferreira, A. and Atkinson, J. (2007), “Intelligent search engines”, McGraw-Hill supplemental articles, available at: http://books.mcgraw-hill.com/EST10/site/supparticles/Intelligent-search-engines-YB070360.pdf (accessed November 2014).
  17. Fu, G. , Jones, C.B. and Abdelmoty, A.I. (2005), “Ontology-based spatial query expansion in information retrieval”, On the Move to Meaningful Internet Systems 2005: CoopIS, DOA, and ODBASE (Vol. 3761 of the series Lecture Notes in Computer Science ), Springer, Berlin Heidelberg, pp. 1466-1482. · DOI
  18. Hirsh, S.G. (1999), “Children⊁s relevance criteria and information seeking on electronic resources”, Journal of the American Society for Information Science , Vol. 50 No. 14, pp. 1265-1283. · DOI · Paper page
  19. Jansen, B.J. and Rieh, S.Y. (2010), “The seventeen theoretical constructs of information searching and information retrieval”, Journal of the American Society for Information Science and Technology , Vol. 61 No. 8, pp. 1517-1534. · DOI
  20. Jeon, J. , Croft, W.B. and Lee, J.H. (2005), “Finding semantically similar questions based on their answers”, Proceedings of the 28th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, ACM, Bethesda, MD, pp. 617-618. · DOI
  21. Kuhlthau, C. (1991), “Inside the information search process: information seeking from the user’s perspective”, Journal of the American Society for Information Science , Vol. 42 No. 5, pp. 361-371. · DOI · Paper page
  22. Lawrence, S. and Giles, C.L. (1998), “Context and page analysis for improved web search”, IEEE Internet Computing , Vol. 2 No. 4, pp. 38-46. · DOI
  23. Maglaughlin, K.L. and Sonnenwald, D.H. (2002), “User perspectives on relevance criteria: a comparison among relevant, partially relevant, and not-relevant judgments”, Journal of the American Society for Information Science and Technology , Vol. 53 No. 5, pp. 327-342. · DOI
  24. Pokorny, J. (2004), “Web searching and information retrieval”, IEEE Computer Software , Vol. 6 No. 4, pp. 43-48. · DOI
  25. Savolainen, R. and Kari, J. (2005), “User-defined relevance criteria in web searching”, Journal of Documentation , Vol. 62 No. 6, pp. 685-707. · DOI
  26. Schamber, L. (1991), “Users’ criteria for evaluation in multimedia information seeking and use situations”, Unpublished Doctoral Dissertation, Syracuse University, Syracuse, New York, NY.
  27. Schatz, B.R. , Johnson, E.H. , Cochrane, P.A. and Chen, H. (1996), “Interactive term suggestion for users of digital libraries: using subject thesauri and co-occurrence lists for information retrieval”, Proceedings of the First ACM International Conference on Digital Libraries, ACM, Bethesda, MD , pp. 126-133. · doi:10.1145/226931.226956
  28. Spink, A. , Greisdorf, H. and Bateman, J. (1998), “From highly relevant to not relevant: examining different regions of relevance”, Information Processing and Management , Vol. 34 No. 5, pp. 599-621. · DOI · Paper page
  29. Spink, A. , Ozmutlu, H.C. and Ozmutlu, S. (2002), “Multitasking information seeking and searching processes”, Journal of the American Society for Information Science and Technology , Vol. 53 No. 8, pp. 639-652. · DOI · Paper page
  30. Sugiyama, K. , Hatano, K. and Yoshikawa, M. (2004), “Adaptive web search based on user profile constructed without any effort from user”, Proceedings of WWW 2004, New York, NY, 17-22 May , pp. 675-684. · DOI · Paper page
  31. Tang, R. and Solomon, P. (1998), “Towards an understanding of the dynamics of relevance judgments: an analysis of one person’s search behaviour”, Information Processing & Management , Vol. 34 Nos 2/3, pp. 237-256. · DOI
  32. Taylor, A. (2008), “Examination of relevance criteria choices and the information search process”, Journal of Documentation , Vol. 65 No. 5, pp. 719-744. · doi:10.1108/00220410910983083
  33. Taylor, A. , Zhang, X. and Amadio, W.J. (2009), “Examination of relevance criteria choices and the information search process”, Journal of Documentation , Vol. 65 No. 5, pp. 719-744. · doi:10.1108/00220410910983083
  34. Wang, J. and Davison, B.D. (2008), “Explorations in tag suggestion and query expansion”, Proceeding of the 2008 ACM Workshop on Search in Social Media (SSM ’08), ACM, New York, NY, pp. 43-50. · doi:10.1145/1458583.1458592
  35. Wen, J.R. , Nie, J.Y. and Zhang, H.J. (2002), “Query clustering using user logs”, ACM Transactions on Information Systems , Vol. 20 No. 1, pp. 59-81. · DOI · Paper page
  36. Widyantoro, D.H. and Yen, J. (2001), “A fuzzy ontology-based abstract search engine and its user studies”, The 10th IEEE International Conference on Fuzzy Systems, IEEE, Piscataway, NJ, Vol. 3, pp. 1291-1294. · doi:10.1109/fuzz.2001.1008895
  37. Schamber, L. and Bateman, J. (1996), “User criteria in relevance evaluation: toward development of measurement scale”, Proceedings of the ASIS Annual Meeting , Vol. 33 No. 1, pp. 218-225.
  38. Saracevic, T. (2006), “Relevance: a review of the literature and framework for thinking on the notion in information science. Part II: nature and manifestations of relevance”, Advances in Librarianship , Vol. 30 No. 1, pp. 3-69, available at: www.scils.rutgers.edu/∼tefko/Advances_librar_2006.pdf (accessed 13 November 2014).

Metadata

Authors
Fattahi, R., Parirokh, M., Dayyani, M. H., Khosravi, A., & Zareivenovel, M.
Venue
The Electronic Library
Publisher
Emerald
Published
2016-04-04
Volume
34
Issue
2
Pages
302-314
ISSN
0264-0473
Cited by
6
References
2
Language
en
Licence
www.emerald.com/insight/site-policies (tdm)

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Effectiveness of Google keyword suggestion on users’ relevance judgment: A mixed method approach to query expansion. https://app.cochat.ai/papers/10.1108/el-03-2015-0035
The paper
Fattahi, R., Parirokh, M., Dayyani, M. H., Khosravi, A., & Zareivenovel, M. (2016). Effectiveness of Google keyword suggestion on users’ relevance judgment: A mixed method approach to query expansion. The Electronic Library. https://doi.org/10.1108/el-03-2015-0035

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.