The DOI matches its Crossref record; 2 of the 2 references it cites were found in a registry or an index.
Effectiveness of Google keyword suggestion on users’ relevance judgment: A mixed method approach to query expansion
Abstract
Purpose: – One of the most effective ways information retrieval (IR) systems including Web search engines can improve relevance performance is to provide their users with tools for facilitating query expansion. Search engines such as Google provide users with keyword suggest tools. This paper aims to investigate users’ criteria in relevance judgment regarding Google’s keywords suggest tool and to see how such keywords would lead to more relevant results from the viewpoint of users. Design/methodology/approach: – Through a mixed method approach, quantitative and qualitative data were collected from 60 postgraduate students at Ferdowsi University of Mashhad, Iran, using four different instruments (questionnaire, thinking aloud technique, query logs and interviews). Findings: – Among other criteria, the “relation between suggested keywords and the information need” (with the mean rate of 3.53 of four) was considered the most important by searchers in selecting suggested keywords for query expansion. Also, the “relation between suggested Keywords and the retrieved items” (with the mean rate of 3.62) was considered the second most important criterion in judging the relevance of the retrieved results. The participants agreed that the suggested keywords by Google improved the retrieval relevance. The content analysis of the participants’ aloud-thinking sessions and the interviews approved such findings. Originality/value: – This research makes a contribution to the need of designers of IR systems regarding the use of add words for query expansion. It also helps librarians how to instruct searchers with expanding their queries to retrieve more relevant results. Another contribution of the study is the identification of a number of new relevance judgment criteria for Web-based environments.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Until or unless chatbots adopt Google's seemingly effective keyword suggestion approach (Fattahi et al. 2016), virtual reference librarians will offer users a clear advantage.”
From AI on the Shoulders of Giants: Using Kuhlthau’s Information Search Process to Improve AI Support for Information-Seeking (2025) · cites this paper as Fattahi et al. 2016
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
How users assess Web pages for information seeking
Tombros, A., Ruthven, I., & Jose, J. M. (2005) · Journal of the American Society for Information Science and Technology · doi:10.1002/asi.20106 · Cited by 202
The internet and information retrieval research: a brief review
Chowdhury, G. G. (1999) · Journal of Documentation · doi:10.1108/eum0000000007144 · Cited by 30
Research works printed but not linked: 38
- Abam, Z. (2010), “The study of users’ browsing behavior and relevance judgment in web-based meta-link environment”, PhD Dissertation, Library and Information Science, Ferdowsi University of Mashhsad, Mashhsad
- Ahn, M. (2003), “Exploring factors affecting users’ link-following decision and evaluation behavior during web browsing”, PhD Thesis, School of Information Studies, University of Pittsburgh, Pittsburgh, PA.
- Andreou, A. (2005), “Ontologies and query expansion”, Master’s thesis, School of Informatics, University of Edinburgh, Edinburgh.
- Anick, P. (2003), “Using terminological feedback for web search refinement: a log-based study”, Proceedings of the 26th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, ACM, Bethesda, MD, pp. 88-95. · DOI · Paper page
- Baeza-Yates, R. , Hurtado, C.A. and Mendoza, M. (2004), “Query recommendation using query logs in search engines”, in Lindner, W. , Mesiti, M. , Türker, C. , Tzitzikas, Y. and Vakali, A. (Eds), EDBT Workshops , Springer, New York, NY, pp. 588-596. · DOI · Paper page
- Barry, C.L. (1994), “User-defined relevance criteria: an exploratory study”, Journal of the American Society for Information Science , Vol. 45 No. 3, pp. 149-159. · DOI · Paper page
- Barry, C.L. and Schamber, L. (1998), “User’s criteria for relevance evaluation: a cross-situational comparison”, Information Processing and Management , Vol. 34 Nos 2/3, pp. 219-236. · DOI · Paper page
- Borland, P. (2003), “The concept of relevance”, Journal of the American Society for Information Science & Technology , Vol. 54 No. 10, pp. 913-925. · DOI · Paper page
- Chen, Z. and Xu, Y. (2005), “User-oriented relevance judgment: a conceptual model”, paper presented at Proceedings of the 38th Hawaii International Conference on System Sciences, Hawaii, HI.
- Choi, Y. and Rasmussen, E.M. (2002), “User’s relevance criteria in image retrieval in American history”, Information Processing and Management , Vol. 38 No. 50, pp. 695-726. · doi:10.1016/s0306-4573(01)00059-0
- Chowdhury, A. and Soboroff, I. (2002), “Automatic evaluation of World Wide Web search services”, Proceedings of the 25th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, ACM, Bethesda, MD, pp. 421-422. · doi:10.1145/564376.564474
- Chu, H. (2010), “Factor affecting relevance judgment: a report from TREC legal track”, Journal of Documentation , Vol. 67 No. 2, pp. 264-278. · doi:10.1108/00220411111109467
- Efthimiadis, E.N. (2000), “Interactive query expansion: a user-based evaluation in a relevance feedback environment”, Journal of the American Society for Information Science , Vol. 51 No. 11, pp. 989-1003. · DOI
- Ellis, D. , Ford, N. and Furner, J. (1998), “In search of the unknown user: indexing and hypertext and the world wide web”, Journal of Documentation , Vol. 54 No. 1, pp. 28-47. · DOI
- Fattahi, R. , Wilson, C.S. and Cole, F. (2008), “An alternative approach to natural language query expansion in search engines: text analysis of non-topical terms in web documents”, Information Processing and Management , Vol. 44 No. 4, pp. 1503-1516. · DOI
- Ferreira, A. and Atkinson, J. (2007), “Intelligent search engines”, McGraw-Hill supplemental articles, available at: http://books.mcgraw-hill.com/EST10/site/supparticles/Intelligent-search-engines-YB070360.pdf (accessed November 2014).
- Fu, G. , Jones, C.B. and Abdelmoty, A.I. (2005), “Ontology-based spatial query expansion in information retrieval”, On the Move to Meaningful Internet Systems 2005: CoopIS, DOA, and ODBASE (Vol. 3761 of the series Lecture Notes in Computer Science ), Springer, Berlin Heidelberg, pp. 1466-1482. · DOI
- Hirsh, S.G. (1999), “Children⊁s relevance criteria and information seeking on electronic resources”, Journal of the American Society for Information Science , Vol. 50 No. 14, pp. 1265-1283. · DOI · Paper page
- Jansen, B.J. and Rieh, S.Y. (2010), “The seventeen theoretical constructs of information searching and information retrieval”, Journal of the American Society for Information Science and Technology , Vol. 61 No. 8, pp. 1517-1534. · DOI
- Jeon, J. , Croft, W.B. and Lee, J.H. (2005), “Finding semantically similar questions based on their answers”, Proceedings of the 28th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, ACM, Bethesda, MD, pp. 617-618. · DOI
- Kuhlthau, C. (1991), “Inside the information search process: information seeking from the user’s perspective”, Journal of the American Society for Information Science , Vol. 42 No. 5, pp. 361-371. · DOI · Paper page
- Lawrence, S. and Giles, C.L. (1998), “Context and page analysis for improved web search”, IEEE Internet Computing , Vol. 2 No. 4, pp. 38-46. · DOI
- Maglaughlin, K.L. and Sonnenwald, D.H. (2002), “User perspectives on relevance criteria: a comparison among relevant, partially relevant, and not-relevant judgments”, Journal of the American Society for Information Science and Technology , Vol. 53 No. 5, pp. 327-342. · DOI
- Pokorny, J. (2004), “Web searching and information retrieval”, IEEE Computer Software , Vol. 6 No. 4, pp. 43-48. · DOI
- Savolainen, R. and Kari, J. (2005), “User-defined relevance criteria in web searching”, Journal of Documentation , Vol. 62 No. 6, pp. 685-707. · DOI
- Schamber, L. (1991), “Users’ criteria for evaluation in multimedia information seeking and use situations”, Unpublished Doctoral Dissertation, Syracuse University, Syracuse, New York, NY.
- Schatz, B.R. , Johnson, E.H. , Cochrane, P.A. and Chen, H. (1996), “Interactive term suggestion for users of digital libraries: using subject thesauri and co-occurrence lists for information retrieval”, Proceedings of the First ACM International Conference on Digital Libraries, ACM, Bethesda, MD , pp. 126-133. · doi:10.1145/226931.226956
- Spink, A. , Greisdorf, H. and Bateman, J. (1998), “From highly relevant to not relevant: examining different regions of relevance”, Information Processing and Management , Vol. 34 No. 5, pp. 599-621. · DOI · Paper page
- Spink, A. , Ozmutlu, H.C. and Ozmutlu, S. (2002), “Multitasking information seeking and searching processes”, Journal of the American Society for Information Science and Technology , Vol. 53 No. 8, pp. 639-652. · DOI · Paper page
- Sugiyama, K. , Hatano, K. and Yoshikawa, M. (2004), “Adaptive web search based on user profile constructed without any effort from user”, Proceedings of WWW 2004, New York, NY, 17-22 May , pp. 675-684. · DOI · Paper page
- Tang, R. and Solomon, P. (1998), “Towards an understanding of the dynamics of relevance judgments: an analysis of one person’s search behaviour”, Information Processing & Management , Vol. 34 Nos 2/3, pp. 237-256. · DOI
- Taylor, A. (2008), “Examination of relevance criteria choices and the information search process”, Journal of Documentation , Vol. 65 No. 5, pp. 719-744. · doi:10.1108/00220410910983083
- Taylor, A. , Zhang, X. and Amadio, W.J. (2009), “Examination of relevance criteria choices and the information search process”, Journal of Documentation , Vol. 65 No. 5, pp. 719-744. · doi:10.1108/00220410910983083
- Wang, J. and Davison, B.D. (2008), “Explorations in tag suggestion and query expansion”, Proceeding of the 2008 ACM Workshop on Search in Social Media (SSM ’08), ACM, New York, NY, pp. 43-50. · doi:10.1145/1458583.1458592
- Wen, J.R. , Nie, J.Y. and Zhang, H.J. (2002), “Query clustering using user logs”, ACM Transactions on Information Systems , Vol. 20 No. 1, pp. 59-81. · DOI · Paper page
- Widyantoro, D.H. and Yen, J. (2001), “A fuzzy ontology-based abstract search engine and its user studies”, The 10th IEEE International Conference on Fuzzy Systems, IEEE, Piscataway, NJ, Vol. 3, pp. 1291-1294. · doi:10.1109/fuzz.2001.1008895
- Schamber, L. and Bateman, J. (1996), “User criteria in relevance evaluation: toward development of measurement scale”, Proceedings of the ASIS Annual Meeting , Vol. 33 No. 1, pp. 218-225.
- Saracevic, T. (2006), “Relevance: a review of the literature and framework for thinking on the notion in information science. Part II: nature and manifestations of relevance”, Advances in Librarianship , Vol. 30 No. 1, pp. 3-69, available at: www.scils.rutgers.edu/∼tefko/Advances_librar_2006.pdf (accessed 13 November 2014).
Cited by
Ravuri, B., & Mardis, M. A. (2025) · Library Trends · doi:10.1353/lib.2025.a961195 · Cited by 9
Metadata
- Authors
- Fattahi, R., Parirokh, M., Dayyani, M. H., Khosravi, A., & Zareivenovel, M.
- Venue
- The Electronic Library
- Publisher
- Emerald
- Published
- 2016-04-04
- Volume
- 34
- Issue
- 2
- Pages
- 302-314
- ISSN
- 0264-0473
- Cited by
- 6
- References
- 2
- Language
- en
- Licence
- www.emerald.com/insight/site-policies (tdm)
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Effectiveness of Google keyword suggestion on users’ relevance judgment: A mixed method approach to query expansion. https://app.cochat.ai/papers/10.1108/el-03-2015-0035
- The paper
- Fattahi, R., Parirokh, M., Dayyani, M. H., Khosravi, A., & Zareivenovel, M. (2016). Effectiveness of Google keyword suggestion on users’ relevance judgment: A mixed method approach to query expansion. The Electronic Library. https://doi.org/10.1108/el-03-2015-0035
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.