Enhancing credibility judgment of web search results
The DOI matches its Crossref record; 30 of the 33 references it cites were found in a registry or an index.
Abstract
In this paper, we propose a system for helping users to judge the credibility of Web search results and to search for credible Web pages. Conventional Web search engines present only titles, snippets, and URLs for users, which give few clues to judge the credibility of Web search results. Moreover, ranking algorithms of the conventional Web search engines are often based on relevance and popularity of Web pages. Towards credibility-oriented Web search, our proposed system provides users with the following three functions: (1) calculation and visualization of several scores of Web search results on the main credibility aspects, (2) prediction of user's credibility judgment model through user's credibility feedback for Web search results, and (3) re-ranking of Web search results based on user's predicted credibility model. Experimental results suggest that our system enables users - in particular, users with knowledge about search topics - to find credible Web pages from a list of Web search results more efficiently than conventional Web search interfaces.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Yamamoto and Tanaka [68] built a system that shows scores of trustworthiness attributes of web sites and re-ranks the search results.”
From Designing for Responsible Trust in AI Systems: A Communication Perspective (2022) · cites this paper as [68]
In the citing paper
“The other is to provide independent augmenting tooling to truthfully highlight an AI system's trustworthiness cues, which the creators may have downplayed or hidden [29, 53, 68].”
From Designing for Responsible Trust in AI Systems: A Communication Perspective (2022) · cites this paper as [68]
Citations
Cites
The Elaboration Likelihood Model of Persuasion
Petty, R. E., & Cacioppo, J. T. (1986) · Advances in Experimental Social Psychology · doi:10.1016/s0065-2601(08)60214-2 · Cited by 8,822 · cited 1×
“According to ELM theory proposed by Petty et al. [18], if people have enough knowledge to understand information about a topic, they often pay attention to it.”
MethodThe Influence of Source Credibility on Communication Effectiveness
Hovland, C. I., & Weiss, W. (1951) · Public Opinion Quarterly · doi:10.1086/266350 · Cited by 3,543 · cited 2×
“Credibility has been studied in communication and social psychology since 1950s [11].”
BackgroundLexRank: Graph-based Lexical Centrality as Salience in Text Summarization
Erkan, G., & Radev, D. R. (2004) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1523 · Cited by 1,700 · cited 1×
“We use the LexRank algorithm to calculate content typicality [7].”
MethodImproving retrieval performance by relevance feedback
Salton, G., & Buckley, C. (1990) · Journal of the American Society for Information Science · doi:10.1002/(sici)1097-4571(199006)41:4<288::aid-asi8>3.0.co;2-h · Cited by 1,495
Propagation of trust and distrust
Guha, R., Kumar, R., Raghavan, P., & Tomkins, A. (2004) · Proceedings of the 13th international conference on World Wide Web · doi:10.1145/988672.988727 · Cited by 1,464 · cited 1×
“Guha et al. developed a framework to predict trust between two people on a large trust network through link analysis [10].”
BackgroundFinding high-quality content in social media
Agichtein, E., Castillo, C., Donato, D., Gionis, A., & Mishne, G. (2008) · Proceedings of the international conference on Web search and web data mining - WSDM '08 · doi:10.1145/1341531.1341557 · Cited by 1,230 · cited 1×
“Agichtein et al. evaluated authority and hub scores of users by using a graph structure representing the interactions between the users and then evaluated the quality of answers on the basis of those users’ scores [2].”
MethodThe elements of computer credibility
Fogg, B. J., & Tseng, H. (1999) · Proceedings of the SIGCHI conference on Human factors in computing systems the CHI is the limit - CHI '99 · doi:10.1145/302979.303001 · Cited by 802 · cited 4×
“However, researchers in the above fields indicate that information credibility is a subjective quality, and its interpretation depends on the receivers and types of information [9, 24, 14].”
BackgroundInfluentialPersonalizing search via automated analysis of interests and activities
Teevan, J., Dumais, S. T., & Horvitz, E. (2005) · Proceedings of the 28th annual international ACM SIGIR conference on Research and development in information retrieval · doi:10.1145/1076034.1076111 · Cited by 800 · cited 1×
“In information retrieval, a lot of researchers have developed methods for relevance feedback [22] or personalized searches [27].”
Metzger, M. J., Flanagin, A. J., Eyal, K., Lemus, D. R., & Mccann, R. M. (2003) · Annals of the International Communication Association · doi:10.1080/23808985.2003.11679029 · Cited by 768
What makes Web sites credible?: a report on a large quantitative study
Fogg, B. J., Marshall, J., Laraki, O., Osipovich, A., Varma, C., Fang, N., Paul, J., Rangnekar, A., Shon, J., Swani, P., et al. (2001) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/365024.365037 · Cited by 685 · cited 1×
“conducted a large-scale survey to investigate what factors affect users’ perception of the credibility of Web sites [8].”
BackgroundJudgment of information quality and cognitive authority in the Web
Rieh, S. Y. (2002) · Journal of the American Society for Information Science and Technology · doi:10.1002/asi.10017.abs · Cited by 585 · cited 1×
“Recently, researchers have started to discuss information quality, and some have argued that the importance of credibility is a factor in information quality [21, 20].”
BackgroundCollege student Web use, perceptions of information credibility, and verification behavior
Metzger, M. J., Flanagin, A. J., & Zwarun, L. (2003) · Computers & Education · doi:10.1016/s0360-1315(03)00049-6 · Cited by 448 · cited 2×
“On the other hand, some researchers reported that a lot of Web users trust Web information to some extent [15].”
BackgroundA content-driven reputation system for the wikipedia
Adler, B. T., & de Alfaro, L. (2007) · Proceedings of the 16th international conference on World Wide Web · doi:10.1145/1242572.1242608 · Cited by 440
Source Credibility Dimensions in Marketing Communication - A Generalized Solution
Eisend, M. (2006) · Cited by 275
Trust and mistrust of online health sites
Sillence, E., Briggs, P., Fishwick, L., & Harris, P. (2004) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/985692.985776 · Cited by 211 · cited 1×
“For example, Sillence et al. reported that there are more than 20,000 medical Web sites on the Web, but over half are not checked by medical experts [23].”
BackgroundDenning, P., Horning, J., Parnas, D., & Weinstein, L. (2005) · Communications of the ACM · doi:10.1145/1101779.1101804 · Cited by 205 · cited 1×
“Denning et al. warned about the credibility of Wikipedia, which is one of the most popular websites [4].”
BackgroundChucking the Checklist: A Contextual Approach to Teaching Undergraduates Web-Site Evaluation
Meola, M. (2004) · portal: Libraries and the Academy · doi:10.1353/pla.2004.0055 · Cited by 200 · cited 2×
“In library science, some researchers developed a guideline to evaluate information credibility [13].”
BackgroundMethodMetzger, M. J., Flanagin, A. J., Eyal, K., Lemus, D. R., & Mccann, R. M. (2003) · Communication Yearbook · doi:10.1207/s15567419cy2701_10 · Cited by 192
BrowseRank: letting web users vote for page importance
Liu, Y., Gao, B., Liu, T.-Y., Zhang, Y., Ma, Z., He, S., & Li, H. (2008) · Proceedings of the 31st annual international ACM SIGIR conference on Research and development in information retrieval · doi:10.1145/1390334.1390412 · Cited by 172 · cited 1×
“However, most has focused on ranking a set of data on the basis of relevancy or popularity by analyzing similarity between query and documents, link structures [3], and user-behavior data [12].”
Background(2012) · Encyclopedia of the Sciences of Learning · doi:10.1007/978-1-4419-1428-6_5254 · Cited by 168
Lifting the veil: improving accountability and social transparency in Wikipedia with wikidashboard
Suh, B., Chi, E. H., Kittur, A., & Pendleton, B. A. (2008) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/1357054.1357214 · Cited by 135 · cited 1×
“Suh et al. developed W IKI D ASHBOARD , a system to visualize edit histories on Wikipedia articles [25].”
BackgroundQuality-aware collaborative question answering: methods and evaluation
Suryanto, M. A., Lim, E., Sun, A., & Chiang, R. H. L. (2009) · ACM International Conference on Web Search and Data Mining (WSDM) · doi:10.1145/1498759.1498820 · Cited by 121 · cited 1×
“Suryanto et al. focused on expertise of answerers on Q&A sites to evaluate answers posted on them [26].”
BackgroundHighlighting disputed claims on the web
Ennals, R., Trushkowsky, B., & Agosta, J. M. (2010) · Proceedings of the 19th international conference on World wide web · doi:10.1145/1772690.1772726 · Cited by 113 · cited 1×
“Ennals et al. developed D ISPUTE F INDER , the system that searches the Web for counter sentences when users specify questionable sentences on browsed Web pages [6].”
BackgroundOn ranking controversies in wikipedia
Vuong, B.-Q., Lim, E., Sun, A., Le, M.-T., & Lauw, H. W. (2008) · ACM International Conference on Web Search and Data Mining (WSDM) · doi:10.1145/1341531.1341556 · Cited by 104 · cited 1×
“Vuong et al. developed a method to find controversial Wikipedia articles [28].”
MethodZheng, X. S., Chakraborty, I., Lin, J. J.-W., & Rauschenberger, R. (2009) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/1518701.1518703 · Cited by 88 · cited 1×
“Then we also need controlled experiments where images in Web pages are eliminated, because visual designs of Web pages often affect participants’ quality judgment as Zheng et al. pointed out [29].”
BackgroundTrustworthiness Analysis of Web Search Results
Nakamura, S., Konishi, S., Jatowt, A., Ohshima, H., Kondo, H., Tezuka, T., Oyama, S., & Tanaka, K. (2007) · Lecture notes in computer science · doi:10.1007/978-3-540-74851-9_4 · Cited by 70 · cited 2×
“In particular, for Web search engines, which are used as gates to vast Web information, many users perceive their search re-sults to be somewhat credible, and some users think that the Web search engines rank Web pages on the basis of credibility [16].”
BackgroundSelf-citationPage quality: in search of an unbiased web ranking
Cho, J., Roy, S., & Adams, R. E. (2005) · Proceedings of the 2005 ACM SIGMOD international conference on Management of data · doi:10.1145/1066157.1066220 · Cited by 69 · cited 1×
“However, most has focused on ranking a set of data on the basis of relevancy or popularity by analyzing similarity between query and documents, link structures [3], and user-behavior data [12].”
BackgroundStiff, J. B., & Mongeau, P. A. (1994) · Cited by 63
Pirolli, P., Wollny, E., & Suh, B. (2009) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/1518701.1518929 · Cited by 50 · cited 1×
“Pirolli et al. used WikiDashboard to study how the system affects users’ credibility judgments on Wikipedia articles [19].”
BackgroundQuality Evaluation of Search Results by Typicality and Speciality of Terms Extracted from Wikipedia
Nakatani, M., Jatowt, A., Ohshima, H., & Tanaka, K. (2009) · Lecture notes in computer science · doi:10.1007/978-3-642-00887-0_50 · Cited by 8 · cited 1×
“To measure how Web pages in a list of search results cover technical topics about a given query, we use the method to search Wikipedia for technical terms that Nakatani et al. developed [17].”
MethodSelf-citation
Research works printed but not linked: 3
- 10.5555/1622487.1622501 · doi:10.5555/1622487.1622501
- 10.5555/2392444.2392450 · doi:10.5555/2392444.2392450
- 10.1002/aris.144.v41:1 · doi:10.1002/aris.144.v41:1
Cited by
Social Data: Biases, Methodological Pitfalls, and Ethical Boundaries
Olteanu, A., Castillo, C., Diaz, F., & Kiciman, E. (2016) · doi:10.2139/ssrn.2886526 · Cited by 763
Social Data: Biases, Methodological Pitfalls, and Ethical Boundaries
Olteanu, A., Castillo, C., Diaz, F., & Kıcıman, E. (2019) · Frontiers in Big Data · doi:10.3389/fdata.2019.00013 · Cited by 763
Tweeting is believing?: understanding microblog credibility perceptions
Morris, M. R., Counts, S., Roseway, A., Hoff, A., & Schwarz, J. (2012) · Proceedings of the ACM 2012 conference on Computer Supported Cooperative Work · doi:10.1145/2145204.2145274 · Cited by 411
“However, users are often forced to make credibility judgments about online content before having the opportunity to view a full web page – Schwarz and Morris [34] and Yamamoto and Tanaka [41] have proposed techniques for supplementing search results to support credibility assessment.”
MethodDesigning for Responsible Trust in AI Systems: A Communication Perspective
Liao, Q. V., & Sundar, S. S. (2022) · 2022 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3531146.3533182 · Cited by 165
“Practical means, including design guidelines [15, 16] and tooling [53, 68], have also come out of this line of work both to facilitate technology creators’ design choices and empower web users to make better judgments.”
BackgroundUnderstanding and predicting Web content credibility using the Content Credibility Corpus
Kąkol, M., Nielek, R., & Wierzbicki, A. (2017) · Information Processing & Management · doi:10.1016/j.ipm.2017.04.003 · Cited by 108
“Yamamoto and Tanaka (2011) proposed using the following six factors: update frequency, freshness, topic coverage, content typicality, referential importance and social acceptance.”
Background
Metadata
- Authors
- Yamamoto, Y., & Tanaka, K.
- Affiliations
- Kyoto University (JP)
- Venue
- Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
- Publisher
- ACM
- Published
- 2011-05-07
- Pages
- 1235-1244
- OpenAlex
- W1974267095
- Cited by
- 75
- References
- 33
- Citations by year
- 2026: 2 · 2025: 1 · 2024: 1 · 2023: 3 · 2022: 4 · 2021: 3 · 2020: 5 · 2019: 9 · 2018: 6 · 2017: 7 · 2016: 5 · 2015: 4 · 2014: 6 · 2013: 7 · 2012: 12
- FWCI
- 6.01
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- CHI Conference on Human Factors in Computing Systems (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
- Social Sciences › Sociology and Political Science › Misinformation and Its Impacts
- Physical Sciences › Computer Science › Information Systems › Recommender Systems and Techniques
- Keywords
- credibility judgments, web search results, re-ranking, search result ranking
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.