Augmenting web pages and search results to support credibility assessment
The DOI matches its Crossref record; 31 of the 37 references it cites were found in a registry or an index.
Abstract
The presence (and, sometimes, prominence) of incorrect and misleading content on the Web can have serious consequences for people who increasingly rely on the internet as their information source for topics such as health, politics, and financial advice. In this paper, we identify and collect several page features (such as popularity among specialized user groups) that are currently difficult or impossible for end users to assess, yet provide valuable signals regarding credibility. We then present visualizations designed to augment search results and Web pages with the most promising of these features. Our lab evaluation finds that our augmented search results are particularly effective at increasing the accuracy of users'" credibility assessments, highlighting the potential of data aggregation and simple interventions to help people make more informed decisions as they search for information online.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“For GUIs, augmenting search results and webpages using visualization techniques to show various metadata including website's overall popularity, popularity among domain experts, number of ads, number of awards received by website, e.g., HON award for health websites, have been successful in improving the accuracy of the end user's credibility assessment of online information [71].”
From Use of Intelligent Voice Assistants by Older Adults with Low Technology Use (2020) · cites this paper as [71]
In the citing paper
“Schwarz and Morris [53] developed visualization to augment web search results, displaying metrics that reflect the quality of content in a web site and making visible otherwise hidden information that provides supporting evidence for its level of trustworthiness”
From Designing for Responsible Trust in AI Systems: A Communication Perspective (2022) · cites this paper as [53]
In the citing paper
“The other is to provide independent augmenting tooling to truthfully highlight an AI system's trustworthiness cues, which the creators may have downplayed or hidden [29, 53, 68].”
From Designing for Responsible Trust in AI Systems: A Communication Perspective (2022) · cites this paper as [53]
Citations
Cites
The anatomy of a large-scale hypertextual Web search engine
Brin, S., & Page, L. (1998) · Computer Networks and ISDN Systems · doi:10.1016/s0169-7552(98)00110-x · Cited by 16,004
(2006) · BiblioBoard Library Catalog (Open Research Library) · Cited by 3,173
Confronting the Challenges of Participatory Culture: Media Education for the 21st Century
Jenkins, H., Purushotma, R., Weigel, M., Clinton, K., & Robison, A. J. (2009) · The MIT Press eBooks · doi:10.7551/mitpress/8435.001.0001 · Cited by 2,258
Accurately interpreting clickthrough data as implicit feedback
Joachims, T., Granka, L., Pan, B., Hembrooke, H., & Gay, G. (2005) · Proceedings of the 28th annual international ACM SIGIR conference on Research and development in information retrieval · doi:10.1145/1076034.1076063 · Cited by 1,408 · cited 1×
“…area for further research, in order to answer questions such as whether exposure to such visualizations helps users learn over time to be more discerning Web consumers, or whether augmented search results will change users’ tendency to predominantly click through to the top-ranked result [20].”
BackgroundOpen information extraction from the web
Banko, M., Cafarella, M., Soderland, S. G., Broadhead, M., & Etzioni, O. (2007) · International Joint Conference on Artificial Intelligence · Cited by 1,398
Combating Web Spam with TrustRank
Gyöngyi, Z., Garcia-Molina, H., & Pedersen, J. (2004) · Proceedings 2004 VLDB Conference · doi:10.1016/b978-012088469-8.50052-8 · Cited by 1,085 · cited 2×
“Two of the better known algorithms, TrustRank [16] and CredibleRank [4], use the link structure of the Web to determine a credibility score.”
BackgroundMethodWhat makes Web sites credible?: a report on a large quantitative study
Fogg, B. J., Marshall, J., Laraki, O., Osipovich, A., Varma, C., Fang, N., Paul, J., Rangnekar, A., Shon, J., Swani, P., et al. (2001) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/365024.365037 · Cited by 685 · cited 3×
“…since the sparseness of search results pages would boost the likely impact of such visualizations according to Prominence-Interpretation theory [11], and since once users have seen a Web page itself, it is difficult to overcome the first impressions they form based on the professional…”
BackgroundMethodHilligoss, B., & Rieh, S. Y. (2008) · Information Processing & Management · doi:10.1016/j.ipm.2007.10.001 · Cited by 611 · cited 2×
“Hilligoss and Rieh [18] present a framework which identifies aspects of credibility assessment independent of media, information type, and environment.”
BackgroundMethodPew Internet and American Life Project
Barrett, C. (2020) · Encyclopedia of Behavioral Medicine · doi:10.1007/978-3-030-39903-0_1474 · Cited by 594
How do users evaluate the credibility of Web sites?: a study with over 2,500 participants
Fogg, B. J., Soohoo, C., Danielson, D. R., Marable, L., Stanford, J., & Tauber, E. R. (2003) · Proceedings of the 2003 conference on Designing for user experiences · doi:10.1145/997078.997097 · Cited by 565 · cited 5×
“[11, 14], ultimately determining that in practice the “look and feel” of a site has the greatest impact on users’ credibility assessments.”
BackgroundMethodInfluentialDo security toolbars actually prevent phishing attacks?
Wu, M., Miller, R. C., & Garfinkel, S. L. (2006) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/1124772.1124863 · Cited by 557
What are you looking for?: an eye-tracking study of information usage in web search
Cutrell, E., & Guan, Z. (2007) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/1240624.1240690 · Cited by 460 · cited 1×
“These timings were selected based on prior findings [6] that users typically spend 10 seconds reading search results.”
MethodDesigning trust into online experiences
Shneiderman, B. (2000) · Communications of the ACM · doi:10.1145/355112.355124 · Cited by 426 · cited 1×
“…how people evaluate credibility so as to help Webmasters and designers create sites that will appear more credible (regardless of their true information quality), e.g., Schneiderman’s guidelines for designing trust online [30] and Ivory and Hearst’s tool for high quality site design [19].”
MethodUsing information scent to model user information needs and actions and the Web
Chi, E. H., Pirolli, P., Chen, K., & Pitkow, J. (2001) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/365024.365325 · Cited by 418
Prominence-interpretation theory: explaining how people assess credibility online
Fogg, B. J. (2003) · CHI '03 extended abstracts on Human factors in computing systems - CHI '03 · doi:10.1145/765891.765951 · Cited by 335 · cited 3×
“Fogg proposed Prominence-Interpretation Theory [13] to model how people assess credibility online.”
BackgroundMethodResultCharacterizing the influence of domain expertise on web search behavior
White, R. W., Dumais, S. T., & Teevan, J. (2009) · Proceedings of the Second ACM International Conference on Web Search and Data Mining · doi:10.1145/1498759.1498819 · Cited by 312
(2010) · International journal of communication · Cited by 301
International Journal of Communications
(2020) · International Journal of Communications · doi:10.46300/9107 · Cited by 247
Investigating the querying and browsing behavior of advanced search engine users
White, R. W., & Morris, D. (2007) · Proceedings of the 30th annual international ACM SIGIR conference on Research and development in information retrieval · doi:10.1145/1277741.1277787 · Cited by 171 · cited 1×
“…a heuristic-based approach to defining expertise (users who visit a set of white-listed URLs identified by a professional in the target topic area) is effective at differentiating users according to several standards of expert behavior used in the information retrieval community (e.g., [2, 32]).”
BackgroundTeaching undergrads WEB evaluation: A guide for library instruction
Kapoun, J. (2020) · College & Research Libraries News · doi:10.5860/crln.59.7.522 · Cited by 163
Factors and effects of information credibility
McKnight, D. H., & Kacmar, C. J. (2007) · International Conference on Electronic Commerce · doi:10.1145/1282100.1282180 · Cited by 156 · cited 4×
“…results pages would boost the likely impact of such visualizations according to Prominence-Interpretation theory [11], and since once users have seen a Web page itself, it is difficult to overcome the first impressions they form based on the professional appearance of a page’s design [14, 27].”
BackgroundMethodInfluentialCan you ever trust a wiki?: impacting perceived trustworthiness in wikipedia
Kittur, A., Suh, B., & Chi, E. H. (2008) · Proceedings of the 2008 ACM conference on Computer supported cooperative work · doi:10.1145/1460563.1460639 · Cited by 122 · cited 2×
“Examples of this type of intervention include augmenting Wikipedia pages with a visualization of edit history [23], and creating certifications or whitelists for trustworthy sites (such as HON certification [www.hon.ch]).”
BackgroundBLEWS: Using Blogs to Provide Context for News Articles
Gamon, M., Basu, S., Belenko, D., Fisher, D., & Hurst, M. (2021) · Proceedings of the International AAAI Conference on Web and Social Media · doi:10.1609/icwsm.v2i1.18616 · Cited by 98 · cited 1×
“For instance, Dispute Finder [8] identifies contentious topics by looking for text such as “X is disputed,” and BLEWS [15] provides insight into online news by analyzing the content and sentiment of blogs referencing particular articles.”
BackgroundStatistical profiles of highly-rated web sites
Ivory, M. Y., & Hearst, M. A. (2002) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/503376.503442 · Cited by 85 · cited 1×
“…how people evaluate credibility so as to help Webmasters and designers create sites that will appear more credible (regardless of their true information quality), e.g., Schneiderman’s guidelines for designing trust online [30] and Ivory and Hearst’s tool for high quality site design [19].”
Domain-specific search strategies for the effective retrieval of healthcare and shopping information
Bhavnani, S. K. (2002) · CHI '02 Extended Abstracts on Human Factors in Computing Systems · doi:10.1145/506443.506508 · Cited by 85 · cited 1×
“Domain Type: Because users tend to focus on the page contents rather than the browser’s address bar [33], the domain type (.com, .gov, etc.) of a page may not be salient; we collected this as a categorical feature. (ρ[1000]=0.”
BackgroundNew Literacies of Online Reading Comprehension
Leu, D. J., Everett‐Cacopardo, H., Zawilinski, L., McVerry, G., & O’Byrne, W. I. (2012) · Wiley eBooks · doi:10.1002/9781405198431.wbeal0865 · Cited by 50
Ennals, R., Byler, D., Agosta, J. M., & Rosario, B. (2010) · Proceedings of the 4th workshop on Information credibility · doi:10.1145/1772938.1772952 · Cited by 49 · cited 1×
“For instance, Dispute Finder [8] identifies contentious topics by looking for text such as “X is disputed,” and BLEWS [15] provides insight into online news by analyzing the content and sentiment of blogs referencing particular articles.”
BackgroundCountering web spam with credibility-based link analysis
Caverlee, J., & Liu, L. (2007) · ACM Symposium on Principles of Distributed Computing (PODC) · doi:10.1145/1281100.1281124 · Cited by 41 · cited 4×
“Two of the better known algorithms, TrustRank [16] and CredibleRank [4], use the link structure of the Web to determine a credibility score.”
BackgroundMethodResultInfluentialUnderstanding Web Credibility: A Synthesis of the Research Literature
Lazar, J. K., & Meiselwitz, G. (2007) · now publishers, Inc. eBooks · doi:10.1561/9781601980816 · Cited by 29
Understanding Web Credibility: A Synthesis of the Research Literature
Lazar, J. K., Meiselwitz, G., & Feng, J. (2007) · Foundations and Trends® in Human–Computer Interaction · doi:10.1561/1100000007 · Cited by 25
Ferebee, S. S. (2007) · Lecture notes in computer science · doi:10.1007/978-3-540-77006-0_23 · Cited by 13 · cited 1×
“Factors such as user involvement [10], user task [28], and experience affect prominence, while a user’s assumptions, knowledge level, and context affect interpretation [13].”
Background
Research works printed but not linked: 8
- 10.5555/1625275.1625705 · doi:10.5555/1625275.1625705
- Dignan , L. Gartner Puts Phishing Tab at $3.2 Billion . ZDNet. December 17, 2007 . Dignan, L. Gartner Puts Phishing Tab at $3.2 Billion. ZDNet. December 17, 2007.
- 10.5555/1780402.1780433 · doi:10.5555/1780402.1780433
- 10.5555/640634 · doi:10.5555/640634
- 10.5555/1316689.1316740 · doi:10.5555/1316689.1316740
- Lohr , S. This Boring Headline is Written for Google . The New York Times , April 9, 2006 . Lohr, S. This Boring Headline is Written for Google. The New York Times, April 9, 2006.
- 10.1002/asi.v58:13 · doi:10.1002/asi.v58:13
- Pew Research . Growing Number of Americans Say Obama is a Muslim . August 18, 2010 . Pew Research. Growing Number of Americans Say Obama is a Muslim. August 18, 2010.
Cited by
Information credibility on twitter
Castillo, C., Mendoza, M., & Poblete, B. (2011) · Proceedings of the 20th international conference on World wide web · doi:10.1145/1963405.1963500 · Cited by 2,593
“In a recent user study, it was found that providing information to users about the estimated credibility of online content was very useful and valuable to them [30].”
BackgroundSocial Data: Biases, Methodological Pitfalls, and Ethical Boundaries
Olteanu, A., Castillo, C., Diaz, F., & Kiciman, E. (2016) · doi:10.2139/ssrn.2886526 · Cited by 763
Social Data: Biases, Methodological Pitfalls, and Ethical Boundaries
Olteanu, A., Castillo, C., Diaz, F., & Kıcıman, E. (2019) · Frontiers in Big Data · doi:10.3389/fdata.2019.00013 · Cited by 763
Tweeting is believing?: understanding microblog credibility perceptions
Morris, M. R., Counts, S., Roseway, A., Hoff, A., & Schwarz, J. (2012) · Proceedings of the ACM 2012 conference on Computer Supported Cooperative Work · doi:10.1145/2145204.2145274 · Cited by 411
“However, users are often forced to make credibility judgments about online content before having the opportunity to view a full web page – Schwarz and Morris [34] and Yamamoto and Tanaka [41] have proposed techniques for supplementing search results to support credibility assessment.”
MethodSelf-citationPredicting information credibility in time-sensitive social media
Castillo, C., Mendoza, M., & Poblete, B. (2013) · Internet Research · doi:10.1108/intr-05-2012-0095 · Cited by 339
Metadata
- Authors
- Schwarz, J., & Morris, M.
- Affiliations
- Microsoft (United States) (US)
- ORCID
- Morris, Meredith
- Venue
- Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
- Publisher
- ACM
- Published
- 2011-05-07
- Pages
- 1245-1254
- OpenAlex
- W2164498220
- Cited by
- 181
- References
- 37
- Citations by year
- 2026: 1 · 2025: 5 · 2024: 7 · 2023: 11 · 2022: 10 · 2021: 16 · 2020: 13 · 2019: 14 · 2018: 5 · 2017: 11 · 2016: 15 · 2015: 13 · 2014: 20 · 2013: 21 · 2012: 15
- FWCI
- 52.46
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- CHI Conference on Human Factors in Computing Systems (publishedVersion)
- Topics
- Social Sciences › Sociology and Political Science › Misinformation and Its Impacts
- Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
- Physical Sciences › Computer Science › Computer Graphics and Computer-Aided Design › Data Visualization and Analytics
- Keywords
- credibility assessment, misinformation, information visualization, data aggregation, online information seeking, user decision support
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.