CoChat

Auditing Search Engines for Differential Satisfaction Across Demographics

Open access
Mehrotra, Rishabh, Anderson, Ashton, Diaz, Fernando, Sharma, Amit, Wallach, Hanna, Yilmaz, Emine · 2017 · Proceedings of the 26th International Conference on World Wide Web Companion - WWW '17 Companion
Cited by 68

The DOI matches its Crossref record; 10 of the 10 references it cites were found in a registry or an index.

Abstract

Many online services, such as search engines, social media platforms, and digital marketplaces, are advertised as being available to any user, regardless of their age, gender, or other demographic factors. However, there are growing concerns that these services may systematically underserve some groups of users. In this paper, we present a framework for internally auditing such services for differences in user satisfaction across demographic groups, using search engines as a case study. We first explain the pitfalls of naively comparing the behavioral metrics that are commonly used to evaluate search engines. We then propose three methods for measuring latent differences in user satisfaction from observed differences in evaluation metrics. To develop these methods, we drew on ideas from the causal inference literature and the multilevel modeling literature. Our framework is broadly applicable to other online services, and provides general insight into interpreting their evaluation metrics.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1145/3041021.3054197

Cites

10 of 10 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 10 references

Research works printed but not linked: 32

  1. H. Ajutsu, G. E. Legge, J. A. Ross, and K. J. Scheubel. Psychophysics of reading--X. Effects of age-related changes in vision. Journal of Gerontology, 46(6):325--331, 1991. · DOI · Paper page
  2. P. N. Bennett, K. Svore, and S. T. Dumais. Classification-enhanced ranking. In WWW, 2010. · DOI · Paper page
  3. B. Bin, M. Shokouhi, M. Kosinski, and T. Graepel. Inferring the demographics of search users: Social data meets search queries. In WWW, 2013. · DOI · Paper page
  4. A. Borisov, I. Markov, M. de Rijke, and P. Serdyukov. A context-aware time model for web search. In SIGIR, 2016. · DOI
  5. G. Buscher, L. van Elst, and A. Dengel. Segment-level time as implicit feedback: A comparison to eye tracking. In SIGIR, 2009. · DOI
  6. B. Carterette, E. Kanoulas, and E. Yilmaz. Incorporating variability in user behavior into systems based evaluation. In CIKM, 2012. · doi:10.1145/2396761.2396782
  7. S. L. Connelly, L. Hasher, and R. T. Zacks. Age and reading: The impact of distraction. Psychology and Aging, 6(4):533--541, 1991. · DOI · Paper page
  8. N. Diakopoulos. Algorithmic accountability. Digital Journalism, 3(3):398--415, 2015. · DOI · Paper page
  9. D. Downey, S. Dumais, and E. Horvitz. Heads and tails: Studies of web search with common and rare queries. In SIGIR, 2007. · doi:10.1145/1277741.1277939
  10. R. Epstein and R. E. Robertson. The search engine manipulation effect (SEME) and its possible impact on the outcomes of elections. PNAS, 2015. · DOI · Paper page
  11. H. A. Feild, J. Allan, and R. Jones. Predicting searcher frustration. In SIGIR, 2010. · DOI · Paper page
  12. N. Ford, D. Miller, and N. Moss. Web search strategies and human individual differences: Cognitive and demographic factors, internet attitudes, and approaches. JASIST, 56(7):741--756, 2005. · DOI
  13. G. M. Fulgoni. The "professional respondent" problem in online survey panels today. Presented at the Marketing Research Association Annual Conference, 2005.
  14. A. Gelman, J. B. Carlin, H. S. Stern, D. B. Dunson, A. Vehtari, and D. B. Rubin. Bayesian Data Analysis. CRC Press, 2014. · doi:10.1201/b16018 · Paper page
  15. A. Gelman and J. Hill. Data Analysis Using Regression and Multilevel/Hierarchical Models. Cambridge University Press, 2006. · DOI · Paper page
  16. A. Hassan, X. Shi, N. Craswell, and B. Ramsey. Beyond clicks: Query reformulation as a predictor of search satisfaction. In CIKM, 2013. · DOI · Paper page
  17. A. Hassan and R. W. White. Personalized models of search satisfaction. In CKIM, 2013. · DOI
  18. Y. Hu, Y. Koren, and C. Volinsky. Collaborative filtering for implicit feedback datasets. In ICDM, 2008. · DOI · Paper page
  19. J. Jiang, A. Hassan, Z. Shi, and R. W. White. Understanding and predicting graded search satisfaction. In WSDM, 2015. · DOI
  20. Y. Kim, A. Hassan, R. W. White, and I. Zitouni. Modeling dwell time to predict click-level satisfaction. In WSDM, 2014. · DOI · Paper page
  21. A. Large, J. Beheshti, and T. Rahman. Gender differences in collaborative web searching behavior: An elementary school study. Information Processing and Management, 2002. · DOI · Paper page
  22. L. Lorigo, B. Pan, H. Hembrooke, T. Joachims, L. Granka, and G. Gay. The influence of task and gender on search and evaluation behavior using Google. Information Processing and Management, 2006. · DOI · Paper page
  23. E. R. Núnez-Valdéz, J. M. C. Lovelle, O. S. Martínez, V. García-Díaz, P. O. de Pablos, and C. Marín. Implicit feedback techniques on recommender systems applied to electronic books. Computers in Human Behavior, 28(4):1186--1193, 2012. · DOI
  24. F. Radlinski, M. Szummer, and N. Craswell. Inferring query intent from reformulations and clicks. In WWW, 2010. · doi:10.1145/1772690.1772859
  25. M. Roy and M. T. H. Chi. Gender differences in patterns of searching the web. Journal of Educational Computing Research, 29(3), 2003. · DOI
  26. D. B. Rubin. Matched Sampling for Causal Effects. Cambridge University Press, 2006. · DOI · Paper page
  27. Y. Wang and E. Agichtein. Query ambiguity revisited: Clickthrough measures for distinguishing informational and ambiguous queries. In NAACL, 2010.
  28. I. Weber and C. Castillo. The demographics of web search. In SIGIR, 2010. · DOI · Paper page
  29. R. W. White. Interactions with Search Systems. Cambridge University Press, 2016. · DOI · Paper page
  30. R. W. White and S. Dumais. Characterizing and predicting search engine switching behavior. In CIKM, 2009. · DOI
  31. P. Yin, P. Luo, W.-C. Lee, and M. Wang. Silence is also evidence: Interpreting dwell time for recommendation from psychological perspective. In KDD, 2013. · DOI
  32. M. Zhou. Gender difference in web search perceptions and behavior: Does it vary by task performance? Computers and Education, 78:174--184, 2014. · DOI

Cited by

See all 68

Metadata

Authors
Mehrotra, R., Anderson, A., Diaz, F., Sharma, A., Wallach, H., & Yilmaz, E.
Venue
Proceedings of the 26th International Conference on World Wide Web Companion - WWW '17 Companion
Publisher
ACM Press
Published
2017
Pages
626-633
OpenAlex
W2612327332
Cited by
68
References
10
FWCI
5.29
Type
conference paper
Language
en
Open access
gold

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Auditing Search Engines for Differential Satisfaction Across Demographics. https://app.cochat.ai/papers/10.1145/3041021.3054197
The paper
Mehrotra, R., Anderson, A., Diaz, F., Sharma, A., Wallach, H., & Yilmaz, E. (2017). Auditing Search Engines for Differential Satisfaction Across Demographics. Proceedings of the 26th International Conference on World Wide Web Companion - WWW '17 Companion. https://doi.org/10.1145/3041021.3054197

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.