Worst Practices for Designing Production Information Access Systems
Abstract
Information access systems have become a core part of everyday life for a large variety of users. Behind these large systems are decades of academic information access research. Unfortunately, there is a significant gap between studying and implementing information access systems. In this presentation, I will concentrate on open problems in production systems that academia is better suited for addressing than industry.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Metrics may themselves be designed using a statistical model, subject to the same biases presented in section 7.3.
“Finally, in some cases, metrics may themselves be designed using a statistical model, subject to the same biases presented in section 7.3 (Diaz, 2016).”
From Social Data: Biases, Methodological Pitfalls, and Ethical Boundaries (2019) · cites this paper as Diaz, 2016
Research on social data is arguably human subjects research because data elements in social datasets represent people or groups of people.
“given that data elements in social datasets represent people or groups of people (Varshney, 2015; Diaz, 2016), research on social data is, arguably, human subjects research (Metcalf and Crawford, 2016).”
From Social Data: Biases, Methodological Pitfalls, and Ethical Boundaries (2019) · cites this paper as Diaz, 2016
Cites
Barocas, S., & Selbst, A. D. (2014) · doi:10.2139/ssrn.2477899 · Cited by 3,048 · see citing works
Certifying and Removing Disparate Impact
Feldman, M., Friedler, S. A., Moeller, J., Scheidegger, C., & Venkatasubramanian, S. (2015) · Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2783258.2783311 · Cited by 1,827 · see citing works
Active exploration for learning rankings from clickthrough data
Radlinski, F., & Joachims, T. (2007) · ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/1281192.1281254 · Cited by 154 · see citing works
Integration of news content into web results
Díaz, F. (2009) · ACM International Conference on Web Search and Data Mining (WSDM) · doi:10.1145/1498759.1498825 · Cited by 134 · see citing works
Automated Experiments on Ad Privacy Settings: A Tale of Opacity, Choice, and DiscriminationPreprint
Datta, A. J., Tschantz, M. C., & Datta, A. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1408.6491 · Cited by 129 · see citing works
Robust models of mouse movement on dynamic web search results pages
Díaz, F., White, R. W., Buscher, G., & Liebling, D. J. (2013) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/2505515.2505717 · Cited by 62 · see citing works
White, R. W., & Horvitz, E. (2013) · Journal of the American Medical Informatics Association · doi:10.1136/amiajnl-2012-001473 · Cited by 57 · see citing works
Sunlight: Fine-grained Targeting Detection at Scale with Statistical Confidence
Lecuyer, M., Spahn, R., Spiliopolous, Y., Chaintreau, A., Geambasu, R., & Hsu, D. (2015) · Proceedings of the 22nd ACM SIGSAC Conference on Computer and Communications Security · doi:10.1145/2810103.2813614 · Cited by 48 · see citing works
Big Data, Machine Learning, and the Social Sciences: Fairness, Accountability, and Transparency
Wallach, H. (2014) · Cited by 21 · see citing works
Yom‐Tov, E., & Díaz, F. (2011) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/2009916.2009970 · Cited by 20 · see citing works
The shoe-fitting fluoroscope as a radiation hazard.
Lewis, L. L., & Caplan, P. (1950) · PubMed · Cited by 11 · see citing works
Radiation exposure from a shoe-fitting fluoroscope.
Bushong, S. C., & West, W. D. (1970) · PubMed · Cited by 3 · see citing works
Flagged first, then by citations
1 of the 13 references couldn’t be matched to a known paper.
Cited by
Sorted by citations
Social Data: Biases, Methodological Pitfalls, and Ethical Boundaries
Olteanu, A., Castillo, C., Diaz, F., & Kıcıman, E. (2019) · Frontiers in Big Data · doi:10.3389/fdata.2019.00013 · Cited by 763 · see citing works
“Finally, in some cases, metrics may themselves be designed using a statistical model, subject to the same biases presented in section 7.3 (Diaz, 2016).”
BackgroundInfluentialSocial Data: Biases, Methodological Pitfalls, and Ethical Boundaries
Olteanu, A., Castillo, C., Diaz, F., & Kiciman, E. (2016) · doi:10.2139/ssrn.2886526 · Cited by 763 · see citing works
Self-citationThe influence of search engine optimization on Google's results
Lewandowski, D., Sünkler, S., & Yagci, N. (2021) · ACM Web Science Conference (WebSci) · doi:10.1145/3447535.3462479 · Cited by 55 · see citing works
Public awareness and attitudes towards search engine optimization
Lewandowski, D., & Schultheiß, S. (2022) · Behaviour and Information Technology · doi:10.1080/0144929x.2022.2056507 · Cited by 55 · see citing works
Learning fair representations via rebalancing graph structure
Zhang, G., Cheng, D., Yuan, G., & Zhang, S. (2023) · Information Processing & Management · doi:10.1016/j.ipm.2023.103570 · Cited by 54 · see citing works
Fairness and Discrimination in Retrieval and Recommendation
Ekstrand, M. D., Burke, R., & Díaz, F. (2019) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3331184.3331380 · Cited by 47 · see citing works
Fairness and discrimination in recommendation and retrieval
Ekstrand, M. D., Burke, R., & Díaz, F. (2019) · ACM Conference on Recommender Systems (RecSys) · doi:10.1145/3298689.3346964 · Cited by 46 · see citing works
Zimmerman, S. M., Thorpe, A., Fox, C., & Kruschwitz, U. (2019) · Conference on Human Information Interaction and Retrieval · doi:10.1145/3295750.3298952 · Cited by 41 · see citing works
Towards Search Strategies for Better Privacy and Information
Zimmerman, S. M., Thorpe, A., Chamberlain, J., & Kruschwitz, U. (2020) · Conference on Human Information Interaction and Retrieval · doi:10.1145/3343413.3377958 · Cited by 30 · see citing works
A Critical Review of Online Social Data
Olteanu, A., Kıcıman, E., & Castillo, C. (2018) · ACM International Conference on Web Search and Data Mining (WSDM) · doi:10.1145/3159652.3162004 · Cited by 20 · see citing works
What does Google recommend when you want to compare insurance offerings?
Lewandowski, D., & Sünkler, S. (2019) · Aslib Journal of Information Management · doi:10.1108/ajim-07-2018-0172 · Cited by 15 · see citing works
How Does Search Engine Optimization Affect Outcomes of Electronic Marketing Strategy?
Aryshandy, G., Fernando, Y., Dharmaputra, R. T., Ikhsan, R. B., & Wahyuni-TD, I. S. (2021) · International Conference on Cybernetics and Intelligent System (ICORIS) · doi:10.1109/icoris52787.2021.9649450 · Cited by 4 · see citing works
Towards a Framework for Harm Prevention in Web Search
Zimmerman, S. M., Herzog, S. M., Elsweiler, D., Chamberlain, J., & Kruschwitz, U. (2020) · Open Access at Essex (University of Essex) · Cited by 4 · see citing works
Missing Data Can Be a Geographic Phenomenon
Sutton, D., Basiri, A., Li, Z., & Brunsdon, C. (2026) · Annals of the American Association of Geographers · doi:10.1080/24694452.2026.2640220 · Cited by 1 · see citing works
Exploring Potential Pathways to Address Bias and Ethics in IR
Zimmerman, S. M. (2018) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3209978.3210218
Dijital Pazarlama Perspektifinden Arama Motoru Optimizasyonu
Abdil, A. R. I. K. (2026) · Özgür Yayınları eBooks · doi:10.58830/ozgur.pub1533.c6195
- Authors
- Diaz, F.
- Affiliations
- Microsoft
- Venue
- ACM SIGIR Forum
- Published
- 2016-06-27
- Volume
- 50
- Issue
- 1
- Pages
- 2-11
- ISSN
- 0163-5840 (ISSN-L)
- OpenAlex
- W2462066163
- Cited by
- 15
- References
- 13
- Citations by year
- 2026: 2 · 2023: 1 · 2022: 1 · 2021: 2 · 2020: 2 · 2019: 5 · 2018: 2
- FWCI
- 5.21
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- ACM SIGIR Forum (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
- Physical Sciences › Computer Science › Human-Computer Interaction › Usability and User Interface Design
- Physical Sciences › Computer Science › Information Systems › Recommender Systems and Techniques
- Keywords
- information access systems, production systems, research-practice gap, open problems, information retrieval systems, system design
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
