Beyond Ranking: Optimizing Whole-Page Presentation
The DOI matches its Crossref record; 40 of the 41 references it cites were found in a registry or an index.
Abstract
Modern search engines aggregate results from different verticals: webpages, news, images, video, shopping, knowledge cards, local maps, etc. Unlike "ten blue links", these search results are heterogeneous in nature and not even arranged in a list on the page. This revolution directly challenges the conventional "ranked list" formulation in ad hoc search. Therefore, finding proper presentation for a gallery of heterogeneous results is critical for modern search engines.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Citations
Cites
The Hungarian method for the assignment problem
Kuhn, H. W. (1955) · Naval Research Logistics Quarterly · doi:10.1002/nav.3800020109 · Cited by 13,120
Term-weighting approaches in automatic text retrieval
Salton, G., & Buckley, C. (1988) · Information Processing & Management · doi:10.1016/0306-4573(88)90021-0 · Cited by 9,740 · cited 1×
“Design and evaluation of document ranking functions are at the central stage of IR research, dating from vector space models [37] and language modeling ranking functions [40] to more recent machine learning ranking [29] and top document reranking [11, 22, 35].”
BackgroundFriedman, J. H. (2002) · Computational Statistics & Data Analysis · doi:10.1016/s0167-9473(01)00065-2 · Cited by 7,067 · cited 1×
“Gradient boosted decision trees (GBDT) is a very effective method for learning nonlinear functions [18].”
MethodAdvances in Neural Information Processing Systems 20
Bonilla, E. V., Chai, K. M. A., & Williams, C. K. I. (2008) · neural information processing systems · Cited by 2,846
A contextual-bandit approach to personalized news article recommendation
Li, L., Chu, W., Langford, J., & Schapire, R. E. (2010) · Proceedings of the 19th international conference on World wide web · doi:10.1145/1772690.1772758 · Cited by 2,562 · cited 1×
“[28] for evaluating online recommendation systems.”
The use of MMR, diversity-based reranking for reordering documents and producing summaries
Carbonell, J., & Goldstein, J. (1998) · Proceedings of the 21st annual international ACM SIGIR conference on Research and development in information retrieval · doi:10.1145/290941.291025 · Cited by 2,329
The Use of MMR, Diversity-Based Reranking for Reordering Documents and Producing Summaries
Carbonell, J., & Goldstein, J. (2017) · ACM SIGIR Forum · doi:10.1145/3130348.3130369 · Cited by 2,156
Learning to rank for information retrieval
Liu, T.-Y. (2010) · Proceedings of the 33rd international ACM SIGIR conference on Research and development in information retrieval · doi:10.1145/1835449.1835676 · Cited by 1,856
Learning to Rank for Information Retrieval
Liu, T.-Y. (2009) · Foundations and Trends® in Information Retrieval · doi:10.1561/1500000016 · Cited by 1,568
A Study of Smoothing Methods for Language Models Applied to Ad Hoc Information Retrieval
Zhai, C., & Lafferty, J. (2017) · ACM SIGIR Forum · doi:10.1145/3130348.3130377 · Cited by 1,537
Accurately interpreting clickthrough data as implicit feedback
Joachims, T., Granka, L., Pan, B., Hembrooke, H., & Gay, G. (2005) · Proceedings of the 28th annual international ACM SIGIR conference on Research and development in information retrieval · doi:10.1145/1076034.1076063 · Cited by 1,408 · cited 3×
“Eye-tracking experiments [20, 19] and click log analyses [24, 15] observe that users follow sequential order in browsing blue-link-only SERPs.”
BackgroundTHE PROBABILITY RANKING PRINCIPLE IN IR
ROBERTSON, S. E. (1977) · Journal of Documentation · doi:10.1108/eb026647 · Cited by 1,103 · cited 2×
“The presentation is optimal with respect to the user’s effort when she sequentially and independently examines results from top to bottom [36].”
BackgroundAn experimental comparison of click position-bias models
Craswell, N., Zoeter, O., Taylor, M., & Ramsey, B. (2008) · Proceedings of the international conference on Web search and web data mining - WSDM '08 · doi:10.1145/1341531.1341545 · Cited by 942 · cited 2×
“Eye-tracking experiments [20, 19] and click log analyses [24, 15] observe that users follow sequential order in browsing blue-link-only SERPs.”
BackgroundA study of smoothing methods for language models applied to Ad Hoc information retrieval
Zhai, C., & Lafferty, J. (2001) · Proceedings of the 24th annual international ACM SIGIR conference on Research and development in information retrieval · doi:10.1145/383952.384019 · Cited by 889 · cited 1×
“Design and evaluation of document ranking functions are at the central stage of IR research, dating from vector space models [37] and language modeling ranking functions [40] to more recent machine learning ranking [29] and top document reranking [11, 22, 35].”
BackgroundHearst, M. A. (2009) · doi:10.1017/cbo9781139644082 · Cited by 797
Eye-tracking analysis of user behavior in WWW search
Granka, L. A., Joachims, T., & Gay, G. (2004) · Proceedings of the 27th annual international ACM SIGIR conference on Research and development in information retrieval · doi:10.1145/1008992.1009079 · Cited by 696
(2007) · The MIT Press eBooks · doi:10.7551/mitpress/7443.001.0001 · Cited by 525
Sources of evidence for vertical selection
Arguello, J., Diaz, F., Callan, J., & Crespo, J.-F. (2009) · Proceedings of the 32nd international ACM SIGIR conference on Research and development in information retrieval · doi:10.1145/1571941.1571997 · Cited by 200 · cited 1×
“Given a query, the task of vertical selection is to accurately determine which candidate verticals provide potentially relevant results [7, 8].”
BackgroundYue, Y., Patel, R., & Roehrig, H. (2010) · Proceedings of the 19th international conference on World wide web · doi:10.1145/1772690.1772793 · Cited by 186 · cited 1×
“Studies observe more interesting user interaction patterns, notably vertical bias [12, 26, 31] and presentation bias [39].”
BackgroundGlobal Ranking Using Continuous Conditional Random Fields
Qin, T., Liu, T., Zhang, X., Wang, D., & Li, H. (2008) · Cited by 122
Different Users, Different Opinions: Predicting Search Satisfaction with Mouse Movement Information
Liu, Y., Chen, Y., Tang, J., Sun, J., Zhang, M., Ma, S., & Zhu, X. (2015) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/2766462.2767721 · Cited by 78 · cited 1×
“The second step is a utility funciton which defines a user satisfaction metric Finding the right user satisfaction metric based on page-wise user responses is not the focus of this paper, and can itself be a substantial research topic in interactive information systems [21, 30, 38].”
BackgroundLearning to aggregate vertical results into web search results
Arguello, J., Díaz, F., & Callan, J. (2011) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/2063576.2063611 · Cited by 72 · cited 4×
“Previous approaches formulate it as a ranking problem [5, 34, 23].”
BackgroundMethodInfluentialBeyond ten blue links: enabling user click modeling in federated web search
Chen, D., Chen, W., Wang, H., Chen, Z., & Yang, Q. (2012) · Proceedings of the fifth ACM international conference on Web search and data mining · doi:10.1145/2124295.2124351 · Cited by 71 · cited 4×
“Human eyeballs are spontaneously attracted by graphical results, causing a significant attention bias known as the vertical bias [12, 26, 31].”
BackgroundInfluentialA Methodology for Evaluating Aggregated Search Results
Arguello, J., Díaz, F., Callan, J., & Carterette, B. (2011) · Lecture notes in computer science · doi:10.1007/978-3-642-20161-5_15 · Cited by 65 · cited 1×
“In the presence of vertical results, user satisfaction towards an entire SERP cannot be reliably inferred from preference judgments for pairs of results [6].”
BackgroundInfluence of Vertical Result in Web Search Examination
Liu, Z., Liu, Y., Zhou, K., Zhang, M., & Ma, S. (2015) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/2766462.2767714 · Cited by 65 · cited 2×
“Human eyeballs are spontaneously attracted by graphical results, causing a significant attention bias known as the vertical bias [12, 26, 31].”
BackgroundRobust models of mouse movement on dynamic web search results pages
Diaz, F., White, R., Buscher, G., & Liebling, D. (2013) · Proceedings of the 22nd ACM international conference on Information & Knowledge Management · doi:10.1145/2505515.2505717 · Cited by 61 · cited 1×
“As a result, global ranking that takes into account relations between documents is developed [32], and click modeling of federated search becomes even more elaborate [12, 17].”
BackgroundOn composition of a federated web search result page
Ponnuswami, A. K., Pattabiraman, K., Wu, Q., Gilad-Bachrach, R., & Kanungo, T. (2011) · ACM International Conference on Web Search and Data Mining (WSDM) · doi:10.1145/1935826.1935922 · Cited by 53 · cited 2×
“Previous approaches formulate it as a ranking problem [5, 34, 23].”
BackgroundMethodVertical selection in the presence of unlabeled verticals
Arguello, J., Díaz, F., & Paiement, J.-F. (2010) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/1835449.1835564 · Cited by 51 · cited 1×
“Given a query, the task of vertical selection is to accurately determine which candidate verticals provide potentially relevant results [7, 8].”
BackgroundGlobal ranking by exploiting user clicks
Ji, S., Zhou, K., Liao, C. Y., Zheng, Z., Xue, G.-R., Chapelle, O., Sun, G., & Hongyuan Zha, B. (2009) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/1571941.1571950 · Cited by 39 · cited 2×
“to determine the optimal presentation, which is in the same spirit of reranking/global ranking [35, 22].”
BackgroundPredicting Search Satisfaction Metrics with Interleaved Comparisons
Schuth, A., Hofmann, K., & Radlinski, F. (2015) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/2766462.2767695 · Cited by 38 · cited 1×
“Finding the right user satisfaction metric based on page-wise user responses is not the focus of this paper, and can itself be a substantial research topic in interactive information systems [21, 30, 38].”
BackgroundWhole-page optimization and submodular welfare maximization with online bidders
Devanur, N. R., Huang, Z., Korula, N., Mirrokni, V., & Yan, Q. (2013) · ACM Conference on Electronic Commerce (EC) · doi:10.1145/2482540.2482603 · Cited by 37 · cited 1×
“Our work is related to the whole-page optimization for sponsored search [16] or online ads bidding [33], where the focus is to optimize revenue of the search service provider.”
BackgroundOptimizing two-dimensional search results presentation
Chierichetti, F., Kumar, R., & Raghavan, P. (2011) · ACM International Conference on Web Search and Data Mining (WSDM) · doi:10.1145/1935826.1935873 · Cited by 27 · cited 1×
“[14] considers 2-D grid presentation for image and shopping results.”
BackgroundWhole page optimization: how page elements interact with the position auction
Metrikov, P., Díaz, F., Lahaie, S. M., & Rao, J. M. (2014) · ACM Conference on Economics and Computation (EC) · doi:10.1145/2600057.2602871 · Cited by 21 · cited 1×
“Our work is related to the whole-page optimization for sponsored search [16] or online ads bidding [33], where the focus is to optimize revenue of the search service provider.”
BackgroundEvaluating whole-page relevance
Bailey, P., Craswell, N., White, R. W., Chen, L., Satyanarayana, A., & Tahaghoghi, S. M. M. (2010) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/1835449.1835606 · Cited by 20 · cited 1×
“[9] propose the notion of whole-page relevance.”
A unified search federation system based on online user feedback
Jie, L., Lamkhede, S., Sapra, R., Hsu, E., Song, H., & Chang, Y. (2013) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/2487575.2488198 · Cited by 20 · cited 6×
“We adopt the same content features as used in [23] to facilitate direct comparison in experiments (Section 6): • Global result set features : features derived from all returned results.”
BackgroundMethodInfluentialSelf-citationRelevance Ranking for Vertical Search Engines
(2014) · Elsevier eBooks · doi:10.1016/c2012-0-02862-1 · Cited by 19
Effects of task and domain on searcher attention
Lagun, D., & Agichtein, E. (2014) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/2600428.2609516 · Cited by 15 · cited 2×
“Human eyeballs are spontaneously attracted by graphical results, causing a significant attention bias known as the vertical bias [12, 26, 31].”
BackgroundWhole-page optimization and submodular welfare maximization with online bidders
Devanur, N. R., Huang, Z., Korula, N., et al. (2013) · doi:10.1145/2492002.2482603 · Cited by 6
Relevance Ranking for Vertical Search Engines
(2014) · Elsevier eBooks · doi:10.1016/b978-0-12-407171-1.00010-1 · Cited by 6 · cited 1×
“Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page.”
BackgroundThe Evaluation of Search User Interfaces
Hearst, M. A. (2009) · Cambridge University Press eBooks · doi:10.1017/cbo9781139644082.003 · Cited by 3 · cited 1×
“Finding the right user satisfaction metric based on page-wise user responses is not the focus of this paper, and can itself be a substantial research topic in interactive information systems [21, 30, 38].”
Background
Research works printed but not linked: 3
- 10.5555/1996889.1996909 · doi:10.5555/1996889.1996909
- 10.5555/1296180 · doi:10.5555/1296180
- 10.5555/2600139 · doi:10.5555/2600139
Cited by
Unbiased Learning-to-Rank with Biased Feedback
Joachims, T., Swaminathan, A., & Schnabel, T. (2017) · Proceedings of the Tenth ACM International Conference on Web Search and Data Mining · doi:10.1145/3018661.3018699 · Cited by 531
“Our approach uses inverse propensity scoring (IPS), originally employed in survey sampling [7] and causal inference from observational studies [19], but more recently also in whole page optimization [29], IR evaluation with manual judgments [20], and recommender evaluation [13, 21].”
MethodFinding Faster Configurations Using FLASH
Nair, V., Yu, Z., Menzies, T., Siegmund, N., & Apel, S. (2020) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2018.2870895 · Cited by 166
“Among numerous other problems which are similar to performance configuration optimization, the problem of how to maximize conversions on landing pages or click-through rates on search-engine result pages [4], [36], [37] has gathered interest.”
BackgroundZhao, X., Xia, L., Tang, J., & Yin, D. (2019) · ACM SIGWEB Newsletter · doi:10.1145/3320496.3320500 · Cited by 106
“In [Wang et al. 2016; Wang et al. 2018], a framework is proposed to learn the optimal page presentation to render heterogeneous results onto SERP.”
BackgroundSelf-citationA General Framework for Counterfactual Learning-to-Rank
Agarwal, A., Takatsu, K., Zaitsev, I., & Joachims, T. (2019) · Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval · doi:10.1145/3331184.3331202 · Cited by 115
“tual approach uses inverse propensity scoring (IPS), originally employed in survey sampling [8] and causal inference from observational studies [17], but more recently also in whole page optimization [27], IR evaluation with manual judgments [18], and recommender evaluation [12, 19]. This approach is similar in spirit…”
MethodAn Efficient Bandit Algorithm for Realtime Multivariate Optimization
Hill, D. N., Nassif, H., Liu, Y., Iyer, A., & Vishwanathan, S. V. N. (2017) · Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/3097983.3098184 · Cited by 113
“Our approach is most similar to [21] where the authors used a model of pairwise interactions to optimize whole-page presentation.”
Method
Metadata
- Authors
- Wang, Y., Yin, D., Jie, L., Wang, P., Yamada, M., Chang, Y., & Mei, Q.
- Affiliations
- University of Michigan (US)
- Yahoo (United States) (US)
- Snap (United States) (US)
- Kyoto University (JP)
- Venue
- Proceedings of the Ninth ACM International Conference on Web Search and Data Mining
- Publisher
- ACM
- Published
- 2016-02-08
- Pages
- 103-112
- OpenAlex
- W2271350114
- Cited by
- 66
- References
- 41
- Citations by year
- 2026: 2 · 2025: 3 · 2024: 4 · 2023: 4 · 2022: 1 · 2021: 8 · 2020: 4 · 2019: 12 · 2018: 16 · 2017: 9 · 2016: 3
- FWCI
- 11.32
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- ACM International Conference on Web Search and Data Mining (WSDM) (publishedVersion)
- Funders
- National Science Foundation (IIS-1054199)
- Topics
- Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
- Physical Sciences › Computer Science › Information Systems › Recommender Systems and Techniques
- Physical Sciences › Computer Science › Computer Graphics and Computer-Aided Design › Data Visualization and Analytics
- Keywords
- search result presentation, ad-hoc search, search result ranking
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.