Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat
Sign Up
Cited by 14

The DOI matches its Crossref record; the references it cites are not checked yet.

A systematic review of multidimensional relevance estimation in information retrieval

Peikos, Georgios, Pasi, Gabriella·2024·WIREs Data Mining and Knowledge Discovery
DOI 10.1002/widm.1541
Open this paper in CoChat — free account

Abstract

In information retrieval, relevance is perceived as a multidimensional and dynamic concept influenced by user, task, and domain factors. Relying on this perspective, researchers have introduced multidimensional relevance models addressing diverse search tasks across numerous knowledge domains. Through our systematic review of 72 studies, we categorize research based on domain specificity and the distinct relevance aspects employed for estimating multidimensional relevance. Moreover, we highlight the approaches used to aggregate scores related to these factors and rank information items. Our insights underline the importance of concise definitions and unified methods for estimating relevance factors within and across domains. Finally, we identify benchmark collections for evaluations based on multiple relevance aspects while underscoring the necessity for new ones. Our findings suggest that large language models hold considerable promise for shaping future research in this field, mainly due to their relevance labeling abilities. This article is categorized under: Application Areas > Science and Technology Technologies > Computational Intelligence

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1002/widm.1541

Cites

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

The citation lists haven’t been fetched yet

Cited by 14

Research works printed but not linked: 118

  1. 10.1109/CIBCB.2018.8404965 · DOI
  2. 10.1016/j.ipm.2021.102552 · DOI
  3. 10.1108/EL-01-2017-0007 · DOI
  4. 10.1007/978-3-540-71496-5_40 · DOI
  5. 10.1145/2637002.2637016 · DOI
  6. 10.1145/3539618.3591933 · DOI
  7. 10.1007/978-3-319-18503-3_21 · DOI
  8. 10.1109/WI-IAT59888.2023.00022 · DOI
  9. 10.1016/S0306-4573(97)00078-2 · DOI · Paper page
  10. 10.1016/j.inffus.2021.11.010 · DOI
  11. 10.1145/2888422.2888424 · DOI
  12. 10.1145/1935826.1935849 · DOI
  13. Borlund P., 2003, The IIR evaluation model: A framework for evaluation of interactive information retrieval systems, Information Research, 8
  14. 10.1007/978-3-642-25631-8_16 · DOI
  15. 10.1016/S0169-7552(98)00110-X · DOI · Paper page
  16. Burges C. J., 2010, From RankNet to LambdaRank to LambdaMart: An overview, Learning, 11, 81
  17. 10.1145/3366423.3380122 · DOI
  18. Chall J. S., 1995, Readability revisited: The new Dale–Chall readability formula
  19. Chapelle O., 2011, Proceedings of the Learning to Rank Challenge, 1
  20. 10.1016/j.inffus.2014.09.001 · DOI
  21. 10.1108/00220411111109467 · DOI
  22. 10.6028/NIST.SP.1266.misinfo-overview · DOI
  23. 10.1145/2063576.2063639 · DOI
  24. Cool C., 1993, National Online Meeting, 77
  25. 10.7758/9781610448864 · DOI · Paper page
  26. 10.1016/0020-0271(71)90024-6 · DOI · Paper page
  27. Cosijn E., 2009, Encyclopedia of library and information sciences, 4512
  28. 10.1016/S0306-4573(99)00072-2 · DOI
  29. 10.1145/1076034.1076106 · DOI
  30. 10.1007/978-3-319-60777-1 · DOI
  31. 10.1007/978-3-642-00958-7_25 · DOI
  32. 10.1016/j.ipm.2011.07.001 · DOI
  33. 10.1145/2009916.2009933 · DOI
  34. Daoud M., 2013, Modeling geographic, temporal, and proximity contexts for improving geotemporal search, Journal of the Association for Information Science and Technology
  35. 10.1002/asi.24470 · DOI
  36. 10.1145/2661829.2661925 · DOI
  37. 10.1145/2484028.2484066 · DOI
  38. 10.1007/978-3-319-65813-1_27 · DOI
  39. 10.1145/3578337.3605136 · DOI · Paper page
  40. 10.1007/s10791-008-9046-z · DOI
  41. 10.1145/3178876.3186165 · DOI
  42. 10.1016/j.engappai.2022.105211 · DOI
  43. 10.1016/j.jbi.2023.104478 · DOI
  44. Gerani S., 2010, Proceeding of the 33rd International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2010, 403
  45. 10.1007/978-3-642-28997-2_22 · DOI
  46. Giachanou A., 2016, Advances in information retrieval—Proceedings of the 38th European Conference on IR Research, ECIR 2016, 466
  47. 10.1016/0020-0271(66)90006-4 · DOI
  48. 10.1016/j.is.2018.02.002 · DOI
  49. Ingwersen P., 2005, The turn: Integration of information seeking and retrieval in context
  50. 10.1109/WI-IAT.2012.233 · DOI
  51. 10.4108/icst.iniscom.2015.258417 · DOI
  52. 10.1145/2124295.2124350 · DOI
  53. Karmaker Santu S. K., 2017, Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR '17, 475
  54. 10.1145/344599.344630 · DOI · Paper page
  55. 10.1007/978-3-319-44406-2_16 · DOI
  56. 10.1002/asi.23868 · DOI
  57. 10.1145/3442381.3449943 · DOI
  58. 10.1145/3006299.3006315 · DOI
  59. 10.1016/j.ipm.2021.102522 · DOI
  60. Ma Y., 2023, Incorporating structural information into legal case retrieval, ACM Transactions on Information Systems, 42, 1, 10.1145/3609796 · DOI
  61. 10.1007/s11280-021-00962-8 · DOI
  62. 10.1145/3132847.3132870 · DOI
  63. 10.1145/3583069 · DOI
  64. 10.1016/S0953-5438(98)00012-5 · DOI
  65. 10.1002/asi.23094 · DOI
  66. 10.1007/978-3-319-08786-3_14 · DOI
  67. Omidvar‐Tehrani B., 2022, Proceedings of the 31st ACM International Conference on Information & Knowledge Management, CIKM '22, 3410
  68. 10.1007/s00799-011-0070-z · DOI
  69. 10.2196/10986 · DOI
  70. 10.1002/(SICI)1097-4571(199404)45:3<135::AID-ASI3>3.0.CO;2-1 · DOI
  71. 10.1007/978-3-319-91473-2_30 · DOI
  72. Putri D. G. P., 2020, Social informatics, 297
  73. 10.1007/978-3-030-85251-1_9 · DOI
  74. 10.1145/3397271.3401277 · DOI
  75. 10.1007/978-3-030-72113-8_36 · DOI
  76. 10.1145/2505515.2505667 · DOI
  77. Rekabsaz N., 2014, Information access evaluation. Multilinguality, multimodality, and interaction, 141
  78. 10.1145/1552291.1552293 · DOI
  79. Robertson S. E., 1994, Proceedings of the Third Text REtrieval Conference, TREC 1994, Gaithersburg, Maryland, USA, November 2–4, 1994, volume 500–225 of NIST Special Publication, 109
  80. 10.1109/AICCSA.2017.77 · DOI
  81. Saracevic T., 1997, Proceedings of the annual meeting‐american society for information science, 313
  82. 10.1002/asi.20682 · DOI
  83. Saracevic T., 2016, The notion of relevance in information science: Everybody knows what relevance is. But, what is it really
  84. 10.1145/3011141.3011172 · DOI
  85. 10.1016/0306-4573(90)90050-C · DOI · Paper page
  86. 10.1007/s10115-020-01470-3 · DOI
  87. 10.1145/3397271.3401120 · DOI
  88. 10.1145/1321440.1321515 · DOI · Paper page
  89. 10.1109/ICPC.2019.00054 · DOI
  90. 10.1007/978-3-030-37188-3_5 · DOI
  91. 10.2196/12522 · DOI · Paper page
  92. 10.1086/601800 · DOI
  93. 10.1016/j.ipm.2016.11.004 · DOI
  94. 10.1007/978-3-642-19047-6_6 · DOI
  95. Tsagkias M., 2021, ACM SIGIR Forum, 1
  96. 10.1007/978-3-031-20891-1_10 · DOI
  97. Upadhyay R., 2023, Machine learning and knowledge discovery in databases: Research track—European Conference, ECML PKDD 2023, 355
  98. 10.1145/3209978.3210130 · DOI
  99. 10.1109/TLT.2021.3075196 · DOI
  100. 10.1145/3343413.3377955 · DOI
  101. Doorn J., 2016, Proceedings of the 39th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR '16, 769
  102. 10.1007/s10506-017-9195-8 · DOI
  103. Vargas S., 2012, Proceedings of the 35th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR '12, 75
  104. Vickery B. C., 1959, The structure of information retrieval systems, Proceedings of the International Conference on Scientific Information, 2, 1275
  105. Vickery B. C., 1959, Subject analysis for information retrieval, Proceedings of the International Conference on Scientific Information, 2, 855
  106. 10.1145/1076034.1076107 · DOI
  107. 10.1109/T-AFFC.2011.19 · DOI
  108. 10.1002/asi.24799 · DOI
  109. Wiggers G., 2018, Proceedings of the 17th Dutch‐Belgian Information Retrieval Workshop, 5
  110. 10.1109/TCBB.2016.2578337 · DOI
  111. 10.1002/asi.20361 · DOI
  112. 10.1155/2022/9009204 · DOI
  113. Yang H., 2021, Proceedings of the 30th ACM International Conference on Information & Knowledge Management, CIKM '21, 4293
  114. Yulianti E., 2018, The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval, SIGIR '18, 335
  115. 10.1007/978-3-319-08434-3_29 · DOI
  116. 10.1007/978-3-319-28940-3_19 · DOI
  117. 10.1145/3437963.3441796 · DOI
  118. 10.1145/2851613.2851694 · DOI

Metadata

Authors
Peikos, G., & Pasi, G.
Venue
WIREs Data Mining and Knowledge Discovery
Publisher
Wiley
Published
2024-05-07
Volume
14
Issue
5
ISSN
  • 1942-4787
  • 1942-4795
DOI
10.1002/widm.1541
Cited by
14
Language
en
Licence
creativecommons.org/licenses/by-nc/4.0/ (vor)

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: A systematic review of multidimensional relevance estimation in information retrieval. https://app.cochat.ai/papers/10.1002/widm.1541
The paper
Peikos, G., & Pasi, G. (2024). A systematic review of multidimensional relevance estimation in information retrieval. WIREs Data Mining and Knowledge Discovery. https://doi.org/10.1002/widm.1541

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.