CoChat

Lightweight and Direct Document Relevance Optimization for Generative Information Retrieval

Open access
Mekonnen, Kidist Amde, Tang, Yubao, de Rijke, Maarten · 2025 · Proceedings of the 48th International ACM SIGIR Conference on Research and Development in Information Retrieval

The DOI matches its Crossref record; 88 of the 88 references it cites were found in a registry or an index; 7 of the 10 sources its claims cite were checked for support.

Abstract

Generative information retrieval (GenIR) is a promising neural retrieval paradigm that formulates document retrieval as a document identifier (docid) generation task, allowing for end-to-end optimization toward a unified global retrieval objective. To address token-level misalignment, we propose direct document relevance optimization (DDRO), which aligns token-level docid generation with document-level relevance estimation through direct optimization via pairwise ranking, eliminating the need for explicit reward modeling and reinforcement learning. Experimental results show DDRO achieves a 7.4% improvement in MRR@10 for MS MARCO and a 19.9% improvement for Natural Questions.

Overview

CoChat Summary

TL;DR

DDRO aligns token-level docid generation with document-level relevance by adding pairwise ranking after supervised fine-tuning, avoiding reward modeling and reinforcement learning. On MS MARCO and NQ320k it improves early ranking over GenRRL and established baselines, including 7.4% and 19.9% MRR@10 gains over RL-based methods.

Key findings

  • DDRO outperforms RLRF/GenRRL: on MS MARCO, DDRO (TU) beats GenRRL (Sum) by 15.06% in R@1 and 7.4% in MRR@10; on NQ, DDRO (PQ) beats GenRRL (Sum) by 34.69% in R@1 and 19.87% in MRR@10.
  • Against established baselines: on MS MARCO, DDRO (TU) improves over the best baseline ROGER-Ultron by 15.63% in R@1 and 8.03% in MRR@10; on NQ, DDRO (PQ) improves over ROGER-Ultron (TU) by 36.27% in R@1 and 23.58% in MRR@10.
  • Ablation shows pairwise ranking matters: removing it lowers NQ DDRO (PQ) from 48.92 to 44.19 R@1 and from 55.51 to 50.48 MRR@10; on MS MARCO, DDRO (TU) drops from 66.46 to 64.60 R@5 and from 74.01 to 72.90 R@10.
  • Docid design interacts with dataset: DDRO (TU) is strongest on MS MARCO (38.24 R@1, 50.07 MRR@10), while DDRO (PQ) is strongest on NQ (48.92 R@1, 55.51 MRR@10).

Methods

  • Two docid types are constructed: TU (reverse URL if semantically rich, otherwise title + domain) and PQ (product quantization codes from a T5-based encoder).
  • Supervised fine-tuning uses T5-base with indexing and retrieval tasks, passage-to-docid pairs, tf-idf-selected term-to-docid pairs, and pseudo-queries generated by DocT5Query (10 per document).
  • DDRO then directly optimizes pairwise ranking with a reference-policy constraint, initialized from the SFT model and trained with a modified Hugging Face TRL DPOTrainer; experiments use MS MARCO document ranking (320k documents, 360k query-document pairs) and NQ320k (320k query-document pairs), evaluated by R@1/5/10 and MRR@10 with paired t-tests at p < 0.05.

Limitations

  • Stated: evaluation primarily focuses on a subset of the MS MARCO document dataset, leaving scalability to larger, domain-specific, and multilingual corpora unexplored.
  • Stated: future work includes listwise ranking, advanced hard negative mining, and multi-objective optimization; a comprehensive comparison with scalable GenIR baselines such as RIPOR and PAG remains future work.
  • Evident from the text: only two benchmark datasets are used, and document summaries are excluded as docids because they depend on external summarization models, which limits direct comparison with GenRRL (Sum).

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1145/3726302.3730023

Cites

88 of 88 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Figures

Figure 1 The proposed workflow comprises three key stages: (1) Construction of document identifiers (docids) including URL/title, domain, and product quantization codes; (2) Supervised fine-tuning of the retrieval model 𝜋 ref 𝜃 using diverse data pairs; and (3) Freezing the trained reference policy model 𝜋 ref 𝜃 and performing direct learning-to-rank (L2R) optimization on a policy model 𝜋 𝜃 .
Figure 1 The proposed workflow comprises three key stages: (1) Construction of document identifiers (docids) including URL/title, domain, and product quantization codes; (2) Supervised fine-tuning of the retrieval model 𝜋 ref 𝜃 using diverse data pairs; and (3) Freezing the trained reference policy model 𝜋 ref 𝜃 and performing direct learning-to-rank (L2R) optimization on a policy model 𝜋 𝜃 .CC-BY · From the PDF
Figure 3 Architecture of the DDRO model, which fine-tunes the retrieval model through direct learning-to-rank (L2R) optimization using relevance feedback. Unlike GenRRL [68], DDRO directly optimizes with relevance judgment data, avoiding reinforcement learning, explicit reward modeling, and extensive hyperparameter tuning. For clarity , the model 𝜋 ref 𝜃 from the SFT phase is referred to as 𝜋 ref , with its parameters frozen during this phase.
Figure 3 Architecture of the DDRO model, which fine-tunes the retrieval model through direct learning-to-rank (L2R) optimization using relevance feedback. Unlike GenRRL [68], DDRO directly optimizes with relevance judgment data, avoiding reinforcement learning, explicit reward modeling, and extensive hyperparameter tuning. For clarity , the model 𝜋 ref 𝜃 from the SFT phase is referred to as 𝜋 ref , with its parameters frozen during this phase.CC-BY · From the PDF
Figure 4 Comparison of R@1 and MRR@10 metrics for the MS MARCO dataset across various retrieval models.
Figure 4 Comparison of R@1 and MRR@10 metrics for the MS MARCO dataset across various retrieval models.CC-BY · From the PDF
(a) R@1 - Natural Questions
(a) R@1 - Natural QuestionsCC-BY · From the PDF
Figure 5 Comparison of R@1 and MRR@10 metrics for the Natural Questions dataset across various retrieval models.
Figure 5 Comparison of R@1 and MRR@10 metrics for the Natural Questions dataset across various retrieval models.CC-BY · From the PDF

Verification

Claims the paper makes

2 Valid · 3 Not valid · 1 Not checked · 50 Own results and interpretations

18 more claims

Create a free account to see all 56 claims and to verify the 2 references still open.

See all claims — free account
  • It may state two separate claims as one. 25
  • Probably not a substantial claim. 11
  • Borderline: one of the tests for a substantial claim is unclear. 2
  • The quote only partly states this claim. 2

Create a free account to see these claims.

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

3 of 3

Citations

Cites

Show all 88 references

Research works printed but not linked: 12

  1. Christopher JC Burges. 2010. From RankNet to LambdaRank to LambdaMart: An Overview. Learning, Vol. 11, 23--581 (2010), 81.
  2. Yanyan Lan, Yadong Zhu, Jiafeng Guo, Shuzi Niu, and Xueqi Cheng. 2014. Position-aware ListMLE: A Sequential Learning Process for Ranking. In Proceedings of the Thirtieth Conference on Uncertainty in Artificial Intelligence (Quebec City, Quebec, Canada) (UAI'14). AUAI Press, Arlington, Virginia, USA, 449--458.
  3. 10.5555/3600270.3602281 · doi:10.5555/3600270.3602281
  4. 10.5555/3666122.3666574 · doi:10.5555/3666122.3666574
  5. Stephen E. Robertson, Steve Walker, Susan Jones, Micheline Hancock-Beaulieu, and Mike Gatford. 1994. Okapi at TREC-3. In Proceedings of the Third Text REtrieval Conference (TREC-3). National Institute of Standards and Technology (NIST), Gaithersburg, Maryland, USA, 109-126. https://www.microsoft.com/en-us/research/publication/okapi-at-trec-3/
  6. 10.5555/3495724.3495977 · doi:10.5555/3495724.3495977
  7. 10.5555/3666122.3668132 · doi:10.5555/3666122.3668132
  8. Yubao Tang, Ruqing Zhang, Jiafeng Guo, Maarten de Rijke, Wei Chen, and Xueqi Cheng. 2024a. Generative Retrieval Meets Multi-Graded Relevance. In Advances in Neural Information Processing Systems, A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang (Eds.), Vol. 37. Curran Associates, Inc., 72790-72817. https://proceedings.neurips.cc/paper_files/paper/2024/file/853e781cb2af58956ed5c89aa59da3fc-Paper-Conference.pdf
  9. Yubao Tang, Ruqing Zhang, Zhaochun Ren, Jiafeng Guo, and Maarten de Rijke. 2024 e. Recent Advances in Generative Information Retrieval. In Advances in Information Retrieval, Nazli Goharian, Nicola Tonellotto, Yulan He, Aldo Lipani, Graham McDonald, Craig Macdonald, and Iadh Ounis (Eds.). Springer Nature Switzerland, Cham, 363-368.
  10. 10.1145/3589335.3641239 · DOI
  11. 10.5555/3600270.3601857 · doi:10.5555/3600270.3601857
  12. 10.5555/3600270.3602126 · doi:10.5555/3600270.3602126

Cited by

See all 3

Metadata

Authors
Mekonnen, K. A., Tang, Y., & de Rijke, M.
Affiliations
  • University of Amsterdam (NL)
  • University of Amsterdam, Amsterdam, Netherlands
Venue
Proceedings of the 48th International ACM SIGIR Conference on Research and Development in Information Retrieval
Publisher
ACM
Published
2025-07-13
Pages
1327-1338
OpenAlex
W4412376958
Cited by
3
References
72
Citations by year
2026: 2 · 2025: 1
FWCI
3.43
Type
conference paper
Language
en
Open access
Licence
cc-by
Locations
  • International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) (publishedVersion, cc-by)
  • UvA-DARE (University of Amsterdam) (publishedVersion, cc-by)
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • arXiv (Cornell University) (cc-by)
Funders
  • Funder: Dutch Research Council (Identifiers: 024.004.022, NWA.1389.20.\-183, KICH3.LTP.20.006)
  • Funder: Ahold Delhaize (Identifier: AIRLab Amsterdam)
  • Funder: EU (Identifier: 101070212.)
Topics
  • Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Information Systems › Recommender Systems and Techniques
Keywords
pairwise ranking, reward modeling, MS MARCO, Natural Questions

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (gpt-6-sol and qwen3.8-omni-flash).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Lightweight and Direct Document Relevance Optimization for Generative Information Retrieval. https://app.cochat.ai/papers/10.1145/3726302.3730023
The paper
Mekonnen, K. A., Tang, Y., & de Rijke, M. (2025). Lightweight and Direct Document Relevance Optimization for Generative Information Retrieval. Proceedings of the 48th International ACM SIGIR Conference on Research and Development in Information Retrieval. https://doi.org/10.1145/3726302.3730023

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.