CoChat
Sign Up
PreprintCited by 7Information Retrieval and Search BehaviorTopic ModelingNatural Language Processing Techniques

Benchmark for Complex Answer Retrieval

Open access
Nanni, Federico, Mitra, Bhaskar, Magnusson, Matt, Dietz, Laura·2017·arXiv
DOI 10.48550/arxiv.1705.04803arXiv 1705.04803
Open this paper in CoChat — free accountOpen PDF

Abstract

Retrieving paragraphs to populate a Wikipedia article is a challenging task. The new TREC Complex Answer Retrieval (TREC CAR) track introduces a comprehensive dataset that targets this retrieval scenario. We present early results from a variety of approaches -- from standard information retrieval methods (e.g., tf-idf) to complex systems that using query expansion using knowledge bases and deep neural networks. The goal is to offer future participants of this track an overview of some promising approaches to tackle this problem.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.1705.04803

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Retrieval over the live web differs substantially from traditional information retrieval evaluations.

    “A key component of generative research synthesis is retrieval over the live web, which differs substantially from traditional information retrieval evaluations [46, 32].”

    From DeepScholar-Bench: A Live Benchmark and Automated Evaluation for Generative Research Synthesis (2025) · cites this paper as [32]

    The Introduction describes a benchmark for complex-answer retrieval, and the dataset passage describes ranking paragraphs from a provided Wikipedia collection. Neither passage compares live-web retrieval with traditional information-retrieval evaluations; searches for that comparison found none in our copy. (LLM, conf=0.73)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

No references could be resolved yet.

Research works printed but not linked: 22

  1. Elif Aktolga, James Allan, and David A. Smith. 2011. Passage reranking for question answering using syntactic structures and answer types. In Advances in Information Retrieval. Springer.
  2. James Allan, Bruce Cro , Alistair Moffat, and Mark Sanderson. 2012. Frontiers, challenges, and opportunities for information retrieval: Re- port from SWIRL 2012 the second strategic workshop on information retrieval in Lorne. In ACM SIGIR Forum, Vol. 46. ACM, 2-32.
  3. Sören Auer, Christian Bizer, Georgi Kobilarov, Jens Lehmann, Richard Cyganiak, and Zachary Ives. 2007. Dbpedia: A nucleus for a web of open data. Springer.
  4. Siddhartha Banerjee and Prasenjit Mitra. 2015. WikiKreator: Improving Wikipedia Stubs Auto- matically.. In ACL (1). 867-877.
  5. Michael Bendersky, W Bruce Cro , and Yan- lei Diao. 2011. ality-biased ranking of web documents. In WSDM.
  6. Michael Bendersky and Oren Kurland. 2008. Uti- lizing passage-based language models for doc- ument retrieval. In Advances in Information Retrieval. Springer.
  7. Michael Bendersky, Donald Metzler, and W Bruce Cro . 2012. Effective query formu- lation with multiple information sources. In WSDM.
  8. Roi Blanco and Hugo Zaragoza. 2010. Finding support sentences for entities. In SIGIR.
  9. Jeffrey Dalton, Laura Dietz, and James Allan. 2014. Entity ery Feature Expansion Using Knowledge Base Links. In SIGIR.
  10. Laura Dietz and Michael Schuhmacher. 2015. An interface sketch for queripidia: ery- driven knowledge portfolios from the web. In Proc. Workshop on Exploiting Semantic Annota- tions in IR. ACM, 43-46.
  11. Paolo Ferragina and Ugo Scaiella. 2010. Tagme: on-the-fly annotation of short text fragments (by wikipedia entities). In CIKM. ACM.
  12. Alexander Kotov and ChengXiang Zhai. 2012. Tapping into knowledge base for concept feed- back: leveraging conceptnet to improve search results for difficult queries. In WSDM.
  13. Victor Lavrenko and W Bruce Cro . 2001. Rel- evance based language models. In SIGIR.
  14. Hang Li. 2014. Learning to rank for information retrieval and natural language processing. Syn- thesis Lectures on Human Language Technologies 7, 3 (2014).
  15. Xitong Liu and Hui Fang. 2015. Latent entity space: a novel retrieval approach for entity- bearing queries. Information Retrieval Journal 18, 6 (2015).
  16. Bhaskar Mitra and Nick Craswell. 2017. Neural Models for Information Retrieval. arXiv preprint arXiv:1705.01509 (2017). · arXiv
  17. Jeffrey Pennington, Richard Socher, and Christopher D Manning. 2014. Glove: Global Vectors for Word Representation.. In EMNLP, Vol. 14.
  18. Hadas Raviv, Oren Kurland, and David Carmel. 2016. Document Retrieval Using Entity-Based Language Models. In SIGIR.
  19. Ridho Reinanda, Edgar Meij, and Maarten de Ri- jke. 2015. Mining, ranking and recommending entity aspects. In SIGIR.
  20. Petar Ristoski and Heiko Paulheim. 2016. Rdf2vec: Rdf graph embeddings for data mining. In ISWC. Springer.
  21. Chenyan Xiong and Jamie Callan. 2015. Es- drank: Connecting query and documents through external semi-structured data. In CIKM.
  22. Ye Zhang, Md Mustafizur Rahman, Alex Bray- lan, Brandon Dang, Heng-Lu Chang, Henna Kim, inten McNamara, Aaron Angert, Ed- ward Banner, Vivek Khetan, and others. 2016. Neural Information Retrieval: A Literature Re- view. arXiv preprint arXiv:1611.06792 (2016). · arXiv

Cited by

Sorted by citations

Authors
Nanni, F., Mitra, B., Magnusson, M., & Dietz, L.
Affiliations
  • University of Mannheim (DE)
  • University College London (GB)
  • University of New Hampshire (US)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2017
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.1705.04803
arXiv
1705.04803
OpenAlex
W2950757134
Cited by
7
References
0
Citations by year
2019: 1 · 2018: 6
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
Keywords
query expansion, knowledge base, deep neural network, benchmark dataset

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review