CoChat
Sign Up
PreprintCited by 325Topic ModelingNatural Language Processing TechniquesInformation Retrieval and Search Behavior

ELI5: Long Form Question Answering

Open access
Fan, Angela, Jernite, Yacine, Perez, Ethan, Grangier, David, Weston, Jason, Auli, Michael·2019·arXiv
DOI 10.48550/arxiv.1907.09190arXiv 1907.09190
Open this paper in CoChat — free accountOpen PDF

Abstract

We introduce the first large-scale corpus for long-form question answering, a task requiring elaborate and in-depth answers to open-ended questions. The dataset comprises 270K threads from the Reddit forum ``Explain Like I'm Five'' (ELI5) where an online community provides answers to questions which are comprehensible by five year olds. Compared to existing datasets, ELI5 comprises diverse questions requiring multi-sentence answers. We provide a large set of web documents to help answer the question. Automatic and human evaluations show that an abstractive model trained with a multi-task objective outperforms conventional Seq2Seq, language modeling, as well as a strong extractive baseline. However, our best model is still far from human performance since raters prefer gold responses in over 86% of cases, leaving ample opportunity for future improvement.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.1907.09190

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Existing benchmarks like NaturalQuestions, TriviaQA, and ELI5 focus on specific aspects but lack comprehensive assessment.

    “Existing benchmarks like NaturalQuestions (Kwiatkowski et al., 2019), TriviaQA (Joshi et al., 2017), and ELI5 (Fan et al., 2019) focus on specific aspects but lack comprehensive assessment.”

    From Fact, Fetch, and Reason: A Unified Evaluation of Retrieval-Augmented Generation (2024) · cites this paper as Fan et al., 2019

    The Introduction identifies ELI5 as a dataset focused on long-form question answering, supporting its use as an example of a benchmark with a specific focus. The paper does not say that ELI5 lacks comprehensive assessment; its Evaluation Methods section describes several ways it assesses answers. Our copy provides section labels but no page numbers. (LLM, conf=0.82)

    Experimental — accuracy not yet measured

  • ELI5 targets explainability without rigorous multi-hop reasoning evaluation.

    “NaturalQuestions tests retrieval precision, TriviaQA emphasizes factual correctness, and ELI5 targets explainability without rigorous multi-hop reasoning evaluation.”

    From Fact, Fetch, and Reason: A Unified Evaluation of Retrieval-Augmented Generation (2024) · cites this paper as Fan et al., 2019

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

No citation lists for this paper

No citation lists for this paper yet — checked again daily.

References 29 · Cited by 325

Research works printed but not linked: 24

  1. Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. 2017. Enriching word vectors with subword information. TACL, 5:135-146.
  2. Danqi Chen, Adam Fisch, Jason Weston, and Antoine Bordes. 2017. Reading wikipedia to answer open- domain questions. In ACL. · DOI · Paper page
  3. Eunsol Choi, He He, Mohit Iyyer, Mark Yatskar, Wen tau Yih, Yejin Choi, Percy Liang, and Luke Zettle- moyer. 2018. Quac: Question answering in context. In EMNLP.
  4. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understand- ing. CoRR, abs/1810.04805. · arXiv
  5. Matthew Dunn, Levent Sagun, Mike Higgins, V. Ugur Güney, Volkan Cirik, and Kyunghyun Cho. 2017. Searchqa: A new q&a dataset augmented with con- text from a search engine. CoRR, abs/1704.05179. · arXiv
  6. Angela Fan, David Grangier, and Michael Auli. 2018. Controllable abstractive summarization. In ACL Workshop on Neural Machine Translation and Gen- eration.
  7. Jonas Gehring, Michael Auli, David Grangier, Denis Yarats, and Yann N Dauphin. 2017. Convolutional Sequence to Sequence Learning. In Proc. of ICML.
  8. Jeff Johnson, Matthijs Douze, and Hervé Jégou. 2017. Billion-scale similarity search with gpus. CoRR, abs/1702.08734. · arXiv
  9. Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer. 2017. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehen- sion. In ACL. · DOI · Paper page
  10. Tomas Kocisky, Jonathan Schwarz, Phil Blunsom, Chris Dyer, Karl Moritz Hermann, Gabor Melis, and Edward Grefenstette. 2018. The narrativeqa reading comprehension challenge. TACL.
  11. Chin-Yew Lin. 2004. Rouge: a package for automatic evaluation of summaries. In ACL Workshop on Text Summarization Branches Out.
  12. Peter J. Liu, Mohammad Saleh, Etienne Pot, Ben Goodrich, Ryan Sepassi, Lukasz Kaiser, and Noam Shazeer. 2018. Generating wikipedia by summariz- ing long sequences. In ICLR.
  13. Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, and Li Deng. 2016. Ms marco: A human generated machine read- ing comprehension dataset. CoRR.
  14. Myle Ott, Sergey Edunov, David Grangier, and Michael Auli. 2018. Scaling neural machine trans- lation. In WMT, pages 1-9. Association for Compu- tational Linguistics.
  15. Romain Paulus, Caiming Xiong, and Richard Socher. 2017. A deep reinforced model for abstractive sum- marization. arXiv preprint arXiv:1705.04304. · arXiv
  16. Pranav Rajpurkar, Robin Jia, and Percy Liang. 2018. Know what you don't know: Unanswerable ques- tions for squad. In ACL.
  17. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. Squad: 100,000+ questions for machine comprehension of text. In EMNLP. · DOI · Paper page
  18. Siva Reddy, Danqi Chen, and Christopher D Manning. 2018. Coqa: A conversational question answering challenge. arXiv preprint arXiv:1808.07042. · arXiv
  19. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. In ACL.
  20. Minjoon Seo, Aniruddha Kembhavi, Ali Farhadi, and Hannaneh Hajishirzi. 2017. Bidirectional attention flow for machine comprehension. In ICLR.
  21. Anastasios Tombros and Mark Sanderson. 1998. Ad- vantages of query biased summaries in information retrieval. In SIGIR. · DOI
  22. Adam Trischler, Tong Wang, Xingdi Yuan, Justin Har- ris, Alessandro Sordoni, Philip Bachman, and Ka- heer Suleman. 2017. Newsqa: A machine compre- hension dataset. In ACL Workshop on Representa- tion Learning for NLP.
  23. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. NIPS.
  24. Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Ben- gio, William W Cohen, Ruslan Salakhutdinov, and Christopher D Manning. 2018. Hotpotqa: A dataset for diverse, explainable multi-hop question answer- ing. arXiv preprint arXiv:1809.09600. · arXiv

Authors
Fan, A., Jernite, Y., Perez, E., Grangier, D., Weston, J., & Auli, M.
Affiliations
  • Meta (United States) (US)
  • New York University (US)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2019
Pages
3558-3567
ISSN
0736-587X (ISSN-L)
DOI
10.48550/arxiv.1907.09190
arXiv
1907.09190
OpenAlex
W2950681488
Cited by
325
References
29
Citations by year
2026: 27 · 2025: 59 · 2024: 35 · 2023: 67 · 2022: 63 · 2021: 46 · 2020: 21 · 2019: 7
FWCI
12.82
Type
conference paper
Language
en
Open access
Licence
Locations
  • Proceedings of the conference - Association for Computational Linguistics. Meeting (publishedVersion, cc-by)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
  • Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
Keywords
long-form question answering, Reddit, corpus construction, abstractive summarization, open-ended questions, multi-task learning, human evaluation

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review