The DOI matches its Crossref record; 81 of the 81 references it cites were found in a registry or an index.
Constructing Datasets for Multi-hop Reading Comprehension Across Documents
Open accessAbstract
Most Reading Comprehension methods limit themselves to queries which can be answered using a single sentence, paragraph, or document. Enabling models to combine disjoint pieces of textual evidence would extend the scope of machine comprehension methods, but currently no resources exist to train and test this capability. We propose a novel task to encourage the development of models for text understanding across multiple documents and to investigate the limits of existing methods. In our task, a model learns to seek and combine evidence — effectively performing multihop, alias multi-step, inference. We devise a methodology to produce datasets for this task, given a collection of query-answer pairs and thematically linked documents. Two datasets from different domains are induced, and we identify potential pitfalls and devise circumvention strategies. We evaluate two previously proposed competitive models and find that one can integrate information across documents. However, both models struggle to select relevant information; and providing documents guaranteed to be relevant greatly improves their performance. While the models outperform several strong baselines, their best accuracy reaches 54.5% on an annotated test set, compared to human performance at 85.0%, leaving ample room for improvement.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
The Measurement of Observer Agreement for Categorical Data
Landis, J. R., & Koch, G. G. (1977) · Biometrics · doi:10.2307/2529310 · Cited by 81,148
Gene Ontology: tool for the unification of biology
Ashburner, M., Ball, C. A., Blake, J. A., Botstein, D., Butler, H., Cherry, J. M., Davis, A. P., Dolinski, K., Dwight, S. S., Eppig, J. T., et al. (2000) · Nature Genetics · doi:10.1038/75556 · Cited by 43,206
Glove: Global Vectors for Word Representation
Pennington, J., Socher, R., & Manning, C. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1162 · Cited by 34,071
UniProt: the Universal Protein knowledgebase
Apweiler, R. (2004) · Nucleic Acids Research · doi:10.1093/nar/gkh131 · Cited by 7,498
SQuAD: 100,000+ Questions for Machine Comprehension of Text
Rajpurkar, P., Zhang, J., Lopyrev, K., & Liang, P. (2016) · Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d16-1264 · Cited by 6,208 · cited 4×
“IA has thus been used for a wealth of research to build datasets posing queries about a single sentence (Morales et al., 2016; Levy et al., 2017) or article (Yang et al., 2015a; Hewlett et al., 2016; Rajpurkar et al., 2016). However, no attempt has been made to construct a cross-document multi-step RC dataset based on…”
BackgroundMethodInfluentialThe Reactome pathway Knowledgebase
Fabregat, A., Sidiropoulos, K., Garapati, P., Gillespie, M., Hausmann, K., Haw, R., Jassal, B., Jupe, S., Korninger, F., McKay, S., et al. (2016) · Nucleic Acids Research · doi:10.1093/nar/gkv1351 · Cited by 5,605
Freebase: a collaboratively created graph database for structuring human knowledge
Bollacker, K., Evans, C., Paritosh, P., Sturge, T., & Taylor, J. (2008) · Proceedings of the 2008 ACM SIGMOD international conference on Management of data · doi:10.1145/1376616.1376746 · Cited by 4,852
Distant supervision for relation extraction without labeled data
Mintz, M., Bills, S., Snow, R., & Jurafsky, D. (2009) · Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP: Volume 2 - ACL-IJCNLP '09 · doi:10.3115/1690219.1690287 · Cited by 2,752
Richardson, M., & Domingos, P. (2006) · Machine Learning · doi:10.1007/s10994-006-5833-1 · Cited by 2,676
Teaching Machines to Read and Comprehend
Hermann, K. M., Kočiský, T., Grefenstette, E., Espeholt, L., Kay, W., Suleyman, M., & Blunsom, P. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1506.03340 · Cited by 2,521
MS MARCO: A Human Generated MAchine Reading COmprehension Dataset
Bajaj, P., Campos, D., Craswell, N., Deng, L., Gao, J., Liu, X., Majumder, R., McNamara, A., Mitra, B., Nguyen, T., et al. (2016) · arXiv · doi:10.48550/arxiv.1611.09268 · Cited by 2,473
DrugBank 4.0: shedding new light on drug metabolism
Law, V., Knox, C., Djoumbou, Y., Jewison, T., Guo, A. C., Liu, Y., Maciejewski, A., Arndt, D., Wilson, M., Neveu, V., et al. (2014) · Nucleic Acids Research · doi:10.1093/nar/gkt1068 · Cited by 1,993
TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension
Joshi, M., Choi, E., Weld, D., & Zettlemoyer, L. (2017) · Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p17-1147 · Cited by 1,690
TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension
Joshi, M., Choi, E., Weld, D. S., & Zettlemoyer, L. (2017) · arXiv · doi:10.48550/arxiv.1705.03551 · Cited by 1,650
Learning Logical Definitions from Relations
Quinlan, J. R. (1990) · Machine Learning · doi:10.1023/a:1022699322624 · Cited by 1,644
Semantic Parsing on Freebase from Question-Answer Pairs
Berant, J., Chou, A., Frostig, R., & Liang, P. (2013) · Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d13-1160 · Cited by 1,517
Bidirectional Attention Flow for Machine Comprehension
Seo, M. J., Kembhavi, A., Farhadi, A., & Hajishirzi, H. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1611.01603 · Cited by 1,445
Adversarial Examples for Evaluating Reading Comprehension Systems
Jia, R., & Liang, P. (2017) · Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d17-1215 · Cited by 1,410
Open information extraction from the web
Banko, M., Cafarella, M., Soderland, S. G., Broadhead, M., & Etzioni, O. (2007) · International Joint Conference on Artificial Intelligence · Cited by 1,398
Sukhbaatar, S., Szlam, A., Weston, J., & Fergus, R. (2015) · arXiv · doi:10.48550/arxiv.1503.08895 · Cited by 1,345
Modeling Relations and Their Mentions without Labeled Text
Riedel, S., Yao, L., & McCallum, A. (2010) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-15939-8_10 · Cited by 1,270
Self-citationA Survey of Automatic Query Expansion in Information Retrieval
Carpineto, C., & Romano, G. (2012) · ACM Computing Surveys · doi:10.1145/2071389.2071390 · Cited by 1,109
WikiQA: A Challenge Dataset for Open-Domain Question Answering
Yang, Y., Yih, W.-T., & Meek, C. (2015) · Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d15-1237 · Cited by 829
SQuAD: 100,000+ Questions for Machine Comprehension of Text
Rajpurkar, P., Zhang, J., Lopyrev, K., & Liang, P. (2016) · arXiv · doi:10.48550/arxiv.1606.05250 · Cited by 678
Question Answering with Subgraph Embeddings
Bordes, A., Chopra, S., & Weston, J. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1067 · Cited by 635
Relational retrieval using a combination of path-constrained random walks
Lao, N., & Cohen, W. W. (2010) · Machine Learning · doi:10.1007/s10994-010-5205-8 · Cited by 614
Constructing biological knowledge bases by extracting information from text sources.
Craven, M. W., & Kumlien, J. (1999) · PubMed · Cited by 582
MS MARCO: A Human Generated MAchine Reading COmprehension Dataset.
Nguyen, T. G., Rosenberg, M., Xia, S., Gao, J., Tiwary, S., Majumder, R., & Deng, L. (2016) · neural information processing systems · Cited by 539
Overview of BioCreAtIvE: critical assessment of information extraction for biology
Hirschman, L., Yeh, A., Blaschke, C., & Valencia, A. (2005) · BMC Bioinformatics · doi:10.1186/1471-2105-6-s1-s1 · Cited by 530
Large-scale Simple Question Answering with Memory Networks
Bordes, A., Usunier, N., Chopra, S., & Weston, J. (2015) · arXiv · doi:10.48550/arxiv.1506.02075 · Cited by 528
Discovery of inference rules for question-answering
LIN, D., & PANTEL, P. (2001) · Natural Language Engineering · doi:10.1017/s1351324901002765 · Cited by 526
A Thorough Examination of the CNN/Daily Mail Reading Comprehension Task
Chen, D., Bolton, J., & Manning, C. D. (2016) · Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p16-1223 · Cited by 491
Zero-Shot Relation Extraction via Reading Comprehension
Levy, O., Seo, M., Choi, E., & Zettlemoyer, L. (2017) · arXiv · doi:10.48550/arxiv.1706.04115 · Cited by 482 · cited 3×
“same property as in q and keeping their answer expressions. As an example, for the WIKI6 We use the WIKIDATA triples present in WIKIREADING. 7 We thus use a disjoint subset of WIKIREADING compared to Levy et al. (2017) to construct WIKIHOP. DATA property country, this would be the set fFrance;Russia;:::g. Graph traver…”
Zero-Shot Relation Extraction via Reading Comprehension
Levy, O., Seo, M., Choi, E., & Zettlemoyer, L. (2017) · Proceedings of the 21st Conference on Computational Natural Language Learning (CoNLL 2017) · doi:10.18653/v1/k17-1034 · Cited by 482 · cited 3×
“same property as in q and keeping their answer expressions. As an example, for the WIKI6 We use the WIKIDATA triples present in WIKIREADING. 7 We thus use a disjoint subset of WIKIREADING compared to Levy et al. (2017) to construct WIKIHOP. DATA property country, this would be the set fFrance;Russia;:::g. Graph traver…”
Random Walk Inference and Learning in A Large Scale Knowledge Base
Lao, N., Mitchell, T. M., & Cohen, W. W. (2018) · Figshare · doi:10.1184/r1/6476267 · Cited by 482
The Goldilocks Principle: Reading Children's Books with Explicit Memory Representations
Hill, F., Bordes, A., Chopra, S., & Weston, J. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1511.02301 · Cited by 467
Dynamic Coattention Networks For Question Answering
Xiong, C., Zhong, V., & Socher, R. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1611.01604 · Cited by 450
Gurulingappa, H., Rajput, A. M., Roberts, A., Fluck, J., Hofmann-Apitius, M., & Toldo, L. (2012) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2012.04.008 · Cited by 446
Cross-Sentence N-ary Relation Extraction with Graph LSTMs
Peng, N., Poon, H., Quirk, C., Toutanova, K., & Yih, W.-T. (2017) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00049 · Cited by 443
Swanson, D. R. (1986) · The Library Quarterly · doi:10.1086/601720 · Cited by 370
SearchQA: A New Q&A Dataset Augmented with Context from a Search Engine
Dunn, M., Sagun, L., Higgins, M., Guney, V. U., Cirik, V., & Cho, K. (2017) · arXiv · doi:10.48550/arxiv.1704.05179 · Cited by 359
Swiss-Prot: Juggling between evolution and stability
Bairoch, A. (2004) · Briefings in Bioinformatics · doi:10.1093/bib/5.1.39 · Cited by 357
Adaptive Computation Time for Recurrent Neural Networks
Graves, A. (2016) · arXiv · doi:10.48550/arxiv.1603.08983 · Cited by 335 · cited 2×
“…thereof (Peng et al., 2015; Weissenborn, 2016; Xiong et al., 2016; Liu and Perez, 2017), which often is iterated over several times (Sordoni et al., 2016; Mark et al., 2016; Seo et al., 2016; Hu et al., 2017) and may contain train-able stopping mechanisms (Graves, 2016; Shen et al., 2017b).”
BackgroundSegura-Bédmar, I., Martínez, P., & Herrero-Zazo, M. (2013) · LA Referencia (Red Federada de Repositorios Institucionales de Publicaciones Científicas) · Cited by 297
Text Understanding with the Attention Sum Reader Network
Kadlec, R., Schmid, M., Bajgar, O., & Kleindienst, J. (2016) · Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p16-1086 · Cited by 295
ReasoNet: Learning to Stop Reading in Machine Comprehension
Shen, Y., Huang, P.-S., Gao, J., & Chen, W. (2017) · Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/3097983.3098177 · Cited by 262
Towards AI-Complete Question Answering: A Set of Prerequisite Toy Tasks
Weston, J., Bordes, A., Chopra, S., Rush, A. M., van Merriënboer, B., Joulin, A., & Mikolov, T. (2016) · International Conference on Learning Representations · Cited by 261
Wikidata: a new platform for collaborative data collection
Vrandečić, D. (2012) · Proceedings of the 21st International Conference on World Wide Web · doi:10.1145/2187980.2188242 · Cited by 241
Chains of Reasoning over Entities, Relations, and Text using Recurrent Neural Networks
Das, R., Neelakantan, A., Belanger, D., & McCallum, A. (2017) · Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers · doi:10.18653/v1/e17-1013 · Cited by 231
Compositional Vector Space Models for Knowledge Base Completion
Neelakantan, A., Roth, B., & McCallum, A. (2015) · Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.3115/v1/p15-1016 · Cited by 230
Crowdsourcing Multiple Choice Science Questions
Welbl, J., Liu, N. F., & Gardner, M. (2017) · Proceedings of the 3rd Workshop on Noisy User-generated Text · doi:10.18653/v1/w17-4413 · Cited by 188
Self-citationLearning First-Order Horn Clauses from Web Text
Schoenmackers, S., Etzioni, O., Weld, D. S., & Davis, J. (2010) · Lirias · Cited by 168
Making Neural QA as Simple as Possible but not Simpler
Weissenborn, D., Wiese, G., & Seiffe, L. (2017) · Proceedings of the 21st Conference on Computational Natural Language Learning (CoNLL 2017) · doi:10.18653/v1/k17-1028 · Cited by 164
Ask the Right Questions: Active Question Reformulation with Reinforcement LearningPreprint
Buck, C., Bulian, J., Ciaramita, M., Gajewski, W., Gesmundo, A., Houlsby, N., & Wang, W. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1705.07830 · Cited by 155
Improving Information Extraction by Acquiring External Evidence with Reinforcement Learning
Narasimhan, K., Yala, A., & Barzilay, R. (2016) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d16-1261 · Cited by 150
Combining Retrieval, Statistics, and Inference to Answer Elementary Science Questions
Clark, P. E., Etzioni, O., Khot, T., Sabharwal, A., Tafjord, O., Turney, P. D., & Khashabi, D. (2016) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v30i1.10325 · Cited by 140
Who did What: A Large-Scale Person-Centered Cloze Dataset
Onishi, T., Wang, H., Bansal, M., Gimpel, K., & McAllester, D. (2016) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d16-1241 · Cited by 133
WikiReading: A Novel Large-scale Language Understanding Task over Wikipedia
Hewlett, D., Lacoste, A., Jones, L., Polosukhin, I., Fandrianto, A., Han, J. J., Kelcey, M., & Berthelot, D. (2016) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p16-1145 · Cited by 132
The Effect of Different Writing Tasks on Linguistic Style: A Case Study of the ROC Story Cloze Task
Schwartz, R., Sap, M., Konstas, I., Zilles, L., Choi, Y., & Smith, N. A. (2017) · Conference on Computational Natural Language Learning · doi:10.18653/v1/k17-1004 · Cited by 126
2011. Overview of genia event task in bionlp shared task 2011
Kim, J.-D., Takagi, T., Wang, Y., & Yonezawa, A. (2012) · Cited by 124
DISCOVERY AND EXPLANATION OF DRUG-DRUG INTERACTIONS VIA TEXT MINING
Percha, B., Garten, Y., & Altman, R. B. (2011) · Pacific Symposium on Biocomputing (PSB) · doi:10.1142/9789814366496_0040 · Cited by 120
Gated End-to-End Memory Networks
Liu, F., & Pérez, J. (2017) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/e17-1001 · Cited by 116
Improving Learning and Inference in a Large Knowledge-Base using Latent Syntactic Cues
Gardner, M., Talukdar, P. P., Kisiel, B., & Mitchell, T. M. (2013) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d13-1080 · Cited by 93
Query-Reduction Networks for Question AnsweringPreprint
Seo, M., Min, S., Farhadi, A., & Hajishirzi, H. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1606.04582 · Cited by 91
The LAMBADA dataset: Word prediction requiring a broad discourse context
Paperno, D., Kruszewski, G., Lazaridou, A., Pham, N. Q., Bernardi, R., Pezzelle, S., Baroni, M., Boleda, G., & Fernandez, R. (2016) · Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p16-1144 · Cited by 87
Question Answering over Knowledge Base using Factual Memory Networks
Jain, S. (2016) · doi:10.18653/v1/n16-2016 · Cited by 75
A knowledge-intensive approach to learning relational concepts
Pazzani, M. J., Brunk, C. A., & Silverstein, G. (1991) · Elsevier eBooks · doi:10.1016/b978-1-55860-200-7.50089-1 · Cited by 69
Towards Neural Network-based ReasoningPreprint
Peng, B., Lu, Z., Li, H., & Wong, K. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1508.05508 · Cited by 69
Higher-order Lexical Semantic Models for Non-factoid Answer Reranking
Fried, D., Jansen, P. A., Hahn-Powell, G., Surdeanu, M., & Clark, P. E. (2015) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00133 · Cited by 58
Scaling textual inference to the web
Schoenmackers, S., Etzioni, O., & Weld, D. S. (2008) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/1613715.1613727 · Cited by 55
Richards, B. L., & Mooney, R. J. (1991) · Elsevier eBooks · doi:10.1016/b978-1-55860-200-7.50092-1 · Cited by 51
BioNLP Shared Task 2011: Supporting Resources
Stenetorp, P., Topić, G., Pyysalo, S., Ohta, T., Kim, J.-D., & Tsujii, J. (2011) · Cited by 49
Self-citationFraming QA as Building and Ranking Intersentence Answer Justifications
Jansen, P. A., Sharp, R., Surdeanu, M., & Clark, P. E. (2017) · Computational Linguistics · doi:10.1162/coli_a_00287 · Cited by 40
Results of the fifth edition of the BioASQ Challenge
Nentidis, A., Bougiatiotis, K., Krithara, A., Παλιούρας, Γ., & Kakadiaris, I. A. (2017) · doi:10.18653/v1/w17-2306 · Cited by 37
Besting the Quiz Master: Crowdsourcing Incremental Classification Games
Boyd-Graber, J. L., Satinoff, B., He, H., & Daumé, H. (2012) · Empirical Methods in Natural Language Processing · Cited by 35
Mnemonic Reader for Machine Comprehension.Preprint
Hu, M., Peng, Y., & Qiu, X. (2017) · arXiv (Cornell University) · Cited by 28
Neural Question Answering at BioASQ 5B
Wiese, G. K., Weissenborn, D., & Neves, M. (2017) · doi:10.18653/v1/w17-2309 · Cited by 28
Separating Answers from Queries for Neural Reading ComprehensionPreprint
Weissenborn, D. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1607.03316 · Cited by 19
Learning to Answer Questions from Wikipedia Infoboxes
Morales, A., Premtoon, V., Avery, C., Felshin, S., & Katz, B. (2016) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d16-1199 · Cited by 16
WebNav: A New Large-Scale Task for Natural Language based Sequential Decision Making.Preprint
Nogueira, R. F., & Cho, K. (2016) · arXiv (Cornell University) · Cited by 8
Learning to Reason With Adaptive ComputationPreprint
Neumann, M., Stenetorp, P., & Riedel, S. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1610.07647 · Cited by 8
Self-citation
Cited by
Longformer: The Long-Document Transformer
Beltagy, I., Peters, M. E., & Cohan, A. (2020) · arXiv · doi:10.48550/arxiv.2004.05150 · Cited by 2,281
HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering
Yang, Z., Qi, P., Zhang, S., Bengio, Y., Cohen, W., Salakhutdinov, R., & Manning, C. D. (2018) · Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d18-1259 · Cited by 1,804
“Recent datasets like QAngaroo (Welbl et al., 2018) and C OM - PLEX W EB Q UESTIONS (Talmor and Berant, 2018) explore different approaches of using pre-existing knowledge bases (KB) with pre-defined logic rules to generate valid QA pairs, to test QA models’ capability of performing multi-hop…”
BackgroundMethodInfluentialBig Bird: Transformers for Longer Sequences
Zaheer, M., Guruganesh, G., Dubey, A., Ainslie, J., Alberti, C., Ontanon, S., Pham, P., Ravula, A., Wang, Q., Yang, L., et al. (2020) · arXiv · doi:10.48550/arxiv.2007.14062 · Cited by 1,349
CoQA: A Conversational Question Answering Challenge
Reddy, S., Chen, D., & Manning, C. D. (2019) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00266 · Cited by 975
Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answering
Mihaylov, T., Clark, P., Khot, T., & Sabharwal, A. (2018) · Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d18-1260 · Cited by 649
Metadata
- Authors
- Welbl, J., Stenetorp, P., & Riedel, S.
- Venue
- Transactions of the Association for Computational Linguistics
- Publisher
- MIT Press - Journals
- Published
- 2018-12
- Volume
- 6
- Pages
- 287-302
- ISSN
- 2307-387X
- OpenAlex
- W2963866616
- Cited by
- 456
- References
- 81
- Type
- dataset
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.