The DOI matches its Crossref record; 13 of the 13 references it cites were found in a registry or an index.
Building a question answering test collection
Abstract
The TREC-8 Question Answering (QA) Track was the first large-scale evaluation of domain-independent question answering systems. In addition to fostering research on the QA task, the track was used to investigate whether the evaluation methodology used for document retrieval is appropriate for a different natural language processing task. As with document relevance judging, assessors had legitimate differences of opinions as to whether a response actually answers a question, but comparative evaluation of QA systems was stable despite these differences. Creating a reusable QA test collection is fundamentally more difficult than creating a document retrieval test collection since the QA task has no equivalent to document identifiers.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Encyclopedia of Statistical Sciences.
Aspinall, L. J., Kotz, S., & Johnson, N. L. (1983) · Applied Statistics · doi:10.2307/2347959 · Cited by 1,258
The TREC-8 Question Answering Track Evaluation
Voorhees, E. M., & Tice, D. M. (1999) · Cited by 352
Self-citationQuestion Answering from Frequently Asked Question Files: Experiences with the FAQ Finder System
Burke, R., Hammond, K. J., Kulyukin, V., Lytinen, S. L., Tomuro, N., & Schoenberg, S. (1997) · doi:10.1609/aimag.v18i2.1294 · Cited by 333
Relevance and Information Behavior.
Schamber, L. (1994) · Cited by 321
Overview of the Sixth Text REtrieval Conference (TREC-6)
Voorhees, E. M., & Harman, D. (2000) · Information Processing & Management · doi:10.1016/s0306-4573(99)00043-6 · Cited by 317
Self-citationVariations in relevance judgments and the measurement of retrieval effectiveness
Voorhees, E. M. (1998) · Proceedings of the 21st annual international ACM SIGIR conference on Research and development in information retrieval · doi:10.1145/290941.291017 · Cited by 303
Self-citationThe DARPA High-Performance Knowledge Bases Project
Cohen, P. R., Schrag, R. C., Jones, E. K., Pease, A., Lin, A., Starr, B., Gunning, D., & Burke, M. (1998) · Cited by 150
Relevance assessments and retrieval system evaluation
Lesk, M., & Salton, G. (1968) · Information Processing & Management · doi:10.1016/0020-0271(68)90029-6 · Cited by 144
Kupiec, J. M. (1993) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/160688.160717 · Cited by 142
SRI International FASTUS system
Appelt, D. E., Hobbs, J. R., Bear, J., Israel, D. J., Kameyama, M., MARTIN, D. J., Myers, K. L., & Tyson, M. (1995) · Message Understanding Conference (MUC) · doi:10.3115/1072399.1072420 · Cited by 132
Research works printed but not linked: 7
- 10.1016/S0306-4573(99)00013-8 · doi:10.1016/s0306-4573(99)00013-8
- Boris Katz . From sentence processing to information access on the world wide web. Paper presented at the AAAI Spnng Symposium on Natural Language Processing for the World Wide Web , 1997 . Electronic version at http ://www.ai.ait.edu/ people/boris/webaccess. Boris Katz. From sentence processing to information access on the world wide web. Paper presented at the AAAI Spnng Symposium on Natural Language Processing for the World Wide Web, 1997. Electronic version at http ://www.ai.ait.edu/ people/boris/webaccess.
- Joel Martin and Chris Lankester . Ask Me Tomorrow: The University of Ottawa question answering system . In Proceedings of the Eighth Text REtrieval Conference (TREC- 8) , pages 575 - 583 , November 1999 . Notebook draft. Joel Martin and Chris Lankester. Ask Me Tomorrow: The University of Ottawa question answering system. In Proceedings of the Eighth Text REtrieval Conference (TREC- 8), pages 575-583, November 1999. Notebook draft.
- 10.1002/asi.4630310402 · doi:10.1002/asi.4630310402
- B. Webber . Question answering. In Stuart C. Shapiro, editor, Encyclopedia of Articficial Intelligence , volume 2 , pages 814 .-822. Wiley, 1987. B. Webber. Question answering. In Stuart C. Shapiro, editor, Encyclopedia of Articficial Intelligence, volume 2, pages 814.-822. Wiley, 1987.
- Terry Winograd . Five lectures on artificial intelligence . In A. Zampolli, editor, Lingusitic Structures Processing , volume 5 of Fundamental Studies in Computer Science , pages 399 - 520 . North Holland , 1977 . Terry Winograd. Five lectures on artificial intelligence. In A. Zampolli, editor, Lingusitic Structures Processing, volume 5 of Fundamental Studies in Computer Science, pages 399-520. North Holland, 1977.
- W. A. Woods . Lunar rocks in natural english: Explorations in natural language question answering . In A. Zampolli, editor, Lingusitic Structures Processing , volume 5 of Fundamental Studies in Computer Science , pages 521 - 569 . North Holland , 1977 . W. A. Woods. Lunar rocks in natural english: Explorations in natural language question answering. In A. Zampolli, editor, Lingusitic Structures Processing, volume 5 of Fundamental Studies in Computer Science, pages 521-569. North Holland, 1977.
Cited by
SQuAD: 100,000+ Questions for Machine Comprehension of Text
Rajpurkar, P., Zhang, J., Lopyrev, K., & Liang, P. (2016) · Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d16-1264 · Cited by 6,208
“The annual evaluations at the Text REtreival Conference (TREC) (Voorhees and Tice, 2000) led to many advances in open-domain QA, many of which were used in IBM Watson for Jeopardy! (Ferrucci et al.”
BackgroundSimCSE: Simple Contrastive Learning of Sentence Embeddings
Gao, T., Yao, X., & Chen, D. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.552 · Cited by 2,851
TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension
Joshi, M., Choi, E., Weld, D., & Zettlemoyer, L. (2017) · Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p17-1147 · Cited by 1,690
TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension
Joshi, M., Choi, E., Weld, D. S., & Zettlemoyer, L. (2017) · arXiv · doi:10.48550/arxiv.1705.03551 · Cited by 1,650
Making Pre-trained Language Models Better Few-shot Learners
Gao, T., Fisch, A., & Chen, D. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.295 · Cited by 1,380
(2016) · doi:10.18653/v1/d16-1147 · Cited by 1,035
Language-agnostic BERT Sentence Embedding
Feng, F., Yang, Y., Cer, D., Arivazhagan, N., & Wang, W. (2022) · Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2022.acl-long.62 · Cited by 758
SQuAD: 100,000+ Questions for Machine Comprehension of Text
Rajpurkar, P., Zhang, J., Lopyrev, K., & Liang, P. (2016) · arXiv · doi:10.48550/arxiv.1606.05250 · Cited by 678
Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivity
Lu, Y., Bartolo, M., Moore, A., Riedel, S., & Stenetorp, P. (2021) · arXiv · doi:10.48550/arxiv.2104.08786 · Cited by 615
Calibrate Before Use: Improving Few-Shot Performance of Language Models
Zhao, T. Z., Wallace, E., Feng, S., Klein, D., & Singh, S. (2021) · arXiv · doi:10.48550/arxiv.2102.09690 · Cited by 513
Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivity
Lu, Y., Bartolo, M., Moore, A., Riedel, S., & Stenetorp, P. (2022) · Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2022.acl-long.556 · Cited by 610
SentEval: An Evaluation Toolkit for Universal Sentence Representations
Conneau, A., & Kiela, D. (2018) · arXiv · doi:10.48550/arxiv.1803.05449 · Cited by 477
“We use a set of binary classification tasks (see Table 1) that covers various types of sentence classification, including sentiment analysis (MR and both binary and fine-grained SST) (Pang and Lee, 2005; Socher et al., 2013), question-type (TREC) (Voorhees and Tice, 2000), product reviews (CR) (Hu and Liu, 2004), subject…”
BackgroundInfluentialNatural Language Processing and Text Mining
(2007) · Springer eBooks · doi:10.1007/978-1-84628-754-1 · Cited by 474
SentEval: An Evaluation Toolkit for Universal Sentence Representations
Conneau, A., & Kiela, D. (2018) · Language Resources and Evaluation Conference (LREC) · doi:10.63317/2gscircifffd · Cited by 449
Mining product reputations on the Web
Morinaga, S., Yamanishi, K., Tateishi, K., & Fukushima, T. (2002) · Proceedings of the eighth ACM SIGKDD international conference on Knowledge discovery and data mining · doi:10.1145/775047.775098 · Cited by 413
Open question answering over curated and extracted knowledge bases
Fader, A., Zettlemoyer, L., & Etzioni, O. (2014) · Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining · doi:10.1145/2623330.2623677 · Cited by 407
R3: Reinforced Ranker-Reader for Open-Domain Question Answering
Wang, S., Yu, M., Guo, X., Wang, Z., Klinger, T., Zhang, W., Chang, S., Tesauro, G., Zhou, B., & Jiang, J. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.12053 · Cited by 351
DeCLUTR: Deep Contrastive Learning for Unsupervised Textual Representations
Giorgi, J., Nitski, O., Wang, B., & Bader, G. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.72 · Cited by 350
A Deep Look into neural ranking models for information retrieval
Guo, J., Fan, Y., Pang, L., Yang, L., Ai, Q., Zamani, H., Wu, C., Croft, W. B., & Cheng, X. (2020) · Information Processing & Management · doi:10.1016/j.ipm.2019.102067 · Cited by 349
Clark, J. H., Choi, E., Collins, M., Garrette, D., Kwiatkowski, T., Nikolaev, V., & Palomaki, J. (2020) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00317 · Cited by 342
Scaling question answering to the Web
Kwok, C. C. T., Etzioni, O., & Weld, D. S. (2001) · Proceedings of the 10th international conference on World Wide Web · doi:10.1145/371920.371973 · Cited by 326
An Introduction to Neural Information Retrieval
Mitra, B., & Craswell, N. (2018) · Foundations and Trends® in Information Retrieval · doi:10.1561/1500000061 · Cited by 317
Open Question Answering with Weakly Supervised Embedding Models
Bordes, A., Weston, J., & Usunier, N. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-662-44848-9_11 · Cited by 300
Graph-based Natural Language Processing and Information Retrieval
Mihalcea, R., & Radev, D. (2011) · Cambridge University Press eBooks · doi:10.1017/cbo9780511976247 · Cited by 259
Coarse-to-Fine Question Answering for Long Documents
Choi, E., Hewlett, D., Uszkoreit, J., Polosukhin, I., Lacoste, A., & Berant, J. (2017) · Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p17-1020 · Cited by 177
Making Pre-trained Language Models Better Few-shot Learners
Gao, T., Fisch, A., & Chen, D. (2020) · arXiv · doi:10.48550/arxiv.2012.15723 · Cited by 124
Contrastive Learning Models for Sentence Representations
Xu, L., Xie, H., Li, Z., Wang, F. L., Wang, W., & Li, Q. (2023) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3593590 · Cited by 50
Tomayto, Tomahto. Beyond Token-level Answer Equivalence for Question Answering Evaluation
Bulian, J., Buck, C., Gajewski, W., Börschinger, B., & Schuster, T. (2022) · Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2022.emnlp-main.20 · Cited by 44
What In-Context Learning "Learns" In-Context: Disentangling Task Recognition and Task Learning
Pan, J., Gao, T., Chen, H., & Chen, D. (2023) · arXiv · doi:10.48550/arxiv.2305.09731 · Cited by 29
What In-Context Learning “Learns” In-Context: Disentangling Task Recognition and Task Learning
Pan, J., Gao, T., Chen, H., & Chen, D. (2023) · Findings of the Association for Computational Linguistics: ACL 2023 · doi:10.18653/v1/2023.findings-acl.527 · Cited by 29
Crowdsourced Rumour Identification During Emergencies
McCreadie, R., Macdonald, C., & Ounis, I. (2015) · Proceedings of the 24th International Conference on World Wide Web · doi:10.1145/2740908.2742573 · Cited by 25
LitSearch: A Retrieval Benchmark for Scientific Literature Search
Ajith, A., Xia, M., Chevalier, A., Goyal, T., Chen, D., & Gao, T. (2024) · arXiv · doi:10.48550/arxiv.2407.18940 · Cited by 13
ConGen: Unsupervised control and generalization distillation for sentence representation
Limkonchotiwat, P., Ponwitayarat, W., Lowphansirikul, L., Udomcharoenchaikit, C., Chuangsuwanich, E., & Nutanong, S. (2022) · Findings of the Association for Computational Linguistics: EMNLP 2022 · doi:10.18653/v1/2022.findings-emnlp.483 · Cited by 3
Metadata
- Authors
- Voorhees, E. M., & Tice, D. M.
- Venue
- Proceedings of the 23rd annual international ACM SIGIR conference on Research and development in information retrieval
- Publisher
- ACM
- Published
- 2000-07
- Pages
- 200-207
- OpenAlex
- W2028175314
- Cited by
- 435
- References
- 13
- FWCI
- 11.86
- Type
- conference paper
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.