The DOI matches its Crossref record; 18 of the 18 references it cites were found in a registry or an index.
Scaling question answering to the Web
Abstract
The wealth of information on the web makes it an attractive resource for seeking quick answers to simple, factual questions such as "who was the first American in space?" or "what is the second tallest mountain in the world?" Yet today's most advanced web search services (e.g., Google and AskJeeves) make it surprisingly tedious to locate answers to such questions. In this paper, we extend question-answering techniques, first studied in the information retrieval literature, to the web and experimentally evaluate their performance. First we introduce MULDER, which we believe to be the first general-purpose, fully-automated question-answering system available on the web. Second, we describe MULDER's architecture, which relies on multiple search-engine queries, natural-language parsing, and a novel voting procedure to yield reliable answers coupled with high recall. Finally, we compare MULDER's performance to that of Google and AskJeeves on questions drawn from the TREC-8 question track. We find that MULDER's recall is more than a factor of three higher than that of AskJeeves. In addition, we find that Google requires 6.6 times as much user effort to achieve the same level of recall as MULDER. 1.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
The anatomy of a large-scale hypertextual Web search engine
Brin, S., & Page, L. (1998) · Computer Networks and ISDN Systems · doi:10.1016/s0169-7552(98)00110-x · Cited by 16,004
Building a Large Annotated Corpus of English: The Penn Treebank
Marcus, M. (1993) · doi:10.21236/ada273556 · Cited by 7,405
Introduction to WordNet: An On-line Lexical Database*
Miller, G. A., Beckwith, R., Fellbaum, C., Gross, D., & Miller, K. J. (1990) · International Journal of Lexicography · doi:10.1093/ijl/3.4.235 · Cited by 4,197
Authoritative sources in a hyperlinked environment
Kleinberg, J. M. (1998) · Symposium on Discrete Algorithms · doi:10.5555/314613.315045 · Cited by 1,814
Focused crawling: a new approach to topic-specific Web resource discovery
Chakrabarti, S., van den Berg, M., & Dom, B. (1999) · Computer Networks · doi:10.1016/s1389-1286(99)00052-3 · Cited by 1,486
Query Expansion using Lexical-Semantic Relations
Voorhees, E. M. (1994) · SIGIR ’94 · doi:10.1007/978-1-4471-2099-5_7 · Cited by 947
Nymble: a high-performance learning name-finder
Bikel, D. M., Miller, S., Schwartz, R., & Weischedel, R. (1997) · Proceedings of the fifth conference on Applied natural language processing - · doi:10.3115/974557.974586 · Cited by 726
Grouper: a dynamic clustering interface to Web search results
Zamir, O., & Etzioni, O. (1999) · Computer Networks · doi:10.1016/s1389-1286(99)00054-7 · Cited by 720
Self-citationA new statistical parser based on bigram lexical dependencies
Collins, M. J. (1996) · Proceedings of the 34th annual meeting on Association for Computational Linguistics - · doi:10.3115/981863.981888 · Cited by 614
Building a Large Annotated Corpus of English: The
MarcusMitchell, P., Ann, M., & SantoriniBeatrice (1993) · Computational Linguistics · doi:10.5555/972470.972475 · Cited by 489
Automatic Query expansion using SMART : TREC 3
Buckley, C., Salton, G., Allan, J., & Singhal, A. (1994) · Cited by 464
Building a question answering test collection
Voorhees, E. M., & Tice, D. M. (2000) · Proceedings of the 23rd annual international ACM SIGIR conference on Research and development in information retrieval · doi:10.1145/345508.345577 · Cited by 435
Question Answering from Frequently Asked Question Files: Experiences with the FAQ Finder System
Burke, R., Hammond, K. J., Kulyukin, V., Lytinen, S. L., Tomuro, N., & Schoenberg, S. (1997) · doi:10.1609/aimag.v18i2.1294 · Cited by 333
Statistical Techniques for Natural Language Parsing
Eugene Charniak, D. (1997) · doi:10.1609/aimag.v18i4.1320 · Cited by 232
An Introduction to the Principles of Transformational Syntax
Morin, J.-Y., Akmajian, A., & Heny, F. W. (1976) · Modern Language Journal · doi:10.2307/324616 · Cited by 203
Kupiec, J. M. (1993) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/160688.160717 · Cited by 142
Measuring index quality using random walks on the Web
Henzinger, M. R., Heydon, A., Mitzenmacher, M., & Najork, M. (1999) · Computer Networks · doi:10.1016/s1389-1286(99)00016-x · Cited by 134
The Use of Predictive Annotation for Question Answering in TREC8.
Prager, J. M., Radev, D., Brown, E. W., Coden, A. R., & Samn, V. (1999) · Cited by 83
Cited by
Web-scale information extraction in knowitall: (preliminary results)
Etzioni, O., Cafarella, M., Downey, D., Kok, S., Popescu, A.-M., Shaked, T., Soderland, S., Weld, D. S., & Yates, A. (2004) · Proceedings of the 13th international conference on World Wide Web · doi:10.1145/988672.988687 · Cited by 747
Self-citationWeb page classification: Features and algorithms
Qi, X., & Davison, B. D. (2009) · ACM Computing Surveys · doi:10.1145/1459352.1459357 · Cited by 435
Sparse, Dense, and Attentional Representations for Text Retrieval
Luan, Y., Eisenstein, J., Toutanova, K., & Collins, M. (2021) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00369 · Cited by 363
R3: Reinforced Ranker-Reader for Open-Domain Question Answering
Wang, S., Yu, M., Guo, X., Wang, Z., Klinger, T., Zhang, W., Chang, S., Tesauro, G., Zhou, B., & Jiang, J. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.12053 · Cited by 351
Modelling vague places with knowledge from the Web
Jones, C. B., Purves, R. S., Clough, P. D., & Joho, H. (2008) · International Journal of Geographical Information Science · doi:10.1080/13658810701850547 · Cited by 345
Metadata
- Authors
- Kwok, C. C. T., Etzioni, O., & Weld, D. S.
- Venue
- Proceedings of the 10th international conference on World Wide Web
- Publisher
- ACM
- Published
- 2001-04
- Pages
- 150-161
- OpenAlex
- W2055354597
- Cited by
- 326
- References
- 18
- FWCI
- 27.63
- Type
- conference paper
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.