The DOI matches its Crossref record; 18 of the 18 references it cites were found in a registry or an index.
Scaling to very very large corpora for natural language disambiguation
Open accessAbstract
The amount of readily available on-line text has reached hundreds of billions of words and continues to grow. Yet for most core natural language tasks, algorithms continue to be optimized, tested and compared after training on corpora consisting of only one million words or less. In this paper, we evaluate the performance of different learning methods on a prototypical natural language disambiguation task, confusion set disambiguation, when trained on orders of magnitude more labeled data than has previously been used. We are fortunate that for this particular application, correctly labeled training data is free. Since this will often not be the case, we examine methods for effectively exploiting very large corpora when labeled data comes at a cost.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Breiman, L. (1996) · Machine Learning · doi:10.1007/bf00058655 · Cited by 16,325
Unsupervised word sense disambiguation rivaling supervised methods
Yarowsky, D. (1995) · Proceedings of the 33rd annual meeting on Association for Computational Linguistics - · doi:10.3115/981658.981684 · Cited by 2,385
A sequential algorithm for training text classifiers
Lewis, D., & Gale, W. A. (1994) · arXiv (Cornell University) · doi:10.5555/188490.188495 · Cited by 1,731
Editorial: On Machine Learning
Langley, P. (1986) · Machine Learning · doi:10.1023/a:1022687019898 · Cited by 951
A method for disambiguating word senses in a large corpus
Gale, W. A., Church, K., & Yarowsky, D. (1992) · Computers and the Humanities · doi:10.1007/bf00136984 · Cited by 532
Committee-Based Sampling For Training Probabilistic Classifiers
Dagan, I., & Engelson, S. P. (1995) · Machine Learning Proceedings 1995 · doi:10.1016/b978-1-55860-377-6.50027-x · Cited by 499
Tagging English text with a probabilistic model
Mérialdo, B. (1994) · Cited by 474
Yarowsky, D. (1994) · Proceedings of the 32nd annual meeting on Association for Computational Linguistics - · doi:10.3115/981732.981745 · Cited by 356
Learning to classify text from labeled and unlabeled documents
Nigam, K., McCallum, A., Thrun, S., & Mitchell, T. M. (1998) · Cited by 327
A Winnow-Based Approach to Context-Sensitive Spelling Correction
Golding, A. R., & Roth, D. (1999) · Machine Learning · doi:10.1023/a:1007545901558 · Cited by 291
Cited by
80 Million Tiny Images: A Large Data Set for Nonparametric Object and Scene Recognition
Torralba, A., Fergus, R., & Freeman, W. T. (2008) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2008.128 · Cited by 1,921
Cheap and fast---but is it good?: evaluating non-expert annotations for natural language tasks
Snow, R., O'Connor, B., Jurafsky, D., & Ng, A. Y. (2008) · Proceedings of the Conference on Empirical Methods in Natural Language Processing - EMNLP '08 · doi:10.3115/1613715.1613751 · Cited by 1,887
Scaling Laws for Neural Language Models
Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., & Amodei, D. (2020) · arXiv · doi:10.48550/arxiv.2001.08361 · Cited by 9,550
“Some early [BB01, Goo01] work found power-law scalings between performance and dataset size.”
BackgroundMarketing Analytics for Data-Rich Environments
Wedel, M., & Kannan, P. K. (2016) · Journal of Marketing · doi:10.1509/jm.15.0413 · Cited by 1,261
“Some of the debate surrounding that question originates in research at Microsoft, in which Banko and Brill (2001) showed that in the context of text mining, algorithms of different complexity performed similarly, but adding data greatly improved performance.”
Introduction to the Special Issue on the Web as Corpus
Kilgarriff, A., & Grefenstette, G. (2003) · Computational Linguistics · doi:10.1162/089120103322711569 · Cited by 929
ECG-based heartbeat classification for arrhythmia detection: A survey
Luz, E. J. D. S., Schwartz, W. R., Cámara-Chávez, G., & Menotti, D. (2016) · Computer Methods and Programs in Biomedicine · doi:10.1016/j.cmpb.2015.12.008 · Cited by 878
Human Decisions and Machine Predictions
Kleinberg, J., Lakkaraju, H., Leskovec, J., Ludwig, J., & Mullainathan, S. (2018) · The Quarterly Journal of Economics · doi:10.1093/qje/qjx032 · Cited by 1,040
Web-scale information extraction in knowitall: (preliminary results)
Etzioni, O., Cafarella, M., Downey, D., Kok, S., Popescu, A.-M., Shaked, T., Soderland, S., Weld, D. S., & Yates, A. (2004) · Proceedings of the 13th international conference on World Wide Web · doi:10.1145/988672.988687 · Cited by 747
Large-scale deep unsupervised learning using graphics processors
Raina, R., Madhavan, A., & Ng, A. Y. (2009) · Proceedings of the 26th Annual International Conference on Machine Learning · doi:10.1145/1553374.1553486 · Cited by 742
An empirical survey of data augmentation for time series classification with neural networks
Iwana, B. K., & Uchida, S. (2021) · PLOS ONE · doi:10.1371/journal.pone.0254841 · Cited by 725
Deep neural network based malware detection using two dimensional binary program features
Saxe, J., & Berlin, K. (2015) · 2015 10th International Conference on Malicious and Unwanted Software (MALWARE) · doi:10.1109/malware.2015.7413680 · Cited by 711
Big data for development: a review of promises and challenges
Hilbert, M. (2016) · Development Policy Review · doi:10.1111/dpr.12142 · Cited by 679
Evaluation and accurate diagnoses of pediatric diseases using artificial intelligence
Liang, H., Tsui, B. Y., Ni, H., Valentim, C. C. S., Baxter, S. L., Liu, G., Cai, W., Kermany, D. S., Sun, X., Chen, J., et al. (2019) · Nature Medicine · doi:10.1038/s41591-018-0335-9 · Cited by 659
Big-Data Science in Porous Materials: Materials Genomics and Machine Learning
Jablonka, K. M., Ongari, D., Moosavi, S. M., & Smit, B. (2020) · Chemical Reviews · doi:10.1021/acs.chemrev.0c00004 · Cited by 599
Human Decisions and Machine Predictions
Kleinberg, J., Lakkaraju, H., Leskovec, J., Ludwig, J., & Mullainathan, S. (2017) · doi:10.3386/w23180 · Cited by 545
Data-intensive text processing with MapReduce
Lin, J., & Dyer, C. (2009) · Proceedings of Human Language Technologies: The 2009 Annual Conference of the North American Chapter of the Association for Computational Linguistics, Companion Volume: Tutorial Abstracts on - NAACL '09 · doi:10.3115/1620950.1620951 · Cited by 445
Mandera, P., Keuleers, E., & Brysbaert, M. (2017) · Journal of Memory and Language · doi:10.1016/j.jml.2016.04.001 · Cited by 428
Deep Learning Scaling is Predictable, Empirically
Hestness, J., Narang, S., Ardalani, N., Diamos, G., Jun, H., Kianinejad, H., Patwary, M. M. A., Yang, Y., & Zhou, Y. (2017) · arXiv · doi:10.48550/arxiv.1712.00409 · Cited by 427
A Survey on Data Augmentation for Text Classification
Bayer, M., Kaufhold, M.-A., & Reuter, C. (2023) · ACM Computing Surveys · doi:10.1145/3544558 · Cited by 425
Neural network analysis of sleep stages enables efficient diagnosis of narcolepsy
Stephansen, J. B., Olesen, A. N., Olsen, M., Ambati, A., Leary, E. B., Moore, H. E., Carrillo, O., Lin, L., Han, F., Yan, H., et al. (2018) · Nature Communications · doi:10.1038/s41467-018-07229-3 · Cited by 413
An unsupervised approach to recognizing discourse relations
Marcu, D., & Echihabi, A. (2001) · Proceedings of the 40th Annual Meeting on Association for Computational Linguistics - ACL '02 · doi:10.3115/1073083.1073145 · Cited by 382
Using the Web to Obtain Frequencies for Unseen Bigrams
Keller, F., & Lapata, M. (2003) · Computational Linguistics · doi:10.1162/089120103322711604 · Cited by 350
Active learning for logistic regression: an evaluation
Schein, A. I., & Ungar, L. H. (2007) · Machine Learning · doi:10.1007/s10994-007-5019-5 · Cited by 346
RobotReviewer: evaluation of a system for automatically assessing bias in clinical trials
Marshall, I. J., Kuiper, J., & Wallace, B. C. (2016) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocv044 · Cited by 317
A survey on data‐efficient algorithms in big data era
Adadi, A. (2021) · Journal of Big Data · doi:10.1186/s40537-021-00419-9 · Cited by 331
Securing the Internet of Things in the Age of Machine Learning and Software-Defined Networking
Restuccia, F., D'Oro, S., & Melodia, T. (2018) · IEEE Internet of Things Journal · doi:10.1109/jiot.2018.2846040 · Cited by 286
Zhou, Z.-H., Chawla, N. V., Jin, Y., & Williams, G. J. (2014) · IEEE Computational Intelligence Magazine · doi:10.1109/mci.2014.2350953 · Cited by 275
Towards felicitous decision making: An overview on challenges and trends of Big Data
Wang, H., Xu, Z., Fujita, H., & Liu, S. (2016) · Information Sciences · doi:10.1016/j.ins.2016.07.007 · Cited by 268
Google Earth Engine and Artificial Intelligence (AI): A Comprehensive Review
Yang, L., Driscol, J., Sarigai, S., Wu, Q., Chen, H., & Lippitt, C. D. (2022) · Remote Sensing · doi:10.3390/rs14143253 · Cited by 264
Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Tasks
Wang, Y., Mishra, S., Alipoormolabashi, P., Kordi, Y., Mirzaei, A., Naik, A., Ashok, A., Dhanasekaran, A. S., Arunkumar, A., Stap, D., et al. (2022) · Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2022.emnlp-main.340 · Cited by 247
“While the conventional wisdom in supervised learning is that more training instances usually helps (Banko and Brill, 2001; Sun et al., 2017; Hestness et al., 2017), in our setup, the model’s performance saturates when only 64 instances per task are used for training.”
BackgroundWhen Words Sweat: Identifying Signals for Loan Default in the Text of Loan Applications
Netzer, O., Lemaire, A., & Herzenstein, M. (2019) · Journal of Marketing Research · doi:10.1177/0022243719852959 · Cited by 238
Syntactic Structure from Deep Learning
Linzen, T., & Baroni, M. (2021) · Annual Review of Linguistics · doi:10.1146/annurev-linguistics-032020-051035 · Cited by 202
Pandl, K. D., Thiebes, S., Schmidt-Kraepelin, M., & Sunyaev, A. (2020) · IEEE Access · doi:10.1109/access.2020.2981447 · Cited by 115
Harmonization of quality metrics and power calculation in multi-omic studies
Tarazona, S., Balzano-Nogueira, L., Gómez-Cabrero, D., Schmidt, A., Imhof, A., Hankemeier, T., Tegnér, J., Westerhuis, J. A., & Conesa, A. (2020) · Nature Communications · doi:10.1038/s41467-020-16937-8 · Cited by 105
Marvin, H. J. P., Bouzembrak, Y., Janssen, E. M., van der Zande, M., Murphy, F., Sheehan, B., Mullins, M., & Bouwmeester, H. (2017) · Nanotoxicology · doi:10.1080/17435390.2016.1278481 · Cited by 82
Toward Robust Anxiety Biomarkers: A Machine Learning Approach in a Large-Scale Sample
Boeke, E. A., Holmes, A. J., & Phelps, E. A. (2020) · Biological Psychiatry: Cognitive Neuroscience and Neuroimaging · doi:10.1016/j.bpsc.2019.05.018 · Cited by 50
Artificial intelligence, robotics, ethics, and the military: a Canadian perspective
Wasilow, S., & Thorpe, J. B. (2019) · AI Magazine · doi:10.1609/aimag.v40i1.2848 · Cited by 42
“Research has shown that the amount of data used to train machine learning algorithms has a greater effect on prediction accuracy than the type of machine learning method used (Banko and Brill 2001).”
BackgroundSciRide Finder: a citation-based paradigm in biomedical literature search
Volanakis, A., & Krawczyk, K. (2018) · Scientific Reports · doi:10.1038/s41598-018-24571-0 · Cited by 19
(2018) · University of Groningen research database (University of Groningen / Centre for Information Technology) · Cited by 9
Metadata
- Authors
- Banko, M., & Brill, E.
- Venue
- Proceedings of the 39th Annual Meeting on Association for Computational Linguistics - ACL '01
- Published
- 2001
- Pages
- 26-33
- OpenAlex
- W2157963512
- Cited by
- 675
- References
- 18
- FWCI
- 12.29
- Type
- conference paper
- Language
- en
- Open access
- bronze
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.