The DOI matches its Crossref record; 31 of the 31 references it cites were found in a registry or an index.
Arabic Dialect Identification
Open accessAbstract
The written form of the Arabic language, Modern Standard Arabic (MSA), differs in a non-trivial manner from the various spoken regional dialects of Arabic—the true “native languages” of Arabic speakers. Those dialects, in turn, differ quite a bit from each other. However, due to MSA's prevalence in written form, almost all Arabic data sets have predominantly MSA content. In this article, we describe the creation of a novel Arabic resource with dialect annotations. We have created a large monolingual data set rich in dialectal Arabic content called the Arabic On-line Commentary Data set (Zaidan and Callison-Burch 2011). We describe our annotation effort to identify the dialect level (and dialect itself) in each of more than 100,000 sentences from the data set by crowdsourcing the annotation task, and delve into interesting annotator behaviors (like over-identification of one's own dialect). Using this new annotated data set, we consider the task of Arabic dialect identification: Given the word sequence forming an Arabic sentence, determine the variety of Arabic in which it is written. We use the data to train and evaluate automatic classifiers for dialect identification, and establish that classifiers using dialectal data significantly and dramatically outperform baselines that use MSA-only data, achieving near-human classification accuracy. Finally, we apply our classifiers to discover dialectical data from a large Web crawl consisting of 3.5 million pages mined from on-line Arabic newspapers.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
The Measurement of Observer Agreement for Categorical Data
Landis, J. R., & Koch, G. G. (1977) · Biometrics · doi:10.2307/2529310 · Cited by 81,148
Measuring nominal scale agreement among many raters.
Fleiss, J. L. (1971) · Psychological Bulletin · doi:10.1037/h0031619 · Cited by 8,733
An empirical study of smoothing techniques for language modeling
Chen, S. F., & Goodman, J. (1999) · Computer Speech & Language · doi:10.1006/csla.1999.0128 · Cited by 2,066
N-gram-based text categorization
Cavnar, W. B., & Trenkle, J. M. (1994) · Cited by 2,060
Modern Arabic: Structures, Functions and Varieties
Kaye, A. S., & Holes, C. (1997) · Language · doi:10.2307/416614 · Cited by 758
Introduction to Arabic Natural Language Processing
Habash, N. Y. (2010) · Synthesis Lectures on Human Language Technologies · doi:10.1007/978-3-031-02139-8 · Cited by 753
Comparison of four approaches to automatic language identification of telephone speech
Zissman, M. A. (1996) · IEEE Transactions on Speech and Audio Processing · doi:10.1109/tsa.1996.481450 · Cited by 618
Versteegh, K. (2014) · Edinburgh University Press eBooks · doi:10.1515/9780748645299 · Cited by 414
Using “annotator rationales” to improve machine learning for text categorization
Zaidan, O. F., Eisner, J. M., & Piatko, C. (2007) · Cited by 393
Self-citationHabash, N., Soudi, A., & Buckwalter, T. (2007) · Text, Speech and Language Technology · doi:10.1007/978-1-4020-6046-5_2 · Cited by 305
Introduction to Arabic Natural Language Processing
Habash, N. Y. (2010) · Synthesis lectures on human language technologies · doi:10.2200/s00277ed1v01y201008hlt010 · Cited by 301
Creating Speech and Language Data With Amazon’s Mechanical Turk
Callison-Burch, C., & Dredze, M. H. (2010) · Cited by 294
Self-citationBassiouney, R. (2009) · Edinburgh University Press eBooks · doi:10.3366/edinburgh/9780748623730.001.0001 · Cited by 246
Machine Translation of Arabic Dialects
Zbib, R. M., Malchiodi, E., Devlin, J., et al. (2012) · Cited by 179
A short reference grammar of Iraqi Arabic
Erwin, W. M. (1963) · Americanae (AECID Library) · Cited by 179
Modeling annotators: a generative approach to learning from annotator rationales
Zaidan, O. F., & Eisner, J. M. (2008) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/1613715.1613721 · Cited by 171
Self-citationConventional Orthography for Dialectal Arabic
Habash, N. Y., Diab, M., & Rambow, O. (2012) · Language Resources and Evaluation Conference (LREC) · doi:10.63317/3vf7xvvb69fg · Cited by 149
Callison-Burch, C., & Association for Computing Machinery-Digital Library (2010) · Cited by 144
Self-citationMAGEAD: a morphological analyzer and generator for the Arabic dialects
Habash, N. Y., & Rambow, O. (2006) · International Conference on Computational Linguistics · doi:10.3115/1220175.1220261 · Cited by 139
Habash, N. Y., Rambow, O., Chiang, D., Diab, M., Hwa, R., Sima’an, K., Lacey, V., Lévy, R., Nichols, C., & Shareef, S. (2006) · Columbia Academic Commons (Columbia University) · doi:10.7916/d85t3tzg · Cited by 110
Habash, N. Y. (2008) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.3115/1557690.1557706 · Cited by 108
Natural Language Processing for Social Media
Farzindar, A., & Inkpen, D. Z. (2015) · Synthesis lectures on human language technologies · doi:10.2200/s00659ed1v01y201508hlt030 · Cited by 101
Modeling latent biographic attributes in conversational genres
Garera, N., & Yarowsky, D. (2009) · doi:10.3115/1690219.1690245 · Cited by 99
Odden, D. A. (1996) · Language · doi:10.2307/416687 · Cited by 92
Dialectal to Standard Arabic Paraphrasing to Improve Arabic-English Statistical Machine Translation
Salloum, W., & Habash, N. Y. (2011) · doi:10.7916/d8697bg2 · Cited by 80
Sidner, C. L., Schultz, T., Stone, M., & Zhai, C. (2007) · North American Chapter of the Association for Computational Linguistics · Cited by 76
Kaye, A. S., & Versteegh, K. (2000) · Journal of the American Oriental Society · doi:10.2307/604908 · Cited by 47
Cross-Lingual Short-Text Document Classification for Facebook Comments
Faqeeh, M., Abdulla, N. A., Al‐Ayyoub, M., Jararweh, Y., & Quwaider, M. (2014) · International Conference on Future Internet of Things and Cloud (FiCloud) · doi:10.1109/ficloud.2014.99 · Cited by 45
Nationalism and the Arabic Language: A Historical Overview
Suleiman, Y. (2013) · doi:10.4324/9781315026190-10 · Cited by 30
Compression-based arabic text classification
Taamneh, H., Keshek, E. A., Issa, M. B., Al‐Ayyoub, M., & Jararweh, Y. (2014) · IEEE/ACS International Conference on Computer Systems and Applications (AICCSA) · doi:10.1109/aiccsa.2014.7073253 · Cited by 18
Crowdsourcing annotation for machine learning in natural language processing tasks
Callison-Burch, C., & Zaidan, O. F. (2012) · Cited by 6
Self-citation
Cited by
ARBERT & MARBERT: Deep Bidirectional Transformers for Arabic
Abdul-Mageed, M., Elmadany, A., & Nagoudi, E. M. B. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.551 · Cited by 397
Kazi, S., Khoja, S., & Daud, A. (2025) · Computer Science Review · doi:10.1016/j.cosrev.2025.100756 · Cited by 9
Metadata
- Authors
- Zaidan, O. F., & Callison-Burch, C.
- Venue
- Computational Linguistics
- Publisher
- MIT Press
- Published
- 2014-03
- Volume
- 40
- Issue
- 1
- Pages
- 171-202
- ISSN
- 0891-2017 (ISSN-L)
- 1530-9312
- OpenAlex
- W2160802179
- Cited by
- 294
- References
- 31
- FWCI
- 27.24
- Type
- journal article
- Language
- en
- Open access
- bronze
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.