The DOI matches its Crossref record; 30 of the 30 references it cites were found in a registry or an index.
Code Mixing: A Challenge for Language Identification in the Language of Social Media
Open accessAbstract
In social media communication, multilingual speakers often switch between languages, and, in such an environment, automatic language identification becomes both a necessary and challenging task.In this paper, we describe our work in progress on the problem of automatic language identification for the language of social media.We describe a new dataset that we are in the process of creating, which contains Facebook posts and comments that exhibit code mixing between Bengali, English and Hindi.We also present some preliminary word-level language identification experiments using this dataset.Different techniques are employed, including a simple unsupervised dictionary-based approach, supervised word-level classification with and without contextual clues, and sequence labelling using Conditional Random Fields.We find that the dictionary-based approach is surpassed by supervised classification and sequence labelling, and that it is important to take contextual clues into consideration.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
The WEKA data mining software: an update
Hall, M., Frank, E., Holmes, G., Pfahringer, B., Reutemann, P., & Witten, I. H. (2009) · ACM SIGKDD Explorations Newsletter · doi:10.1145/1656274.1656278 · Cited by 17,750 · cited 1×
“WEKA: We use the Weka toolkit (Hall et al., 2009) for our experiments in decision tree training.”
MethodA Practical Guide to Support Vector Classication
Hsu, C., Chang, C.-C., & Lin, C. (2008) · Cited by 5,046
Language identification in the limit
Gold, E. M. (1967) · Information and Control · doi:10.1016/s0019-9958(67)91165-5 · Cited by 3,572 · cited 1×
“The problem of language identification has been investigated for half a century (Gold, 1967) and that of computational analysis of code switching for several decades (Joshi, 1982), but there has been less work on automatic language identification for multilingual code-mixed texts.”
BackgroundN-gram-based text categorization
Cavnar, W. B., & Trenkle, J. M. (1994) · Cited by 2,060
Code Switching in Conversation: Language, Interaction and Identity
Lanvers, U., & Auer, P. (2000) · The Modern Language Review · doi:10.2307/3736717 · Cited by 1,293
SemEval-2016 Task 4: Sentiment Analysis in Twitter
Nakov, P., Ritter, A., Rosenthal, S., Sebastiani, F., & Stoyanov, V. (2016) · Proceedings of the 10th International Workshop on Semantic Evaluation (SemEval-2016) · doi:10.18653/v1/s16-1001 · Cited by 966
Improved Part-of-Speech Tagging for Online Conversational Text with Word Clusters
Owoputi, O., O'Connor, B., Dyer, C., Gimpel, K., Schneider, N., & Smith, N. A. (2018) · Figshare · doi:10.1184/r1/6473408 · Cited by 662
The BNC Handbook: Exploring the British National Corpus with SARA
Aston, G., & Burnard, L. (1998) · Cited by 652
SemEval-2014 Task 9: Sentiment Analysis in Twitter
Rosenthal, S., Ritter, A., Nakov, P., & Stoyanov, V. (2014) · Proceedings of the 8th International Workshop on Semantic Evaluation (SemEval 2014) · doi:10.3115/v1/s14-2009 · Cited by 617
Emotions in Multiple Languages
Dewaele, J.-M. (2010) · Palgrave Macmillan UK eBooks · doi:10.1057/9780230289505 · Cited by 612
One Speaker, Two Languages: Cross-Disciplinary Perspectives on Code-Switching
Scates, C., Milroy, L., & Muysken, P. (1996) · Rocky Mountain Review of Language and Literature · doi:10.2307/1348244 · Cited by 498
LIBLINEAR: A Library for Large Linear Classification
FanRong-En, ChangKai-Wei, HsiehCho-Jui, WangXiang-Rui, & LinChih-Jen (2008) · Journal of Machine Learning Research · doi:10.5555/1390681.1442794 · Cited by 328
Processing of sentences with intra-sentential code-switching
Joshi, A. K. (1982) · International Conference on Computational Linguistics · doi:10.3115/991813.991836 · Cited by 246
DCU: Aspect-based Polarity Classification for SemEval Task 4
Wagner, J., Arora, P., Cortes, S., Barman, U., Bogdanova, D., Foster, J., & Tounsi, L. (2014) · Proceedings of the 8th International Workshop on Semantic Evaluation (SemEval 2014) · doi:10.3115/v1/s14-2036 · Cited by 222
Self-citationLanguage Matters In Twitter: A Large Scale Study
Hong, L., Convertino, G., & Chi, E. (2021) · Proceedings of the International AAAI Conference on Web and Social Media · doi:10.1609/icwsm.v5i1.14184 · Cited by 211
Language Identification: The Long and the Short of the Matter
Baldwin, T., & Lui, M. (2010) · North American Chapter of the Association for Computational Linguistics · Cited by 181
Automatically Constructing a Normalisation Dictionary for Microblogs
Han, B., Cook, P. F., & Baldwin, T. (2012) · Cited by 178
Generalized Expectation Criteria for Semi-Supervised Learning with Weakly Labeled Data
Mann, G., & McCallum, A. (2010) · ScholarWorks@UMassAmherst (University of Massachusetts Amherst) · doi:10.5555/1756006.1756038 · Cited by 173
Overview for the First Shared Task on Language Identification in Code-Switched Data
Solorio, T., Blair, E., Maharjan, S., Bethard, S. J., Diab, M., Ghoneim, M. A., Hawwari, A., AlGhamdi, F., Hirschberg, J., Chang, A., et al. (2014) · doi:10.3115/v1/w14-3907 · Cited by 157
Labeling the Languages of Words in Mixed-Language Documents using Weakly Supervised Methods
King, B., & Abney, S. P. (2013) · Cited by 152
Cantonese‐English code‐switching research in Hong Kong: a Y2K review
Li, D. C. S. (2000) · World Englishes · doi:10.1111/1467-971x.00181 · Cited by 146
Learning to predict code-switching points
Solorio, T., & Liu, Y. (2008) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/1613715.1613841 · Cited by 131
Part-of-speech tagging for English-Spanish code-switched text
Solorio, T., & Liu, Y. (2008) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/1613715.1613852 · Cited by 127
Generalized Expectation Criteria for Semi-Supervised Learning of Conditional Random Fields
Mann, G., & McCallum, A. (2008) · Scholarworks (University of Massachusetts Amherst) · Cited by 120
Reconsidering Language Identification for Written Language Resources
Hughes, B., Baldwin, T., Bird, S., Nicholson, J., & MacKinlay, A. (2006) · Language Resources and Evaluation Conference (LREC) · doi:10.63317/4y7hvqnp6mr3 · Cited by 107
Language Identification for Creating Language-Specific Twitter Collections
Bergsma, S., & McNamee, P. (2013) · Cited by 91
Chinese-English Code-switching in Blogs by Macao Young People
San, H. K. (2009) · ERA · Cited by 23
A tagging algorithm for mixed language identification in a noisy domain
Rosner, M., & Farrugia, P.-J. (2007) · Interspeech (USB) · doi:10.21437/interspeech.2007-77 · Cited by 22
Languages, Code-Switching Practice and Primary Functions of Facebook among University Students
Shafie, L. A., & Nayan, S. (2013) · Studies in English Language Teaching · doi:10.22158/selt.v1n1p187 · Cited by 21
DCU-Symantec at the WMT 2013 Quality Estimation Shared Task
Rubino, R., Wagner, J., Foster, J., et al. (2013) · Arrow@dit (Dublin Institute of Technology) · Cited by 7
Self-citation
Cited by
TweetEval: Unified Benchmark and Comparative Evaluation for Tweet Classification
Barbieri, F., Camacho-Collados, J., Espinosa Anke, L., & Neves, L. (2020) · Findings of the Association for Computational Linguistics: EMNLP 2020 · doi:10.18653/v1/2020.findings-emnlp.148 · Cited by 674
A Dataset of Hindi-English Code-Mixed Social Media Text for Hate Speech Detection
Bohra, A., Vijay, D., Singh, V., Akhtar, S. S., & Shrivastava, M. (2018) · Proceedings of the Second Workshop on Computational Modeling of People’s Opinions, Personality, and Emotions in Social Media · doi:10.18653/v1/w18-1105 · Cited by 265
Bhuiyan, M. H. M., Kumar, M. K. B., & Staicu, C.-A. (2026) · Proceedings of the 2026 IEEE/ACM 48th International Conference on Software Engineering · doi:10.1145/3744916.3787766 · Cited by 5
“Language Detection Accuracy: Detecting the natural language of short, informal GitHub text is challenging, especially for code-mixed messages or short comments [6, 9, 48].”
Metadata
- Authors
- Barman, U., Das, A., Wagner, J., & Foster, J.
- Venue
- Proceedings of the First Workshop on Computational Approaches to Code Switching
- Published
- 2014
- Pages
- 13-23
- OpenAlex
- W2250548009
- Cited by
- 267
- References
- 30
- FWCI
- 10.87
- Type
- conference paper
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.