When POS data sets don’t add up: Combatting sample bias
Open accessAbstract
Several works in Natural Language Processing have recently looked into part-of-speech (POS) annotation of Twitter data and typically used their own data sets.Since conventions on Twitter change rapidly, models often show sample bias.Training on a combination of the existing data sets should help overcome this bias and produce more robust models than any trained on the individual corpora.Unfortunately, combining the existing corpora proves difficult: many of the corpora use proprietary tag sets that have little or no overlap.Even when mapped to a common tag set, the different corpora systematically differ in their treatment of various tags and tokens.This includes both preprocessing decisions, as well as default labels for frequent tokens, thus exhibiting data bias and label bias, respectively.Only if we address these biases can we combine the existing data sets to also overcome sample bias.We present a systematic study of several Twitter POS data sets, the problems of label and data bias, discuss their effects on model performance, and show how to overcome them to learn models that perform well on various test sets, achieving relative error reduction of up to 21%.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Several aspects related to how data samples are collected from sources have been questioned, including their representativeness and completeness, which is problematic for both type I and type II research.
“Beyond source selection bias, several aspects related to how data samples are collected from these sources have been questioned, including their representativeness and completeness (González-Bailón et al., 2014b; Hovy et al., 2014), which is problematic for both type I and type II research.”
From Social Data: Biases, Methodological Pitfalls, and Ethical Boundaries (2019) · cites this paper as Hovy et al., 2014
The Introduction and Conclusion describe sample bias in Twitter POS datasets: models perform well on data from the same sample but worse on other datasets, supporting the concern about representativeness. The paper does not address sample completeness or classify the consequences by type I and type II research; the latter appears to be the citing authors’ framing. (LLM, conf=0.85)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
No references could be resolved yet.
Research works printed but not linked: 8
- Jacob Eisenstein. 2013. What to do about bad language on the internet. In NAACL.
- Kevin Gimpel, Nathan Schneider, Brendan O'Connor, Di- panjan Das, Daniel Mills, Jacob Eisenstein, Michael Heilman, Dani Yogatama, Jeffrey Flanigan, and Noah A. Smith. 2011. Part-of-speech tagging for twitter: Anno- tation, features, and experiments. In ACL.
- Olutobi Owoputi, Brendan O'Connor, Chris Dyer, Kevin Gimpel, Nathan Schneider, and Noah A Smith. 2013. Improved part-of-speech tagging for online conversa- tional text with word clusters. In NAACL.
- Slav Petrov, Dipanjan Das, and Ryan McDonald. 2012. A universal part-of-speech tagset. In LREC.
- Dennis Reidsma and Jean Carletta. 2008. Reliability measurement without limits. Computational Linguistics, 34(3):319-326.
- Alan Ritter, Sam Clark, Oren Etzioni, et al. 2011. Named entity recognition in tweets: an experimental study. In EMNLP.
- Daniel Zeman. 2008. Reusable tagset conversion using tagset drivers. In LREC.
- Daniel Zeman. 2010. Hard problems of tagset conversion. In Proceedings of the Second International Conference on Global Interoperability for Language Resources.
Cited by
Sorted by citations
Social Data: Biases, Methodological Pitfalls, and Ethical Boundaries
Olteanu, A., Castillo, C., Diaz, F., & Kıcıman, E. (2019) · Frontiers in Big Data · doi:10.3389/fdata.2019.00013 · Cited by 763 · see citing works
Social Data: Biases, Methodological Pitfalls, and Ethical Boundaries
Olteanu, A., Castillo, C., Diaz, F., & Kiciman, E. (2016) · doi:10.2139/ssrn.2886526 · Cited by 763 · see citing works
- Authors
- Hovy, D., Plank, B., & Søgaard, A.
- Affiliations
- University of Copenhagen (DK)
- Venue
- Proceedings of the Language Resources and Evaluation Conference
- Published
- 2014-05
- Pages
- 4472-4475
- ISSN
- 2522-2686
- OpenAlex
- W2251406234
- Cited by
- 14
- References
- 0
- Citations by year
- 2023: 2 · 2019: 1 · 2017: 2 · 2016: 1 · 2015: 6 · 2014: 2
- FWCI
- 2.16
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by-nc
- Locations
- Language Resources and Evaluation Conference (LREC) (publishedVersion, cc-by-nc)
- Archivio istituzionale della Ricerca - Bocconi (submittedVersion)
- Research at the University of Copenhagen (University of Copenhagen) (submittedVersion)
- Language Resources and Evaluation
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Sentiment Analysis and Opinion Mining
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Keywords
- part-of-speech tagging, Twitter, sampling bias, label bias, data bias, social media text, robustness
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
