Robust Text Classification in the Presence of Confounding Bias
Open accessAbstract
As text classifiers become increasingly used in real-time applications, it is critical to consider not only their accuracy but also their robustness to changes in the data distribution. In this paper, we consider the case where there is a confounding variable Z that influences both the text features X and the class variable Y. For example, a classifier trained to predict the health status of a user based on their online communications may be confounded by socioeconomic variables. When the influence of Z changes from training to testing data, we find that classifier accuracy can degrade rapidly. Our approach, based on Pearl's back-door adjustment, estimates the underlying effect of a text variable on the class variable while controlling for the confounding variable. Although our goal is prediction, not causal inference, we find that such adjustments are essential to building text classifiers that are robust to confounding variables. On three diverse text classifications tasks, we find that covariate adjustment results in higher accuracy than competing baselines over a range of confounding relationships (e.g., in one setting, accuracy improves from 60% to 81%).
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
These efforts include employing techniques from the causal inference literature that can lead to more robust research results, or calibrating non-representative social data samples.
“These efforts include employing techniques from the causal inference literature that can lead to more robust research results (Landeiro and Culotta, 2016; Proserpio et al., 2016), or calibrating non-representative social data samples (Zagheni and Weber, 2015).”
From Social Data: Biases, Methodological Pitfalls, and Ethical Boundaries (2019) · cites this paper as Landeiro and Culotta, 2016
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
The interpretation of interaction in contingency tables
Simpson, E. H. (1951) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.2517-6161.1951.tb00088.x · Cited by 2,301 · cited 1× · see citing works
“As another way of considering the effect of BA, recall the notion of Simpson’s paradox (Simpson 1951).”
BackgroundAutomatic Indexing: An Experimental Inquiry
Maron, M. E. (1961) · Journal of the ACM · doi:10.1145/321075.321084 · Cited by 570 · cited 2× · see citing works
“While researchers have investigated automatic text classification algorithms for over fifty years (Maron 1961), they have mostly applied them to topical categorization of documents.”
MethodThe Phantom Menace: Omitted Variable Bias in Econometric Research
Clarke, K. A. (2005) · Conflict Management and Peace Science · doi:10.1080/07388940500339183 · Cited by 484 · cited 1× · see citing works
“This may inflate the coefficients of age-related terms in the clas-sifier (a result of omitted-variable bias (Clarke 2005)).”
Background
Flagged first, then by citations
Research works printed but not linked: 22
- Bareinboim, E.; Tian, J.; and Pearl, J. 2014. Recovering from selection bias in causal and statistical inference. In Proceedings of The Twenty-Eighth Conference on Artificial Intelligence (CE Brodley and P. Stone, eds.). AAAI Press, Menlo Park, CA.
- Chamlertwat, W.; Bhattarakosol, P.; Rungkasiri, T.; and Haruechaiyasak, C. 2012. Discovering consumer insight from twitter via sentiment analysis. J. UCS 18(8):973-992.
- Dehejia, R. H., and Wahba, S. 2002. Propensity score- matching methods for nonexperimental causal studies. Re- view of Economics and statistics 84(1):151-161.
- Dredze, M. 2012. How social media will change public health. IEEE Intelligent Systems 27(4):81-84.
- Elkan, C. 2001. The foundations of cost-sensitive learning. In International joint conference on artificial intelligence, 973-978.
- Fukuchi, K.; Sakuma, J.; and Kamishima, T. 2013. Predic- tion with model-based neutrality. In Machine Learning and Knowledge Discovery in Databases. Springer. 499-514.
- Hajian, S., and Domingo-Ferrer, J. 2013. A methodology for direct and indirect discrimination prevention in data min- ing. Knowledge and Data Engineering, IEEE Transactions on 25(7):1445-1459.
- Hirst, G.; Riabinin, Y.; and Graham, J. 2010. Party status as a confound in the automatic classification of political speech by ideology. In Proceedings of the 10th International Con- ference on Statistical Analysis of Textual Data (JADT 2010), 731-742.
- Kuroki, M., and Pearl, J. 2014. Measurement bias and effect restoration in causal inference. Biometrika 101(2):423-437.
- Lazer, D.; Pentland, A. S.; Adamic, L.; Aral, S.; Barabasi, A. L.; Brewer, D.; Christakis, N.; Contractor, N.; Fowler, J.; Gutmann, M.; et al. 2009. Life in the network: the coming age of computational social science. Science (New York, NY) 323(5915):721.
- Li, P.; Wu, Q.; and Burges, C. J. 2007. Mcrank: Learning to rank using multiple classification and gradient boosting. In Advances in neural information processing systems, 897- 904.
- Maas, A. L.; Daly, R. E.; Pham, P. T.; Huang, D.; Ng, A. Y.; and Potts, C. 2011. Learning word vectors for sentiment analysis. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Lan- guage Technologies, 142-150.
- Marathe, M., and Ramakrishnan, N. 2013. Recent advances in computational epidemiology. IEEE intelligent systems 28(4):96.
- Pedreshi, D.; Ruggieri, S.; and Turini, F. 2008. Discrimination-aware data mining. In Proceedings of the 14th ACM SIGKDD international conference on Knowledge discovery and data mining, 560-568. ACM.
- Pourhoseingholi, M. A.; Baghestani, A. R.; and Vahedi, M. 2012. How to control confounding effects by statistical anal- ysis. Gastroenterology and Hepatology from bed to bench 5(2):79.
- Rosenbaum, P. R., and Rubin, D. B. 1983. The central role of the propensity score in observational studies for causal effects. Biometrika 70(1):41-55.
- Rubin, D. B. 2006. Matched sampling for causal effects. Cambridge University Press.
- Sugiyama, M.; Krauledat, M.; and Müller, K.-R. 2007. Co- variate shift adaptation by importance weighted cross vali- dation. The Journal of Machine Learning Research 8:985- 1005.
- Sutton, C.; Sindelar, M.; and McCallum, A. 2006. Reducing weight undertraining in structured discriminative learning. In Proceedings of the main conference on Human Language Technology Conference of the North American Chapter of the Association of Computational Linguistics, 89-95. Asso- ciation for Computational Linguistics.
- Volkova, S.; Wilson, T.; and Yarowsky, D. 2013. Explor- ing demographic language variations to improve multilin- gual sentiment analysis in social media. In EMNLP, 1815- 1827.
- Zadrozny, B. 2004. Learning and evaluating classifiers un- der sample selection bias. In Proceedings of the twenty-first international conference on Machine learning, 114. ACM.
- Zemel, R.; Wu, Y.; Swersky, K.; Pitassi, T.; and Dwork, C. 2013. Learning fair representations. In Proceedings of the 30th International Conference on Machine Learning (ICML-13), 325-333.
Cited by
Sorted by citations
Social Data: Biases, Methodological Pitfalls, and Ethical Boundaries
Olteanu, A., Castillo, C., Diaz, F., & Kıcıman, E. (2019) · Frontiers in Big Data · doi:10.3389/fdata.2019.00013 · Cited by 763 · see citing works
Social Data: Biases, Methodological Pitfalls, and Ethical Boundaries
Olteanu, A., Castillo, C., Diaz, F., & Kiciman, E. (2016) · doi:10.2139/ssrn.2886526 · Cited by 763 · see citing works
- Authors
- Landeiro, V., & Culotta, A.
- Affiliations
- Illinois Institute of Technology (US)
- ORCID
- Culotta, Aron
- Venue
- Proceedings of the AAAI Conference on Artificial Intelligence
- Published
- 2016-02-21
- Volume
- 30
- Issue
- 1
- ISSN
- 2159-5399 (ISSN-L)
- 2374-3468
- OpenAlex
- W2436695585
- Cited by
- 34
- References
- 3
- Citations by year
- 2025: 1 · 2023: 5 · 2022: 1 · 2021: 4 · 2020: 4 · 2019: 9 · 2018: 4 · 2017: 5 · 2016: 1
- FWCI
- 3.31
- Type
- conference paper
- Language
- en
- Open access
- diamond
- doi.org
- Locations
- Proceedings of the AAAI Conference on Artificial Intelligence (publishedVersion)
- AAAI Publications (The Association for the Advancement of Artificial Intelligence (AAAI)) (submittedVersion)
- Funders
- National Science Foundation (1526674)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Text and Document Classification Technologies
- Physical Sciences › Mathematics › Statistics and Probability › Advanced Causal Inference Techniques
- Social Sciences › General Social Sciences › Computational and Text Analysis Methods
- Keywords
- text classification, confounding, distribution shift, causal inference, covariate adjustment, robustness, socioeconomic confounding
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
