The DOI matches its Crossref record; 28 of the 28 references it cites were found in a registry or an index.
Facing Imbalanced Data--Recommendations for the Use of Performance Metrics
Open accessAbstract
Recognizing facial action units (AUs) is important for situation analysis and automated video annotation. Previous work has emphasized face tracking and registration and the choice of features classifiers. Relatively neglected is the effect of imbalanced data for action unit detection. While the machine learning community has become aware of the problem of skewed data for training classifiers, little attention has been paid to how skew may bias performance metrics. To address this question, we conducted experiments using both simulated classifiers and three major databases that differ in size, type of FACS coding, and degree of skew. We evaluated influence of skew on both threshold metrics (Accuracy, F-score, Cohen's kappa, and Krippendorf's alpha) and rank metrics (area under the receiver operating characteristic (ROC) curve and precision-recall curve). With exception of area under the ROC curve, all were attenuated by skewed distributions, in many cases, dramatically so. While ROC was unaffected by skew, precision-recall curves suggest that ROC may mask poor performance. Our findings suggest that skew is a critical factor in evaluating performance metrics. To avoid or minimize skew-biased estimates of performance, we recommend reporting skew-normalized scores along with the obtained ones.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
A Coefficient of Agreement for Nominal Scales
Cohen, J. (1960) · Educational and Psychological Measurement · doi:10.1177/001316446002000104 · Cited by 42,398
Content Analysis: An Introduction to its Methodology.
Shelley, M., & Krippendorff, K. (1984) · Journal of the American Statistical Association · doi:10.2307/2288384 · Cited by 24,312
An introduction to ROC analysis
Fawcett, T. (2006) · Pattern Recognition Letters · doi:10.1016/j.patrec.2005.10.010 · Cited by 21,879
Content Analysis: An Introduction to Its Methodology
Krippendorff, K. (2019) · doi:10.4135/9781071878781 · Cited by 15,448
He, H., & Garcia, E. A. (2009) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2008.239 · Cited by 10,359
Lucey, P., Cohn, J. F., Kanade, T., Saragih, J., Ambadar, Z., & Matthews, I. (2010) · 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition - Workshops · doi:10.1109/cvprw.2010.5543262 · Cited by 4,303
Self-citationA Survey of Affect Recognition Methods: Audio, Visual, and Spontaneous Expressions
Zeng, Z., Pantic, M., Roisman, G. I., & Huang, T. S. (2009) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2008.52 · Cited by 2,797
Comprehensive database for facial expression analysis
Kanade, T., Cohn, J. F., & Tian, Y. (2002) · Proceedings Fourth IEEE International Conference on Automatic Face and Gesture Recognition (Cat. No. PR00580) · doi:10.1109/afgr.2000.840611 · Cited by 2,591
Self-citationEditorial: special issue on learning from imbalanced data sets
Chawla, N. V., Japkowicz, N., & Kotcz, A. (2004) · ACM SIGKDD Explorations Newsletter · doi:10.1145/1007730.1007733 · Cited by 2,063
Active Appearance Models Revisited
Matthews, I., & Baker, S. (2004) · International Journal of Computer Vision · doi:10.1023/b:visi.0000029666.37597.d3 · Cited by 1,535
Cited by
Learning from class-imbalanced data: Review of methods and applications
Haixiang, G., Yijing, L., Shang, J., Mingyun, G., Yuanyue, H., & Bing, G. (2017) · Expert Systems with Applications · doi:10.1016/j.eswa.2016.12.035 · Cited by 2,377
AffectNet: A Database for Facial Expression, Valence, and Arousal Computing in the Wild
Mollahosseini, A., Hasani, B., & Mahoor, M. H. (2019) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2017.2740923 · Cited by 2,338
Emotional expressions reconsidered: Challenges to inferring emotion from human facial movements
Barrett, L. F., Adolphs, R., Marsella, S., Martinez, A. M., & Pollak, S. D. (2019) · Psychological Science in the Public Interest · doi:10.1177/1529100619832930 · Cited by 1,748
Luque, A., Carrasco, A., Martín, A., & de las Heras, A. (2019) · Pattern Recognition · doi:10.1016/j.patcog.2019.02.023 · Cited by 1,128
Using machine learning approaches for multi-omics data analysis: A review
Reel, P. S., Reel, S., Pearson, E., Trucco, E., & Jefferson, E. (2021) · Biotechnology Advances · doi:10.1016/j.biotechadv.2021.107739 · Cited by 903
Automatic Analysis of Facial Affect: A Survey of Registration, Representation, and Recognition
Sariyanidi, E., Gunes, H., & Cavallaro, A. (2015) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2014.2366127 · Cited by 685
A Review and Meta-Analysis of Multimodal Affect Detection Systems
D'mello, S. K., & Kory, J. (2015) · ACM Computing Surveys · doi:10.1145/2682899 · Cited by 628
Applications of multi‐omics analysis in human diseases
Chen, C., Wang, J., Pan, D., Wang, X., Xu, Y., Yan, J., Wang, L., Yang, X., Yang, M., & Liu, G. (2023) · MedComm · doi:10.1002/mco2.315 · Cited by 545
Fraud detection: A systematic literature review of graph-based anomaly detection approaches
Pourhabibi, T., Ong, K.-L., Kam, B. H., & Boo, Y. L. (2020) · Decision Support Systems · doi:10.1016/j.dss.2020.113303 · Cited by 434
Yu, H., Yang, L. T., Zhang, Q., Armstrong, D., & Deen, M. J. (2021) · Neurocomputing · doi:10.1016/j.neucom.2020.04.157 · Cited by 412
Machine Learning for the Diagnosis of Parkinson's Disease: A Review of Literature
Mei, J., Desrosiers, C., & Frasnelli, J. (2021) · Frontiers in Aging Neuroscience · doi:10.3389/fnagi.2021.633752 · Cited by 401
Automatic Analysis of Facial Actions: A Survey
Martinez, B., Valstar, M. F., Jiang, B., & Pantic, M. (2019) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2017.2731763 · Cited by 350
Engagement detection in online learning: a review
Dewan, M. A. A., Murshed, M., & Lin, F. (2019) · Smart Learning Environments · doi:10.1186/s40561-018-0080-z · Cited by 350
Automated Detection of Engagement Using Video-Based Estimation of Facial Expressions and Heart Rate
Monkaresi, H., Bosch, N., Calvo, R. A., & D'Mello, S. K. (2017) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2016.2515084 · Cited by 329
Why Cohen’s Kappa should be avoided as performance measure in classification
Delgado, R., & Tibau, X.-A. (2019) · PLOS ONE · doi:10.1371/journal.pone.0222916 · Cited by 324
Deep Pain: Exploiting Long Short-Term Memory Networks for Facial Expression Classification
Rodriguez, P., Cucurull, G., Gonzalez, J., Gonfaus, J. M., Nasrollahi, K., Moeslund, T. B., & Roca, F. X. (2022) · IEEE Transactions on Cybernetics · doi:10.1109/tcyb.2017.2662199 · Cited by 297
On the class overlap problem in imbalanced data classification
Vuttipittayamongkol, P., Elyan, E., & Petrovski, A. (2021) · Knowledge-Based Systems · doi:10.1016/j.knosys.2020.106631 · Cited by 290
Large-Scale Meta-Analysis of Human Medial Frontal Cortex Reveals Tripartite Functional Organization
de la Vega, A., Chang, L. J., Banich, M. T., Wager, T. D., & Yarkoni, T. (2016) · The Journal of Neuroscience · doi:10.1523/jneurosci.4402-15.2016 · Cited by 288
Neighbourhood-based undersampling approach for handling imbalanced and overlapped data
Vuttipittayamongkol, P., & Elyan, E. (2020) · Information Sciences · doi:10.1016/j.ins.2019.08.062 · Cited by 286
Delving into egocentric actions
Li, Y., Ye, Z., & Rehg, J. M. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298625 · Cited by 284
Stafford, I. S., Kellermann, M., Mossotto, E., Beattie, R. M., MacArthur, B. D., & Ennis, S. (2020) · npj Digital Medicine · doi:10.1038/s41746-020-0229-3 · Cited by 272
Steven Eyobu, O., & Han, D. S. (2018) · Sensors · doi:10.3390/s18092892 · Cited by 259
Deep learning for depression recognition with audiovisual cues: A review
He, L., Niu, M., Tiwari, P., Marttinen, P., Su, R., Jiang, J., Guo, C., Wang, H., Ding, S., Wang, Z., et al. (2022) · Information Fusion · doi:10.1016/j.inffus.2021.10.012 · Cited by 253
A Combined Model- and Learning-Based Framework for Interaction-Aware Maneuver Prediction
Bahram, M., Hubmann, C., Lawitzky, A., Aeberhard, M., & Wollherr, D. (2016) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2015.2506642 · Cited by 175
Automatic Detection of Learning-Centered Affective States in the Wild
Bosch, N., D'Mello, S., Baker, R., Ocumpaugh, J., Shute, V., Ventura, M., Wang, L., & Zhao, W. (2015) · Proceedings of the 20th International Conference on Intelligent User Interfaces · doi:10.1145/2678025.2701397 · Cited by 178
“Other metrics such as Cohen’s Kappa also provide unstable estimates when class distractions are widely skewed [24].”
BackgroundNicholls, H. L., John, C. R., Watson, D. S., Munroe, P. B., Barnes, M. R., & Cabrera, C. P. (2020) · Frontiers in Genetics · doi:10.3389/fgene.2020.00350 · Cited by 155
Koay, A. M. Y., Ko, R. K. L., Hettema, H., & Radke, K. (2023) · Journal of Intelligent Information Systems · doi:10.1007/s10844-022-00753-1 · Cited by 131
Identifying incipient dementia individuals using machine learning and amyloid imaging
Mathotaarachchi, S., Pascoal, T. A., Shin, M., Benedet, A. L., Kang, M. S., Beaudry, T., Fonov, V. S., Gauthier, S., & Rosa-Neto, P. (2017) · Neurobiology of Aging · doi:10.1016/j.neurobiolaging.2017.06.027 · Cited by 118
Le Mau, T., Hoemann, K., Lyons, S. H., Fugate, J. M. B., Brown, E. N., Gendron, M., & Barrett, L. F. (2021) · Nature Communications · doi:10.1038/s41467-021-25352-6 · Cited by 74
Spontaneous facial expression in unscripted social interactions can be measured automatically
Girard, J. M., Cohn, J. F., Jeni, L. A., Sayette, M. A., & De la Torre, F. (2015) · Behavior Research Methods · doi:10.3758/s13428-014-0536-1 · Cited by 69
“Because highly skewed class distributions severely attenuate measures of classifier performance (Jeni, Cohn, & De la Torre, 2013), AUs that occurred less than about 3% of the time were excluded from analysis.”
Self-citationMonllaó Olivé, D., Huynh, D. Q., Reynolds, M., Dougiamas, M., & Wiese, D. (2020) · Journal of Computing in Higher Education · doi:10.1007/s12528-019-09230-1 · Cited by 64
Estimating smile intensity: A better way
Girard, J. M., Cohn, J. F., & De la Torre, F. (2015) · Pattern Recognition Letters · doi:10.1016/j.patrec.2014.10.004 · Cited by 44
Self-citation
Metadata
- Authors
- Jeni, L. A., Cohn, J. F., & De La Torre, F.
- Venue
- 2013 Humaine Association Conference on Affective Computing and Intelligent Interaction
- Publisher
- IEEE
- Published
- 2013-09
- Volume
- 2013
- Pages
- 245-251
- OpenAlex
- W2068258779
- Cited by
- 790
- References
- 28
- FWCI
- 11.93
- Type
- conference paper
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.