CoChat
Sign Up
Cited by 15Computational and Text Analysis MethodsTopic ModelingSentiment Analysis and Opinion Mining

Uncertainty-Aware Machine Support for Paper Reviewing on the Interspeech 2019 Submission Corpus

Open access
Stappen, Lukas, Rizos, Georgios, Hasan, Madina, Hain, Thomas, Schuller, Björn W.·2020·Interspeech 2020
DOI 10.21437/interspeech.2020-2862
Open this paper in CoChat — free accountOpen PDF

Abstract

The evaluation of scientific submissions through peer review is both the most fundamental component of the publication process, as well as the most frequently criticised and questioned.Academic journals and conferences request reviews from multiple reviewers per submission, which an editor, or area chair aggregates into the final acceptance decision.Reviewers are often in disagreement due to varying levels of domain expertise, confidence, levels of motivation, as well as due to the heavy workload and the different interpretations by the reviewers of the score scale.Herein, we explore the possibility of a computational decision support tool for the editor, based on Natural Language Processing, that offers an additional aggregated recommendation.We provide a comparative study of state-of-the-art text modelling methods on the newly crafted, largest review dataset of its kind based on Interspeech 2019, and we are the first to explore uncertainty-aware methods (soft labels, quantile regression) to address the subjectivity inherent in this problem.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.21437/interspeech.2020-2862

Cites

4 of 4 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 32

  1. D. A. Kronick, "Peer review in 18th-century scientific journalism," Journal of the American Medical Association, vol. 263, no. 10, pp. 1321-1322, 1990.
  2. D. Chavalarias and J. P. Ioannidis, "Science mapping analysis char- acterizes 235 biases in biomedical research," Journal of Clinical Epidemiology, vol. 63, no. 11, pp. 1205-1215, 2010.
  3. J. S. Gans and G. B. Shepherd, "How are the mighty fallen: Re- jected classic articles by leading economists," Journal of Economic Perspectives, vol. 8, no. 1, pp. 165-179, 1994.
  4. J. L. Jackson, M. Srinivasan, J. Rea, K. E. Fletcher, and R. L. Kravitz, "The validity of peer review in a general medicine journal," PloS one, vol. 6, no. 7, 2011.
  5. N. Lawrence and C. Cortes. (2014) The NIPS experiment. [Online]. Available: http://inverseprobability.com/2014/12/16/ the-nips-experiment
  6. K. Wang and X. Wan, "Sentiment analysis of peer review texts for scholarly papers," in The 41st International ACM SIGIR Confer- ence on Research & Development in Information Retrieval. ACM, 2018, pp. 175-184.
  7. T. Ghosal, R. Verma, A. Ekbal, and P. Bhattacharyya, "Deepsen- tipeer: Harnessing sentiment in review texts to recommend peer review decisions," in Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. ACL, 2019, pp. 1120-1130.
  8. T. Mikolov, I. Sutskever, K. Chen, G. S. Corrado, and J. Dean, "Distributed representations of words and phrases and their compo- sitionality," in Advances in Neural Information Processing Systems (NeurIPS), 2013, pp. 3111-3119.
  9. D. Cer, Y. Yang, S.-y. Kong, N. Hua, N. Limtiaco, R. S. John, N. Constant, M. Guajardo-Cespedes, S. Yuan, C. Tar et al., "Uni- versal sentence encoder for english," in Proceedings of the 2018 Conference on Empirical Methods in Natural Language Process- ing: System Demonstrations (EMNLP). ACL, 2018, pp. 169-174.
  10. C. J. Hutto and E. Gilbert, "Vader: A parsimonious rule-based model for sentiment analysis of social media text," in 18th Interna- tional AAAI Conference on Weblogs and Social Media. AAAI, 2014.
  11. Z. Lan, M. Chen, S. Goodman, K. Gimpel, P. Sharma, and R. Sori- cut, "Albert: A lite bert for self-supervised learning of language representations," in International Conference on Learning Repre- sentations (ICLR), 2019.
  12. P. Bojanowski, E. Grave, A. Joulin, and T. Mikolov, "Enriching word vectors with subword information," Transactions of the Asso- ciation for Computational Linguistics, vol. 5, pp. 135-146, 2017.
  13. A. Ando, S. Kobashikawa, H. Kamiyama, R. Masumura, Y. Ijima, and Y. Aono, "Soft-target training with ambiguous emotional ut- terances for dnn-based speech emotion classification," in 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2018, pp. 4964-4968.
  14. G. Hinton, O. Vinyals, and J. Dean, "Distilling the knowledge in a neural network," arXiv preprint arXiv:1503.02531, 2015. · arXiv
  15. G. Ostrovski, W. Dabney, and R. Munos, "Autoregressive quantile networks for generative modeling," in Proceedings of the 35th International Conference on Machine Learning (ICLM), 2018.
  16. W. Dabney, G. Ostrovski, D. Silver, and R. Munos, "Implicit quantile networks for distributional reinforcement learning," in Proceedings of the 35th International Conference on Machine Learning (ICLM), 2018, pp. 1096-1105.
  17. Y. Romano, E. Patterson, and E. J. Candès, "Conformalized quan- tile regression," in Advances in Neural Information Processing Systems (NeurIPS), 2019.
  18. F. Rodrigues and F. C. Pereira, "Beyond expectation: Deep joint mean and quantile regression for spatiotemporal problems," IEEE Transactions on Neural Networks and Learning Systems, 2020.
  19. A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, "Attention is all you need," in Advances in Neural Information Processing Systems (NeurIPS), 2017, pp. 5998-6008.
  20. L. Stappen, F. Brunn, and B. Schuller, "Cross-lingual zero-and few- shot hate speech detection utilising frozen transformer language models and axel," arXiv preprint arXiv:2004.13850, 2020. · arXiv
  21. B. W. Schuller, A. Batliner, C. Bergler, E.-M. Messner, A. Hamil- ton, S. Amiriparian, A. Baird, G. Rizos, M. Schmitt, L. Stappen et al., "The interspeech 2020 computational paralinguistics chal- lenge: Elderly emotion, breathing & masks." ISCA, 2020.
  22. G. Lample and A. Conneau, "Cross-lingual language model pre- training," arXiv preprint arXiv:1901.07291, 2019. · arXiv
  23. Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, and V. Stoyanov, "Roberta: A robustly optimized bert pretraining approach," arXiv preprint arXiv:1907.11692, 2019. · arXiv
  24. T. Mikolov, K. Chen, G. Corrado, and J. Dean, "Efficient esti- mation of word representations in vector space," arXiv preprint arXiv:1301.3781, 2013. · arXiv
  25. J. Pennington, R. Socher, and C. Manning, "Glove: Global Vectors for Word Representation," in Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2014.
  26. L. Stappen, N. Cummins, E.-M. Meßner, H. Baumeister, J. Dineley, and B. Schuller, "Context modelling using hierarchical attention networks for sentiment and self-assessed emotion detection in spoken narratives," in ICASSP 2019-2019 IEEE International Con- ference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2019, pp. 6680-6684.
  27. Y. Kim, "Convolutional neural networks for sentence classification," in Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP). ACL, Oct. 2014.
  28. J. P. Chiu and E. Nichols, "Named entity recognition with bidirec- tional lstm-cnns," Transactions of the Association for Computa- tional Linguistics, vol. 4, pp. 357-370, 2016.
  29. Z. Zhang, D. Robinson, and J. Tepper, "Detecting hate speech on twitter using a convolution-gru based deep neural network," in European Semantic Web Conference. Springer, 2018, pp. 745- 760.
  30. G. Rizos, K. Hemker, and B. Schuller, "Augment to prevent: Short- text data augmentation in deep learning for hate-speech classifica- tion," in Proceedings of the 28th ACM International Conference on Information and Knowledge Management (CIKM). ACM, 2019, pp. 991-1000.
  31. Z. Zhang, B. Wu, and B. Schuller, "Attention-augmented end-to- end multi-task learning for emotion prediction from speech," in ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2019, pp. 6705- 6709.
  32. A. Kendall and Y. Gal, "What uncertainties do we need in bayesian deep learning for computer vision?" in Advances in Neural Infor- mation Processing Systems (NeurIPS), 2017, pp. 5574-5584.

Cited by

Sorted by citations

Authors
Stappen, L., Rizos, G., Hasan, M., Hain, T., & Schuller, B. W.
Affiliations
  • University of Augsburg (DE)
  • Imperial College London (GB)
  • University of Sheffield (GB)
ORCID
Venue
Interspeech 2020
Publisher
ISCA
Published
2020-10-25
Pages
1808-1812
ISSN
2308-457X (ISSN-L)
DOI
10.21437/interspeech.2020-2862
OpenAlex
W3095601272
Cited by
15
References
4
Citations by year
2025: 1 · 2024: 2 · 2023: 3 · 2022: 1 · 2021: 8
FWCI
4.29
Type
conference paper
Language
en
Open access
Locations
  • Interspeech (USB) (publishedVersion)
  • White Rose Research Online (University of Leeds, The University of Sheffield, University of York) (acceptedVersion)
  • OPUS (Augsburg University) (submittedVersion)
Funders
  • Imperial College London
  • Engineering and Physical Sciences Research Council (2021037)
Topics
  • Social Sciences › General Social Sciences › Computational and Text Analysis Methods
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Artificial Intelligence › Sentiment Analysis and Opinion Mining
Keywords
peer review, decision support system, soft labels, quantile regression, reviewer disagreement, text modeling

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review