CoChat
Sign Up
PreprintCited by 466Topic ModelingText and Document Classification TechnologiesSentiment Analysis and Opinion Mining

Comparing BERT against traditional machine learning text classification

Open access
González-Carvajal, Santiago, Garrido-Merchán, Eduardo C.·2020·arXiv
DOI 10.48550/arxiv.2005.13012arXiv 2005.13012
Open this paper in CoChat — free accountOpen PDF

Abstract

The BERT model has arisen as a popular state-of-the-art machine learning model in the recent years that is able to cope with multiple NLP tasks such as supervised text classification without human supervision. Its flexibility to cope with any type of corpus delivering great results has make this approach very popular not only in academia but also in the industry. Although, there are lots of different approaches that have been used throughout the years with success. In this work, we first present BERT and include a little review on classical NLP approaches. Then, we empirically test with a suite of experiments dealing different scenarios the behaviour of BERT against the traditional TF-IDF vocabulary fed to machine learning algorithms. Our purpose of this work is to add empirical evidence to support or refuse the use of BERT as a default on NLP tasks. Experiments show the superiority of BERT and its independence of features of the NLP problem such as the language of the text adding empirical evidence to use BERT as a default technique to be used in NLP problems.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2005.13012

Cites

3 of 3 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • When using single text vectorization methods alone, pre-trained transformer models (SPECTER2, all-MiniLM-L6-v2) outperformed simpler text vectorization techniques (e.g., TF-IDF and LSA of TF-IDF).

    “When using single text vectorization methods alone, pre-trained, transformer models (SPECTER2, all-MiniLM-L6-v2) outperformed simpler text vectorization techniques (e.g., TF-IDF and LSA of TF-IDF), highlighting the bene fi ts of contextualized embeddings 34 -though at the cost of higher computational cost in large-scale screenings.”

    From AIM review tool: artificial intelligence for smarter systematic review screening (2026) · cites this paper as [34]

    The Abstract and §5 (Conclusions and Further Work; pages not provided) report that BERT outperformed TF-IDF-based traditional classifiers, supporting the general comparison. The paper does not report results for SPECTER2, all-MiniLM-L6-v2, or LSA of TF-IDF, so it cannot substantiate the claim about those specific methods. (LLM, conf=0.87)

    Experimental — accuracy not yet measured

  • Contextualized embeddings come at the cost of higher computational cost in large-scale screenings.

    “When using single text vectorization methods alone, pre-trained, transformer models (SPECTER2, all-MiniLM-L6-v2) outperformed simpler text vectorization techniques (e.g., TF-IDF and LSA of TF-IDF), highlighting the bene fi ts of contextualized embeddings 34 -though at the cost of higher computational cost in large-scale screenings.”

    From AIM review tool: artificial intelligence for smarter systematic review screening (2026) · cites this paper as [34]

    The Experiments and Conclusions sections compare BERT with TF-IDF-based classifiers and report better classification results for BERT, but they do not compare computational costs. Searches for computational cost, processing time, and resource demands found no passage addressing the claimed cost in large-scale screening. (LLM, conf=0.72)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 22

  1. Aggarwal, C. C., & Zhai, C. (2012). Mining text data. Germany: Springer.
  2. Besançon, R., De Chalendar, G., Ferret, O., Gara, F., Mesnard, O., Laïb, M., & Semmar, N. (2010). LIMA: A multilingual framework for linguistic analysis and linguistic resources development and evaluation. In Proceedings of the Seventh International Conference on Language Resources and Evaluation, 3697-3704.
  3. Bikel, D., & Zitouni, I. (2012). Multilingual natural language processing applications: From theory to practice. USA: IBM Press.
  4. Cambria, E., & White, B. (2014). Jumping NLP curves: A review of natural language processing research. IEEE Computational Intelligence Magazine, 9(2), 48-57.
  5. Farzindar, A., & Inkpen, D. (2015). Natural language processing for social media. Synthesis Lectures on Human Language Technologies, 8(2), 1-166.
  6. Garrido-Merchán, E. C., Molina, M., & Mendoza, F. M. (2020). An artificial consciousness model and its relations with philosophy of mind. arXiv preprint:2011.14475. · arXiv
  7. Garrido-Merchán, E. (2015). Context-adaptive supervised expansion of polarized lexicons. Master Thesis, Comillas Pontifical University.
  8. Jin, D., Jin, Z., Zhou, J. T., & Szolovits, P. (2019). Is BERT really robust? Natural language attack on text classification and entailment. arXiv preprint:1907.11932, 2. · arXiv
  9. Khurana, D., Koli, A., Khatter, K., & Singh, S. (2023). Natural language processing: State of the art, current trends and challenges. Multimedia Tools and Applications, 82(3), 3713-3744.
  10. Manning, C., & Schutze, H. (1999). Foundations of statistical natural language processing. USA: MIT Press.
  11. Merchán, E. C. G., & Molina, M. (2020). A machine consciousness architecture based on deep learning and Gaussian processes. In Hybrid Artificial Intelligent Systems: 15th International Conference, 15, 350-361.
  12. Otter, D. W., Medina, J. R., & Kalita, J. K. (2020). A survey of the usages of deep learning for natural language processing. IEEE Transactions on Neural Networks and Learning Systems, 32(2), 604-624.
  13. Robertson, S. (2004). Understanding inverse document frequency: On theoretical arguments for IDF. Journal of Documentation, 60(5), 503-520.
  14. Rother, K., & Rettberg, A. (2018). ULMFiT at germeval-2018: A deep neural language model for the classification of hate speech in German tweets. In 14th Conference on Natural Language Processing, 113-119.
  15. Stamatatos, E. (2011). Plagiarism detection using stopword n-grams. Journal of the American Society for Information Science and Technology, 62(12), 2512-2527.
  16. Tenney, I., Das, D., & Pavlick, E. (2019). BERT rediscovers the classical NLP pipeline. arXiv preprint:1905.05950. · arXiv
  17. Uchida, H., & Zhu, M. (2001). The universal networking language beyond machine translation. In International Symposium on Language in Cyberspace, 26-27.
  18. Usherwood, P., & Smit, S. (2019). Low-shot classification: A comparison of classical and deep transfer machine learning approaches. arXiv preprint:1907.07543. · arXiv
  19. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., : : : & Polosukhin, I. (2017). Attention is all you need. Advances in Neural Information Processing Systems, 30.
  20. Wu, Y., Schuster, M., Chen, Z., Le, Q. V., Norouzi, M., Macherey, W., : : : & Dean, J. (2016). Google's neural machine translation system: Bridging the gap between human and machine translation. arXiv preprint:1609.08144. · arXiv
  21. Zhang, Y. Z., Gong, L., & Wang, Y. C. (2005). An improved TF-IDF approach for text classification. Journal of Zhejiang University-Science A, 6, 49-55.
  22. Zhang, Y., Jin, R., & Zhou, Z. H. (2010). Understanding bag-of- words model: A statistical framework. International Journal of Machine Learning and Cybernetics, 1, 43-52.

Cited by

Sorted by citations

Authors
González-Carvajal, S., & Garrido-Merchán, E. C.
Affiliations
Universidad Politécnica de Madrid (ES)
Venue
arXiv
Publisher
arXiv
Published
2020
Volume
2
Issue
4
Pages
352-356
ISSN
2810-9503 (ISSN-L)
DOI
10.48550/arxiv.2005.13012
arXiv
2005.13012
OpenAlex
W3030030185
Cited by
466
References
3
Citations by year
2026: 72 · 2025: 125 · 2024: 95 · 2023: 87 · 2022: 60 · 2021: 26 · 2020: 1
FWCI
32.69
Type
journal article
Language
en
Open access
Licence
Locations
  • Journal of Computational and Cognitive Engineering (publishedVersion, cc-by)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Funders
Universidad Autónoma de Madrid
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Artificial Intelligence › Text and Document Classification Technologies
  • Physical Sciences › Computer Science › Artificial Intelligence › Sentiment Analysis and Opinion Mining
Keywords
BERT, text classification, TF-IDF, transformer models, language independence, feature representation

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review