CoChat
Sign Up
PreprintCited by 887Topic ModelingHate Speech and Cyberbullying DetectionText and Document Classification Technologies

FastText.zip: Compressing text classification models

Open access
Joulin, Armand, Grave, Edouard, Bojanowski, Piotr, Douze, Matthijs, Jégou, Hérve, Mikolov, Tomas·2016·arXiv
DOI 10.48550/arxiv.1612.03651arXiv 1612.03651
Open this paper in CoChat — free accountOpen PDF

Abstract

We consider the problem of producing compact architectures for text classification, such that the full model fits in a limited amount of memory. After considering different solutions inspired by the hashing literature, we propose a method built upon product quantization to store word embeddings. While the original technique leads to a loss in accuracy, we adapt this method to circumvent quantization artefacts. Our experiments carried out on several benchmarks show that our approach typically requires two orders of magnitude less memory than fastText while being only slightly inferior with respect to accuracy. As a result, it outperforms the state of the art by a good margin in terms of the compromise between memory usage and accuracy.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.1612.03651

Cites

3 of 3 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Language identification is cross-validated using fastText on recognized text and MMS-LID on raw waveform to make language annotations robust against ASR errors, short utterances, and mixed-language cases.

    “Furthermore, language identification (LID) is cross-validated using fastText [29, 30] on the recognized text and MMS-LID [31] on the raw waveform, making the final language annotations robust against ASR errors, short utterances, and mixed-language cases.”

    From MOSS-Audio Technical Report (2026) · cites this paper as [30]

  • DeepLog considers all new log events as anomalies because they cannot be predicted by the model.

    “Moreover, DeepLog [5], using the indexes of log templates to predict the next log event, considers all new log events as anomalies because they cannot be predicted by the model.”

    From Log-based Anomaly Detection Without Log Parsing (2021) · cites this paper as [5]

    The Introduction (p. 1) and Related Work (p. 2) discuss text classification and model compression, not DeepLog. Searches for DeepLog and for unseen log templates and anomaly detection found no passage addressing whether new log events are treated as anomalies. (LLM, conf=0.76)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 39

  1. Alekh Agarwal, Olivier Chapelle, Miroslav Dud´ ık, and John Langford. A reliable effective terascale linear learning system. Journal of Machine Learning Research , 15(1):1111-1133, 2014.
  2. Francis Bach, Rodolphe Jenatton, Julien Mairal, and Guillaume Obozinski. Optimization with sparsity-inducing penalties. Foundations and Trends R © in Machine Learning , 4(1):1-106, 2012.
  3. Ashwinkumar Badanidiyuru, Baharan Mirzasoleiman, Amin Karbasi, and Andreas Krause. Streaming submodular maximization: Massive data summarization on the fly. In SIGKDD , pp. 671-680. ACM, 2014.
  4. Mohammad Hossein Bateni, Mohammad Taghi Hajiaghayi, and Morteza Zadimoghaddam. Submodular secretary problem and extensions. In Approximation, Randomization, and Combinatorial Optimization. Algorithms and Techniques , pp. 39-52. Springer, 2010.
  5. Moses S. Charikar. Similarity estimation techniques from rounding algorithms. In STOC , pp. 380388, May 2002.
  6. Welin Chen, David Grangier, and Michael Auli. Strategies for training large vocabulary neural language models. arXiv preprint arXiv:1512.04906 , 2015. · arXiv
  7. Flavio Chierichetti, Ravi Kumar, and Andrew Tomkins. Max-cover in map-reduce. In International Conference on World Wide Web , 2010.
  8. Matthieu Courbariaux, Itay Hubara, Daniel Soudry, Ran El-Yaniv, and Yoshua Bengio. Binarized neural networks: Training neural networks with weights and activations constrained to +1 or -1. arXiv preprint arXiv:1602.02830 , 2016. · arXiv
  9. M. Datar, N. Immorlica, P. Indyk, and V.S. Mirrokni. Locality-sensitive hashing scheme based on pstable distributions. In Proceedings of the Symposium on Computational Geometry , pp. 253-262, 2004.
  10. Scott Deerwester, Susan T Dumais, George W Furnas, Thomas K Landauer, and Richard Harshman. Indexing by latent semantic analysis. Journal of the American society for information science , 1990.
  11. Misha Denil, Babak Shakibi, Laurent Dinh, Marc-Aurelio Ranzato, and Nando et all de Freitas. Predicting parameters in deep learning. In NIPS , pp. 2148-2156, 2013.
  12. Uriel Feige. A threshold of ln n for approximating set cover. JACM , 45(4):634-652, 1998.
  13. Tiezheng Ge, Kaiming He, Qifa Ke, and Jian Sun. Optimized product quantization for approximate nearest neighbor search. In CVPR , June 2013.
  14. Yunchao Gong and Svetlana Lazebnik. Iterative quantization: A procrustean approach to learning binary codes. In CVPR , June 2011.
  15. Yunchao Gong, Liu Liu, Ming Yang, and Lubomir Bourdev. Compressing deep convolutional networks using vector quantization. arXiv preprint arXiv:1412.6115 , 2014. · arXiv
  16. Edouard Grave, Armand Joulin, Moustapha Ciss´ e, David Grangier, and Herv´ e J´ egou. Efficient softmax approximation for gpus. arXiv preprint arXiv:1609.04309 , 2016. · arXiv
  17. Song Han, Huizi Mao, and William J Dally. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding. In ICLR , 2016.
  18. Thorsten Joachims. Text categorization with support vector machines: Learning with many relevant features . Springer, 1998.
  19. Yann LeCun, John S Denker, and Sara A Solla. Optimal brain damage. NIPS , 2:598-605, 1990.
  20. Zhouhan Lin, Matthieu Courbariaux, Roland Memisevic, and Yoshua Bengio. Neural networks with few multiplications. arXiv preprint arXiv:1510.03009 , 2015. · arXiv
  21. Andrew McCallum and Kamal Nigam. A comparison of event models for naive bayes text classification. In AAAI workshop on learning for text categorization , 1998.
  22. Lukas Meier, Sara Van De Geer, and Peter B¨ uhlmann. The group lasso for logistic regression. Journal of the Royal Statistical Society: Series B (Statistical Methodology) , 70(1):53-71, 2008.
  23. Tomas Mikolov. Statistical language models based on neural networks. In PhD thesis . VUT Brno, 2012.
  24. Tomas Mikolov, Ilya Sutskever, Anoop Deoras, Hai-Son Le, Stefan Kombrink, and J Cernocky. Subword language modeling with neural networks. preprint , 2012.
  25. Behnam Neyshabur and Nathan Srebro. On symmetric and asymmetric lshs for inner product search. In ICML , pp. 1926-1934, 2015.
  26. Mohammad Norouzi and David Fleet. Cartesian k-means. In CVPR , June 2013.
  27. Alexandre Sablayrolles, Matthijs Douze, Herv´ e J´ egou, and Nicolas Usunier. How should we evaluate supervised hashing? arXiv preprint arXiv:1609.06753 , 2016. · arXiv
  28. Jorge S´ anchez and Florent Perronnin. High-dimensional signature compression for large-scale image classification. In CVPR , 2011.
  29. Anshumali Shrivastava and Ping Li. Asymmetric LSH for sublinear time maximum inner product search. In NIPS , pp. 2321-2329, 2014.
  30. Andreas Stolcke. Entropy-based pruning of backoff language models. arXiv preprint cs/0006025 , 2000. · arXiv
  31. David Talbot and Thorsten Brants. Randomized language models via perfect hash functions. In ACL , 2008.
  32. Bart Thomee, David A Shamma, Gerald Friedland, Benjamin Elizalde, Karl Ni, Douglas Poland, Damian Borth, and Li-Jia Li. Yfcc100m: The new data in multimedia research. In Communications of the ACM , 2016.
  33. Jingdong Wang, Heng Tao Shen, Jingkuan Song, and Jianqiu Ji. Hashing for similarity search: A survey. arXiv preprint arXiv:1408.2927 , 2014. · arXiv
  34. Jun Wang, Wei Liu, Sanjiv Kumar, and Shih-Fu Chang. Learning to hash for indexing big data - A survey. CoRR , abs/1509.05472, 2015. · arXiv
  35. Sida Wang and Christopher D Manning. Baselines and bigrams: Simple, good sentiment and topic classification. In ACL , 2012.
  36. Kilian Q Weinberger, Anirban Dasgupta, John Langford, Alex Smola, and Josh Attenberg. Feature hashing for large scale multitask learning. In ICML , 2009.
  37. Yair Weiss, Antonio Torralba, and Rob Fergus. Spectral hashing. In NIPS , December 2009.
  38. Yijun Xiao and Kyunghyun Cho. Efficient character-level document classification by combining convolution and recurrent layers. arXiv preprint arXiv:1602.00367 , 2016. · arXiv
  39. Xiang Zhang, Junbo Zhao, and Yann LeCun. Character-level convolutional networks for text classification. In NIPS , 2015.

Cited by

Sorted by citations

Authors
Joulin, A., Grave, E., Bojanowski, P., Douze, M., Jégou, H., & Mikolov, T.
ORCID
Venue
arXiv
Publisher
arXiv
Published
2016
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.1612.03651
arXiv
1612.03651
OpenAlex
W2563351168
Cited by
887
References
3
Citations by year
2026: 2 · 2025: 50 · 2024: 104 · 2023: 142 · 2022: 137 · 2021: 164 · 2020: 138 · 2019: 102 · 2018: 37 · 2017: 11
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Artificial Intelligence › Hate Speech and Cyberbullying Detection
  • Physical Sciences › Computer Science › Artificial Intelligence › Text and Document Classification Technologies
Keywords
Twitter, online communities, destructive behavior, emoji use, linguistic patterns, COREX, topic extraction, Word Mover's Distance, sentence similarity, data annotation, data augmentation

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review