MTEB: Massive Text Embedding Benchmark
Open accessAbstract
Text embeddings are commonly evaluated on a small set of datasets from a single task not covering their possible applications to other tasks.It is unclear whether state-of-the-art embeddings on semantic textual similarity (STS) can be equally well applied to other tasks like clustering or reranking.This makes progress in the field difficult to track, as various models are constantly being proposed without proper evaluation.To solve this problem, we introduce the Massive Text Embedding Benchmark (MTEB).MTEB spans 8 embedding tasks covering a total of 58 datasets and 112 languages.Through the benchmarking of 33 models on MTEB, we establish the most comprehensive benchmark of text embeddings to date.We find that no particular text embedding method dominates across all tasks.This suggests that the field has yet to converge on a universal text embedding method and scale it up sufficiently to provide state-of-the-art results on all embedding tasks.MTEB comes with open-source code and a public leaderboard at https://github.com/ embeddings-benchmark/mteb.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The MoE approach has demonstrated excellent performance across a range of tasks within natural language processing and computer vision domains.
“The MoE approach has demonstrated excellent performance across a range of tasks within natural language processing77 and computer vision domains.”
From MoMA: a mixture-of-multimodal-agents architecture for enhancing clinical prediction modelling (2025) · cites this paper as [77]
The Introduction describes MTEB as a benchmark for text-embedding models, and §4.2 analyzes those models’ performance across tasks. Searches for “MoE,” “mixture of experts,” and expert routing found no passage assessing the MoE approach in natural language processing; our copy therefore does not verify the claim attributed to this source. (LLM, conf=0.76)
Experimental — accuracy not yet measuredThe Massive Text Embedding Benchmark shows that text-embedding-ada-002 is not clearly much better or worse than a variety of SentenceTransformers models, including all-MiniLM-L6-v2, across a range of tasks.
“we consult the Massive Text Embedding Benchmark [63], which shows that the most powerful OpenAI embeddings model ( text-embedding-ada-002 ) is not clearly much better or worse than a variety of SentenceTransformers models (including all-MiniLM-L6-v2 ) across a range of tasks.”
From Homogenization Effects of Large Language Models on Human Creative Ideation (2024) · cites this paper as [63]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Jakubův, J., Chvalovský, K., Goertzel, Z., et al. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.itp.2023.19 · Cited by 76,648 · see citing works
Scikit-learn: Machine Learning in Python
Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Müller, A., Nothman, J., Louppe, G., et al. (2012) · arXiv · doi:10.48550/arxiv.1201.0490 · Cited by 71,016 · see citing works
Glove: Global Vectors for Word Representation
Pennington, J., Socher, R., & Manning, C. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1162 · Cited by 34,071 · see citing works
Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks
Reimers, N., & Gurevych, I. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1410 · Cited by 12,778 · see citing works
Self-citationExploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., & Liu, P. J. (2019) · arXiv · doi:10.48550/arxiv.1910.10683 · Cited by 10,929 · see citing works
Transformers: State-of-the-Art Natural Language Processing
Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., et al. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations · doi:10.18653/v1/2020.emnlp-demos.6 · Cited by 8,741 · see citing works
GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding
Wang, A., Singh, A., Michael, J., Hill, F., Levy, O., & Bowman, S. (2018) · Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP · doi:10.18653/v1/w18-5446 · Cited by 4,222 · see citing works
Learning Word Vectors for Sentiment Analysis
Maas, A. L., Daly, R. E., Pham, P. T., Huang, D., Ng, A. Y., & Potts, C. (2011) · Cited by 3,710 · see citing works
SciBERT: A Pretrained Language Model for Scientific Text
Beltagy, I., Lo, K., & Cohan, A. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1371 · Cited by 3,218 · see citing works
SimCSE: Simple Contrastive Learning of Sentence Embeddings
Gao, T., Yao, X., & Chen, D. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.552 · Cited by 2,851 · see citing works
PaLM: Scaling Language Modeling with Pathways
Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H. W., Sutton, C., Gehrmann, S., et al. (2022) · arXiv · doi:10.48550/arxiv.2204.02311 · Cited by 2,590 · see citing works
Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding
Saharia, C., Chan, W., Saxena, S., Li, L., Whang, J., Denton, E., Ghasemipour, S. K. S., Ayan, B. K., Mahdavi, S. S., Lopes, R. G., et al. (2022) · arXiv · doi:10.48550/arxiv.2205.11487 · Cited by 2,528 · see citing works
LXMERT: Learning Cross-Modality Encoder Representations from Transformers
Tan, H., & Bansal, M. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1514 · Cited by 2,319 · see citing works
Hidden factors and hidden topics
McAuley, J., & Leskovec, J. (2013) · ACM Conference on Recommender Systems (RecSys) · doi:10.1145/2507157.2507163 · Cited by 1,565 · see citing works
GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models
Nichol, A., Dhariwal, P., Ramesh, A., Shyam, P., Mishkin, P., McGrew, B., Sutskever, I., & Chen, M. (2021) · arXiv · doi:10.48550/arxiv.2112.10741 · Cited by 1,357 · see citing works
V-Measure: A Conditional Entropy-Based External Cluster Evaluation Measure
Rosenberg, A., & Hirschberg, J. (2007) · Cited by 1,216 · see citing works
SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems
Wang, A., Pruksachatkun, Y., Nangia, N., Singh, A., Michael, J., Hill, F., Levy, O., & Bowman, S. R. (2019) · arXiv · doi:10.48550/arxiv.1905.00537 · Cited by 1,051 · see citing works
MPNet: Masked and Permuted Pre-training for Language UnderstandingPreprint
Song, K., Tan, X., Qin, T., Lu, J., & Liu, T. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2004.09297 · Cited by 713 · see citing works
MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformers
Wang, W., Wei, F., Dong, L., Bao, H., Yang, N., & Zhou, M. (2020) · arXiv · doi:10.48550/arxiv.2002.10957 · Cited by 703 · see citing works
Text and Code Embeddings by Contrastive Pre-TrainingPreprint
Neelakantan, A., Xu, T., Puri, R., Radford, A., Han, J. M., Tworek, J., Yuan, Q.-M., Tezak, N., Kim, J. W., Hallacy, C., et al. (2022) · arXiv.org · doi:10.48550/arxiv.2201.10005 · Cited by 657 · see citing works
SemEval-2012 Task 6: A Pilot on Semantic Textual Similarity
Agirre, E., Cer, D. M., Diab, M., & González-Agirre, A. (2012) · Cited by 653 · see citing works
A Survey of Text Clustering Algorithms
Aggarwal, C. C., & Zhai, C. (2012) · Springer eBooks · doi:10.1007/978-1-4614-3223-4_4 · Cited by 583 · see citing works
SPECTER: Document-level Representation Learning using Citation-informed Transformers
Cohan, A., Feldman, S., Beltagy, I., Downey, D., & Weld, D. S. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.207 · Cited by 572 · see citing works
Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models
Srivastava, A., Rastogi, A., Rao, A., Shoeb, A. A. M., Abid, A., Fisch, A., Brown, A. R., Santoro, A., Gupta, A., Garriga-Alonso, A., et al. (2022) · arXiv · doi:10.48550/arxiv.2206.04615 · Cited by 569 · see citing works
Self-citationGLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding
Wang, A., Singh, A., Michael, J., Hill, F., Levy, O., & Bowman, S. R. (2018) · arXiv · doi:10.48550/arxiv.1804.07461 · Cited by 562 · see citing works
Top2Vec: Distributed Representations of TopicsPreprint
Angelov, D. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2008.09470 · Cited by 539 · see citing works
SemEval-2016 Task 1: Semantic Textual Similarity, Monolingual and Cross-Lingual Evaluation
Agirre, E., Banea, C., Cer, D. M., Diab, M., González-Agirre, A., Mihalcea, R. F., Rigau, G., & Wiebe, J. (2016) · International Workshop on Semantic Evaluation (SemEval) · doi:10.18653/v1/s16-1081 · Cited by 527 · see citing works
MIND: A Large-scale Dataset for News Recommendation
Wu, F., Qiao, Y., Chen, J.-H., Wu, C., Qi, T., Lian, J., Liu, D., Xie, X., Gao, J., Wu, W., et al. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.331 · Cited by 527 · see citing works
SemEval-2014 Task 10: Multilingual Semantic Textual Similarity
Agirre, E., Banea, C., Cardie, C., Cer, D. M., Diab, M., González-Agirre, A., Guo, W., Mihalcea, R. F., Rigau, G., & Wiebe, J. (2014) · International Workshop on Semantic Evaluation (SemEval) · doi:10.3115/v1/s14-2010 · Cited by 492 · see citing works
SemEval-2015 Task 2: Semantic Textual Similarity, English, Spanish and Pilot on Interpretability
Agirre, E., Banea, C., Cardie, C., Cer, D. M., Diab, M., González-Agirre, A., Guo, W., López-Gazpio, I., Maritxalar, M., Mihalcea, R. F., et al. (2015) · International Workshop on Semantic Evaluation (SemEval) · doi:10.18653/v1/s15-2045 · Cited by 488 · see citing works
CodeSearchNet Challenge: Evaluating the State of Semantic Code Search
Husain, H., Wu, H.-H., Gazit, T., Allamanis, M., & Brockschmidt, M. (2019) · arXiv · doi:10.48550/arxiv.1909.09436 · Cited by 483 · see citing works
SentEval: An Evaluation Toolkit for Universal Sentence Representations
Conneau, A., & Kiela, D. (2018) · arXiv · doi:10.48550/arxiv.1803.05449 · Cited by 477 · see citing works
LXMERT: Learning Cross-Modality Encoder Representations from TransformersPreprint
Tan, H. X., & Bansal, M. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1908.07490 · Cited by 433 · see citing works
Improving language models by retrieving from trillions of tokens
Borgeaud, S., Mensch, A., Hoffmann, J., Cai, T., Rutherford, E., Millican, K., Driessche, G. V. D., Lespiau, J.-B., Damoc, B., Clark, A., et al. (2021) · arXiv · doi:10.48550/arxiv.2112.04426 · Cited by 382 · see citing works
Crosslingual Generalization through Multitask Finetuning
Muennighoff, N., Wang, T., Sutawika, L., Roberts, A., Biderman, S., Le Scao, T., Bari, M. S., Shen, S., Yong, Z. X., Schoelkopf, H., et al. (2023) · Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2023.acl-long.891 · Cited by 361 · see citing works
Self-citation*SEM 2013 shared task: Semantic Textual Similarity
Agirre, E., Cer, D. M., Diab, M., González-Agirre, A., & Guo, W. (2013) · Joint Conference on Lexical and Computational Semantics · Cited by 360 · see citing works
Datasets: A Community Library for Natural Language Processing
Lhoest, Q., Villanova del Moral, A., Jernite, Y., Thakur, A., von Platen, P., Patil, S., Chaumond, J., Drame, M., Plu, J., Tunstall, L., et al. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing: System Demonstrations · doi:10.18653/v1/2021.emnlp-demo.21 · Cited by 343 · cited 1× · see citing works
“(c) Extensibility: New datasets for existing tasks can be benchmarked in MTEB via a single file that specifies the task and a Hugging Face dataset name where the data has been uploaded (Lhoest et al., 2021).”
MethodCARER: Contextualized Affect Representations for Emotion Recognition
Saravia, E., Liu, H.-C. T., Huang, Y.-H., Wu, J., & Chen, Y.-S. (2018) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d18-1404 · Cited by 337 · see citing works
SGPT: GPT Sentence Embeddings for Semantic SearchPreprint
Muennighoff, N. (2022) · arXiv.org · doi:10.48550/arxiv.2202.08904 · Cited by 285 · see citing works
Self-citationLanguage-agnostic BERT Sentence EmbeddingPreprint
Feng, F., Yang, Y., Cer, D. M., et al. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2007.01852 · Cited by 194 · see citing works
(2022) · doi:10.18653/v1/2022.emnlp-main.669 · Cited by 177 · see citing works
Unsupervised Dense Information Retrieval with Contrastive Learning
Izacard, G., Caron, M., Hosseini, L., Riedel, S., Bojanowski, P., Joulin, A., & Grave, E. (2021) · arXiv · doi:10.48550/arxiv.2112.09118 · Cited by 172 · see citing works
A Continuously Growing Dataset of Sentential Paraphrases
Lan, W., Qiu, S., He, H., & Xu, W. H. (2017) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d17-1126 · Cited by 157 · see citing works
SemEval-2015 Task 1: Paraphrase and Semantic Similarity in Twitter (PIT)
Xu, W. H., Callison-Burch, C., & Dolan, B. (2015) · International Workshop on Semantic Evaluation (SemEval) · doi:10.18653/v1/s15-2001 · Cited by 145 · see citing works
Dependency Based Embeddings for Sentence Classification Tasks
Komninos, A., & Manandhar, S. (2016) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n16-1175 · Cited by 128 · see citing works
(2021) · doi:10.18653/v1/2021.naacl-main.46 · Cited by 115 · see citing works
Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks
Reimers, N., & Gurevych, I. (2019) · arXiv · doi:10.48550/arxiv.1908.10084 · Cited by 86 · see citing works
Self-citationAdversarial Domain Adaptation for Duplicate Question Detection
Shah, D., Lei, T., Moschitti, A., Romeo, S. O., & Nakov, P. (2018) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d18-1131 · Cited by 69 · see citing works
Overview of the Second BUCC Shared Task: Spotting Parallel Sentences in Comparable Corpora
Zweigenbaum, P., Sharoff, S., & Rapp, R. (2017) · doi:10.18653/v1/w17-2512 · Cited by 62 · see citing works
SantaCoder: don't reach for the stars!
Allal, L. B., Li, R., Kocetkov, D., Mou, C., Akiki, C., Ferrandis, C. M., Muennighoff, N., Mishra, M., Gu, A., Dey, M., et al. (2023) · arXiv · doi:10.48550/arxiv.2301.03988 · Cited by 61 · see citing works
Self-citationSentence-T5: Scalable Sentence Encoders from Pre-trained Text-to-Text ModelsPreprint
Ni, J., Ábrego, G. H., Constant, N., et al. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2108.08877 · Cited by 59 · see citing works
Datasets: A Community Library for Natural Language ProcessingPreprint
Lhoest, Q., del Moral, A. V., Jernite, Y., Thakur, A., von Platen, P., Patil, S. S., Chaumond, J., Drame, M., Plu, J., Tunstall, L., et al. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2109.02846 · Cited by 54 · see citing works
Unsupervised Corpus Aware Language Model Pre-training for Dense Passage RetrievalPreprint
Gao, L., & Callan, J. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2108.05540 · Cited by 52 · see citing works
Bitext Mining Using Distilled Sentence Representations for Low-Resource Languages
Heffernan, K. S., Çelebi, O., & Schwenk, H. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.154 · Cited by 47 · see citing works
Task-Oriented Intrinsic Evaluation of Semantic Textual Similarity
(2016) · Cited by 45 · see citing works
Wang, K., Reimers, N., & Gurevych, I. (2021) · TUbilio (Technical University of Darmstadt) · doi:10.48550/arxiv.2104.06979 · Cited by 43 · see citing works
Self-citationCross-Modal Retrieval in the Cooking Context: Learning Semantic Text-Image EmbeddingsPreprint
Carvalho, M. C., Cadène, R., Picard, D., Soulier, L., Thome, N., & Cord, M. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1804.11146 · Cited by 41 · see citing works
Vilio: State-of-the-art Visio-Linguistic Models applied to Hateful MemesPreprint
Muennighoff, N. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2012.07788 · Cited by 41 · see citing works
Self-citationSimCSE: Simple Contrastive Learning of Sentence EmbeddingsPreprint
Gao, T., Yao, X., & Chen, D. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2104.08821 · Cited by 36 · see citing works
Overview of the Third BUCC Shared Task: Spotting Parallel Sentences in Comparable Corpora
Zweigenbaum, P., Sharoff, S., & Rapp, R. (2018) · HAL (Le Centre pour la Communication Scientifique Directe) · Cited by 35 · see citing works
Crosslingual Generalization through Multitask Finetuning
Muennighoff, N., Wang, T., Sutawika, L., Roberts, A., Biderman, S., Scao, T. L., Bari, M. S., Shen, S., Yong, Z.-X., Schoelkopf, H., et al. (2022) · arXiv · doi:10.48550/arxiv.2211.01786 · Cited by 29 · see citing works
Self-citationFitzGerald, J., Hench, C., Peris, C., Mackie, S., Rottmann, K., Sanchez, A., Nash, A., Urbach, L., Kakarala, V., Singh, R., et al. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2204.08582 · Cited by 22 · see citing works
Papanokechi (2026) · Zenodo (CERN European Organization for Nuclear Research) · doi:10.5281/zenodo.21334049 · Cited by 21 · see citing works
MTOP: A Comprehensive Multilingual Task-Oriented Semantic Parsing BenchmarkPreprint
Li, H., Arora, A., Chen, S., Gupta, A., Gupta, S., & Mehdad, Y. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2008.09335 · Cited by 15 · see citing works
Making a MIRACL: Multilingual Information Retrieval Across a Continuum of LanguagesPreprint
Zhang, X., Thakur, N., Ogundepo, O., et al. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2210.09984 · Cited by 12 · see citing works
Liu, X., Wang, C., Leng, Y., & Zhai, C. (2018) · doi:10.1145/3283812.3283815 · Cited by 11 · see citing works
Bitext Mining Using Distilled Sentence Representations for Low-Resource LanguagesPreprint
Heffernan, K. S., Çelebi, O., & Schwenk, H. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2205.12654 · Cited by 8 · see citing works
M-VADER: A Model for Diffusion with Multimodal ContextPreprint
Weinbach, S., Bellagente, M., Eichenberg, C., et al. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2212.02936 · Cited by 7 · see citing works
TWEAC: Transformer with Extendable QA Agent ClassifiersPreprint
Geigle, G., Reimers, N., Rücklé, A., & Gurevych, I. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2104.07081 · Cited by 6 · see citing works
Self-citationXOR QA: Cross-lingual Open-Retrieval Question AnsweringPreprint
Asai, A., Kasai, J., Clark, J. H., et al. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.11856 · Cited by 5 · see citing works
Large Dual Encoders Are Generalizable RetrieversPreprint
Ni, J., Qu, C., Lü, J., et al. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2112.07899 · Cited by 5 · see citing works
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805
BLOOM: A 176B-Parameter Open-Access Multilingual Language Model
Workshop, B., :, Scao, T. L., Fan, A., Akiki, C., Pavlick, E., Ilić, S., Hesslow, D., Castagné, R., Luccioni, A. S., et al. (2022) · arXiv · doi:10.48550/arxiv.2211.05100
Self-citation
Flagged first, then by citations
Research works printed but not linked: 7
- Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural computation, 9(8):1735- 1780.
- Jui-Ting Huang, Ashish Sharma, Shuying Sun, Li Xia, David Zhang, Philip Pronin, Janani Padmanab- han, Giuseppe Ottaviano, and Linjun Yang. 2020. Embedding-based retrieval in facebook search. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 2553-2561.
- James O'Neill, Polina Rozenshtein, Ryuichi Kiryo, Mo- toko Kubota, and Danushka Bollegala. 2021. I wish i would have loved this one, but i didn't -a multilin- gual dataset for counterfactual detection in product reviews. · DOI
- Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9.
- Nandan Thakur, Nils Reimers, Andreas Rücklé, Ab- hishek Srivastava, and Iryna Gurevych. 2021. Beir: A heterogenous benchmark for zero-shot evaluation of information retrieval models. · DOI · Paper page
- Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in neural information processing systems, 30.
- Pierre Zweigenbaum, Serge Sharoff, and Reinhard Rapp. 2016. Towards preparation of the second bucc shared task: Detecting parallel sentences in comparable cor- pora. In Proceedings of the Ninth Workshop on Building and Using Comparable Corpora. European Language Resources Association (ELRA), Portoroz, Slovenia, pages 38-43.
Cited by
Sorted by citations
Huang, L., Yu, W., Ma, W., Zhong, W., Feng, Z., Wang, H., Chen, Q., Peng, W., Feng, X., Qin, B., et al. (2025) · ACM Transactions on Information Systems · doi:10.1145/3703155 · Cited by 2,142 · see citing works
Exploring the Potential of Large Language Models (LLMs)in Learning on Graphs
Chen, Z., Mao, H., Li, H., Jin, W., Wen, H., Wei, X., Wang, S., Yin, D., Fan, W., Liu, H., et al. (2024) · ACM SIGKDD Explorations Newsletter · doi:10.1145/3655103.3655110 · Cited by 209 · see citing works
Homogenization Effects of Large Language Models on Human Creative Ideation
Anderson, B. R., Shah, J. H., & Kreminski, M. (2024) · Creativity and Cognition · doi:10.1145/3635636.3656204 · Cited by 188 · see citing works
Highway Construction Safety Analysis Using Large Language Models
Smetana, M., Salles de Salles, L., Sukharev, I., & Khazanovich, L. (2024) · Applied Sciences · doi:10.3390/app14041352 · Cited by 87 · see citing works
Choi, M., Pei, J., Kumar, S., Shu, C., & Jurgens, D. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.699 · Cited by 38 · see citing works
OctoPack: Instruction Tuning Code Large Language Models
Muennighoff, N., Liu, Q., Zebaze, A., Zheng, Q., Hui, B., Zhuo, T. Y., Singh, S., Tang, X., von Werra, L., & Longpre, S. (2023) · arXiv · doi:10.48550/arxiv.2308.07124 · Cited by 17 · see citing works
“Large-scale benchmarking has proven useful in many areas of natural language processing (Wang et al., 2019; Kiela et al., 2021; Srivastava et al., 2022; Muennighoff et al., 2022a).”
BackgroundSelf-citationInfLLM: Training-Free Long-Context Extrapolation for LLMs with an Efficient Context Memory
Xiao, C., Zhang, P., Han, X., Xiao, G., Lin, Y., Zhang, Z., Liu, Z., & Sun, M. (2024) · Advances in Neural Information Processing Systems 37 · doi:10.52202/079017-3801 · Cited by 14 · see citing works
LongMemEval: Benchmarking Chat Assistants on Long-Term Interactive Memory
Wu, D., Wang, H., Yu, W., Zhang, Y., Chang, K.-W., & Yu, D. (2024) · arXiv · doi:10.48550/arxiv.2410.10813 · Cited by 3 · see citing works
“For the retriever, we choose dense retrieval with the 1.5B Stella V5 model (Zhang, 2023), given its high performance on MTEB (Muennighoff et al., 2023).”
BrowseComp-Plus: A More Fair and Transparent Evaluation Benchmark of Deep-Research Agent
Chen, Z., Ma, X., Zhuang, S., Nie, P., Zou, K., Liu, A., Green, J., Patel, K., Meng, R., Su, M., et al. (2025) · arXiv · doi:10.48550/arxiv.2508.06600 · Cited by 3 · see citing works
MoMA: a mixture-of-multimodal-agents architecture for enhancing clinical prediction modelling
Gao, J., Rahman, M., Caskey, J., Oguss, M., O’Rourke, A., Brown, R., Stey, A., Mayampurath, A., Churpek, M. M., Chen, G., et al. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-02219-4 · Cited by 2 · see citing works
“We adhere to the Transparent Reporting of a Multivariable Prediction Model for Individual Prognosis or Diagnosis - Large Language Models (TRIPOD-LLM) guidelines 75 and completed the accompanying checklist, meeting all reporting requirements, as detailed in Supplementary Note 4.”
BrowseComp-Plus: A Fair and Disentangled Evaluation Benchmark for Deep Search Agents
Chen, Z., Ma, X., Zhuang, S., Nie, P., Zou, K., Sharifymoghaddam, S., Liu, A., Green, J., Patel, K., Meng, R., et al. (2026) · Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2026.acl-long.1023 · Cited by 2 · see citing works
No Free Labels: Limitations of LLM-as-a-Judge Without Human Grounding
Krumdick, M., Lovering, C., Reddy, V., Ebner, S., & Tanner, C. (2025) · arXiv · doi:10.48550/arxiv.2503.05061 · Cited by 1 · see citing works
REMem: Reasoning with Episodic Memory in Language Agent
Shu, Y., Jonnalagedda, S. P., Gao, X., Gutiérrez, B. J., Qi, W., Das, K., Sun, H., & Su, Y. (2026) · arXiv · doi:10.48550/arxiv.2602.13530 · Cited by 1
SWE-Marathon: Can Agents Autonomously Complete Ultra-Long-Horizon Software Work?
Desai, R., Hu, J., Cabezas, J., Harsola, N., Shukla, P., Chaim, R. B., Assadi, A. E., Kamath, O. M., Faldu, F., Hebbar, P., et al. (2026) · arXiv · doi:10.48550/arxiv.2606.07682
SciRepEval: A Multi-Format Benchmark for Scientific Document Representations
Singh, A., D'Arcy, M., Cohan, A., Downey, D., & Feldman, S. (2022) · arXiv · doi:10.48550/arxiv.2211.13308 · Cited by 76
- Authors
- Muennighoff, N., Tazi, N., Magne, L., & Reimers, N.
- ORCID
- Muennighoff, Niklas
- Venue
- Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics
- Published
- 2023
- Pages
- 2014-2037
- OpenAlex
- W4386576685
- Cited by
- 548
- References
- 73
- Citations by year
- 2026: 156 · 2025: 237 · 2024: 114 · 2023: 41
- FWCI
- 97.02
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Conference of the European Chapter of the Association for Computational Linguistics (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- Centre National de la Recherche Scientifique
- Grand Équipement National De Calcul Intensif
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- text embeddings, benchmark, semantic textual similarity, multilingual embeddings, embedding evaluation, clustering, re-ranking, cross-task generalization, open-source benchmark
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
