CoChat
Sign Up
PreprintCited by 155Topic ModelingMultimodal Machine Learning ApplicationsDomain Adaptation and Few-Shot Learning

Improving Text Embeddings with Large Language Models

Open access
Wang, Liang, Yang, Nan, Huang, Xiaolong, Yang, Linjun, Majumder, Rangan, Wei, Furu·2024·arXiv
DOI 10.48550/arxiv.2401.00368arXiv 2401.00368
Open this paper in CoChat — free accountOpen PDF

Abstract

In this paper, we introduce a novel and simple method for obtaining high-quality text embeddings using only synthetic data and less than 1k training steps. Unlike existing methods that often depend on multi-stage intermediate pre-training with billions of weakly-supervised text pairs, followed by fine-tuning with a few labeled datasets, our method does not require building complex training pipelines or relying on manually collected datasets that are often constrained by task diversity and language coverage. We leverage proprietary LLMs to generate diverse synthetic data for hundreds of thousands of text embedding tasks across 93 languages. We then fine-tune open-source decoder-only LLMs on the synthetic data using standard contrastive loss. Experiments demonstrate that our method achieves strong performance on highly competitive text embedding benchmarks without using any labeled data. Furthermore, when fine-tuned with a mixture of synthetic and labeled data, our model sets new state-of-the-art results on the BEIR and MTEB benchmarks.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2401.00368

Cites

0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • We compute judgments using self-consistency, taking the majority vote over five separate inferences generated with a temperature of 0.7, to reduce overall variance.

    “To reduce the overall variance in this analysis, we compute our judgments using self-consistency (Wang et al., 2023), taking the majority vote over five separate inferences generated with a temperature of 0.7.”

    From No Free Labels: Limitations of LLM-as-a-Judge Without Human Grounding (2025) · cites this paper as Wang et al., 2023

    The paper’s §4.2 describes fine-tuning and evaluation, while §3.2 gives a training temperature of 0.02; neither addresses self-consistency for judgments. Searches for self-consistency, majority voting, five inferences, and a temperature of 0.7 found no supporting passage in our copy. (LLM, conf=0.76)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

The publisher hasn’t deposited this paper’s reference list.

Research works printed but not linked: 40

  1. Sanjeev Arora, Yingyu Liang, and Tengyu Ma. 2017. A simple but tough-to-beat baseline for sentence embed- dings. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. Open- Review.net.
  2. Mikel Artetxe and Holger Schwenk. 2019. Mas- sively multilingual sentence embeddings for zero- shot cross-lingual transfer and beyond. Transactions of the Association for Computational Linguistics, 7:597-610.
  3. Luiz Henrique Bonifacio, Hugo Abonizio, Marzieh Fadaee, and Rodrigo Nogueira. 2022. Inpars: Unsu- pervised dataset generation for information retrieval. Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Infor- mation Retrieval.
  4. Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. In Ad- vances in Neural Information Processing Systems 33: Annual Conference on Neural Information Process- ing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual.
  5. Daniel Fernando Campos, Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, Li Deng, and Bhaskar Mitra. 2016. Ms marco: A human generated machine reading compre- hension dataset. ArXiv preprint, abs/1611.09268. · arXiv
  6. Zhuyun Dai, Vincent Y Zhao, Ji Ma, Yi Luan, Jianmo Ni, Jing Lu, Anton Bakalov, Kelvin Guu, Keith Hall, and Ming-Wei Chang. 2022. Promptagator: Few- shot dense retrieval from 8 examples. In The Eleventh International Conference on Learning Representa- tions.
  7. Scott Deerwester, Susan T Dumais, George W Furnas, Thomas K Landauer, and Richard Harshman. 1990. Indexing by latent semantic analysis. Journal of the American society for information science, 41(6):391- 407.
  8. Fangxiaoyu Feng, Yinfei Yang, Daniel Cer, Naveen Ari- vazhagan, and Wei Wang. 2022. Language-agnostic bert sentence embedding. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 878-891.
  9. Tianyu Gao, Howard Yen, Jiatong Yu, and Danqi Chen. 2023. Enabling large language models to generate text with citations. ArXiv preprint, abs/2305.14627. · arXiv
  10. Suriya Gunasekar, Yi Zhang, Jyoti Aneja, Caio Ce- sar Teodoro Mendes, Allison Del Giorno, Sivakanth Gopi, Mojan Javaheripi, Piero C. Kauffmann, Gus- tavo de Rosa, Olli Saarikivi, Adil Salim, S. Shah, Harkirat Singh Behl, Xin Wang, Sébastien Bubeck, Ronen Eldan, Adam Tauman Kalai, Yin Tat Lee, and Yuan-Fang Li. 2023. Textbooks are all you need. ArXiv preprint, abs/2306.11644. · arXiv
  11. Or Honovich, Thomas Scialom, Omer Levy, and Timo Schick. 2022. Unnatural instructions: Tuning lan- guage models with (almost) no human labor. ArXiv preprint, abs/2212.09689. · arXiv
  12. Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. Lora: Low-rank adaptation of large language models. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net.
  13. Gautier Izacard, Mathilde Caron, Lucas Hosseini, Se- bastian Riedel, Piotr Bojanowski, Armand Joulin, and Edouard Grave. 2021. Towards unsupervised dense information retrieval with contrastive learning. ArXiv preprint, abs/2112.09118. · arXiv
  14. Albert Q Jiang, Alexandre Sablayrolles, Arthur Men- sch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guil- laume Lample, Lucile Saulnier, et al. 2023. Mistral 7b. ArXiv preprint, abs/2310.06825. · arXiv
  15. Aditya Kusupati, Gantavya Bhatt, Aniket Rege, Matthew Wallingford, Aditya Sinha, Vivek Ra- manujan, William Howard-Snyder, Kaifeng Chen, Sham M. Kakade, Prateek Jain, and Ali Farhadi. 2022. Matryoshka representation learning. In Neural Information Processing Systems.
  16. Patrick S. H. Lewis, Ethan Perez, Aleksandra Pik- tus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. 2020. Retrieval-augmented generation for knowledge-intensive NLP tasks. In Advances in Neu- ral Information Processing Systems 33: Annual Con- ference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual.
  17. Xianming Li and Jing Li. 2023. Angle-optimized text embeddings. ArXiv preprint, abs/2309.12871. · arXiv
  18. Zehan Li, Xin Zhang, Yanzhao Zhang, Dingkun Long, Pengjun Xie, and Meishan Zhang. 2023. Towards general text embeddings with multi-stage contrastive learning. ArXiv preprint, abs/2308.03281. · arXiv
  19. Xueguang Ma, Liang Wang, Nan Yang, Furu Wei, and Jimmy Lin. 2023. Fine-tuning llama for multi-stage text retrieval. ArXiv preprint, abs/2310.08319. · arXiv
  20. Tomas Mikolov, Kai Chen, Gregory S. Corrado, and Jeffrey Dean. 2013. Efficient estimation of word representations in vector space. In ICLR.
  21. Amirkeivan Mohtashami and Martin Jaggi. 2023. Landmark attention: Random-access infinite con- text length for transformers. ArXiv preprint, abs/2305.16300. · arXiv
  22. Niklas Muennighoff. 2022. Sgpt: Gpt sentence em- beddings for semantic search. ArXiv preprint, abs/2202.08904. · arXiv
  23. Niklas Muennighoff, Nouamane Tazi, Loic Magne, and Nils Reimers. 2023. MTEB: Massive text embedding benchmark. In Proceedings of the 17th Conference of the European Chapter of the Association for Com- putational Linguistics, pages 2014-2037, Dubrovnik, Croatia. Association for Computational Linguistics.
  24. Subhabrata Mukherjee, Arindam Mitra, Ganesh Jawa- har, Sahaj Agarwal, Hamid Palangi, and Ahmed Has- san Awadallah. 2023. Orca: Progressive learning from complex explanation traces of gpt-4. ArXiv preprint, abs/2306.02707. · arXiv
  25. Arvind Neelakantan, Tao Xu, Raul Puri, Alec Rad- ford, Jesse Michael Han, Jerry Tworek, Qiming Yuan, Nikolas A. Tezak, Jong Wook Kim, Chris Hallacy, Johannes Heidecke, Pranav Shyam, Boris Power, Tyna Eloundou Nekoul, Girish Sastry, Gretchen Krueger, David P. Schnurr, Felipe Petroski Such, Kenny Sai-Kin Hsu, Madeleine Thompson, Tabarak Khan, Toki Sherbakov, Joanne Jang, Peter Welin- der, and Lilian Weng. 2022. Text and code embed- dings by contrastive pre-training. ArXiv preprint, abs/2201.10005. · arXiv
  26. Jianmo Ni, Chen Qu, Jing Lu, Zhuyun Dai, Gustavo Her- nandez Abrego, Ji Ma, Vincent Zhao, Yi Luan, Keith Hall, Ming-Wei Chang, and Yinfei Yang. 2022b. Large dual encoders are generalizable retrievers. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 9844-9855, Abu Dhabi, United Arab Emirates. As- sociation for Computational Linguistics.
  27. Rodrigo Nogueira, Wei Yang, Jimmy Lin, and Kyunghyun Cho. 2019. Document expansion by query prediction. ArXiv preprint, abs/1904.08375. · arXiv
  28. OpenAI. 2023. Gpt-4 technical report. ArXiv preprint, abs/2303.08774. · arXiv
  29. Yifu Qiu, Hongyu Li, Yingqi Qu, Ying Chen, QiaoQiao She, Jing Liu, Hua Wu, and Haifeng Wang. 2022. DuReader-retrieval: A large-scale Chinese bench- mark for passage retrieval from web search engine. In Proceedings of the 2022 Conference on Empiri- cal Methods in Natural Language Processing, pages 5326-5338, Abu Dhabi, United Arab Emirates. As- sociation for Computational Linguistics.
  30. Baptiste Rozière, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Tan, Yossi Adi, Jingyu Liu, Tal Remez, Jérémy Rapin, Artyom Kozhevnikov, I. Evtimov, Joanna Bitton, Manish P Bhatt, Cris- tian Cantón Ferrer, Aaron Grattafiori, Wenhan Xiong, Alexandre D'efossez, Jade Copet, Faisal Azhar, Hugo Touvron, Louis Martin, Nicolas Usunier, Thomas Scialom, and Gabriel Synnaeve. 2023. Code llama: Open foundation models for code. ArXiv preprint, abs/2308.12950. · arXiv
  31. Timo Schick and Hinrich Schütze. 2021. Generating datasets with pretrained language models. In Pro- ceedings of the 2021 Conference on Empirical Meth- ods in Natural Language Processing, pages 6943- 6951.
  32. Nandan Thakur, Nils Reimers, Andreas Rücklé, Ab- hishek Srivastava, and Iryna Gurevych. 2021. Beir: A heterogeneous benchmark for zero-shot evaluation of information retrieval models. In Thirty-fifth Con- ference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2).
  33. Hugo Touvron, Louis Martin, Kevin Stone, Peter Al- bert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023. Llama 2: Open founda- tion and fine-tuned chat models. ArXiv preprint, abs/2307.09288. · arXiv
  34. Liang Wang, Nan Yang, Xiaolong Huang, Binxing Jiao, Linjun Yang, Daxin Jiang, Rangan Majumder, and Furu Wei. 2022b. Text embeddings by weakly- supervised contrastive pre-training. ArXiv preprint, abs/2212.03533. · arXiv
  35. Liang Wang, Nan Yang, Xiaolong Huang, Linjun Yang, Rangan Majumder, and Furu Wei. 2024. Multilin- gual e5 text embeddings: A technical report. arXiv preprint arXiv:2402.05672. · arXiv
  36. Shitao Xiao, Zheng Liu, Peitian Zhang, and Niklas Muennighof. 2023. C-pack: Packaged resources to advance general chinese embedding. ArXiv preprint, abs/2309.07597. · arXiv
  37. Xiaohui Xie, Qian Dong, Bingning Wang, Feiyang Lv, Ting Yao, Weinan Gan, Zhijing Wu, Xiangsheng Li, Haitao Li, Yiqun Liu, et al. 2023. T2ranking: A large-scale chinese benchmark for passage ranking. ArXiv preprint, abs/2304.03679. · arXiv
  38. Xin Zhang, Zehan Li, Yanzhao Zhang, Dingkun Long, Pengjun Xie, Meishan Zhang, and Min Zhang. 2023a. Language models are universal embedders. ArXiv preprint, abs/2310.08232. · arXiv
  39. Xinyu Crystina Zhang, Nandan Thakur, Odunayo Ogun- depo, Ehsan Kamalloo, David Alfonso-Hermelo, Xi- aoguang Li, Qun Liu, Mehdi Rezagholizadeh, and Jimmy Lin. 2023b. Miracl: A multilingual retrieval dataset covering 18 diverse languages. Transactions of the Association for Computational Linguistics, 11:1114-1131.
  40. Pierre Zweigenbaum, Serge Sharoff, and Reinhard Rapp. 2018. Overview of the third bucc shared task: Spot- ting parallel sentences in comparable corpora. In Proceedings of 11th Workshop on Building and Us- ing Comparable Corpora, pages 39-42. nDCG@10 Recall@100 BM25 mDPR mE5 base mE5 large E5 mistral-7b full BM25 mDPR mE5 base mE5 large E5 mistral-7b full ar 48.1 49.9 71.6 76.0 73.3 88.9 84.1 95.9 97.3 96.0 bn 50.8 44.3 70.2 75.9 70.3 90.9 81.9 96.6 98.2 96.0 en 35.1 39.4 51.2 52.9 57.3 81.9 76.8 86.4 87.6 90.2 es 31.9 47.8 51.5 52.9 52.2 70.2 86.4 88.6 89.1 87.5 fa 33.3 48.0 57.4 59.0 52.1 73.1 89.8 91.2 92.9 88.0 fi 55.1 47.2 74.4 77.8 74.7 89.1 78.8 96.9 98.1 96.7 fr 18.3 43.5 49.7 54.5 55.2 65.3 91.5 90.0 90.6 92.8 hi 45.8 38.3 58.4 62.0 52.1 86.8 77.6 92.6 93.9 89.9 id 44.9 27.2 51.1 52.9 52.7 90.4 57.3 87.4 87.9 88.4 ja 36.9 43.9 64.7 70.6 66.8 80.5 82.5 96.0 97.1 95.1 ko 41.9 41.9 62.2 66.5 61.8 78.3 73.7 91.6 93.4 89.4 ru 33.4 40.7 61.5 67.4 67.7 66.1 79.7 92.7 95.5 95.0 sw 38.3 29.9 71.1 74.9 68.4 70.1 61.6 95.6 96.7 95.5 te 49.4 35.6 75.2 84.6 73.9 83.1 76.2 98.0 99.2 95.1 th 48.4 35.8 75.2 80.2 74.0 88.7 67.8 98.0 98.9 96.5 zh 18.0 51.2 51.5 56.0 54.0 56.0 94.4 92.1 93.3 90.1 Avg 39.3 41.5 62.3 66.5 62.9 78.7 78.8 93.1 94.3 92.6 Dataset w/ synthetic only w/ synthetic + msmarco w/o synthetic data full data BIOSSES 84.2 81.0 85.4 85.5 SICK-R 78.6 78.5 81.7 82.6 STS12 75.8 74.7 77.9 79.7 STS13 84.3 85.3 88.0 88.4 STS14 80.9 81.2 83.7 84.5 STS15 86.2 86.8 89.5 90.4 STS16 85.0 85.3 86.5 87.7 STS17 87.3 87.7 91.0 91.8 STS22 66.0 67.1 66.2 67.0 STSBenchmark 83.5 84.0 87.8 88.6 SummEval 31.9 32.7 31.9 31.4 SprintDuplicateQuestions 93.5 95.8 96.0 95.7 TwitterSemEval2015 78.0 78.5 81.7 81.6 TwitterURLCorpus 86.5 86.9 87.7 87.8 AmazonCounterfactualClass. 79.6 79.9 77.2 78.7 AmazonPolarityClassification 95.8 95.9 93.9 95.9 AmazonReviewsClassification 56.9 55.5 48.2 55.8 Banking77Classification 86.2 87.0 88.8 88.2 EmotionClassification 49.2 47.6 51.0 49.8 ImdbClassification 94.8 94.9 89.0 94.8 MassiveIntentClassification 79.8 79.9 79.6 80.6 MassiveScenarioClassification 81.7 82.4 82.3 82.4 MTOPDomainClassification 95.6 95.9 95.7 96.1 MTOPIntentClassification 84.9 85.9 83.4 86.1 ToxicConversationsClassification 70.2 70.8 70.9 69.6 TweetSentimentExtractionClass. 63.5 63.4 61.6 63.7 AskUbuntuDupQuestions 64.3 65.3 67.4 67.0 MindSmallReranking 33.1 32.8 32.5 32.6 SciDocsRR 86.0 86.0 85.7 86.3 StackOverflowDupQuestions 52.5 53.7 55.9 54.9 ArxivClusteringP2P 51.4 51.2 47.8 50.5 ArxivClusteringS2S 46.5 44.9 44.6 45.5 BiorxivClusteringP2P 44.5 43.3 36.9 43.5 BiorxivClusteringS2S 40.9 40.1 37.0 40.2 MedrxivClusteringP2P 40.5 39.9 32.6 38.2 MedrxivClusteringS2S 38.0 37.9 32.8 37.5 RedditClustering 56.3 55.9 63.1 57.7 RedditClusteringP2P 66.3 64.8 66.4 66.5 StackExchangeClustering 72.9 72.7 74.5 73.1 StackExchangeClusteringP2P 46.1 45.6 34.3 45.9 TwentyNewsgroupsClustering 52.2 52.5 55.6 54.3 ArguAna 52.2 42.7 62.5 61.9 ClimateFEVER 21.1 28.8 25.2 38.4 CQADupstackAndroidRetrieval 40.8 36.0 44.5 43.0 DBPedia 42.0 43.7 47.7 48.9 FEVER 72.5 83.5 73.1 87.8 FiQA2018 38.1 48.4 54.5 56.6 HotpotQA 48.1 64.0 75.6 75.7 MSMARCO 25.7 45.0 42.9 43.1 NFCorpus 35.5 40.0 35.3 38.6 NQ 53.3 63.5 57.3 63.5 QuoraRetrieval 75.0 79.5 89.5 89.6 SCIDOCS 20.6 15.8 19.0 16.3 SciFact 71.5 71.9 74.7 76.4 Touche2020 25.4 32.5 19.1 26.4 TRECCOVID 82.3 87.3 70.8 87.2 Average 63.1 64.5 64.6 66.6 Table 17: Results for each dataset in the MTEB benchmark. The evaluation metrics and detailed baseline results are available in the original paper (Muennighoff et al., 2023).

Cited by

Sorted by citations

Related

Authors
Wang, L., Yang, N., Huang, X., Yang, L., Majumder, R., & Wei, F.
Affiliations
Microsoft (United States) (US)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2024
Pages
11897-11916
ISSN
0736-587X (ISSN-L)
DOI
10.48550/arxiv.2401.00368
arXiv
2401.00368
OpenAlex
W4402672004
Cited by
149
References
0
Citations by year
2026: 63 · 2025: 63 · 2024: 23
FWCI
43.97
Type
conference paper
Language
en
Open access
Licence
Locations
  • Proceedings of the conference - Association for Computational Linguistics. Meeting (publishedVersion, cc-by)
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • arXiv (Cornell University) (cc-by)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
  • Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
Keywords
text embeddings, synthetic data generation, large language models, contrastive learning, fine-tuning, multilingual embeddings, BEIR

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review