CoChat
Sign Up
PreprintCited by 3

The DOI matches its DataCite record; 2 of the 2 references it cites were found in a registry or an index.

GISTEmbed: Guided In-sample Selection of Training Negatives for Text Embedding Fine-tuning

Solatorio, Aivin V.·2024·arXiv
DOI 10.48550/arxiv.2402.16829arXiv 2402.16829
Open this paper in CoChat — free account

Abstract

Embedding models are integral to AI applications like semantic search, personalized recommendations, and retrieval augmented generation for LLMs, necessitating high-quality training data. However, the limited scalability of manual data curation prompts the need for automated methods to ensure data integrity. Traditional unsupervised triplet mining automates training data generation, crucial for embedding model training, yet inadvertently injects biases and noise, thereby degrading model performance. Addressing this, we introduce GISTEmbed, a novel strategy that enhances in-batch negative selection during contrastive training through a guide model. This approach departs from reliance on random sampling and equal utility assumption of batch negatives, significantly reducing noise from data quality issues and improving model fine-tuning. Benchmarked against the Massive Text Embedding Benchmark (MTEB), GISTEmbed showcases consistent performance improvements across various model sizes and achieves state-of-the-art results in select categories. This framework enables significant enhancements for smaller models by leveraging the capabilities of powerful yet resource-intensive large models. GISTEmbed can potentially revolutionize the creation of highly efficient, smaller models, democratizing access to advanced AI technologies. Making these technologies more accessible and cost-effective, especially for applications constrained by resources, significantly expands the impact and accessibility of state-of-the-art AI solutions across diverse sectors.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2402.16829

Cites

2 of 2 references verified0 citation errors · 0 not checkable

Statistics

Figures

Figure 1 Visualization of the GISTEmbed framework for the dynamic selection of in-batch negatives for contrastive learning of embedding models. A guide model is used during training to dynamically exclude texts in the batch that are likely related to the query-positive pair being evaluated. The framework addresses potential data labeling issues and also relaxes assumptions regarding the formation of in-batch negatives that prior approaches use.
Figure 1 Visualization of the GISTEmbed framework for the dynamic selection of in-batch negatives for contrastive learning of embedding models. A guide model is used during training to dynamically exclude texts in the batch that are likely related to the query-positive pair being evaluated. The framework addresses potential data labeling issues and also relaxes assumptions regarding the formation of in-batch negatives that prior approaches use.CC-BY-NC-ND · From the PDF
Figure 2 Visualization of various in-batch negatives selection strategies for contrastive learning (dashed orange boxes). Each panel contains triplets in a training batch, with the columns representing the queries, assigned positives, and assigned negatives. Panel A shows the original strategy for selecting in-batch negatives where all the assigned negatives in the training data are considered. Panel B visualizes the selection of in-batch negatives for the bi-directional InfoNCE loss which includes the queries as well. The full-sample selection of in-batch negatives is shown in Panel C. While Panel D presents how GISTEmbed, with the guide model-informed selection of in-batch negatives, works. In this example, the query-positive pair (q:Capital cities, p:Ottawa) can be considered semantically related to the other texts in the batch [q:"Where is Manila?", n:"What is the capital of Canada?"] . The guide model serves as a filter to remove these texts when selecting the in-batch negatives for computing the loss.
Figure 2 Visualization of various in-batch negatives selection strategies for contrastive learning (dashed orange boxes). Each panel contains triplets in a training batch, with the columns representing the queries, assigned positives, and assigned negatives. Panel A shows the original strategy for selecting in-batch negatives where all the assigned negatives in the training data are considered. Panel B visualizes the selection of in-batch negatives for the bi-directional InfoNCE loss which includes the queries as well. The full-sample selection of in-batch negatives is shown in Panel C. While Panel D presents how GISTEmbed, with the guide model-informed selection of in-batch negatives, works. In this example, the query-positive pair (q:Capital cities, p:Ottawa) can be considered semantically related to the other texts in the batch [q:"Where is Manila?", n:"What is the capital of Canada?"] . The guide model serves as a filter to remove these texts when selecting the in-batch negatives for computing the loss.CC-BY-NC-ND · From the PDF
Figure 3 Training loss curve for the all-MiniLM-L6-v2 model fine-tuning. The different colors simply show the continuation of training from checkpoints. There is a rapid phase of learning below the 50k steps mark. An inflection point at around 70k steps signal the beginning of saturation in the training.
Figure 3 Training loss curve for the all-MiniLM-L6-v2 model fine-tuning. The different colors simply show the continuation of training from checkpoints. There is a rapid phase of learning below the 50k steps mark. An inflection point at around 70k steps signal the beginning of saturation in the training.CC-BY-NC-ND · From the PDF

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 2 references

Research works printed but not linked: 29

  1. Cao, R., Wang, Y., Liang, Y., Gao, L., Zheng, J., Ren, J., and Wang, Z. Exploring the impact of negative samples of contrastive learning: A case study of sentence embedding. arXiv preprint arXiv:2202.13093, 2022. URL https: //arxiv.org/abs/2202.13093. · arXiv
  2. Chan, C.-M., Chen, W., Su, Y., Yu, J., Xue, W., Zhang, S., Fu, J., and Liu, Z. Chateval: Towards better llm- based evaluators through multi-agent debate, 2023. URL https://arxiv.org/abs/2308.07201. · arXiv
  3. Cheng, Q., Yang, X., Sun, T., Li, L., and Qiu, X. Improv- ing contrastive learning of sentence embeddings from ai feedback. arXiv preprint arXiv:2305.01918, 2023. URL https://arxiv.org/abs/2305.01918. · arXiv
  4. Gao, T., Yao, X., and Chen, D. SimCSE: Simple con- trastive learning of sentence embeddings. In Moens, M.-F., Huang, X., Specia, L., and Yih, S. W.-t. (eds.), Proceedings of the 2021 Conference on Empirical Meth- ods in Natural Language Processing, pp. 6894-6910, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: 10.18653/v1/2021.emnlp-main.552. URL https:// aclanthology.org/2021.emnlp-main.552. · DOI · Paper page
  5. Henderson, M., Al-Rfou, R., Strope, B., Sung, Y.-H., Lukács, L., Guo, R., Kumar, S., Miklos, B., and Kurzweil, R. Efficient natural language response suggestion for smart reply. arXiv preprint arXiv:1705.00652, 2017. URL https://arxiv.org/abs/1705.00652. · arXiv
  6. Hill, F., Cho, K., and Korhonen, A. Learning distributed representations of sentences from unlabelled data. In Knight, K., Nenkova, A., and Rambow, O. (eds.), Pro- ceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguis- tics: Human Language Technologies, pp. 1367-1377, San Diego, California, June 2016. Association for Computa- tional Linguistics. doi: 10.18653/v1/N16-1162. URL https://aclanthology.org/N16-1162. · DOI · Paper page
  7. Huang, J., Tang, D., Zhong, W., Lu, S., Shou, L., Gong, M., Jiang, D., and Duan, N. Whiteningbert: An easy unsupervised sentence embedding approach. arXiv preprint arXiv:2104.01767, 2021. URL https:// arxiv.org/abs/2104.01767. · arXiv
  8. Huang, J., Gu, S., Hou, L., Wu, Y., Wang, X., Yu, H., and Han, J. Large language models can self-improve. In Bouamor, H., Pino, J., and Bali, K. (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 1051-1068, Singapore, De- cember 2023. Association for Computational Linguistics. doi: 10.18653/v1/2023.emnlp-main.67. URL https: //aclanthology.org/2023.emnlp-main.67. · DOI · Paper page
  9. Karpukhin, V., Oguz, B., Min, S., Lewis, P., Wu, L., Edunov, S., Chen, D., and Yih, W.-t. Dense passage retrieval for open-domain question answering. arXiv preprint arXiv:2004.04906, 2020. URL https:// arxiv.org/abs/2004.04906. · arXiv
  10. Kiros, R., Zhu, Y., Salakhutdinov, R. R., Zemel, R., Urtasun, R., Torralba, A., and Fidler, S. Skip-thought vectors. In Cortes, C., Lawrence, N., Lee, D., Sugiyama, M., and Garnett, R. (eds.), Advances in Neural Information Processing Systems, volume 28.
  11. Curran Associates, Inc., 2015. URL https://proceedings.neurips. cc/paper_files/paper/2015/file/ f442d33fa06832082290ad8544a8da27-Paper. pdf.
  12. Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., Küttler, H., Lewis, M., Yih, W.-t., Rocktäschel, T., et al. Retrieval-augmented generation for knowledge- intensive nlp tasks. Advances in Neural Information Pro- cessing Systems, 33:9459-9474, 2020.
  13. Li, X. and Li, J. Angle-optimized text embeddings. arXiv preprint arXiv:2309.12871, 2023. URL https:// arxiv.org/abs/2309.12871. · arXiv
  14. Li, X., Li, Z., Li, J., Xie, H., and Li, Q. 2d matryoshka sentence embeddings. arXiv preprint arXiv:2402.14776, 2024. URL https://arxiv.org/abs/2402. 14776. · arXiv
  15. Li, Z., Zhang, X., Zhang, Y., Long, D., Xie, P., and Zhang, M. Towards general text embeddings with multi-stage contrastive learning. arXiv preprint arXiv:2308.03281, 2023. URL https://arxiv.org/abs/2308. 03281. · arXiv
  16. Ma, X., Wang, Z., Ng, P., Nallapati, R., and Xiang, B. Universal text representation from bert: An empirical study. arXiv preprint arXiv:1910.07973, 2019. URL https://arxiv.org/abs/1910.07973. · arXiv
  17. Mikolov, T., Chen, K., Corrado, G., and Dean, J. Effi- cient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781, 2013. URL https: //arxiv.org/abs/1301.3781. · arXiv
  18. Muennighoff, N. Sgpt: Gpt sentence embeddings for se- mantic search. arXiv preprint arXiv:2202.08904, 2022. URL https://arxiv.org/abs/2202.08904. · arXiv
  19. Ni, J., Qu, C., Lu, J., Dai, Z., Ábrego, G. H., Ma, J., Zhao, V. Y., Luan, Y., Hall, K. B., Chang, M.-W., et al. Large dual encoders are generalizable retriev- ers. arXiv preprint arXiv:2112.07899, 2021. URL https://arxiv.org/abs/2112.07899. · arXiv
  20. Okura, S., Tagami, Y., Ono, S., and Tajima, A. Embedding- based news recommendation for millions of users. In Proceedings of the 23rd ACM SIGKDD international conference on knowledge discovery and data mining, pp. 1933-1942, 2017.
  21. Oord, A. v. d., Li, Y., and Vinyals, O. Representa- tion learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748, 2018. URL https:// arxiv.org/abs/1807.03748. · arXiv
  22. Reimers, N. and Gurevych, I. Sentence-bert: Sentence em- beddings using siamese bert-networks. arXiv preprint arXiv:1908.10084, 2019. URL https://arxiv. org/abs/1908.10084. · arXiv
  23. Rosenberg, A. and Hirschberg, J. V-measure: A conditional entropy-based external cluster evaluation measure. In Proceedings of the 2007 joint conference on empirical methods in natural language processing and computa- tional natural language learning (EMNLP-CoNLL), pp. 410-420, 2007.
  24. Su, H., Shi, W., Kasai, J., Wang, Y., Hu, Y., Ostendorf, M., Yih, W.-t., Smith, N. A., Zettlemoyer, L., and Yu, T. One embedder, any task: Instruction-finetuned text em- beddings. 2022. URL https://arxiv.org/abs/ 2212.09741.
  25. Wu, X., Gao, C., Zang, L., Han, J., Wang, Z., and Hu, S. ES- imCSE: Enhanced sample building method for contrastive learning of unsupervised sentence embedding. In Calzo- lari, N., Huang, C.-R., Kim, H., Pustejovsky, J., Wanner, L., Choi, K.-S., Ryu, P.-M., Chen, H.-H., Donatelli, L., Ji, H., Kurohashi, S., Paggio, P., Xue, N., Kim, S., Hahm, Y., He, Z., Lee, T. K., Santus, E., Bond, F., and Na, S.-H. (eds.), Proceedings of the 29th International Conference on Computational Linguistics, pp. 3898-3907, Gyeongju, Republic of Korea, October 2022. International Com- mittee on Computational Linguistics. URL https: //aclanthology.org/2022.coling-1.342.
  26. Xu, L., Xie, H., Li, Z., Wang, F. L., Wang, W., and Li, Q. Contrastive learning models for sentence representations. ACM Transactions on Intelligent Systems and Technology, 14(4):1-34, 2023.
  27. Yan, Y., Li, R., Wang, S., Zhang, F., Wu, W., and Xu, W. ConSERT: A contrastive framework for self-supervised sentence representation transfer. In Zong, C., Xia, F., Li, W., and Navigli, R. (eds.), Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natu- ral Language Processing (Volume 1: Long Papers), pp. 5065-5075, Online, August 2021. Association for Com- putational Linguistics. doi: 10.18653/v1/2021.acl-long. 393. URL https://aclanthology.org/2021. acl-long.393. · DOI
  28. Zhang, P., Xiao, S., Liu, Z., Dou, Z., and Nie, J.-Y. Retrieve anything to augment large language models. arXiv preprint arXiv:2310.07554, 2023. URL https: //arxiv.org/abs/2310.07554. · arXiv
  29. Zhou, K., Zhang, B., Zhao, X., and Wen, J.-R. Debi- ased contrastive learning of unsupervised sentence repre- sentations. In Muresan, S., Nakov, P., and Villavicen- cio, A. (eds.), Proceedings of the 60th Annual Meet- ing of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 6120-6130, Dublin, Ire- land, May 2022. Association for Computational Linguis- tics. doi: 10.18653/v1/2022.acl-long.423. URL https: //aclanthology.org/2022.acl-long.423. · DOI

Cited by

See all 3

Metadata

Authors
Solatorio, A. V.
Venue
arXiv
Publisher
arXiv
Published
2024
DOI
10.48550/arxiv.2402.16829
arXiv
2402.16829
References
2
Type
Preprint
Licence
creativecommons.org/licenses/by-nc-nd/4.0/legalcode

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: GISTEmbed: Guided In-sample Selection of Training Negatives for Text Embedding Fine-tuning. https://app.cochat.ai/papers/10.48550/arxiv.2402.16829
The paper
Solatorio, A. V. (2024). GISTEmbed: Guided In-sample Selection of Training Negatives for Text Embedding Fine-tuning. arXiv. https://doi.org/10.48550/arxiv.2402.16829