The DOI matches its DataCite record; 7 of the 7 references it cites were found in a registry or an index; none of the 16 sources its claims cite is checked for support yet.
Benchmarking Google Embeddings 2 against Open-Source Models for Multilingual Dense Retrieval and RAG Systems
Open accessAbstract
We benchmark Google Embeddings (GE2), a Vertex-AI-hosted bi-encoder with 2,048-token context and explicit task-type conditioning, against five open-source alternatives: BGE-M3, E5-large, Multilingual-E5-large (mE5-L), LaBSE, and Paraphrase-Multilingual-MPNet (mMPNet). Evaluation covers four BEIR subsets, a synthetic Italian RAG corpus, a chunking ablation considering 5 sizes of tokens with three strategies, and per-query latency on commodity CPU hardware. GE2 ranks first on every task, achieving BEIR avg.nDCG@10 = 0.638 and IT-RAG-Bench nDCG@10 = 0.282, but at 231.6 ms median latency, it is roughly 14x slower than the fastest local models. mE5-L reaches within 0.003 nDCG of GE2 on Italian at 31 ms, making it the preferred option when sub-100 ms SLAs matter. A more striking finding concerns LaBSE, which, despite widespread multilingual deployment scores 0.188 average nDCG@10 on BEIR, below every dedicated retrieval model including mMPNet. Chunking experiments show that all six models saturate at 32-token chunks on our corpus, with semantic chunking providing measurable gains only at 16 tokens.
Overview
CoChat Summary
TL;DR
This paper benchmarks Google Embeddings 2 (GE2) against five open-source embedding models for multilingual dense retrieval and RAG, using BEIR subsets, a synthetic Italian corpus, chunking ablations, and CPU latency measurements. GE2 ranks first on every task but is much slower, while mE5-L is nearly equivalent on short Italian passages at far lower latency.
Key findings
- GE2 achieves the best retrieval quality: BEIR average nDCG@10 = 0.638 and IT-RAG-Bench nDCG@10 = 0.282, ahead of mE5-L at 0.546 on BEIR and 0.279 on Italian.
- GE2’s median query latency is 231.6 ms (σ = 102.6 ms, p95 = 575.5 ms), roughly 14× slower than the fastest local models; LaBSE and mMPNet run at 16.6 ms, while BGE-M3, E5-large, and mE5-L run near 31 ms.
- mE5-L is within 0.003 nDCG of GE2 on Italian at 31 ms, making it the preferred option when sub-100 ms SLAs matter.
- LaBSE scores 0.188 average nDCG@10 on BEIR, below every dedicated retrieval model including mMPNet at 0.243; chunking experiments show all six models saturate at 32-token chunks, with semantic chunking gains only at 16 tokens.
Methods
- Offline retrieval pipeline: documents are chunked, embedded, indexed with FAISS using exhaustive flat search for corpora with N ≤ 5,000, queries retrieve top-k chunks, and chunks are mapped back to parent documents for evaluation.
- Data: four BEIR subsets (FiQA-2018, NFCorpus, SciFact, TREC-COVID) and IT-RAG-Bench, a synthetic Italian corpus of 3,200 passages (1,200 Wikipedia, 800 public-administration FAQ, 1,200 civil-code articles) with 640 templated queries, seed 42, and 200 manually verified queries yielding Cohen’s κ = 0.81.
- Analysis: primary metric nDCG@10, plus MRR and Recall@k; chunking ablation uses fixed, sliding-window, and semantic strategies at sizes {8, 16, 32, 64, 128} whitespace tokens; latency is median over 50 single-query runs after 5 warmups on Apple M-series CPU.
Limitations
- The paper states IT-RAG-Bench is synthetic and dominated by short passages, so the 32-token saturation point may not generalize to longer legal or scientific documents, where the inflection may shift to 128–256 tokens.
- The paper states it covers only four of 18 BEIR subsets, omits MSMARCO and HotpotQA, and BGE-M3’s missing TREC-COVID score introduces minor incomparability; latency is CPU-only on Apple Silicon, single-query and non-batched, and each model-dataset combination is evaluated once without bootstrap resampling, so nDCG differences below 0.01 should be interpreted cautiously.
- The text also makes evident that GE2 latency variance reflects network round-trip and Vertex AI autoscaling rather than model depth, semantic chunking is model-specific because boundary detection uses the evaluated model’s embeddings, and chunk-to-parent relevance mapping introduces a mild optimistic bias for fixed chunking at large sizes.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures
Verification
Claims the paper makes
0 Valid · 1 Not valid · 16 Not checked · 108 Own results and interpretations
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“LaBSE at 0.188 average nDCG is the clearest cautionary result.”
§ Benchmarking Google Embeddings 2 against Open-Source Models for Multilingual Dense Retrieval and RAG Systems > A. English Retrieval (BEIR)
boosted- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The difference in nDCG@10 is 0.003 (0.282 vs. 0.279)”
§ Benchmarking Google Embeddings 2 against Open-Source Models for Multilingual Dense Retrieval and RAG Systems > B. Italian RAG Benchmark (IT-RAG-Bench)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Chunking experiments show that all six models saturate at 32-token chunks on our corpus, with semantic chunking providing measurable gains only at 16 tokens.”
§ Benchmarking Google Embeddings 2 against Open-Source Models for Multilingual Dense Retrieval and RAG Systems > Abstract
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“FiQA shows a similarly large gap (0.582 vs. 0.438)”
§ Benchmarking Google Embeddings 2 against Open-Source Models for Multilingual Dense Retrieval and RAG Systems > A. English Retrieval (BEIR)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“GE2 ranks first on every task, achieving BEIR avg.nDCG@10 = 0.638 and IT-RAG-Bench nDCG@10 = 0.282, but at 231.6 ms median latency it is roughly 14× slower than the fastest local models.”
§ Benchmarking Google Embeddings 2 against Open-Source Models for Multilingual Dense Retrieval and RAG Systems > Abstract
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
65 more claims
Create a free account to see all 125 claims and every source verdict.
- It may state two separate claims as one. 42
- Probably not a substantial claim. 25
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated major: it would touch a result — a value a result rests on, reported and recomputed p-values on opposite sides of the significance level, two passages that conflict with no sign which is right, or a cited work retracted before this paper was published.
In the paper
§ Benchmarking Google Embeddings 2 against Open-Source Models for Multilingual Dense Retrieval and RAG Systems > B. Italian RAG Benchmark (IT-RAG-Bench)
“GE2's modestly higher Recall@10, by 1.2 percentage points, constitutes a more reliable indicator of performance.”
§ Benchmarking Google Embeddings 2 against Open-Source Models for Multilingual Dense Retrieval and RAG Systems > B. Italian RAG Benchmark (IT-RAG-Bench)
“GE2 0.061 0.288 0.476 0.259 0.282 mE5-L 0.051 0.280 0.489 0.243 0.279”
Rated major: it would touch a result — a value a result rests on, reported and recomputed p-values on opposite sides of the significance level, two passages that conflict with no sign which is right, or a cited work retracted before this paper was published.
In the paper
§ Benchmarking Google Embeddings 2 against Open-Source Models for Multilingual Dense Retrieval and RAG Systems > A. Models
“GE2's two most architecturally significant properties are its 2,048-token context window, which is four times longer than all local models, and its asymmetric task-type conditioning.”
§ Benchmarking Google Embeddings 2 against Open-Source Models for Multilingual Dense Retrieval and RAG Systems > Additional information
“BGE-M3 BAAI/bge-m3 1024 8192 ✓ Open”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Benchmarking Google Embeddings 2 against Open-Source Models for Multilingual Dense Retrieval and RAG Systems > C. Latency
“All models attain at least 95% of their peak nDCG@10 performance by a chunk size of 32 tokens and exhibit no further gains at 64 or 128 tokens; the scores are numerically indistinguishable for chunk sizes above 32.”
§ Benchmarking Google Embeddings 2 against Open-Source Models for Multilingual Dense Retrieval and RAG Systems > When does GE2's quality premium materialize?
“E5-large 0.257 fixed-64 0.262 12.5 mMPNet 0.236 fixed-64 0.243 9.4 BGE-M3 0.231 fixed-64 0.238 12.5 LaBSE 0.180 fixed-64 0.189 9.4”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Benchmarking Google Embeddings 2 against Open-Source Models for Multilingual Dense Retrieval and RAG Systems > Abstract
“below every dedicated retrieval model including mMPNet.”
§ Benchmarking Google Embeddings 2 against Open-Source Models for Multilingual Dense Retrieval and RAG Systems > A. Models
“LaBSE and mMPNet are included specifically as representatives of the sentence-similarity model family, to quantify the cost of misapplying them to retrieval tasks.”
Citations
Cites
Billion-scale similarity search with GPUs
Johnson, J., Douze, M., & Jegou, H. (2021) · IEEE Transactions on Big Data · doi:10.1109/tbdata.2019.2921572 · Cited by 4,228
Dense passage retrieval for open-domain question answering
Karpukhin, V., Oguz, B., Min, S., Lewis, P., Wu, L., Edunov, S., Chen, D., & Yih, W.-T. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.550 · Cited by 3,575
MTEB: Massive Text Embedding Benchmark
Muennighoff, N., Tazi, N., Magne, L., & Reimers, N. (2023) · Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2023.eacl-main.148 · Cited by 548
C-Pack: Packed Resources For General Chinese Embeddings
Xiao, S., Liu, Z., Zhang, P., Muennighoff, N., Lian, D., & Nie, J.-Y. (2024) · Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval · doi:10.1145/3626772.3657878 · Cited by 523
C-Pack: Packed Resources For General Chinese Embeddings
Xiao, S., Liu, Z., Zhang, P., Muennighoff, N., Lian, D., & Nie, J.-Y. (2023) · arXiv · doi:10.48550/arxiv.2309.07597 · Cited by 514
Text Embeddings by Weakly-Supervised Contrastive Pre-training
Wang, L., Yang, N., Huang, X., Jiao, B., Yang, L., Jiang, D., Majumder, R., & Wei, F. (2022) · arXiv · doi:10.48550/arxiv.2212.03533 · Cited by 135
MIRACL : A Multilingual Retrieval Dataset Covering 18 Diverse Languages
Zhang, X., Thakur, N., Ogundepo, O., Kamalloo, E., Alfonso-Hermelo, D., Li, X., Liu, Q., Rezagholizadeh, M., & Lin, J. (2023) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00595 · Cited by 91
Research works printed but not linked: 6
- P. Lewis, E. Perez, A. Piktus, F. Petroni, V. Karpukhin, N. Goyal, H. Küttler, M. Lewis, W.-t. Yih, T. Rocktäschel, S. Riedel, and D. Kiela, "Retrieval-augmented generation for knowledge-intensive NLP tasks," in Advances in Neural Information Processing Systems, vol.
- F. Shi, X. Chen, K. Misra, N. Scales, D. Dohan, E. H. Chi, N. Schärli, and D. Zhou, "Large language models can be easily distracted by irrelevant context," in Proceedings of the 40th International Conference on Machine Learning. PMLR, 2023, pp. 31 210-31 227.
- F. Feng, Y. Yang, D. Cer, N. Arivazhagan, and W. Wang, "Language- agnostic bert sentence embedding," in Proceedings of the 60th annual meeting of the association for computational linguistics (volume 1: Long papers), 2022, pp. 878-891.
- N. Thakur, N. Reimers, A. Rücklé, A. Srivastava, and I. Gurevych, "BEIR: A heterogeneous benchmark for zero-shot evaluation of information retrieval models," in Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2021.
- M. A. Hearst, "TextTiling: Segmenting text into multi-paragraph subtopic passages," Computational Linguistics, vol. 23, no. 1, pp. 33-64, 1997.
- D. Rau and A. Søgaard, "Context embeddings for efficient answer generation in RAG," in Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, 2024.
Cited by
Still being checked.
Metadata
- Authors
- Cirillo, S., Desiato, D., Polese, G., & Solimando, G.
- Affiliations
- University of Salerno (IT)
- University of Bari Aldo Moro (IT)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2026
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2605.23618
- OpenAlex
- W7162333213
- Semantic Scholar
- 88b77a2d0624eef2cba5218803cd1e32f748f102
- Cited by
- 0
- References
- 2
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- arXiv (Cornell University) (submittedVersion, cc-by-sa)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Image and Video Retrieval Techniques
- Keywords
- retrieval-augmented generation, BEIR benchmark, chunking strategies, semantic chunking, LaBSE, BGE-M3
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-11-13, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-11-13 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Benchmarking Google Embeddings 2 against Open-Source Models for Multilingual Dense Retrieval and RAG Systems. https://app.cochat.ai/papers/10.48550/arxiv.2605.23618
- The paper
- Cirillo, S., Desiato, D., Polese, G., & Solimando, G. (2026). Benchmarking Google Embeddings 2 against Open-Source Models for Multilingual Dense Retrieval and RAG Systems. arXiv. https://doi.org/10.48550/arxiv.2605.23618
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.
