Comparative benchmark of embedding models for multilingual retrieval (FR/EN/JP) on Wikipedia articles about Japanese culture.
We needed to decide which embedding model to use for rag-ferrite, a self-hosted RAG engine. The corpus is mixed French/English with transcripts (oral style), notes (structured), and technical docs. We wanted to know:
- Does our current model (Qwen3-Embedding-8B) give the best retrieval quality?
- How does Mistral Embed compare, despite being 10x more expensive?
- Does Qwen3's Matryoshka (MRL) truncation to 512d actually lose quality vs full 4096d?
- How fast is each model for bulk ingestion?
| Model | Model ID | Parameters | Dimensions | Cost (per 1M tokens) |
|---|---|---|---|---|
| Qwen3-Embedding-8B @ 512d | qwen/qwen3-embedding-8b |
8B | 512 (MRL-truncated) | ~$0.01 |
| Qwen3-Embedding-8B @ 1024d | qwen/qwen3-embedding-8b |
8B | 1024 (MRL-truncated) | ~$0.01 |
| Qwen3-Embedding-8B @ 4096d | qwen/qwen3-embedding-8b |
8B | 4096 (full) | ~$0.01 |
| BGE-M3 | baai/bge-m3 |
~568M | 1024 (native) | ~$0.01 |
| Mistral Embed (2312) | mistralai/mistral-embed-2312 |
~335M | 1024 (native) | ~€0.10 |
All models accessed via the OpenRouter API (https://openrouter.ai/api/v1/embeddings) using a standard POST request with Authorization: Bearer <key>. No local GPU, no Ollama — everything through the cloud API. The same API key was used for all models to keep conditions identical.
Note on dimensions parameter: Qwen3-Embedding-8B supports the dimensions parameter via Matryoshka Representation Learning (MRL), allowing truncation to 512 or 1024. BGE-M3 and Mistral Embed do not support this parameter — they output at their native dimensionality. When dimensions is not supported, the parameter is omitted from the API call.
30 Wikipedia articles — 10 topics × 3 languages (French, English, Japanese):
| Topic | FR | EN | JP |
|---|---|---|---|
| Tokyo | ✓ | ✓ | ✓ |
| Sushi | ✓ | ✓ | ✓ |
| Tea Ceremony | ✓ | ✓ | ✓ |
| Buddhism | ✓ | ✓ | ✓ |
| Mount Fuji | ✓ | ✓ | ✓ |
| Tsunami | ✓ | ✓ | ✓ |
| Anime | ✓ | ✓ | ✓ |
| Manga | ✓ | ✓ | ✓ |
| Samurai | ✓ | ✓ | ✓ |
| Calligraphy (Shodō) | ✓ | ✓ | ✓ |
Articles fetched via Wikipedia REST API, truncated to ~3000 chars each. Total: 59 chunks (500-char chunking).
30 queries in mixed languages — French, English, and Japanese. Each query has an expected source document (the Wikipedia article that should rank highest).
Example queries:
Quelle est la capitale du Japon ?→ expectstokyo_frWhat is raw fish on rice called?→ expectssushi_en日本の最高峰の山は？→ expectsmount_fuji_ja漫画とは何ですか→ expectsmanga_ja
Full dataset: golden_dataset.json
- Chunk all 30 documents at 500 chars (simple paragraph-based chunking)
- Embed all 59 chunks with each model
- Embed each query
- Score all chunks by cosine similarity to the query
- Check if the expected source document appears in the top-5 results
- Calculate Hit Rate @5 and MRR (Mean Reciprocal Rank)
No BM25, no reranking — pure embedding quality comparison. This isolates the embedding model's contribution.
What it means: Out of 30 queries, how often did the correct document appear somewhere in the top 5 results?
Why it matters: In a RAG system, you typically retrieve 5-10 chunks and feed them to the LLM. If the right chunk isn't in those top 5, the LLM won't have the information it needs. Hit Rate measures recall — does the model find the needle at all?
Example: 100% means the correct document was always in the top 5. 96.7% means it missed once.
What it means: The average of 1/rank for each query, where rank is the position of the correct document in the results (1 = first result, 2 = second, etc.).
Why it matters: Hit Rate tells you if the right answer is somewhere in the top 5. MRR tells you if it's at the top. In practice, users (and LLMs) pay more attention to the first results. A model with MRR 0.84 puts the right answer at rank 1 most of the time. A model with MRR 0.74 might bury it at rank 3-4.
Concrete example:
- All 30 queries return the correct doc at rank 1 → MRR = 1.0 (perfect)
- Correct doc always at rank 2 → MRR = 0.5
- Correct doc always at rank 5 → MRR = 0.2
Scale: MRR ranges from 0 to 1. Above 0.75 is good. Above 0.80 is excellent.
What it means: Wall-clock time to embed all 59 document chunks (queries are negligible — 1 embedding each).
Why it matters: For ingestion of large corpora (thousands of documents), embedding speed directly determines how long a rebuild takes. A model that's 10x slower means 10x longer ingestion.
What it means: The mathematical measure used to rank documents against a query. It computes the cosine of the angle between two vectors — 1.0 means identical direction, 0.0 means orthogonal (unrelated).
Why no raw scores in the results: Cosine similarity scores are not comparable across models (each model has its own scale). Only the ranking (which document comes first) is comparable. That's why we use Hit Rate and MRR, not raw similarity scores.
| Model | Hit Rate @5 | MRR | Embed Time | Dims |
|---|---|---|---|---|
| Mistral Embed 2312 | 100.0% (30/30) | 0.836 | 20.3s | 1024 |
| BGE-M3 | 96.7% (29/30) | 0.797 | 32.7s | 1024 |
| Qwen3-8B @ 1024d | 100.0% (30/30) | 0.750 | 298.4s | 1024 |
| Qwen3-8B @ 512d | 100.0% (30/30) | 0.736 | 212.4s | 512 |
| Qwen3-8B @ 4096d | 100.0% (30/30) | 0.739 | 263.7s | 4096 |
Query distribution: 4 French, 16 English, 10 Japanese.
| Model | FR MRR | EN MRR | JA MRR | FR Hit | EN Hit | JA Hit |
|---|---|---|---|---|---|---|
| Mistral Embed | 1.000 | 0.724 | 0.950 | 100% | 100% | 100% |
| BGE-M3 | 0.875 | 0.714 | 0.900 | 100% | 94% | 100% |
| Qwen3-8B @ 512d | 1.000 | 0.750 | 0.608 | 100% | 100% | 100% |
| Qwen3-8B @ 1024d | 0.875 | 0.740 | 0.717 | 100% | 100% | 100% |
| Qwen3-8B @ 4096d | 0.875 | 0.760 | 0.650 | 100% | 100% | 100% |
Best model per language:
- 🇫🇷 French: Mistral Embed / Qwen3-8B @ 512d (tied at MRR 1.000)
- 🇬🇧 English: Qwen3-8B @ 4096d (MRR 0.760) — but Qwen3 @ 512d is very close (0.750)
- 🇯🇵 Japanese: Mistral Embed (MRR 0.950) — clear winner
Note: The French query count (4) is small. Results for FR should be treated as indicative, not statistically significant. The dataset is intentionally Japanese-culture-focused, which skews the language distribution.
The overall ranking is misleading because Japanese queries heavily favor Mistral and BGE-M3 while penalizing Qwen3. When you remove Japanese queries and look only at French + English (which is what most real-world RAG corpora contain), the ranking flips:
| Model | MRR (All) | MRR (FR+EN only) | MRR (JP only) |
|---|---|---|---|
| Mistral Embed | 0.836 | 0.779 | 0.950 |
| BGE-M3 | 0.797 | 0.746 | 0.900 |
| Qwen3-8B @ 512d | 0.736 | 0.800 | 0.608 |
| Qwen3-8B @ 1024d | 0.750 | 0.767 | 0.717 |
| Qwen3-8B @ 4096d | 0.739 | 0.783 | 0.650 |
Without Japanese, Qwen3-8B @ 512d is the best model (MRR 0.800, beating Mistral at 0.779). The Japanese queries artificially boosted Mistral and BGE-M3 in the overall ranking.
If your corpus contains Japanese or other CJK languages: Mistral Embed is clearly superior (MRR 0.950 on JP vs 0.608 for Qwen3-8B @ 512d).
If your corpus is French/English only: Qwen3-8B @ 512d wins on both quality and cost.
Qwen3-Embedding-8B supports Matryoshka Representation Learning (MRL), allowing truncation from 4096d to 512d or 1024d. The results show that 512d is not a downgrade — it's actually the best configuration for FR+EN retrieval (MRR 0.800 vs 0.783 for 4096d and 0.767 for 1024d).
This confirms that Qwen3's MRL training places the most important semantic information in the first dimensions. 512d is a first-class operating mode, not a degraded fallback.
Practical implication: 512d vectors use 8x less storage and RAM than 4096d, with better retrieval quality.
Mistral Embed dominated Japanese queries (MRR 0.950) and had the fastest embedding time overall (20.3s for 59 chunks). If your corpus includes Japanese, Chinese, or Korean, it's the clear winner — but at ~10x the cost.
For FR/EN-only corpora, the quality advantage disappears and the cost premium is not justified.
BGE-M3 had the second-highest overall MRR (0.797) and was the second-fastest (32.7s). It missed one query in English (Samurai code of honor bushido → expected samurai_fr). Strong on Japanese (0.900), good value at ~$0.01/1M tokens.
Mistral Embed: 20.3s (0.34s/chunk)
BGE-M3: 32.7s (0.55s/chunk)
Qwen3 @ 4096d: 263.7s (4.47s/chunk)
Qwen3 @ 1024d: 298.4s (5.06s/chunk)
Qwen3 @ 512d: 212.4s (3.60s/chunk)
Qwen3-8B is significantly slower — it's an 8B parameter model vs BGE-M3 (~568M) and Mistral Embed (~335M). For bulk ingestion of thousands of documents, this matters. However, embeddings are only computed once at ingestion time; query-time latency (1 embedding per query) is negligible for all models.
| Rank | Count | Queries |
|---|---|---|
| 1 | 24 | Most queries |
| 2 | 5 | Buddhism EN, Sushi FR, Tsunami FR, Anime EN, Manga FR, Tsunami JP, Fuji JP |
| 3 | 2 | Fuji eruption, Calligraphy EN |
| 4 | 1 | Samurai bushido FR |
| Rank | Count | Note |
|---|---|---|
| 1 | 22 | |
| 2-4 | 7 | |
| MISS | 1 | Samurai code of honor bushido → returned wrong source |
| Rank | Count |
|---|---|
| 1 | 22 |
| 2 | 5 |
| 3 | 2 |
| 4 | 1 |
MTEB (Massive Text Embedding Benchmark) multilingual leaderboard as of June 2025:
| Model | MTEB Multilingual | Notes |
|---|---|---|
| Qwen3-Embedding-8B | 70.58 (#1) | MRL-native, 100+ languages |
| BGE-M3 | ~63.0 | Multi-function (dense + sparse + multi-vector), 100+ languages |
| Mistral Embed 2312 | Not on MTEB | Proprietary, limited public benchmarks |
Sources: MTEB Leaderboard, Qwen3 Embedding blog, BGE-M3 paper
Despite Mistral Embed not appearing on MTEB, our practical benchmark shows it outperforms both MTEB leaders on real-world multilingual retrieval. This highlights the gap between academic benchmarks and domain-specific performance.
The right model depends on your corpus language:
Winner: Qwen3-Embedding-8B @ 512d
- Best MRR on FR+EN (0.800), beating both Mistral and BGE-M3
- 100% hit rate
- Cheapest option (~$0.01/1M tokens)
- 512d = 8x less storage/RAM than 4096d, with better quality
- Slower at ingestion (3.6s/chunk) but embeddings are computed once
Winner: Mistral Embed 2312
- Best MRR on Japanese (0.950 vs 0.608 for Qwen3-8B @ 512d)
- Fastest embedding (0.34s/chunk)
- 100% hit rate
- ~10x more expensive (~€0.10/1M tokens), but negligible for small-to-medium corpora
Winner: BGE-M3 on Ollama
- Strong multilingual support (MRR 0.900 on JP)
- Free, runs on local GPU
- Available via Ollama (
bge-m3:latest)
# Install numpy
pip install numpy
# Set API key
export OPENROUTER_API_KEY="your-key-here"
# Run all models
python3 benchmark.py
# Run specific model
python3 benchmark.py --model mistral-embed
# Results saved to results/<model>.jsonembedding-benchmark/
├── README.md ← this file
├── benchmark.py ← benchmark script
├── golden_dataset.json ← 30 queries with expected sources
├── docs/ ← 30 Wikipedia articles (FR/EN/JP)
│ ├── tokyo_fr.txt
│ ├── tokyo_en.txt
│ ├── tokyo_ja.txt
│ └── ... (27 more)
└── results/ ← JSON results per model
├── qwen3-8b.json
├── bge-m3.json
└── ...
MIT — data from Wikipedia (CC BY-SA), benchmark code is ours.
Benchmark run June 30, 2026. Models accessed via OpenRouter API. No local GPU used.