Capturing Semantics with Word Embeddings

1. Overview

How it started

In a previous article I compared the performance of a commercially available LLM to non commercial LLMs in a language identification task using data obtained from tanzil.net (see here). I discovered that: 1. nomic-embed-text:latest and nomic-embed-text-v2-moe:latest achieved comparable performance to gemini-embedding-001 in identifying the language of the quranic text analysed; 2. suprisingly the performance of nomic-embed-text:latest which was developed primarily for the English language (en) was comparable (if not better) to both nomic-embed-text-v2-moe:latest and quran-embedding-prefect-gemini-embedding-001 which are multilingual; 3. a heuristic that uses the count of characters associated with the languages in the text, e.g Ge’ez for Amharic and alfabesi for Turkish, achieved comparable performance to gemini-embedding-001, nomic-embed-text-v2-moe:latest, and nomic-embed-text:latest.

The script used for each text is specific to the language of the text e.g. Arabic script for Arabic and Ge’ez for Amharic and this is instrumental to the success of the heurictic. This raised the following question: Which of the LLMs best captures the semantics of the words - so called word sense? This is aimed at looking beyond the characters in the text

Key Findings

1. gemini-embedding-001 captures meaning best. It has the highest mean precision@k (k=10) and mean reciprocal rank (MRR). Most of its hits are in other languages, so it matches a verse to its translations regardless of language or script.
2. nomic-embed-text-v2-moe:latest comes second overall and it beats Gemini on Bosnian and Albanian, and achieves comparable performance on Spanish and Turkish but performs poorly on Tajik (0.002), Divehi (0.000), Uzbek (0.007), Amharic (0.026), Sindhi (0.042) and Arabic (0.073), even though most of these are listed as supported.
3. nomic-embed-text:latest does not capture meaning across languages at all. Its cross-language precision@10 and cross-language MRR are both about 0.001 or below. Nearly all of its hits come from English queries finding another English translation (precision@10 of 0.694 for English, about 0 for most other languages).
4. All three models do poorly on Hausa and Somali which are low resource languages.

Caveats: - The nomic models were run without their recommended task prefixes (search_document: / search_query:), which may understate their performance.

So What 1. No free lunch: The evidence suggests Gemini best captures the semantics of the quran text because compared to other LLMs it reliably finds the same verse across languages, for example an Arabic verse from an English query. nomic-embed-text-v2-moe:latest is a viable alternative only for English and some Latin-script European languages, such as Bosnian, Albanian, Spanish and Turkish. For Arabic, Persian, Urdu and the Central Asian and South Asian languages, it falls well short of Gemini. 3. It is more than text deep: The earlier success of nomic-embed-text:latest in language identification is likely due to its sensitivity to language and script, not an understanding of meaning. 4. horses for courses: The best model for telling languages apart is not the best for matching meaning. For language identification, the simple character-count heuristic remains a viable option. 5. To draw the short straw: Users querying in low-resource languages such as Hausa and Somali would currently get poor results from any of these models. This is an area of research currently being covered by initiatives such as HausaNLP for Hausa, and SomaliNLP for somali.

import warnings
from numba.core.errors import NumbaWarning
warnings.filterwarnings("ignore", category=NumbaWarning)

from vectoriser import get_qdrant
from query_qdrant import (load_sample, load_sample_where, 
                    get_payload_keys, get_payload, 
                    sample_records, extract_payload, 
                    fetch_records_for_dataframe)

from sklearn.preprocessing import LabelEncoder
from sklearn.decomposition import PCA
from sklearn.metrics import normalized_mutual_info_score, recall_score, precision_score
from sklearn.pipeline import Pipeline

import numpy as np
from itertools import combinations
import pandas as pd
import umap
from matplotlib import pyplot as plt
import seaborn as sb
from umap import UMAP
import umap.plot
import matplotlib.ticker as ticker
import regex
from lang_util import detect_language, language_map
from pandas.io.formats.style import Styler

from qdrant_client import models

1.2 Approach

To answer this question, I took the following approach:

1. Source Data: Each verse of the Quran, in a range of languages and translations from tanzil.net, was embedded with three models: gemini-embedding-001, nomic-embed-text:latest and nomic-embed-text-v2-moe:latest. The embeddings for each model are stored in their own collection on qdrant, alongside the verse text and its attributes (language, translator, chapter and verse). Details of the data and the models are given in Section 2.0.
2. Test Sample: I drew a random 40% sample of records (99,777) from the gemini-embedding-001 collection and downsampled English, which otherwise makes up about 42% of the data, to the average size of the other languages. This left 60,312 records. I then fetched exactly the same records from the two nomic collections, so the three samples are identical and only the embeddings differ between them.
3. Expectation: A verse means the same thing in every translation. For example, chapter 1, verse 1 in English (Hilali translation) has the same meaning as chapter 1, verse 1 in Arabic (Jalalayn). A model that captures meaning should therefore place the translations of a verse close together, regardless of their language or script.
4. Measurement: For each sample verse, I retrieved its 20 nearest neighbours from the full collection of that model. A neighbour counts as a hit if it is the same chapter and verse from a different translation. Each hit is also labelled as same-language (e.g. another English translation) or cross-language (e.g. the Arabic). I compared the models using two rank-based metrics, described in Section 3.1:
   • precision@k, the proportion of the top k neighbours that are hits, which rewards finding many translations of a verse (see here);
   • mean reciprocal rank (MRR), based on the rank of the first hit, which rewards finding one translation quickly (see here).
   Cross-language hits are the key evidence: a model that only distinguishes languages can find other translations in the same language, but only a model that captures meaning can find the same verse in another language.

flowchart TD
    A[("Three Qdrant collections<br/>one per embedding model")] -->|"random 40% sample from the Gemini collection,<br/>English downsampled"| B["60,312 sample records"]
    B -->|"same records fetched from<br/>both nomic collections"| C["Identical samples,<br/>one set of vectors per model"]
    C -->|"top-20 nearest neighbours per verse,<br/>query record excluded"| D["Nearest neighbours"]
    D -->|"hit = same chapter and verse,<br/>different translation"| E["Same-language and<br/>cross-language hits"]
    E --> F["Precision@k<br/>k = 1, 5, 10, 20"]
    E --> G["MRR@20"]
    F --> H["Model comparison,<br/>overall and by language"]
    G --> H

2.0 Data

The vectorised quranic text obtained from tanzil.net is stored in the following collections on a qdrant instance running locally on a homelab cluster. Details of each collection and the vectorisers used is provided below:

Collection (word embeddings)	Number of Records	Vector Dimensions	Vectoriser	Source	Vectoriser No. Parameters (Millions)	Model Type
quran-embedding-prefect-gemini-embedding-001	249,442	768	gemini-embedding-001	Google	Undisclosed	Transformer (Gemini-initialised, decoder-based)
quran-embedding-prefect-nomic-embed-text-latest	249,442	768	nomic-embed-text:latest	Ollama (Local Instance)	137	BERT
quran-embedding-prefect-nomic-embed-text-v2-moe-latest	249,442	768	nomic-embed-text-v2-moe:latest	Ollama (Local Instance)	475.29	BERT - MoE

A list of the languages covered in this study and whether or not they are supported by either of the embedding models is provided below

Code	Language	gemini-embedding-001	nomic v1.5 (nomic-embed-text-latest)	nomic-embed-text-v2-moe
am	Amharic	✓	✗	✓
ar	Arabic	✓	✗	✓
bs	Bosnian	✓	✗	✓
dv	Divehi	✓	✗	✗
en	English	✓	✓	✓
es	Spanish	✓	✗	✓
fa	Persian	✓	✗	✓
ha	Hausa	✓	✗	✓
nl	Dutch	✓	✗	✓
ru	Russian	✓	✗	✓
sd	Sindhi	✓	✗	✗
so	Somali	✓	✗	✓
sq	Albanian	✓	✗	✓
ta	Tamil	✓	✗	✓
tg	Tajik	✓	✗	✓
th	Thai	✓	✗	✓
tr	Turkish	✓	✗	✓
ur	Urdu	✓	✗	✓
uz	Uzbek	✓	✗	✓
zh	Chinese	✓	✗	✓

2.1 Sampling

• Samples were collected from the collections starting with one collection first and then using the text in the first collection to sample records matching the same text from the other two collections.
• The data is largely comprised of text in English (en) therefore some downsampling was applied to prevent skewing the results.

def downsample_majority(df,samp_col="language",random_state=1235813):
    props = df.groupby([samp_col]).size().to_dict()
    sorted_props = sorted(props, key=lambda e:props[e],reverse=True)
    maj = sorted_props[0]
    avg_minority = int(np.ceil(np.mean([props[e] for e in props if e!=maj])))
    A = df[df[samp_col].eq(maj)].sample(avg_minority, random_state=random_state)
    B = df[~df[samp_col].eq(maj)]
    C = pd.concat([A,B]).reset_index(drop=True)

    return C    

collections = {"gemini-embedding-001":'quran-embedding-prefect-gemini-embedding-001',
               "nomic-embed-text:latest":'quran-embedding-prefect-nomic-embed-text-latest', 
                "nomic-embed-text-v2-moe:latest":'quran-embedding-prefect-nomic-embed-text-v2-moe-latest'}

sample_prop = 0.4

init = "gemini-embedding-001"
vectors_main, main_df = sample_records(collections[init],frac=sample_prop,exclude=[])
main_df['language_description'] = main_df['language'].map(language_map)

The table below shows how many records the 40% sample drew from each language. English makes up 42,337 of the 99,777 records (about 42%), because the collection holds far more English translations than translations in any other language. The first row (blank language code) counts records with no language code.

freq = main_df.groupby(['language','language_description'],dropna=False).size()
freq = pd.DataFrame(freq).rename(columns={0:"Number of Occurrences"})
freq

		Number of Occurrences
language	language_description
	NaN	2527
am	Amharic	2490
ar	Arabic	4920
bs	Bosnian	2472
dv	Divehi	2493
en	English	42337
es	Spanish	2538
fa	Persian	2458
ha	Hausa	2457
nl	Dutch	2489
ru	Russian	2478
sd	Sindhi	2521
so	Somali	2547
sq	Albanian	2527
ta	Tamil	2519
tg	Tajik	2517
th	Thai	2592
tr	Turkish	2469
ur	Urdu	4909
uz	Uzbek	2482
zh	Chinese	5035

samples = {}
samples[init] = downsample_majority(main_df)

vectors = {}
# downsample_majority reorders rows, so index vectors by id to keep vectors[init][i] aligned with samples[init].iloc[i]
idx = pd.Index(main_df['id']).get_indexer(samples[init]['id'])
vectors[init] = vectors_main[idx]

samples[init].groupby(['language']).size()
%store init

Stored 'init' (str)

The first five rows of the gemini-embedding-001 sample are shown below. Each row is one record: a verse, or part of a long verse, from one translation. text is the text that was embedded. language, name (the translator), chapter and verse identify it, and the revelation_* columns give the revelation metadata of its chapter (surah). The index column is the record’s position in the full collection before sampling.

samples[init].head()

	index	id	language	name	chapter	verse	text	surah_name	revelation_order	revelation_type	revelation_note	language_description
0	236019	f2627bd6-f23a-54e5-a1a6-5bbdbad95756	en	ahmedraza	88	7	Which neither give strength nor satisfy the hu...	Al-Ghaashiya	68	Meccan	NaN	English
1	248871	ff6d1d6b-bf5d-52c7-962a-966cc590144a	en	qarai	68	26	But when they saw it, they said, ‘We have inde...	Al-Qalam	2	Medinan	Except 17-33 and 48-50, from Medina	English
2	217314	df0c4ee7-8643-57f8-86ad-e5c489867bed	en	sahih	10	77	Moses said, "Do you say [thus] about the truth...	Yunus	51	Meccan	Except 40, 94, 95, 96, from Medina	English
3	4029	042bb1ea-609d-5059-bacc-dd2fe3110639	en	qaribullah	37	63	We have made this (tree) a trial for the evild...	As-Saaffaat	56	Meccan	NaN	English
4	24268	18e67fb4-1203-54f0-bf46-d011a8464c28	en	sahih	6	95	Indeed, Allah is the cleaver of grain and date...	Al-An'aam	55	Meccan	Except 20, 23, 91, 93, 114, 151, 152, 153, fro...	English

A sample equivalent to that collected from quran-embedding-prefect-gemini-embedding-001 was collected from the other two collections detailed in Section 2.0

filter_columns = ['language','name','chapter','verse','text']

colls  = [coll for coll in collections if coll !=init]
for c in colls:
    if collections[c]!= None:
        print("Processing {}".format(c))
        matching_records = fetch_records_for_dataframe(
            samples[init],
            collections[c],
            filter_cols = filter_columns,
            batch_size = 200,
            with_vectors = True
        )
        samp, vec = extract_payload(matching_records, exclude=[])
        samples[c] = pd.DataFrame(samp) 
        vectors[c] = vec

        assert len(samples[c]) == len(samples[init]), (
            f"{c}: expected {len(samples[init])} matching records, got {len(samples[c])} "
            "- some rows had no exact match, so the like-for-like comparison "
            "across collections no longer holds."
        )
        assert (samples[c]['id'].to_numpy() == samples[init]['id'].to_numpy()).all(), (
            f"{c}: records are not in the same order as {init}, so vectors[{c!r}][i] "
            "would not correspond to samples[init].iloc[i]."
        )
    else:
        samples[c] = None
        vectors[c] = None

Processing nomic-embed-text:latest
Processing nomic-embed-text-v2-moe:latest

The first five rows of the matching nomic-embed-text:latest sample are shown below. They are the same records as in the gemini-embedding-001 table above, in the same order, with the same id, language, translator, chapter, verse and text. Only the stored embeddings differ.

samples['nomic-embed-text:latest'].head()

	id	language	name	chapter	verse	text	surah_name	revelation_order	revelation_type	revelation_note
0	f2627bd6-f23a-54e5-a1a6-5bbdbad95756	en	ahmedraza	88	7	Which neither give strength nor satisfy the hu...	Al-Ghaashiya	68	Meccan	NaN
1	ff6d1d6b-bf5d-52c7-962a-966cc590144a	en	qarai	68	26	But when they saw it, they said, ‘We have inde...	Al-Qalam	2	Medinan	Except 17-33 and 48-50, from Medina
2	df0c4ee7-8643-57f8-86ad-e5c489867bed	en	sahih	10	77	Moses said, "Do you say [thus] about the truth...	Yunus	51	Meccan	Except 40, 94, 95, 96, from Medina
3	042bb1ea-609d-5059-bacc-dd2fe3110639	en	qaribullah	37	63	We have made this (tree) a trial for the evild...	As-Saaffaat	56	Meccan	NaN
4	18e67fb4-1203-54f0-bf46-d011a8464c28	en	sahih	6	95	Indeed, Allah is the cleaver of grain and date...	Al-An'aam	55	Meccan	Except 20, 23, 91, 93, 114, 151, 152, 153, fro...

The figure below compares the three samples. Each row of the figure corresponds to a model’s sample, and the columns show the number of records per language (left), per chapter (middle) and per verse number (right). The three rows are identical, as expected, because they contain the same records. The sampling step also checks that the record ids match one for one. The figure that that the samples have identical statistical signature.

fig, ax = plt.subplots(3,3,figsize=(20,20), sharex=False)
ax = ax.flatten()

for x,c in zip([0, 3, 6],collections.keys()):
    samples[c].groupby('language').size().plot.bar(ax=ax[x])
    ax[x].set_title(c)

for x,c in zip([1,4,7],collections.keys()):
    samples[c].groupby('chapter').size().plot.bar(ax=ax[x])
    ax[x].xaxis.set_major_locator(ticker.MaxNLocator(nbins=6))
    ax[x].set_title(c)

for x,c in zip([2,5,8],collections.keys()):
    samples[c].groupby('verse').size().plot.bar(ax=ax[x])
    ax[x].xaxis.set_major_locator(ticker.MaxNLocator(nbins=6))
    ax[x].set_title(c)

for a in ax: 
    a.set_ylabel("Number of Occurrences")

for col in samples:
    if collections[col]!=None:
        samples[col]['language_description'] = samples[col]['language'].map(language_map)

counts = {}
for c in samples:
    if collections[c] != None:
        count = samples[c].fillna('n/a').groupby(['language','language_description'],dropna=False).size()
        counts[c] = count

The table below gives the number of records per language in each model’s sample after downsampling. English drops from 42,337 records to 2,872, close to the average of the other languages, so no single language dominates the results. Arabic, Urdu and Chinese still have about twice as many records as most other languages. The counts are identical for all three models.

pd.DataFrame(counts.values(),index = counts.keys()).T

		gemini-embedding-001	nomic-embed-text:latest	nomic-embed-text-v2-moe:latest
language	language_description
	n/a	2527	2527	2527
am	Amharic	2490	2490	2490
ar	Arabic	4920	4920	4920
bs	Bosnian	2472	2472	2472
dv	Divehi	2493	2493	2493
en	English	2872	2872	2872
es	Spanish	2538	2538	2538
fa	Persian	2458	2458	2458
ha	Hausa	2457	2457	2457
nl	Dutch	2489	2489	2489
ru	Russian	2478	2478	2478
sd	Sindhi	2521	2521	2521
so	Somali	2547	2547	2547
sq	Albanian	2527	2527	2527
ta	Tamil	2519	2519	2519
tg	Tajik	2517	2517	2517
th	Thai	2592	2592	2592
tr	Turkish	2469	2469	2469
ur	Urdu	4909	4909	4909
uz	Uzbek	2482	2482	2482
zh	Chinese	5035	5035	5035

3.0 Analysis

qdrant_client = get_qdrant()

3.1 Metrics:

Precision@k

For each sample record, its top-\(k\) nearest neighbours are retrieved from the whole collection (excluding the record itself). A neighbour is a hit if it is the same chapter and verse from a different translation. Precision@k is the fraction of the \(k\) neighbours that are hits.

Because the metric depends only on the ranking of neighbours and not on raw cosine scores, it is comparable across models whose similarity scores are scaled differently.

Hits are further split into same-language hits (e.g. another English translation of the verse) and cross-language hits (e.g. the Arabic for an English query). A model that mostly encodes language rather than meaning can only score same-language hits; cross-language hits require capturing the meaning of the verse.

Mean Reciprocal Rank (MRR)

For each query it takes \(\frac{1}{\text{rank}}\) of the first hit (1.0 if the nearest neighbour is a hit, 0.5 if it is second, etc), averaged over queries. While precision@k rewards finding many translations of the verse; MRR rewards finding one quickly, which is what matters for a semantic search that shows the top result. Only the top 20 neighbours are retrieved, so a query with no hit in its top 20 scores 0 (i.e. this is MRR@20). MRR is also split by language: the rank of the first same-language hit and the rank of the first cross-language hit.

def query_nearest_neighbours(collection, samp, vector, k=20, batch_size=64):
    """Retrieve the top-k nearest neighbours of each sample record (excluding the record itself),
    with no payload filter or score threshold, one row per (query, neighbour)."""
    rows = []
    for start in range(0, len(samp), batch_size):
        query_ids = samp['id'].iloc[start:start + batch_size].tolist()
        requests = [
            models.QueryRequest(
                query=vector[start + j].tolist(),
                filter=models.Filter(must_not=[models.HasIdCondition(has_id=[query_id])]),
                limit=k,
                with_payload=["chapter", "verse", "name", "language"],
            )
            for j, query_id in enumerate(query_ids)
        ]
        results = qdrant_client.query_batch_points(collection_name=collection, requests=requests)
        rows.extend(
            {"query_id": query_id, "rank": rank, "id": point.id, "score": point.score, **point.payload}
            for query_id, res in zip(query_ids, results)
            for rank, point in enumerate(res.points, start=1)
        )
    return pd.DataFrame(rows)

def label_hits(neighbours, samp):
    """Join each neighbour to its query record and label it as a hit (same chapter and verse from a
    different translation), split into same-language and cross-language hits."""
    query = samp[['id', 'chapter', 'verse', 'name', 'language']].add_prefix('query_')
    df = neighbours.merge(query, on='query_id')

    # chapter/verse are compared as strings since the payload stores them inconsistently
    is_hit = (
        df['chapter'].astype(str).eq(df['query_chapter'].astype(str))
        & df['verse'].astype(str).eq(df['query_verse'].astype(str))
        & df['name'].ne(df['query_name'])
    )
    same_language = df['language'].eq(df['query_language'])
    return df.assign(hit=is_hit, same_language=is_hit & same_language, cross_language=is_hit & ~same_language)


def precision_at_k(hits, ks):
    """Per-query precision@k for each k in `ks`, split into same-language and cross-language hits."""
    per_k = [
        hits[hits['rank'].le(k)]
        .groupby('query_id')[['same_language', 'cross_language']]
        .sum()
        .div(k)
        .assign(k=k)
        for k in ks
    ]
    res = pd.concat(per_k).reset_index()
    return res.assign(precision=res['same_language'] + res['cross_language'])


def reciprocal_rank(hits):
    """Per-query reciprocal rank of the first hit (0 if none was retrieved), overall and for the
    first same-language and first cross-language hit."""
    first_rank = {
        col: hits[hits[col]].groupby('query_id')['rank'].min()
        for col in ['hit', 'same_language', 'cross_language']
    }
    return (
        (1 / pd.DataFrame(first_rank))
        .reindex(hits['query_id'].unique())
        .fillna(0)
        .rename(columns={'hit': 'rr', 'same_language': 'rr_same_language', 'cross_language': 'rr_cross_language'})
        .rename_axis('query_id')
        .reset_index()
    )

ks = [1, 5, 10, 20]

hits = {
    key: label_hits(
        query_nearest_neighbours(collections[key], samples[key], vectors[key], k=max(ks)),
        samples[key],
    )
    for key in samples
}

query_languages = (
    samples[init][['id', 'language_description']]
    .rename(columns={'id': 'query_id'})
    .fillna({'language_description': 'n/a'})
)
precision = pd.concat(precision_at_k(h, ks).assign(model=key) for key, h in hits.items()).merge(query_languages, on='query_id')
mrr = pd.concat(reciprocal_rank(h).assign(model=key) for key, h in hits.items()).merge(query_languages, on='query_id')

The table below shows the mean precision@k for each model at k = 1, 5, 10 and 20. Precision is the share of the top k neighbours that are another translation of the query verse. It is the sum of Same Language (translations in the query’s language) and Cross Language (translations in other languages). In all tables in this section, the colour scale runs from 0 (pale) to 1 (dark green).

gemini-embedding-001 scores highest at every k, followed by nomic-embed-text-v2-moe:latest and nomic-embed-text:latest. Most of Gemini’s hits are cross-language.

summary = precision.groupby(['model', 'k'])[['precision', 'same_language', 'cross_language']].mean()
summary.rename(columns={"precision":"Precision","same_language":"Same Language","cross_language":"Cross Language"}, inplace=True)
summary.style.background_gradient(cmap='YlGn', vmin=0, vmax=1).format(precision=3)

		Precision	Same Language	Cross Language
model	k
gemini-embedding-001	1	0.508	0.202	0.306
	5	0.471	0.081	0.390
	10	0.463	0.061	0.402
	20	0.435	0.040	0.396
nomic-embed-text-v2-moe:latest	1	0.308	0.168	0.141
	5	0.245	0.070	0.176
	10	0.228	0.052	0.177
	20	0.197	0.033	0.164
nomic-embed-text:latest	1	0.098	0.097	0.001
	5	0.052	0.051	0.001
	10	0.041	0.040	0.000
	20	0.025	0.025	0.000

The table below shows the mean reciprocal rank (MRR) of the first hit for each model. MRR (Any Hit) uses the first hit of either kind; the other two columns use the first same-language hit and the first cross-language hit. A value of 1 means the first hit is always the nearest neighbour; lower values mean it tends to appear further down the list, and a query with no hit in its top 20 scores 0.

The ranking matches precision@k. gemini-embedding-001 has the highest MRR and finds a cross-language hit much sooner than the other models. nomic-embed-text:latest almost never finds a cross-language hit in its top 20.

mrr_summary = mrr.groupby('model')[['rr', 'rr_same_language', 'rr_cross_language']].mean()
mrr_summary\
    .rename(columns={"rr":"MRR (Any Hit)","rr_same_language":"Reciprocal Rank (Same Language)", "rr_cross_language":"Reciprocal Rank (Cross Language)"},inplace=True)
mrr_summary.style.background_gradient(cmap='YlGn', vmin=0, vmax=1).format(precision=3)

	MRR (Any Hit)	Reciprocal Rank (Same Language)	Reciprocal Rank (Cross Language)
model
gemini-embedding-001	0.605	0.222	0.459
nomic-embed-text-v2-moe:latest	0.373	0.183	0.224
nomic-embed-text:latest	0.105	0.104	0.001

The figure below summarises the two tables above:

• Left: mean precision@k against k. Precision falls slightly as k grows for every model.
• Middle: precision@10 split into same-language (blue) and cross-language (orange) hits. Cross-language hits make up most of the precision of gemini-embedding-001 and nomic-embed-text-v2-moe:latest, but almost none of nomic-embed-text:latest.
• Right: MRR for any hit (blue), the first same-language hit (orange) and the first cross-language hit (green). The green bar for nomic-embed-text:latest is close to zero: it almost never finds the same verse in another language.

fig, ax = plt.subplots(1, 3, figsize=(24, 8))

summary['Precision'].unstack('model').plot.line(ax=ax[0], marker="o")
ax[0].set_xticks(ks)
ax[0].set_xlabel("k")
ax[0].set_ylabel("Mean precision@k")
ax[0].set_ylim(0, 1)
ax[0].grid(True, linestyle="--", alpha=0.5)
ax[0].set_title("Precision@k - same chapter and verse, different translation")

k_split = 10
summary.xs(k_split, level='k')[['Same Language', 'Cross Language']].plot.bar(ax=ax[1], stacked=True, rot=0)
ax[1].set_ylabel(f"Mean precision@{k_split}")
ax[1].set_ylim(0, 1)
ax[1].grid(True, axis="y", linestyle="--", alpha=0.5)
ax[1].set_title(f"Precision@{k_split} - same-language vs cross-language hits")

mrr_summary.plot.bar(ax=ax[2], rot=0)
ax[2].legend(["Any hit", "First same-language hit", "First cross-language hit"])
ax[2].set_ylabel("MRR@20")
ax[2].set_ylim(0, 1)
ax[2].grid(True, axis="y", linestyle="--", alpha=0.5)
ax[2].set_title("Mean reciprocal rank of the first hit")

plt.tight_layout()

The table below breaks precision@10 (left three columns) and MRR (right three columns) down by the language of the query verse, to show where each model succeeds or fails:

• gemini-embedding-001 is the strongest model for most languages. It is far ahead of the other two on Arabic (precision@10 of 0.545 vs 0.073 for nomic-embed-text-v2-moe:latest), Persian, Tajik, Uzbek, Sindhi, Divehi and Amharic.
• nomic-embed-text-v2-moe:latest beats Gemini on Bosnian (0.502 vs 0.384) and Albanian (0.316 vs 0.267), and comes close on Spanish and Turkish. It scores close to zero on Tajik, Uzbek, Divehi, Amharic and Sindhi.
• nomic-embed-text:latest scores well only on English (0.694). Its MRR for Chinese (0.581) and Urdu (0.175) comes entirely from finding another translation in the same language, since it finds almost no cross-language hits.
• All three models score poorly on Hausa and Somali (precision@10 of 0.053 or below). These are considered low resource language with relatively very little NLP corpora.
• The n/a row (records with no language code) scores 0.730 with Gemini but close to zero with both nomic models. As shown in Appendix A.1 these come from uthmani text which is in Arabic and are not translations. The top ranking hits matching these text are in English, followed by Arabic.

pd.concat(
    {
        'precision@10': precision[precision['k'].eq(10)].pivot_table(
            index='language_description', columns='model', values='precision', aggfunc='mean'),
        'MRR': mrr.pivot_table(index='language_description', columns='model', values='rr', aggfunc='mean'),
    },
    axis=1,
).style.background_gradient(cmap='YlGn', vmin=0, vmax=1).format(precision=4)

	precision@10			MRR
model	gemini-embedding-001	nomic-embed-text-v2-moe:latest	nomic-embed-text:latest	gemini-embedding-001	nomic-embed-text-v2-moe:latest	nomic-embed-text:latest
language_description
Albanian	0.2669	0.3162	0.0009	0.3988	0.4205	0.0020
Amharic	0.2624	0.0264	0.0001	0.3110	0.0440	0.0003
Arabic	0.5454	0.0727	0.0002	0.7270	0.2190	0.0014
Bosnian	0.3835	0.5022	0.0008	0.5099	0.5978	0.0018
Chinese	0.5447	0.2503	0.0615	0.9737	0.9492	0.5813
Divehi	0.1284	0.0003	0.0004	0.1736	0.0008	0.0011
Dutch	0.4812	0.3778	0.0005	0.5075	0.4699	0.0016
English	0.8391	0.7508	0.6939	0.9125	0.8623	0.8642
Hausa	0.0528	0.0018	0.0002	0.0684	0.0036	0.0004
Persian	0.6682	0.2739	0.0005	0.7661	0.4651	0.0023
Russian	0.3904	0.2029	0.0002	0.4367	0.2705	0.0014
Sindhi	0.3404	0.0415	0.0006	0.6528	0.0969	0.0034
Somali	0.0457	0.0037	0.0000	0.0649	0.0079	0.0001
Spanish	0.6634	0.6203	0.0026	0.6836	0.7118	0.0043
Tajik	0.4192	0.0021	0.0008	0.6912	0.0065	0.0029
Tamil	0.5456	0.1869	0.0002	0.5699	0.2425	0.0015
Thai	0.5286	0.2684	0.0001	0.5650	0.3098	0.0002
Turkish	0.5219	0.4553	0.0021	0.5493	0.5369	0.0041
Urdu	0.6176	0.3592	0.0274	0.8859	0.7393	0.1750
Uzbek	0.3797	0.0069	0.0008	0.6657	0.0197	0.0035
n/a	0.7304	0.0004	0.0001	0.7739	0.0011	0.0003

4.0 Lessons Learned

• Horses for courses, revisited: Being good at one task says little about another. The model that good at identifying language (nomic-embed-text:latest) is the worst at capturing meaning across languages. It takes exploration to determine the most suitable models for a given task.

• “Supported” is not the same as “works well”: nomic-embed-text-v2-moe:latest lists Arabic, Tajik, Uzbek and Amharic as being supported, yet scores close to zero on them. The average across all languages hides this, so it is worth breaking results down by language.

• The low resource curse: providing AI solutions to demographies with scarse NLP corpora can be challenging.

4.1 What Next

• It is worth exploring how well models specifically tailored to Hausa and Somali perform on the semantic search task discussed in this paper.

Appendix

A.1 Language of the hits for records with no language code (gemini-embedding-001)

The sampled texts above with a blank language code are all in Arabic script and are likely to come from quran-uthmani-arabic.txt, the original Arabic text of the Quran: its filename does not follow the <language>.<translator>.txt pattern, so no language code is parsed from it. The cells below check this by running the script-based detector from lang_util on these records, then break down gemini-embedding-001’s hits for these queries by the language and translation of the hit.

no_lang = samples[init][samples[init]['language'].eq("")]

# Translator name(s) and detected language of the records with no language code
pd.concat(
    {
        'name': no_lang['name'].value_counts(),
        'detected language': no_lang['text'].map(lambda t: detect_language(t, return_scores=False)['main_language']).value_counts(),
    }
)

name                     2527
detected language  ar    2527
Name: count, dtype: int64

gemini_hits = hits[init]
no_lang_hits = gemini_hits[gemini_hits['query_language'].eq("") & gemini_hits['hit']]

hit_language = (
    no_lang_hits
    .assign(language_description=no_lang_hits['language'].map(language_map).fillna('n/a'))
    .groupby(['language', 'language_description', 'name'])
    .agg(hits=('query_id', 'size'), queries=('query_id', 'nunique'), mean_rank=('rank', 'mean'))
    .assign(share_of_hits=lambda df: df['hits'] / df['hits'].sum())
    .sort_values('hits', ascending=False)
)
hit_language.style.background_gradient(cmap='YlGn', subset=['share_of_hits'], vmin=0, vmax=1).format({'share_of_hits': '{:.3f}', 'mean_rank': '{:.2f}'})

			hits	queries	mean_rank	share_of_hits
language	language_description	name
en	English	sahih	2041	2041	6.84	0.059
		mubarakpuri	2019	2019	7.50	0.058
		shakir	1835	1835	7.83	0.053
		hilali	1796	1796	9.28	0.052
		qarai	1759	1759	8.27	0.051
		pickthall	1750	1750	9.82	0.050
		qaribullah	1725	1725	9.61	0.050
		daryabadi	1723	1723	9.59	0.050
ar	Arabic	jalalayn	1591	1591	5.87	0.046
en	English	yusufali	1572	1572	11.73	0.045
		ahmedraza	1560	1560	9.91	0.045
		maududi	1506	1506	11.39	0.043
		itani	1478	1478	10.53	0.043
		arberry	1333	1333	11.80	0.038
		transliteration	1291	1291	5.72	0.037
		wahiduddin	1146	1146	12.59	0.033
ur	Urdu	qadri	1130	1130	12.10	0.033
		maududi	1035	1035	12.43	0.030
en	English	ahmedali	1009	1009	13.49	0.029
ar	Arabic	muyassar	987	987	8.99	0.028
fa	Persian	ansarian	930	930	12.74	0.027
en	English	sarwar	697	697	14.41	0.020
th	Thai	thai	576	576	14.56	0.017
ta	Tamil	tamil	512	512	14.77	0.015
dv	Divehi	divehi	409	409	12.44	0.012
am	Amharic	sadiq	292	292	14.48	0.008
sd	Sindhi	amroti	215	215	15.07	0.006
tr	Turkish	ates	165	165	15.65	0.005
zh	Chinese	majian	130	130	15.81	0.004
es	Spanish	bornez	111	111	15.84	0.003
zh	Chinese	jian	99	99	15.88	0.003
uz	Uzbek	sodik	65	65	16.15	0.002
ru	Russian	abuadel	58	58	16.36	0.002
tg	Tajik	ayati	39	39	14.59	0.001
ha	Hausa	gumi	35	35	16.34	0.001
sq	Albanian	nahi	14	14	16.79	0.000
so	Somali	abduh	11	11	12.73	0.000
nl	Dutch	keyzer	8	8	18.38	0.000
bs	Bosnian	korkut	3	3	17.00	0.000

The figure below shows the share of gemini-embedding-001’s hits for the no-language queries that fall in each language (left) and the rank of the first hit of each language (right). If these records are the original Arabic, most hits should be the Arabic (ar) tafsirs (Jalalayn and Muyassar) and should appear near the top of the neighbour list.

fig, ax = plt.subplots(1, 2, figsize=(20, 7))

hit_language.groupby('language_description')['share_of_hits'].sum().sort_values(ascending=False).plot.bar(ax=ax[0], rot=45)
ax[0].set_ylabel("Share of hits")
ax[0].set_title(f"{init}: language of hits for queries with no language code")
ax[0].grid(True, axis="y", linestyle="--", alpha=0.5)

first_hit_rank = (
    no_lang_hits
    .assign(language_description=no_lang_hits['language'].map(language_map).fillna('n/a'))
    .groupby(['query_id', 'language_description'])['rank'].min()
    .reset_index()
)
order = first_hit_rank.groupby('language_description')['rank'].median().sort_values().index
sb.boxplot(data=first_hit_rank, x='language_description', y='rank', order=order, ax=ax[1])
ax[1].tick_params(axis='x', rotation=45)
ax[1].set_xlabel("language_description")
ax[1].set_ylabel("Rank of first hit in that language")
ax[1].set_title(f"{init}: rank of first hit by language")
ax[1].grid(True, axis="y", linestyle="--", alpha=0.5)

plt.tight_layout()

no_lang

	index	id	language	name	chapter	verse	text	surah_name	revelation_order	revelation_type	revelation_note	language_description
2881	44534	2dbacb91-6bb7-578c-9edb-36f1851d0b96			20	108	يَوْمَئِذٍۢ يَتَّبِعُونَ ٱلدَّاعِىَ لَا عِوَجَ...	Taa-Haa	45	Meccan	Except 130 and 131, from Medina	NaN
2911	214809	dc8855a4-ee1b-59dc-af9a-ece84a70144d			3	59	إِنَّ مَثَلَ عِيسَىٰ عِندَ ٱللَّهِ كَمَثَلِ ءَ...	Aal-i-Imraan	89	Medinan	NaN	NaN
2912	33492	2267d836-d9a9-55b1-aef4-cd1e50e05964			7	184	أَوَلَمْ يَتَفَكَّرُوا۟ ۗ مَا بِصَاحِبِهِم مِّ...	Al-A'raaf	39	Meccan	Except 163-170, from Medina	NaN
2913	240772	f72c91e8-ae5d-56cd-ab44-51c02a573914			46	22	قَالُوٓا۟ أَجِئْتَنَا لِتَأْفِكَنَا عَنْ ءَالِ...	Al-Ahqaf	66	Meccan	Except 10, 15, 35, from Medina	NaN
2962	99217	65b40877-ee94-5afe-9d5f-ba84754e4129			84	24	فَبَشِّرْهُم بِعَذَابٍ أَلِيمٍ	Al-Inshiqaaq	83	Meccan	NaN	NaN
...	...	...	...	...	...	...	...	...	...	...	...	...
60207	210204	d7c198de-28dc-57b1-b139-9643d5fd0b65			22	9	ثَانِىَ عِطْفِهِۦ لِيُضِلَّ عَن سَبِيلِ ٱللَّه...	Al-Hajj	103	Medinan	Except 52-55, revealed between Mecca and Medina	NaN
60223	171714	b045c307-121f-5689-82f6-af1a342b20e4			42	30	وَمَآ أَصَـٰبَكُم مِّن مُّصِيبَةٍۢ فَبِمَا كَ...	Ash-Shura	62	Meccan	Except 23, 24, 25, 27, from Medina	NaN
60241	14148	0e797d45-2266-543d-9689-fa6938433de1			14	19	أَلَمْ تَرَ أَنَّ ٱللَّهَ خَلَقَ ٱلسَّمَـٰوَٰت...	Ibrahim	72	Meccan	Except 28, 29, from Medina	NaN
60261	3987	04202ef2-44ac-59d4-8039-a68907a2e3fa			87	18	إِنَّ هَـٰذَا لَفِى ٱلصُّحُفِ ٱلْأُولَىٰ	Al-A'laa	8	Meccan	NaN	NaN
60282	171307	afd876af-b230-5b66-959e-717cf96f5c99			50	16	وَلَقَدْ خَلَقْنَا ٱلْإِنسَـٰنَ وَنَعْلَمُ مَا...	Qaaf	34	Meccan	Except 38, from Medina	NaN

2527 rows × 12 columns