Comparison of Gemini Embedding with Multilingual-e5-large & Jina

Word embeddings for Indic languages like Hindi are crucial for advancing Natural Language Processing (NLP) tasks such as machine translation, question answering, and information retrieval. These embeddings capture semantic properties of words, enabling more accurate and context-aware NLP applications. Given the vast number of Hindi speakers and the growing digital content in Indic languages, high-quality embeddings are essential for improving NLP performance in these languages. Customized embeddings can particularly address the unique linguistic features and resource constraints of Indic languages. The newly released Gemini Embedding model represents a significant advancement in multilingual text embeddings, leveraging Google’s powerful Gemini AI framework to deliver state-of-the-art performance across over 100 languages.

Gemini Embedding model excels in tasks such as classification, retrieval, and semantic search, offering enhanced efficiency and accuracy. By supporting larger input sizes and higher-dimensional outputs, Gemini Embedding provides richer text representations, making it highly versatile for diverse applications.

Learning Objectives

• Introduction to Gemini Embeddings and their integration with the Gemini LLM.
• Hands-on tutorial on retrieving Hindi documents using Gemini Embeddings.
• Comparative analysis with Jina AI embeddings and Multilingual-e5-large.
• Insights into capabilities and applications in multilingual text retrieval.

This article was published as a part of the Data Science Blogathon.

What are Gemini Embeddings?

In March 2025, Google released a new experimental Gemini Embedding text model (gemini-embedding-exp-03-07) available in the Gemini API.
Developed from the Gemini model, this advanced embedding model is claimed to have inherited Gemini’s profound grasp of language and subtle contextual nuances, rendering it versatile for diverse applications. It has grabbed the top position on the MTEB Multilingual leaderboard.

Gemini Embedding represents text as dense vectors where semantically similar text inputs
are mapped to vectors near one another in the vector space. Currently it supports more than 100+
languages, and its embeddings can be used for various tasks such as retrieval and classification.

Key Features of Gemini Embeddings

• Robust multilingual capabilities: model showcases outstanding performance in more than 100 languages, excelling not only in high-resource languages like English but also in low-resource languages such as Assamese and Macedonian.
• Handle upto 8000 Input Tokens: This substantial capacity enables the model to seamlessly handle lengthy documents or intricate queries without truncation, thereby maintaining context and meaning in a manner that surpasses many existing embedding models.
• Output dimensions of 3K dimensions: The model generates embeddings with a dimensionality of up to 3,072, offering support for sub-dimensions like 768 and 1,536 to allow for task-specific optimization.
• Impressive Performance. Gemini Embedding tops the Massive Text Embedding Benchmark (MTEB) with a mean task score of 68.32, surpassing its nearest competitors by a substantial margin.

Model Architecture of Gemini Embeddings

At its core, Gemini Embedding is built on a transformer architecture, initialized from the Gemini LLM. This foundation provides the model with a deep understanding of language structure and semantics. The model uses bidirectional attention mechanisms to process input sequences, allowing it to consider the full context of a word or phrase when generating embeddings.

1. An input sequence T of 𝐿 tokens is processed by M, a transformer with bidirectional attention
   initialized from Gemini, producing a sequence of token embeddings.
2. To generate a single embedding representing all the information in the input, a pooling function is applied
3. Finally, a linear projection is applied to scale the embedding to the target dimension, resulting in the final output embedding.

Loss Function. The Gemini Embedding model was trained with a noise-contrastive estimation (NCE) loss with in-batch negatives. The exact loss differs slightly depending on the stage of training. In general, a training example includes a query, a positive target and (optionally) a hard negative target.

Training Strategy

1. Pre-Finetuning: During this stage, the model is trained on a vast and varied dataset comprising query-target pairs. This exposure tunes the large language model’s parameters for encoding tasks, establishing a foundation for its adaptability.
2. Fine-Tuning: In the second stage, the model undergoes fine-tuning using task-specific datasets that include query-positive-hard negative triples. This process employs smaller batch sizes and meticulously curated datasets to boost performance on targeted tasks.

Also Read: Gemini Embedding: Generalizable Embeddings from Gemini

Comparison with Other Multilingual Embedding Models

We compare the retrieval from Hindi Documents with the newly released state of the art Gemini Embeddings and then compare it against Jina AI Embeddings & Multilingual-e5-large embeddings. As shown in the Table below, with respect to the number of max tokens, Gemini Embeddings and Jina AI embeddings are high, enabling the models to handle long documents or intricate queries. In terms. Also as seen from the Table below, the Gemini embeddings have a higher embedding dimension that can capture more nuanced and fine-grained semantic relationships between words, enabling models to represent complex linguistic patterns and subtle distinctions in meaning.

Retrieval with Gemini Embeddings & Comparison with Jina AI Embeddings & Multilingual-e5-large

In the following hands on tutorial, we will compare retrieval from Hindi Documents with the newly released state of the art Gemini Embeddings and then compare it against Jina AI Embeddings & Multilingual-e5-large embeddings.

Step 1. Install Necessary Libraries

!pip install langchain-community
!pip install chromadb

Step 2. Loading the Data

We use Hindi data from a website to assess how the Gemini Embeddings perform with respect to retrieval in Hindi Language.

from langchain_community.document_loaders import WebBaseLoader

loader = WebBaseLoader("https://ckbirlahospitals.com/rbh/blog/pregnancy-early-symptoms-in-hindi")
data = loader.load()

Step 3. Chunking the Data

The code below uses the RecursiveCharacterTextSplitter to split large text documents into smaller chunks of 500 characters each, with no overlap. It then applies this splitting to the data variable and stores the results in all_splits. We use only 10 of the splits because of the rate limits in Gemini Embedding API.

from langchain_text_splitters import RecursiveCharacterTextSplitter

text_splitter = RecursiveCharacterTextSplitter(chunk_size=500, chunk_overlap=0)
all_splits = text_splitter.split_documents(data)
all_splits = all_splits[:10]

Step 4. Storing the Data in a Vector DB

We first create a class “GeminiEmbeddingFunction” that helps in querying the Gemini Embedding API and returns the values of the embeddings for the input query. We then create a function “create_chroma_db” for creating a collection in ChromaDB that will store the data along with the embeddings.

import chromadb
from chromadb import Documents, EmbeddingFunction, Embeddings

class GeminiEmbeddingFunction(EmbeddingFunction):
  def __call__(self, input: Documents) -> Embeddings:
    title = "Custom query"  
    return client.models.embed_content(
        model="gemini-embedding-exp-03-07",
        contents=input).embeddings[0].values
        
 

def create_chroma_db(documents, name):
  chroma_client = chromadb.Client()
  db = chroma_client.create_collection(name=name, embedding_function=GeminiEmbeddingFunction())
  for i, d in enumerate(documents):
    db.add(
      documents=d.page_content,
      ids=str(i)
    )
  return db

db = create_chroma_db(all_splits, "datab")

Step 5. Querying the DB

def get_relevant_passage(query, db):
  passage = db.query(query_texts=[query], n_results=1)['documents'][0][0]

  return passage

passage = get_relevant_passage("आपको प्रेगनेंसी टेस्ट कब करवाना चाहिए?", db)
print(passage)

Step 6. Comparing with Jina AI Embeddings

The code below defines a custom embedding function using a Hugging Face transformer model and a method for processing text inputs to generate embeddings.

1. The AutoTokenizer and AutoModel from transformers are used to load a pretrained model (jinaai/jina-embeddings-v3) and EmbeddingFunction from chromadb is imported for creating custom embeddings.
2. average_pool function: This function aggregates the hidden states from the model by performing a pooling operation on them, averaging over the sequence length while considering the attention mask (ignoring padding tokens).
3. CustomHuggingFace class: It tokenizes the text, feeds it through the model, and computes the embeddings using the average_pool function. The result is returned as a list of embeddings.

from transformers import AutoTokenizer, AutoModel
from chromadb import EmbeddingFunction


tokenizer = AutoTokenizer.from_pretrained('jinaai/jina-embeddings-v3')
model = AutoModel.from_pretrained('jinaai/jina-embeddings-v3')


# the model returns many hidden states per document so we must aggregate them
def average_pool(last_hidden_states, attention_mask):
    last_hidden = last_hidden_states.masked_fill(~attention_mask[...,None].bool(), 0.0)
    return last_hidden.sum(dim=1) / attention_mask.sum(dim=1)[...,None]

class CustomHuggingFace(EmbeddingFunction):
    def __call__(self, texts):
        queries    = [f'query: {text}' for text in texts]         
        batch_dict = tokenizer(texts, max_length=512, padding=True, truncation=True, return_tensors='pt')
        outputs    = model(**batch_dict)        
        embeddings = average_pool(outputs.last_hidden_state, batch_dict['attention_mask'])
        return embeddings.tolist()

Querying

def get_relevant_passage(query, db):
  passage = db.query(query_texts=[query], n_results=1)['documents'][0][0]

  return passage

passage = get_relevant_passage("आपको प्रेगनेंसी टेस्ट कब करवाना चाहिए?", db)
print(passage)

  For selecting the Multilingual-e5-large embeddings, we can just replace the tokenizer and model as “intfloat/multilingual-e5-large-instruct”  

Comparison of Outputs in Retrieval From the Embeddings

Explanation

As seen from the above Hindi outputs, with Gemini Embeddings, we get 5 correct outputs from the 7 queries, while Jina AI embeddings and Multilingual-e5-large, we get 3 correct responses only.

This shows that Gemini Embeddings, as reflective on the MTEB benchmark, can perform well and better than other embeddings models for multilingual languages as well like Hindi.

Conclusions

In conclusion, Gemini Embeddings represent a significant advancement in multilingual NLP, particularly for Indic languages like Hindi. With its robust multilingual capabilities, support for large input sizes, and superior performance on benchmarks like MTEB, Gemini excels in tasks such as retrieval, classification, and semantic search. As demonstrated through hands-on comparisons, Gemini outperforms other models, offering enhanced accuracy and efficiency, making it a valuable tool for advancing NLP in diverse languages.

Key Takeaways

• Importance of Word Embeddings for Indic Languages: High-quality embeddings enhance NLP tasks like translation, QA, and retrieval, addressing linguistic challenges and resource gaps.
• Gemini Embedding Model: Google’s Gemini Embeddings leverage its AI framework for multilingual text processing, covering 100+ languages, including low-resource ones.
• Key Features: Supports 8,000 tokens and 3,072-dimensional embeddings, handling long documents and complex queries efficiently.
• Impressive Performance: Tops the MTEB Multilingual leaderboard with a 68.32 mean task score, proving its superiority in multilingual NLP.

The media shown in this article is not owned by Analytics Vidhya and is used at the Author’s discretion.

Frequently Asked Questions

Nibedita Dutta

Nibedita completed her master’s in Chemical Engineering from IIT Kharagpur in 2014 and is currently working as a Senior Data Scientist. In her current capacity, she works on building intelligent ML-based solutions to improve business processes.