Running OLMo-2 Locally with Gradio and LangChain

Natural Language Processing has grown quickly in recent years. While private models have been leading the way, open-source models have been catching up. OLMo 2 is a big step forward in the open-source world, offering power and accessibility similar to private models. This article provides a detailed discussion of OLMo 2, covering its training, performance, and how to use it locally.

Learning Objectives

• Understand the significance of open-source LLMs and OLMo 2’s role in AI research.
• Explore OLMo 2’s architecture, training methodology, and performance benchmarks.
• Differentiate between open-weight, partially open, and fully open models.
• Learn how to run OLMo 2 locally using Gradio and LangChain.
• Implement OLMo 2 in a chatbot application with Python code examples.

This article was published as a part of the Data Science Blogathon.

Understanding the Need for Open-Source LLMs

The initial dominance of proprietary LLMs created concerns about accessibility, transparency, and control. Researchers and developers were limited in their ability to understand the inner workings of these models, thus hindering further innovation and possibly perpetuating biases. Open-source LLMs have addressed these concerns by providing a collaborative environment where researchers can scrutinize, modify, and improve upon existing models. An open approach is crucial for advancing the field and ensuring that the benefits of LLMs are widely available.

OLMo, initiated by the Allen Institute for AI (AI2), has been at the forefront of this movement. With the release of OLMo 2, they have solidified their commitment to open science by providing not just the model weights, but also the training data, code, recipes, intermediate checkpoints, and instruction-tuned models. This comprehensive release enables researchers and developers to fully understand and reproduce the model’s development process, paving the way for further innovation. Running OLMo 2 Locally with Gradio and LangChain

What is OLMo 2?

OLMo 2 marks a significant upgrade from its forefather, the OLMo-0424. The novel family of parameter models 7B and 13B showcase comparable performance or sometimes better-than-similar fully open models while competing with an open-weight version such as Llama 3.1 over English academic benchmarks. This makes the achievement very remarkable given a reduced total amount of training FLOPs relative to some similar models.

• OLMo-2 Shows Significant Improvement: The OLMo-2 models (both 7B and 13B parameter versions) demonstrate a clear performance jump compared to the earlier OLMo models (OLMo-7B, OLMo-7B-0424, OLMOE-1B-7B-0924). This suggests substantial progress in the model’s architecture, training data, or training methodology.
• Competitive with MAP-Neo-7B: The OLMo-2 models, especially the 13B version, achieve scores comparable to MAP-Neo-7B, which was likely a stronger baseline among the fully open models listed.

Breaking Down OLMo 2’s Training Process

OLMo 2’s architecture builds upon the foundation of the original OLMo, incorporating several key changes to enhance training stability and performance. 

The pretraining process for OLMo 2 is divided into two stages:

• Stage 1: Foundation Training: This stage utilizes the OLMo-Mix-1124 dataset, a massive collection of approximately 3.9 trillion tokens sourced from various open datasets. This stage focuses on building a strong foundation for the model’s language understanding capabilities.
• Stage 2: Refinement and Specialization: This stage employs the Dolmino-Mix-1124 dataset, a curated mixture of high-quality web data and domain-specific data, including academic content, Q&A forums, instruction data, and math workbooks. This stage refines the model’s knowledge and skills in specific areas. The use of “model souping” to combine multiple trained models further enhances the final checkpoint.

As OLMO-2 is Fully Open Model, Let’s see what is the difference between Open Weight Models, Partially Open Models and Fully Open Models:

Open Weight Models

Llama-2-13B, Mistral-7B-v0.3, Llama-3.1-8B, Mistral-Nemo-12B, Qwen-2.5-7B, Gemma-2-9B, Qwen-2.5-14B: These models share a key trait: their weights are publicly available. This allows developers to use them for various NLP tasks. However, critical details about their training process, such as the exact dataset composition, training code, and hyperparameters, are not fully disclosed. This makes them “open weight,” but not fully transparent.

Partially Open Models

StableLM-2-128, Zamba-2-7B: These models fall into a gray area. They offer some additional information beyond just the weights, but not the full picture. StableLM-2-128, for example, lists training FLOPS, suggesting more transparency than purely open-weight models. However, the absence of complete training data and code places it in the “partially open” category.

Fully Open Models

Amber-7B, OLMo-7B, MAP-Neo-7B, OLMo-0424-7B, DCLM-7B, OLMo-2-1124-7B, OLMo-2-1124-13B: These models stand out due to their comprehensive openness. AI2 (Allen Institute for AI), the organization behind the OLMo series, has released everything necessary for full transparency and reproducibility: weights, training data (or detailed descriptions of it), training code, the full training “recipe” (including hyperparameters), intermediate checkpoints, and instruction-tuned versions. This allows researchers to deeply analyze these models, understand their strengths and weaknesses, and build upon them.

Key Differences

Explore OLMo 2

OLMo 2 is an advanced open-source language model designed for efficient and powerful AI-driven conversations. It integrates seamlessly with frameworks like LangChain, enabling developers to build intelligent chatbots and AI applications. Explore its capabilities, architecture, and how it enhances natural language understanding in various use cases.

• Get the Model and Data: Download Here
• Training Code: View
• Evaluation: View

Let’s Run It Locally

Download Ollama here.

To Download Olmo-2 open Cmd and Type

ollama run olmo2:7b

This will download Olmo2 in your system

Install Libraries 

pip install langchain-ollama
pip install gradio

Building a Chatbot with OLMo 2

Leverage the power of OLMo 2 to build an intelligent chatbot with open-weight LLM capabilities. Learn how to integrate it with Python, Gradio, and LangChain for seamless interactions.

Step1: Importing Required Libraries

Load essential libraries, including Gradio for UI, LangChain for prompt handling, and OllamaLLM for leveraging the OLMo 2 model in chatbot responses.

import gradio as gr
from langchain_core.prompts import ChatPromptTemplate
from langchain_ollama.llms import OllamaLLM

Step2: Defining the Response Generation Function

Create a function that takes chat history and user input, formats the prompt, invokes the OLMo 2 model, and updates the conversation history with AI-generated responses.

def generate_response(history, question):
    template = """Question: {question}

    Answer: Let's think step by step."""
    prompt = ChatPromptTemplate.from_template(template)
    model = OllamaLLM(model="olmo2")
    chain = prompt | model
    answer = chain.invoke({"question": question})
    history.append({"role": "user", "content": question})
    history.append({"role": "assistant", "content": answer})
    return history

The generate_response function takes a chat history and a user question as input. It defines a prompt template where the question is inserted dynamically, instructing the AI to think step by step. The function then creates a ChatPromptTemplate and initializes the OllamaLLM model (olmo2). Using LangChain’s pipeline (prompt | model), it generates a response by invoking the model with the provided question. The conversation history is updated, appending the user’s question and AI’s answer. It returns the updated history for further interactions.

Step3: Creating the Gradio Interface

Use Gradio’s Blocks, Chatbot, and Textbox components to design an interactive chat interface, allowing users to input questions and receive responses dynamically.

with gr.Blocks() as iface:
    chatbot = gr.Chatbot(type='messages')
    with gr.Row():
        with gr.Column():
            txt = gr.Textbox(show_label=False, placeholder="Type your question here...")
    txt.submit(generate_response, [chatbot, txt], chatbot)

• Uses gr.Chatbot() for displaying conversations.
• Uses gr.Textbox() for user input.

Step4: Launching the Application

Run the Gradio app using iface.launch(), deploying the chatbot as a web-based interface for real-time interactions.

iface.launch()

This starts the Gradio interface and runs the chatbot as a web app.

Get Code from GitHub Here.

Output

Prompt

Write a Python function that returns True if a given number is a power of 2 without using loops or recursion.

Response

Conclusion

Therefore, OLMo-2 stands out as one of the largest contributions to the open-source LLM ecosystem. It is one of the most powerful performer in the arena of full transparency, with focus on training efficiency. It reflects the growing significance of open collaboration in the world of AI and will pave the way for future progress in accessible and transparent language models.

While OLMo-2-138 is a very strong model, it’s not distinctly dominating on all tasks. Some partially open models and Qwen-2.5-14B, for instance, obtain higher scores on some benchmarks (for example, Qwen-2.5-14B significantly outperforms on ARC/C and WinoG). Besides, OLMo-2 lags significantly behind the very best models at particular challenging tasks like GSM8k (grade school math) and probably AGIEval.

Unlike many other LLMs, OLMo-2 is fully open, providing not only the model weights but also the training data, code, recipes, and intermediate checkpoints. This level of transparency is crucial for research, reproducibility, and community-driven development. It allows researchers to thoroughly understand the model’s strengths, weaknesses, and potential biases.

Key Takeaway

• The OLMo-2 models, especially the 13B parameter version, are showing great performance results on a host of benchmarks, beating other open-weight and even partially open architectures. It appears that full openness is indeed one of the ways to make powerful LLMs.
• The Fully Open models (particularly OLMo) tend to perform well. This supports the argument that having access to the full training process (data, code, etc.) facilitates the development of more effective models.
• The chatbot maintains conversation history, ensuring responses consider previous interactions.
• Gradio’s event-based UI (txt.submit) updates in real-time, making the chatbot responsive and user-friendly.
• OllamaLLM integrates AI models into the pipeline, enabling seamless question-answering functionality.

Frequently Asked Questions

The media shown in this article is not owned by Analytics Vidhya and is used at the Author’s discretion.

Gourav Lohar

Hi I'm Gourav, a Data Science Enthusiast with a medium foundation in statistical analysis, machine learning, and data visualization. My journey into the world of data began with a curiosity to unravel insights from datasets.