GRPO Fine-Tuning on DeepSeek-7B with Unsloth

DeepSeek has taken the world of natural language processing by storm. With its impressive scale and performance, this cutting-edge model excels in tasks like question answering and text summarization. Its ability to handle nuanced understanding makes it a game-changer across industries. Fine-tuning enhances its power, adapting it to niche needs and delivering precise results quickly. Fine-tuning transforms DeepSeek-7B from a generalist to a domain expert by refining it on specialized datasets. This blog explores how GRPO (General Reinforcement Pretraining Optimization) improves fine-tuning with reinforcement learning, and how Unsloth optimizes memory management, speeding up the process for large models like DeepSeek-7B. Together, these methods enable faster, cost-effective fine-tuning, driving next-gen AI applications.

Learning Objectives

By the end of this blog,  you should be able to:

• Learn fundamentals of fine-tuning DeepSeek-7B for enhanced performance on specialized tasks.
• Discover GRPO’s advantages over PPO, boosting training efficiency in fine-tuning.
• Use Unsloth and LoRA for fast, memory-efficient fine-tuning of large models.
• Set up DeepSeek-7B fine-tuning with Unsloth, vLLM, Hugging Face, and optimize GPU performance.
• Implement reward functions like correctness and XML for structured outputs in reinforcement learning.
• Load, save, and reload fine-tuned models using LoRA for memory-efficient, high-performance inference.
• Troubleshoot GPU memory and configuration issues for seamless fine-tuning.
• Explore scaling to larger datasets, new reward functions, and GRPO for multi-modal models.

This article was published as a part of the Data Science Blogathon.

Understanding DeepSeek Models & GRPO Algorithm

What is DeepSeek-R1-Distill-Qwen-7B?

DeepSeek-R1-Distill-Qwen-7B is a state-of-the-art large language model built on top of the Qwen architecture. With a robust and scalable design, it leverages billions of parameters to handle complex NLP tasks such as text generation, question answering, and summarization. The DeepSeek-7B variant is a distilled version of its larger counterparts, which means it retains much of the performance while being more efficient in terms of computation and memory usage. This makes it well-suited for deployment in environments where both inference speed and accuracy are critical. Its architecture employs transformer layers with self-attention mechanisms, making it highly effective in processing long-range dependencies in text.

Key Features and Architecture Overview

 At its core, DeepSeek-7B utilizes a multi-layer transformer architecture that is highly parallelizable, allowing for efficient training on large-scale datasets. Each layer consists of a series of multi-head self-attention modules and feedforward networks. The attention mechanism helps the model focus on relevant parts of the input sequence while processing, making it highly efficient for tasks requiring contextual understanding.

DeepSeek-7B processes token embeddings through positional encoding, attention layers, and a feed-forward layer, enabling efficient scaling to large datasets while maintaining high-quality results. Its deep context-aware understanding enhances generalization across domains after fine-tuning. Methods like LoRA improve training efficiency by applying low-rank updates, making fine-tuning feasible even with limited computational resources.

Introduction to GRPO and How It Improves Fine-Tuning

GRPO (General Reinforcement Pretraining Optimization) is an advanced technique designed to enhance the efficiency of fine-tuning large language models. It combines the principles of reinforcement learning with pretraining to refine the model’s behaviour using reward signals rather than direct supervision. GRPO optimizes the model’s parameters iteratively by using a policy-based optimization approach.

In a typical fine-tuning scenario, the model is trained on a supervised dataset, where it directly learns from ground truth labels. In contrast, GRPO introduces a reinforcement learning (RL) paradigm where the model is trained to maximize a reward signal that guides its behaviour. This process allows the model to adapt more flexibly to task-specific nuances, improving both accuracy and generalization.

The key formula for policy optimization in GRPO can be expressed as:

Where:

This policy-based approach ensures that the model continuously adapts to the feedback provided during training, focusing on improving the reward signal that corresponds to task-specific goals. 

GRPO’s Reward Signal

In GRPO, the reward function can be defined according to specific task requirements, guiding the model to focus on the desired behaviour. The reward can be a function of multiple factors, such as accuracy, formatting, or logical consistency. For instance, a correctness reward function R_correct could be defined as:

This feedback mechanism allows GRPO to progressively refine the model, emphasizing areas that matter most for the given task.

How GRPO Differs from PPO (Proximal Policy Optimization)?

While GRPO introduces policy-based reinforcement learning to optimize the pretraining process, PPO (Proximal Policy Optimization) is another widely used algorithm in reinforcement learning, particularly in the context of fine-tuning large models. PPO is known for its stability and ability to handle high-dimensional action spaces, making it popular for training large-scale models. However, PPO often requires a large amount of data and can be sensitive to hyperparameters like learning rate.

The key difference between GRPO and PPO lies in the nature of policy optimization. In PPO, the policy is updated using a clipped objective to prevent large deviations from the current policy, which can lead to unstable training. The PPO objective function is given by:

Where:

This “clipping” mechanism in PPO helps avoid large policy updates that could lead to instability, but it can also slow down the learning process, especially for large models like DeepSeek-7B.

The clipped objective ensures that the model doesn’t make large, unstable updates by penalizing large deviations in the policy. However, it also introduces a tradeoff between stability and learning speed, especially for larger models where the number of updates and the learning rate must be carefully tuned.

In contrast, GRPO uses a more adaptive and dynamic reward structure that allows it to directly maximize performance on task-specific metrics without relying on a “trust region” approach. The optimization procedure in GRPO doesn’t require clipping, and its reward-based learning mechanism provides a more direct and efficient route to fine-tuning. As a result, GRPO often requires fewer updates to converge to optimal performance.

Gradient Update Rule for the Parameters θ

The gradients for updating the model parameters in GRPO are computed by backpropagating the rewards through the model. If the reward R_t​ at time step t is calculated from the model output, the gradient update rule for the parameters θ is:

This gradient descent approach is more direct and efficient compared to the PPO clipping method, where the gradients are adjusted based on the advantage function. The key differences between PPO and the GRPO algorithm are summarised below:

Unsloth: Enhancing Efficiency in Fine-Tuning

Fine-tuning large language models like DeepSeek-7B is computationally expensive, requiring significant memory and processing power. Unsloth is an optimization framework designed to accelerate training while drastically reducing memory consumption. It is particularly beneficial when using LoRA (Low-Rank Adaptation) and GRPO, as it ensures efficient utilization of GPU resources and enables fine-tuning on consumer-grade hardware.

How Unsloth Optimizes Model Training?

Unsloth introduces several optimizations that improve model fine-tuning efficiency:

• Memory-Efficient Loading: Unsloth supports 4-bit and 8-bit quantization, reducing the memory footprint of models while maintaining performance.
• Fast Training and Inference: By leveraging Flash Attention and paged optimizers, Unsloth significantly accelerates both training and inference.
• Gradient Checkpointing: It supports gradient checkpointing, which reduces the GPU memory required by storing only a subset of activations and recomputing them when needed.
• Seamless Integration with LoRA: Unsloth natively supports LoRA, allowing users to train only a subset of model parameters instead of the entire network.

The model loading process using Unsloth is simple and enables efficient execution. Details of the same is covered in the subsequent section.

Advantages of Using Unsloth

• Reduces GPU memory usage by up to 50%, allowing training on mid-tier GPUs.
• Enables faster training by integrating optimized attention mechanisms.
• Supports vLLM (Very Large Language Models) for inference acceleration.
• Works seamlessly with GRPO, ensuring reinforcement learning-based fine-tuning is resource-efficient.

By incorporating Unsloth into the fine-tuning pipeline, researchers and engineers can maximize the performance of DeepSeek-7B without running into common computational limitations.

Fine-Tuning DeepSeek-7B with GRPO

Building upon the foundation we’ve laid in the previous sections, where we covered the architecture of DeepSeek-7B and the GRPO algorithm, it’s now time to delve into the practical steps required to fine-tune the model. This section will walk you through the necessary steps, from setting up the environment to configuring the GRPO Trainer, including code snippets and detailed explanations for each part of the process.

The DeepSeek-7B model, as discussed in Section 2, is a powerful tool for handling large-scale NLP tasks, and when paired with GRPO (General Reinforcement Pretraining Optimization), it becomes even more efficient. By applying the GRPO approach, we can fine-tune DeepSeek-7B on specific tasks using a reinforcement learning framework. This allows the model to not only produce better results but also adapt to new data more effectively than traditional methods.

Let’s now explore the detailed steps for fine-tuning DeepSeek-7B using GRPO and Unsloth, leveraging LoRA for efficient memory usage during training.

Step 1: Setting Up the Environment

To begin with, fine-tuning DeepSeek-7B, you need to set up the environment. This includes installing dependencies such as Unsloth, vllm, and other necessary packages. Here’s the command to install these packages:

!pip install unsloth vllm datasets
!pip install git+https://github.com/huggingface/trl.git

Explanation:

• Unsloth: A library for efficient language model fine-tuning and memory optimization.
• vllm: Enables fast inference for large models.
• Dataset: A library to work with various NLP datasets, including those from Hugging Face.

Once these are installed, we can proceed to load the model and start fine-tuning.

Step 2: Loading the Model with Unsloth

Now, we’ll load the DeepSeek-7B model using Unsloth. The model will be loaded with LoRA (Low-Rank Adaptation) for efficient fine-tuning. Here’s the code snippet for this step:

from unsloth import FastLanguageModel

model, tokenizer = FastLanguageModel.from_pretrained(
    model_name="unsloth/DeepSeek-R1-Distill-Qwen-7B",
    max_seq_length=512,
    load_in_4bit=True,  # Uses 4-bit quantization for memory efficiency
    fast_inference=True,  # Enables fast inference for quicker processing
    max_lora_rank=32,  # LoRA rank for fine-tuning efficiency
    gpu_memory_utilization=0.6  # Controls memory usage
)

Explanation:

• model_name: We specify the model to be loaded, in this case, DeepSeek-R1-Distill-Qwen-7B.
• max_seq_length: Defines the maximum sequence length for input tokens.
• load_in_4bit: Uses 4-bit quantization, significantly reducing memory usage.
• fast_inference: This enables vLLM to speed up inference times.
• max_lora_rank: The rank for LoRA adaptation, controlling the size of the low-rank matrices.
• gpu_memory_utilization: Adjusts how much GPU memory is used by the model to avoid out-of-memory errors.

Expected Outcome: The model will be loaded into memory with optimized configurations, ready for fine-tuning with LoRA.

Step 3: Applying LoRA for Efficient Fine-Tuning

LoRA is used to optimize memory for large models like DeepSeek-7B. By applying LoRA, we only update low-rank matrices instead of the entire model, which makes fine-tuning memory efficient. Here’s the code snippet:

model = FastLanguageModel.get_peft_model(
    model,
    r=32,  # Rank of LoRA layers, which controls memory and efficiency
    target_modules=["q_proj", "k_proj", "v_proj", "o_proj", "gate_proj", 
    "up_proj", "down_proj"],  # Modules to apply LoRA to
    lora_alpha=32,  # Scaling factor for LoRA
    use_gradient_checkpointing="unsloth",  # Enables gradient checkpointing 
    for long context fine-tuning
    random_state=3407  # Seed for reproducibility
)

Explanation:

• r: The rank of the LoRA matrix. A higher rank can lead to smarter but slower training.
• target_modules: The model layers where LoRA is applied (e.g., q_proj for query projection).
• lora_alpha: The scaling factor used to control the importance of the LoRA layers.
• use_gradient_checkpointing: This reduces memory consumption by only storing intermediate gradients when needed.
• random_state: Ensures reproducibility of the fine-tuning process.

Expected Outcome:
The model is now optimized for memory usage and can be efficiently fine-tuned on large datasets.

Step 4: Preparing the Training Dataset

Fine-tuning DeepSeek-7B requires a dataset formatted in a specific way. Here, we’ll load and transform the dataset from a JSON file format to a Hugging Face Dataset object. Here’s the code:

import json
from datasets import Dataset

def load_and_transform_json(json_path):
    with open(json_path, "r") as f:
        data = json.load(f)
    transformed_data = [{"question": entry["question"], "answer": entry["response"], "prompt": [{"content": SYSTEM_PROMPT, "role": "system"}, {"content": entry["question"], "role": "user"}]} for entry in data]
    return transformed_data

json_file_path = "/content/your_dataset.json"  # Path to your JSON file
dataset = load_and_transform_json(json_file_path)

Explanation:

• load_and_transform_json: Loads a JSON file and transforms it into the required format for training.
• The data includes a question and answer for each entry, along with a system-generated prompt.

Expected Outcome: The dataset is now in the correct format and ready for training. Below is one sample of the dataset.

Step 5: Designing Reward Functions for Structured Output

In reinforcement learning, reward functions guide the model toward desirable outputs. Here, we define reward functions to evaluate the model’s response. For instance, the correctness_reward_func checks if the extracted answer matches the expected answer.

def correctness_reward_func(prompts, completions, answer, **kwargs) -> list[float]:
    responses = [completion[0]['content'] for completion in completions]
    q = prompts[0][-1]['content']
    extracted_responses = [extract_xml_answer(r) for r in responses]
    return [2.0 if r == a else 0.0 for r, a in zip(extracted_responses, answer)]

def int_reward_func(completions, **kwargs) -> list[float]:
    responses = [completion[0]['content'] for completion in completions]
    extracted_responses = [extract_xml_answer(r) for r in responses]
    return [0.5 if r.isdigit() else 0.0 for r in extracted_responses]

def strict_format_reward_func(completions, **kwargs) -> list[float]:
    pattern = r"^<reasoning>\n.*?\n</reasoning>\n<answer>\n.*?\n</answer>\n$"
    responses = [completion[0]["content"] for completion in completions]
    matches = [re.match(pattern, r) for r in responses]
    return [0.5 if match else 0.0 for match in matches]

def soft_format_reward_func(completions, **kwargs) -> list[float]:
    pattern = r"<reasoning>.*?</reasoning>\s*<answer>.*?</answer>"
    responses = [completion[0]["content"] for completion in completions]
    matches = [re.match(pattern, r) for r in responses]
    return [0.5 if match else 0.0 for match in matches]

def xmlcount_reward_func(completions, **kwargs) -> list[float]:
    contents = [completion[0]["content"] for completion in completions]
    return [count_xml(c) for c in contents]

Explanation:

• correctness_reward_func: Compares the extracted response with the expected answer. If they match, it gives a reward of 2.0, else 0.0.
• int_reward_func: Rewards the model for producing numeric responses.
• strict_format_reward_func: Ensures that the model’s output follows a strict XML format, rewarding it for well-formed outputs.
• soft_format_reward_func: Checks if the model’s output loosely adheres to the desired format.
• xmlcount_reward_func: Evaluates how well the output follows the XML structure, with a penalty for poorly structured responses.

Expected Outcome:
These reward functions guide the model toward producing responses that are not only correct but also well-structured and in the desired format.

Step 6: Configuring the GRPO Trainer

Now, we’ll configure the GRPOTrainer to use the training dataset and reward functions. The GRPOConfig object is used to specify training parameters like learning rate and batch size.

from trl import GRPOConfig, GRPOTrainer

training_args = GRPOConfig(
    learning_rate=5e-6,
    per_device_train_batch_size=1,
    num_generations=6,
    max_prompt_length=256,
    max_completion_length=200,
    max_steps=1,
)

trainer = GRPOTrainer(
    model=model,
    processing_class=tokenizer,
    reward_funcs=[correctness_reward_func],
    args=training_args,
    train_dataset=dataset,
)
trainer.train()

Explanation:

• GRPOConfig: Configures various training parameters like learning rate, batch size, and the number of generations to be produced.
• GRPOTrainer: This class is responsible for the actual training process. It takes in the model, tokenizer, reward functions, and training arguments.

Explanation of GRPOConfig Parameters:

• learning_rate: The learning rate for model optimization. A lower value like 5e-6 allows for stable training over many iterations.
• per_device_train_batch_size: Batch size for each training step. Here, it’s set to 1, meaning each GPU will process one example at a time.
• num_generations: Number of generations produced by the model during each fine-tuning step.
• max_prompt_length: Maximum token length for the input prompt.
• max_completion_length: Maximum token length for the model’s output.
• max_steps: The number of training steps to perform.

Expected Outcome:
The model will be trained with the GRPO algorithm using the defined reward functions, fine-tuning the model to perform better on the given dataset.

Saving and Reloading the Fine-Tuned Model

Once the DeepSeek-7B model has been fine-tuned using GRPO and LoRA, it’s important to save the model to disk or cloud storage for future use. In this section, we’ll cover how to save the fine-tuned model and load it again for inference. This ensures that you can persist your progress and avoid retraining from scratch.

Saving the LoRA-Fine-Tuned Model

After the model has been fine-tuned with LoRA and GRPO, you need to save it to a storage location. This is a crucial step to ensure that you can reload the model later without needing to retrain. Here’s how you can save the fine-tuned model, including the LoRA-specific weights, to disk:

# Define the path to save the fine-tuned model
model_save_path = "/content/deepseek_lora_finetuned"

# Save the model and tokenizer
model.save_pretrained(model_save_path)
tokenizer.save_pretrained(model_save_path)

Explanation:

• model.save_pretrained: This saves both the model weights and LoRA-specific layers (such as the low-rank adaptation matrices).
• tokenizer.save_pretrained: Saves the tokenizer, which includes tokenization logic like special tokens and vocabulary.
• model_save_path: The directory where you want to store the model. This can be a local path or a cloud directory (e.g., Google Drive, S3).

Expected Outcome:
The model and tokenizer will be saved to the specified path, making them available for future use. You can later use this saved model to reload the exact fine-tuned version for inference without needing to retrain.

Loading the Model for Future Inference

Once you’ve saved the fine-tuned model, you can easily load it back into memory for inference or further fine-tuning. Here’s the code for loading the saved model and tokenizer, along with the LoRA-specific configuration:

from unsloth import FastLanguageModel

# Define the path where the model is saved
model_save_path = "/content/deepseek_lora_finetuned"

# Reload the model and tokenizer
model, tokenizer = FastLanguageModel.from_pretrained(
    model_save_path,
    max_seq_length=512,
    load_in_4bit=True,  # Ensure it's still using efficient memory settings
    fast_inference=True,  # Enable fast inference
    max_lora_rank=32,  # LoRA rank must match what was used during fine-tuning
    gpu_memory_utilization=0.6
)

Explanation:

• FastLanguageModel.from_pretrained: This function loads the saved model weights and tokenizer from the specified path.
• max_lora_rank: The LoRA rank used during inference must match what was used during fine-tuning to ensure the correct adaptation is applied.
• load_in_4bit and gpu_memory_utilization: Ensures that the model continues to be memory-efficient when loaded for inference.

Expected Outcome:
The model is loaded from the saved directory, along with its LoRA configurations, allowing you to perform inference efficiently. This means the model will leverage the fine-tuned parameters, and you can directly start generating responses or running tasks without reapplying the fine-tuning process.

Below is an example of the output on the dataset used to fine-tune this blog. It was related to process flowsheeting. See how the model reasons and generates the responses to the query. Fine-tuning with the GRPO model incorporates reasoning capabilities, which is reflected in the answer below.

Advanced Option: Saving to Cloud Storage

If you want to save the model to cloud storage (like Google Drive or Amazon S3), you can modify the model_save_path to point to the respective cloud directory. Here’s an example for saving to Google Drive using gdown:

!pip install gdown

import gdown

# Upload the model to Google Drive
gdown.upload(model_save_path, output="path_to_google_drive_folder")

For Amazon S3, you can use the boto3 library to upload the model:

!pip install boto3

import boto3

s3 = boto3.client('s3')

# Upload model to S3
s3.upload_file("/content/deepseek_lora_finetuned", "your-bucket-name", 
"model_directory/deepseek_lora_finetuned")

Explanation:

• gdown.upload: This function uploads the model from your local environment to Google Drive.
• boto3: Amazon’s Python SDK for interacting with AWS services like S3. It allows you to upload your model directly to an S3 bucket.

Expected Outcome:
You can save and access the model from the cloud, making it easy to share and deploy on other environments.

Common Pitfalls and Troubleshooting

When fine-tuning large models like DeepSeek-7B, several common pitfalls can arise, particularly related to GPU memory, training configurations, and reward function tuning. Being aware of these issues and understanding how to troubleshoot them can save a lot of time during the fine-tuning process.

1. GPU Memory Overload

Fine-tuning large models often leads to GPU memory overload, especially when using advanced configurations like LoRA or training with high batch sizes. To mitigate this:

• Reduce batch size or adjust the per_device_train_batch_size parameter in GRPOConfig to fit within your GPU’s memory.
• Use gradient checkpointing by setting use_gradient_checkpointing = “unsloth”, which stores intermediate activations to reduce memory usage.
• Lower the LoRA rank if you encounter memory issues—lower ranks demand less memory.

2. Improper Model Loading

Sometimes, incorrect model loading configurations can cause issues, particularly when loading large models in 4-bit precision or with LoRA. Be sure to:

• Verify that the LoRA rank and other model-specific configurations (like max_lora_rank and gpu_memory_utilization) are correctly set based on your GPU’s capabilities.
• Ensure that vLLM is enabled for fast inference when working with large models to avoid unnecessary delays.

3. Reward Function Mismatches

Fine-tuning with reward functions requires careful consideration. Incorrect or overly strict reward function configurations may hinder learning, making the model perform sub-optimally. To troubleshoot:

1. Review the implementation of reward functions like correctness_reward_func and strict_format_reward_func to ensure they align with your desired output.
2. Fine-tune reward thresholds and scoring mechanisms if the model produces erratic or undesired responses.

4. Data Issues

Data quality and formatting are crucial for successful training. If you’re using custom datasets, transform them into the Hugging Face Dataset format and ensure proper parsing and pre-processing of any JSON-based input. Always check the dataset for any discrepancies or missing fields, especially in complex reward functions like correctness_reward_func, which depends on precise answer matching.

5. Training Configuration Conflicts

Conflicts in training configurations, such as mismatched learning rates, optimizer settings, or gradient accumulation steps, can lead to suboptimal performance or slower convergence. Always ensure that the parameters in GRPO Config are fine-tuned according to the specific requirements of your hardware and training objective. Additionally, a low learning rate with high gradient accumulation steps can help stabilize training for very large models.

By addressing these common pitfalls and monitoring memory usage, data formatting, and reward function effectiveness, you can streamline the fine-tuning process and ensure smoother model training.

BONUS: By now, are you excited to start experimenting with the latest DeepSeek model? Feel free to use the notebook for this blog and develop it for your use case!

Conclusion

In this guide, we explored the process of GRPO Fine-Tuning on DeepSeek-7B (General Reinforcement Pretraining Optimization) and LoRA (Low-Rank Adaptation), combining the strengths of these technologies to optimize large model training. We began by discussing the architecture of DeepSeek-7B and GRPO, outlining the role of Unsloth in memory management and efficient model training. We also demonstrated the practical steps involved, from setting up the environment and loading the model with LoRA to applying reinforcement learning-based reward functions for fine-tuning.

Effective fine-tuning combines GRPO and LoRA: GRPO enhances learning via policy-based updates, while LoRA enables memory-efficient training. We demonstrated defining reward functions, optimizing with GRPOTrainer, and ensuring model usability through saving and reloading. Key challenges include scaling to larger datasets and refining reward functions for better adaptability. Expanding GRPO to multi-modal models could further advance AI capabilities.

Key Takeaways

• DeepSeek-7B and GRPO provide a powerful foundation for fine-tuning large-scale models with reinforcement learning-based optimization.
• LoRA optimizes memory usage and enables efficient fine-tuning on large models by applying low-rank adaptations.
• GRPO differs from traditional methods like PPO by offering policy-based updates, leading to more efficient training.
• Defining well-structured reward functions is crucial in reinforcement learning fine-tuning, guiding the model towards high-quality outputs.
• The process of saving and reloading fine-tuned models ensures reusability and long-term model performance.
• Future improvements can focus on scaling to larger datasets, experimenting with new reward functions, and applying GRPO to multi-modal models (text, images, audio).

The media shown in this article is not owned by Analytics Vidhya and is used at the Author’s discretion.

Frequently Asked Questions

Neil D

Advancing language model research by day and writing about my work online by night. I explore AI breakthroughs and transform complex studies into clear, engaging insights that empower professionals and enthusiasts alike.

Thanks for stopping by my profile!