Pixtral-12B: Mistral AI’s First Multimodal Model

Introduction

Mistral has released its very first multimodal model, namely the Pixtral-12B-2409. This model is built upon Mistral’s 12 Billion parameter, Nemo 12B. What sets this model apart? It can now take both images and text for input. Let’s look more at the model, how it can be used, how well it’s performing the tasks and the other things you need to know.

In this article, you will learn about the Pixtral-12B model. This AI model uses deep learning and a special type of network to create images. We will look at how it works, its uses in machine learning, and how it compares to GPT-3. You’ll also see why its performance is so impressive.

Overview

1. Discover Mistral’s new Pixtral-12B, a multimodal model combining text and image processing for versatile AI applications.
2. Learn how to use Pixtral-12B, Mistral’s latest AI model, designed to handle both text and high-resolution images.
3. Explore the capabilities and use cases of the Pixtral-12B model, featuring a vision adapter for enhanced image understanding.
4. Understand Pixtral-12B’s multimodal features and its potential applications in image captioning, story generation, and more.
5. Get insights into Pixtral-12B’s design, performance, and how to fine-tune it for specific multimodal tasks.

What is Pixtral-12B?

Pixtral-12B is a multimodal model derived from Mistral’s Nemo 12B, with an added 400M-parameter vision adapter. Mistral can be downloaded from a torrent file or on Hugging Face with an Apache 2.0 license. Let’s look at some of the technical features of the Pixtral-12B model:

How to Use Pixtral-12B-2409?

As of September 15th, 2024, the model is currently not available on Mistral’s Le Chat or La Plateforme to use the chat interface directly or access it through API, but we can download the model through a torrent link and use it or even finetune the weights to suit our needs. We can also use the model with the help of Hugging Face. Let’s look at them in detail:

Torrent link to Use:

magnet:?xt=urn:btih:7278e625de2b1da598b23954c13933047126238a&dn=pixtral-12b-
240910&tr=udp%3A%2F%http://2Ftracker.opentrackr.org%3A1337%2Fannounce&tr=udp%
3A%2F%http://2Fopen.demonii.com%3A1337%2Fannounce&tr=http%3A%2F%http://2Ftrac
ker.ipv6tracker.org%3A80%2Fannounce

I’m using an Ubuntu laptop, so I’ll use the Transmission application (it’s pre-installed in most Ubuntu computers). You can use any other application to download the torrent link for the open-source model.

• Click “File” at the top left and select the open URL option. Then, you can paste the link that you copied.

• You can click “Open” and download the Pixtral-12B model. The folder will be downloaded which contains these files:

Hugging Face

This model demands a high GPU, so I suggest you use the paid version of Google Colab or Jupyter Notebook using RunPod. I’ll be using RunPod for the demo of the Pixtral-12B model. If you’re using a RunPod instance with a 40 GB disk, I suggest you use the A100 PCIe GPU.

We’ll be using the Pixtral-12B with the help of vllm. Make sure to do the following installations.

!pip install vllm

!pip install --upgrade mistral_common

Go to this link: of Hugging Face and agree to access the model. Then go to your profile, click on “access_tokens,” and create one. If you don’t have an access token, ensure you have checked the following boxes:

Now run the following code and paste the Access Token to authenticate with  Hugging Face:

from huggingface_hub import notebook_login

notebook_login()

This will take a while as the 25 GB model gets downloaded for use:

from vllm import LLM

from vllm.sampling_params import SamplingParams

model_name = "mistralai/Pixtral-12B-2409"

sampling_params = SamplingParams(max_tokens=8192)

llm = LLM(model=model_name, tokenizer_mode="mistral",max_model_len=70000)

prompt = "Describe this image"

image_url = "https://images.news18.com/ibnlive/uploads/2024/07/suryakumar-yadav-catch-1-2024-07-4a496281eb830a6fc7ab41e92a0d295e-3x2.jpg"

messages = [

{

"role": "user",

"content": [{"type": "text", "text": prompt}, {"type": "image_url", "image_url": {"url": image_url}}]

},

]

I asked the model to describe the following image, which is from the T20 World Cup 2024:

outputs = llm.chat(messages, sampling_params=sampling_params)

print('\n'+ outputs[0].outputs[0].text)

Output

Processed prompts: 100%|██████████| 1/1 [00:06<00:00,  6.52s/it, est. speed
 input: 429.80 toks/s, output: 51.54 toks/s]
The image is a composite of three frames showing a cricket player in action,
 likely from the T20 World Cup. Here's a detailed summary:

1. **Left Frame**:
   - The player is seen mid-jump, looking upwards with his arms outstretched,
 attempting to catch a cricket ball.
   - He is wearing a blue jersey with the name "Surya" and the number "63"
 printed on the back. 
   - His expression is intense and focused, emphasizing the concentration
 required for the catch.

2. **Middle Frame**:
   - The player has his hands cupped together, measurably closer to catching
 the ball.
   - He wears the same blue jersey with red sleeves, which also features the
 "India" team logo and the sponsor's branding, specifically from Adidas.
   - The ball is now very close to his hands, indicating a high likelihood of
 a successful catch.

3. **Right Frame**:
   - The player has successfully caught the ball, raising both hands above
 his head with the ball clasped in his left hand.
   - He displays a triumphant expression with a broad smile, typically
 associated with successfully making a catch.
   - His uniformity and raised hands convey a sense of victory and great
 effort.

The background in all frames features billboards indicating the "T20 World
 Cup," and specifically, a prominent sign for "Emirates," which has
 presumably sponsored the event. There is green grass on the field,
 reflecting the actual playing conditions during the match. 

Overall, the image illustrates a key moment in a cricket game, showcasing the
 player's athletic skill and team spirit.

From the output, we can see that the model was able to identify the image from the T20 World Cup, and it was able to distinguish the frames in the same image to explain what was happening.

prompt = "Write a story describing the whole event that might have happened"

image_url = "https://images.news18.com/ibnlive/uploads/2024/07/suryakumar-yadav-catch-1-2024-07-4a496281eb830a6fc7ab41e92a0d295e-3x2.jpg"

messages = [

{

"role": "user",

"content": [{"type": "text", "text": prompt}, {"type": "image_url", "image_url": {"url": image_url}}]

},

]

outputs = llm.chat(messages, sampling_params=sampling_params)

print('\n'+outputs[0].outputs[0].text)

Output

Processed prompts: 100%|██████████| 1/1 [00:06<00:00,  6.99s/it, est. speed
 input: 401.77 toks/s, output: 52.92 toks/s]
In the heart of a bustling stadium, the T20 World Cup is in full swing. The
 atmosphere is electric, and the crowd is on the edge of their seats, eagerly
 watching the match between India and one of their formidable rivals. The
 focus shifts to the Indian fielders, specifically Suryakumar Yadav, who is
 positioned strategically on the boundary line.

As the ball sails into the air after a powerful shot from the opposing
 batsman, Suryakumar's eyes follow its trajectory. With age-old instincts
 honed by countless hours of practice, he knows that this is his moment. The
 ball is lofted high, giving him just enough time to gauge its path and plan
 his next move.

In the first snapshot, Suryakumar is Hat-cheaper crouched, eyes fixed on the
 ball as it peaks. The anticipation is palpable. His teammates watch
 intently, hoping he can execute the catch perfectly.

The second snapshot captures a moment of pure athleticism. With a burst of
 energy and agility, Suryakumar springs into action. He takes a few swift
 steps forward, his body eager to reach the height required to make the
 catch. The ball is slightly above his head, and he extends his arms, fingers
 spread wide, ready to secure the prize.

In the final snapshot, Suryakumar's face lights up with accomplishment. His
 eyes are focused on the ball, now safely nestled in his palm. The crowd
 explodes with cheers, acknowledging the outstanding effort. His teammates
 rush towards him, celebrating the crucial catch that could turn the tide of
 the match.

This sequence of successful plays not only highlights Suryakumar's individual
 skill but also underscores the strategic teamwork and determined spirit that
 define the Indian cricket team in the prestigious T20 World Cup.

When asked to write a story about the image, the model could gather context on the environment’s characteristics and what exactly happened in the frame.

Conclusion

The Pixtral-12B model significantly advances Mistral’s AI capabilities, blending text and image processing to expand its use cases. Its ability to handle high-resolution 1024 x 1024 images with a detailed understanding of spatial relationships and its strong language capabilities make it an excellent tool for multimodal tasks such as image captioning, story generation, and more.

Despite its powerful features, the model can be further fine-tuned to meet specific needs, whether improving image recognition, enhancing language generation, or adapting it for more specialized domains. This flexibility is a crucial advantage for developers and researchers who want to tailor the model to their use cases.

Mounish V

I'm a tech enthusiast, graduated from Vellore Institute of Technology. I'm working as a Data Science Trainee right now. I am very much interested in Deep Learning and Generative AI.