Introduction
Mistral has released its very first multimodal model, namely the Pixtral-12B-2409. This model is built upon Mistral’s 12 Billion parameter, Nemo 12B. What sets this model apart? It can now take both images and text for input. Let’s look more at the model, how it can be used, how well it’s performing the tasks and the other things you need to know.
Overview
- Discover Mistral’s new Pixtral-12B, a multimodal model combining text and image processing for versatile AI applications.
- Learn how to use Pixtral-12B, Mistral’s latest AI model, designed to handle both text and high-resolution images.
- Explore the capabilities and use cases of the Pixtral-12B model, featuring a vision adapter for enhanced image understanding.
- Understand Pixtral-12B’s multimodal features and its potential applications in image captioning, story generation, and more.
- Get insights into Pixtral-12B’s design, performance, and how to fine-tune it for specific multimodal tasks.
What is Pixtral-12B?
Pixtral-12B is a multimodal model derived from Mistral’s Nemo 12B, with an added 400M-parameter vision adapter. Mistral can be downloaded from a torrent file or on Hugging Face with an Apache 2.0 license. Let’s look at some of the technical features of the Pixtral-12B model:
| Feature | Details |
| Model Size | 12 billion parameters |
| Layers | 40 Layers |
| Vision Adapter | 400 million parameters, utilizing GeLU activation |
| Image Input | Accepts 1024 x 1024 images via URL or base64, segmented into 16 x 16 pixel patches |
| Vision Encoder | 2D RoPE (Rotary Position Embeddings) enhances spatial understanding |
| Vocabulary Size | Up to 131,072 tokens |
| Special Tokens | img, img_break, and img_end |
How to Use Pixtral-12B-2409?
As of September 15th, 2024, the model is currently not available on Mistral’s Le Chat or La Plateforme to use the chat interface directly or access it through API, but we can download the model through a torrent link and use it or even finetune the weights to suit our needs. We can also use the model with the help of Hugging Face. Let’s look at them in detail:
Torrent link to Use:
magnet:?xt=urn:btih:7278e625de2b1da598b23954c13933047126238a&dn=pixtral-12b-
240910&tr=udp%3A%2F%http://2Ftracker.opentrackr.org%3A1337%2Fannounce&tr=udp%
3A%2F%http://2Fopen.demonii.com%3A1337%2Fannounce&tr=http%3A%2F%http://2Ftrac
ker.ipv6tracker.org%3A80%2Fannounce
I’m using an Ubuntu laptop, so I’ll use the Transmission application (it’s pre-installed in most Ubuntu computers). You can use any other application to download the torrent link for the open-source model.
- Click “File” at the top left and select the open URL option. Then, you can paste the link that you copied.
- You can click “Open” and download the Pixtral-12B model. The folder will be downloaded which contains these files:
Hugging Face
This model demands a high GPU, so I suggest you use the paid version of Google Colab or Jupyter Notebook using RunPod. I’ll be using RunPod for the demo of the Pixtral-12B model. If you’re using a RunPod instance with a 40 GB disk, I suggest you use the A100 PCIe GPU.
We’ll be using the Pixtral-12B with the help of vllm. Make sure to do the following installations.
!pip install vllm
!pip install --upgrade mistral_commonGo to this link: https://huggingface.co/mistralai/Pixtral-12B-2409 and agree to access the model. Then go to your profile, click on “access_tokens,” and create one. If you don’t have an access token, ensure you have checked the following boxes:
Now run the following code and paste the Access Token to authenticate with Hugging Face:
from huggingface_hub import notebook_login
notebook_login()#hf_SvUkDKrMlzNWrrSmjiHyFrFPTsobVtltzOThis will take a while as the 25 GB model gets downloaded for use:
from vllm import LLM
from vllm.sampling_params import SamplingParams
model_name = "mistralai/Pixtral-12B-2409"
sampling_params = SamplingParams(max_tokens=8192)
llm = LLM(model=model_name, tokenizer_mode="mistral",max_model_len=70000)
prompt = "Describe this image"
image_url = "https://images.news18.com/ibnlive/uploads/2024/07/suryakumar-yadav-catch-1-2024-07-4a496281eb830a6fc7ab41e92a0d295e-3x2.jpg"
messages = [
{
"role": "user",
"content": [{"type": "text", "text": prompt}, {"type": "image_url", "image_url": {"url": image_url}}]
},
]I asked the model to describe the following image, which is from the T20 World Cup 2024:
outputs = llm.chat(messages, sampling_params=sampling_params)
print('\n'+ outputs[0].outputs[0].text)Output
Processed prompts: 100%|██████████| 1/1 [00:06<00:00, 6.52s/it, est. speed
input: 429.80 toks/s, output: 51.54 toks/s]
The image is a composite of three frames showing a cricket player in action,
likely from the T20 World Cup. Here's a detailed summary:
1. **Left Frame**:
- The player is seen mid-jump, looking upwards with his arms outstretched,
attempting to catch a cricket ball.
- He is wearing a blue jersey with the name "Surya" and the number "63"
printed on the back.
- His expression is intense and focused, emphasizing the concentration
required for the catch.
2. **Middle Frame**:
- The player has his hands cupped together, measurably closer to catching
the ball.
- He wears the same blue jersey with red sleeves, which also features the
"India" team logo and the sponsor's branding, specifically from Adidas.
- The ball is now very close to his hands, indicating a high likelihood of
a successful catch.
3. **Right Frame**:
- The player has successfully caught the ball, raising both hands above
his head with the ball clasped in his left hand.
- He displays a triumphant expression with a broad smile, typically
associated with successfully making a catch.
- His uniformity and raised hands convey a sense of victory and great
effort.
The background in all frames features billboards indicating the "T20 World
Cup," and specifically, a prominent sign for "Emirates," which has
presumably sponsored the event. There is green grass on the field,
reflecting the actual playing conditions during the match.
Overall, the image illustrates a key moment in a cricket game, showcasing the
player's athletic skill and team spirit.
From the output, we can see that the model was able to identify the image from the T20 World Cup, and it was able to distinguish the frames in the same image to explain what was happening.
prompt = "Write a story describing the whole event that might have happened"
image_url = "https://images.news18.com/ibnlive/uploads/2024/07/suryakumar-yadav-catch-1-2024-07-4a496281eb830a6fc7ab41e92a0d295e-3x2.jpg"
messages = [
{
"role": "user",
"content": [{"type": "text", "text": prompt}, {"type": "image_url", "image_url": {"url": image_url}}]
},
]
outputs = llm.chat(messages, sampling_params=sampling_params)
print('\n'+outputs[0].outputs[0].text)Output
Processed prompts: 100%|██████████| 1/1 [00:06<00:00, 6.99s/it, est. speed
input: 401.77 toks/s, output: 52.92 toks/s]
In the heart of a bustling stadium, the T20 World Cup is in full swing. The
atmosphere is electric, and the crowd is on the edge of their seats, eagerly
watching the match between India and one of their formidable rivals. The
focus shifts to the Indian fielders, specifically Suryakumar Yadav, who is
positioned strategically on the boundary line.
As the ball sails into the air after a powerful shot from the opposing
batsman, Suryakumar's eyes follow its trajectory. With age-old instincts
honed by countless hours of practice, he knows that this is his moment. The
ball is lofted high, giving him just enough time to gauge its path and plan
his next move.
In the first snapshot, Suryakumar is Hat-cheaper crouched, eyes fixed on the
ball as it peaks. The anticipation is palpable. His teammates watch
intently, hoping he can execute the catch perfectly.
The second snapshot captures a moment of pure athleticism. With a burst of
energy and agility, Suryakumar springs into action. He takes a few swift
steps forward, his body eager to reach the height required to make the
catch. The ball is slightly above his head, and he extends his arms, fingers
spread wide, ready to secure the prize.
In the final snapshot, Suryakumar's face lights up with accomplishment. His
eyes are focused on the ball, now safely nestled in his palm. The crowd
explodes with cheers, acknowledging the outstanding effort. His teammates
rush towards him, celebrating the crucial catch that could turn the tide of
the match.
This sequence of successful plays not only highlights Suryakumar's individual
skill but also underscores the strategic teamwork and determined spirit that
define the Indian cricket team in the prestigious T20 World Cup.
When asked to write a story about the image, the model could gather context on the environment’s characteristics and what exactly happened in the frame.
Conclusion
The Pixtral-12B model significantly advances Mistral’s AI capabilities, blending text and image processing to expand its use cases. Its ability to handle high-resolution 1024 x 1024 images with a detailed understanding of spatial relationships and its strong language capabilities make it an excellent tool for multimodal tasks such as image captioning, story generation, and more.
Despite its powerful features, the model can be further fine-tuned to meet specific needs, whether improving image recognition, enhancing language generation, or adapting it for more specialized domains. This flexibility is a crucial advantage for developers and researchers who want to tailor the model to their use cases.
Frequently Asked Questions
Q1. What is vLLM?
A. vLLM is a library optimized for efficient inference of large language models, improving speed and memory usage during model execution.
Q2. What’s the use of SamplingParams?
A. SamplingParams in vLLM control how the model generates text, specifying parameters like the maximum number of tokens and sampling techniques for text generation.
Q3. Will the model be available on Mistral’s Le Chat?
A. Yes, Sophia Yang, Head of Mistral Developer Relations, mentioned that the model would soon be available on Le Chat and Le Platform.
mounish12439
16 Sep, 2024
I'm a tech enthusiast, graduated from Vellore Institute of Technology. I'm working as a Data Science Trainee right now. I am very much interested in Deep Learning and Generative AI.
