Getting Started with Groq API: The Fastest Ever Inference Endpoint

Introduction

Real-time AI systems rely heavily on fast inference. Inference APIs from industry leaders like OpenAI, Google, and Azure enable rapid decision-making. Groq’s Language Processing Unit (LPU) technology is a standout solution, enhancing AI processing efficiency. This article delves into Groq’s innovative technology, its impact on AI inference speeds, and how to leverage it using Groq API.

Learning Objectives

• Understand Groq’s Language Processing Unit (LPU) technology and its impact on AI inference speeds
• Learn how to utilize Groq’s API endpoints for real-time, low-latency AI processing tasks
• Explore the capabilities of Groq’s supported models, such as Mixtral-8x7b-Instruct-v0.1 and Llama-70b, for natural language understanding and generation
• Compare and contrast Groq’s LPU system with other inference APIs, examining factors such as speed, efficiency, and scalability

This article was published as a part of the Data Science Blogathon.

What is Groq?

Founded in 2016, Groq is a California-based AI solutions startup with its headquarters located in Mountain View. Groq, which specializes in ultra-low latency AI inference, has advanced AI computing performance significantly. Groq is a prominent participant in the AI technology space, having registered its name as a trademark and assembled a global team committed to democratizing access to AI.

Take your AI innovations to the next level with GenAI Pinnacle. Fine-tune models like Gemini and unlock endless possibilities in NLP, image generation, and more. Dive in today! Explore Now

Language Processing Units

Groq’s Language Processing Unit (LPU), an innovative technology, aims to enhance AI computing performance, particularly for Large Language Models (LLMs). The Groq LPU system strives to deliver real-time, low-latency experiences with exceptional inference performance. Groq achieved over 300 tokens per second per user on Meta AI’s Llama-2 70B model, setting a new industry benchmark.

The Groq LPU system boasts ultra-low latency capabilities crucial for AI support technologies. Specifically designed for sequential and compute-intensive GenAI language processing, it outperforms conventional GPU solutions, ensuring efficient processing for tasks like natural language creation and understanding.

Groq’s first-generation GroqChip, part of the LPU system, features a tensor streaming architecture optimized for speed, efficiency, accuracy, and cost-effectiveness. This chip surpasses incumbent solutions, setting new records in foundational LLM speed measured in tokens per second per user. With plans to deploy 1 million AI inference chips within two years, Groq demonstrates its commitment to advancing AI acceleration technologies.

In summary, Groq’s Language Processing Unit system represents a significant advancement in AI computing technology, offering outstanding performance and efficiency for Large Language Models while driving innovation in AI.

Also Read: Building ML Model in AWS SageMaker

Getting Started with Groq

Right now, Groq is providing free-to-use API endpoints to the Large Language Models running on the Groq LPU – Language Processing Unit. To get started, visit this page and click on login. The page looks like the one below:

Click on Login and choose one of the appropriate methods to sign in to Groq. Then we can create a new API like the one below by clicking on the Create API Key button

Next, assign a name to the API key and click “submit” to create a new API Key. Now, proceed to any code editor/Colab and install the required libraries to begin using Groq.

!pip install groq

This command installs the Groq library, allowing us to infer the Large Language Models running on the Groq LPUs.

Now, let’s proceed with the code.

Code Implementation

# Importing Necessary Libraries
import os
from groq import Groq

# Instantiation of Groq Client
client = Groq(
    api_key=os.environ.get("GROQ_API_KEY"),
)

This code snippet establishes a Groq client object to interact with the Groq API. It begins by retrieving the API key from an environment variable named GROQ_API_KEY and passes it to the argument api_key. Subsequently, the API key initializes the Groq client object, enabling API calls to the Large Language Models within Groq Servers.

Defining our LLM

llm = client.chat.completions.create(
    messages=[
        {
            "role": "system",
            "content": "You are a helpful AI Assistant. You explain ever \
            topic the user asks as if you are explaining it to a 5 year old"
        },
        {
            "role": "user",
            "content": "What are Black Holes?",
        }
    ],
    model="mixtral-8x7b-32768",
)

print(llm.choices[0].message.content)

• The first line initializes an llm object, enabling interaction with the Large Language Model, similar to the OpenAI Chat Completion API.
• The subsequent code constructs a list of messages to be sent to the LLM, stored in the messages variable.
• The first message assigns the role as “system” and defines the desired behavior of the LLM to explain topics as it would to a 5-year-old.
• The second message assigns the role as “user” and includes the question about black holes.
• The following line specifies the LLM to be used for generating the response, set to “mixtral-8x7b-32768,” a 32k context Mixtral-8x7b-Instruct-v0.1 Large language model accessible via the Groq API.
• The output of this code will be a response from the LLM explaining black holes in a manner suitable for a 5-year-old’s understanding.
• Accessing the output follows a similar approach to working with the OpenAI endpoint.

Output

Below shows the output generated by the Mixtral-8x7b-Instruct-v0.1 Large language model:

The completions.create() object can even take in additional parameters like temperature, top_p, and max_tokens.

Generating a Response

Let’s try to generate a response with these parameters:

llm = client.chat.completions.create(
    messages=[
        {
            "role": "system",
            "content": "You are a helpful AI Assistant. You explain ever \
            topic the user asks as if you are explaining it to a 5 year old"
        },
        {
            "role": "user",
            "content": "What is Global Warming?",
        }
    ],
    model="mixtral-8x7b-32768",
    temperature = 1,
    top_p = 1,
    max_tokens = 256,
)

• temperature: Controls the randomness of responses. A lower temperature leads to more predictable outputs, while a higher temperature results in more varied and sometimes more creative outputs
• max_tokens: The maximum number of tokens that the model can process in a single response. This limit ensures computational efficiency and resource management
• top_p: A method of text generation that selects the next token from the probability distribution of the top p most likely tokens. This balances exploration and exploitation during generation

Output

There is even an option to stream the responses generated from the Groq Endpoint. We just need to specify the stream=True option in the completions.create() object for the model to start streaming the responses.

Groq in Langchain

Groq is even compatible with LangChain. To begin using Groq in LangChain, download the library:

!pip install langchain-groq

The above will install the Groq library for LangChain compatibility. Now let’s try it out in code:

# Import the necessary libraries.
from langchain_core.prompts import ChatPromptTemplate
from langchain_groq import ChatGroq

# Initialize a ChatGroq object with a temperature of 0 and the "mixtral-8x7b-32768" model.
llm = ChatGroq(temperature=0, model_name="mixtral-8x7b-32768")

The above code does the following:

• Creates a new ChatGroq object named llm
• Sets the temperature parameter to 0, indicating that the responses should be more predictable
• Sets the model_name parameter to “mixtral-8x7b-32768“, specifying the language model to use

# Define the system message introducing the AI assistant’s capabilities.

# Define the system message introducing the AI assistant's capabilities.
system = "You are an expert Coding Assistant."

# Define a placeholder for the user's input.
human = "{text}"

# Create a chat prompt consisting of the system and human messages.
prompt = ChatPromptTemplate.from_messages([("system", system), ("human", human)])

# Invoke the chat chain with the user's input.
chain = prompt | llm

response = chain.invoke({"text": "Write a simple code to generate Fibonacci numbers in Rust?"})

# Print the Response.
print(response.content)

• The code generates a Chat Prompt using the ChatPromptTemplate class.
• The prompt comprises two messages: one from the “system” (the AI assistant) and one from the “human” (the user).
• The system message presents the AI assistant as an expert Coding Assistant.
• The human message serves as a placeholder for the user’s input.
• The llm method invokes the llm chain to produce a response based on the provided Prompt and the user’s input.

Output

Here is the output generated by the Mixtral Large Language Model:

The Mixtral LLM consistently generates relevant responses. Testing the code in the Rust Playground confirms its functionality. The quick response is attributed to the underlying Language Processing Unit (LPU).

Groq vs Other Inference APIs

Groq’s Language Processing Unit (LPU) system aims to deliver lightning-fast inference speeds for Large Language Models (LLMs), surpassing other inference APIs such as those provided by OpenAI and Azure. Optimized for LLMs, Groq’s LPU system provides ultra-low latency capabilities crucial for AI assistance technologies. It addresses the primary bottlenecks of LLMs, including compute density and memory bandwidth, enabling faster generation of text sequences.

In comparison to other inference APIs, Groq’s LPU system is faster, with the ability to generate up to 18x faster inference performance on Anyscale’s LLMPerf Leaderboard compared to other top cloud-based providers. Groq’s LPU system is also more efficient, with a single core architecture and synchronous networking maintained in large-scale deployments, enabling auto-compilation of LLMs and instant memory access.

The above image displays benchmarks for 70B models. Calculating the output tokens throughput involves averaging the number of output tokens returned per second. Each LLM inference provider processes 150 requests to gather results, and the mean output tokens throughput is calculated using these requests. Improved performance of the LLM inference provider is indicated by a higher throughput of output tokens. It’s clear that Groq’s output tokens per second outperform many of the displayed cloud providers.

Conclusion

In conclusion, Groq’s Language Processing Unit (LPU) system stands out as a revolutionary technology in the realm of AI computing, offering unprecedented speed and efficiency for handling Large Language Models (LLMs) and driving innovation in the field of AI. By leveraging its ultra-low latency capabilities and optimized architecture, Groq is setting new benchmarks for inference speeds, outperforming conventional GPU solutions and other industry-leading inference APIs. With its commitment to democratizing access to AI and its focus on real-time, low-latency experiences, Groq is poised to reshape the landscape of AI acceleration technologies.

Key Takeaways

• Groq’s Language Processing Unit (LPU) system offers unparalleled speed and efficiency for AI inference, particularly for Large Language Models (LLMs), enabling real-time, low-latency experiences
• Groq’s LPU system, featuring the GroqChip, boasts ultra-low latency capabilities essential for AI support technologies, outperforming conventional GPU solutions
• With plans to deploy 1 million AI inference chips within two years, Groq demonstrates its dedication to advancing AI acceleration technologies and democratizing access to AI
• Groq provides free-to-use API endpoints for Large Language Models running on the Groq LPU, making it accessible for developers to integrate into their projects
• Groq’s compatibility with LangChain and LlamaIndex further expands its usability, offering seamless integration for developers seeking to leverage Groq technology in their language-processing tasks

Frequently Asked Questions

The media shown in this article is not owned by Analytics Vidhya and is used at the Author’s discretion.

Dive into the future of AI with GenAI Pinnacle. From training bespoke models to tackling real-world challenges like PII masking, empower your projects with cutting-edge capabilities. Start Exploring.

Ajay

I work as a Developer in the field of Data Science. I constantly spend time learning new things be it related to AI, DataSceine, and CyberSecurity. Deep learning and machine learning are two topics that I find particularly fascinating, and Python is my preferred language for programming. Cyber Security is another field that I'm touching upon recently. I have experience with large-scale data analysis, and I have a solid grasp of a variety of deep learning and machine learning approaches, including neural networks, regression models, and natural language processing. I'm eager to take on new challenges and make a meaningful contribution to the industry, so I'm constantly seeking for ways to enlarge and deepen my knowledge and skills in the subject.