Complete Guide to Pub/Sub in Redis

Introduction

Publish and Subscribe is a messaging mechanism having one or a set of senders sending messages and one or a group of receivers receiving these messages. These senders are called Publishers, responsible for publishing these messages, and the receivers are called Subscribers who subscribe to these Publishers to receive their notifications. Publishers can publish to zero or more Subscribers at a time. Similarly, Subscribers can subscribe to one or more Publishers to receive notifications of all their subscribed Publishers.

The Publishers publish the notifications without knowing who the subscribers are (even if they are no Subscribers, they just push these messages). These messages are pushed into channels. Channel is a specified destination that Publisher chooses to send data and the Subscribers receive messages from these channels. Channels contain information on a specific topic. It is a link between Publishers and Senders. Similarly, the Subscribers subscribe to the different channels of their interest, without knowing who the Publishers are.  The Publish and Subscriber mechanism in Redis is provided by Redis Pub/Sub commands and will look in-depth at them in the following guide

Learning Objectives

• Understand Redis Pub/Sub.
• To learn different Pub/Sub commands
• To learn how to subscribe using patterns
• To understand the differences between Redis Pub/Sub and Redis Streams
• To know the potential applications of Redis Pub/Sub

This article was published as a part of the Data Science Blogathon.

Table of Contents

1. What is Redis Pub/Sub?
2. Getting Started with Pub/Sub
3. Redis Pub/Sub Commands
4. Redis Pub/Sub vs Redis Streams
5. Redis Pub/Sub Applications
6. Conclusion

What is Redis Pub/Sub?

Redis Pub/Sub is a way to use the Publish and Subscribe mechanism in the Redis Server. Here the publishers are not programmed to send messages directly to subscribers, instead, they send these messages to channels, where each channel contains information on a particular topic. The publishers are not aware of who the subscribers are and similarly, subscribers do not know who the publishers are. Redis Pub/Sub comes with a set of commands required to publish messages and read these messages by subscribing to them. It even provides commands for subscribing to many channels through a certain pattern selection and unsubscribing to more than one channel which follows a specific pattern.

Redis supports its Redis Pub/Sub messaging even in its shards, i.e. in a Redis Cluster. Redis Pub/Sub is often used in cloud computing and the IoT space. While a Subscriber is subscribed to a specific channel in Redis, then that specific client/subscriber cannot use the Redis commands, because they will move into Subscriber mode, thus can only listen to the incoming messages from the subscribed channels, which are published by the Publishers. To get out of this mode Ctrl + C is used. We will look into this in the next sections.

Getting Started with Pub/Sub

In this section, we will dive into the Redis Server and look into how the Publish and Subscriber worlds in Redis. To get started you need to have Redis Server up and running which you can do so by downloading the latest Redis Docker Image and running it.

Now, to run the Redis commands, firstly we need to move into the CLI mode which we can do so by running redis-cli.

Now we will create a simple Publisher and Subscribe to that channel and then we will send a message through the Redis Pub/Sub commands and even receive the message. The following can be seen in the below pic

SUBSCRIBE news_bbc

In the right pane, we see that we have subscribed to a channel news_bbc, even though it doesn’t exist through the SUBSCRIBE command. So now the left shell will go into subscriber mode and will wait for the new messages to come. When we subscribed to a new channel, the output contains two messages, one is the “subscribe” and the other is the channel name, because we have subscribed to only one channel, we get an (integer) 1 response. Now let’s publish a message through the left pane and see if it comes on the right pane or not.

#Right Pane
SUBSCRIBE news_bbc

#Left Pane
PUBLISH news_bbc "Today's weather will be hot'"

In the above image, in the left shell, we used the PUBLISH command to PUBLISH a message from that news_bbc channel. After typing this command, we see that a message has arrived in the right window which was published from the left shell. We see that the output generated contains 3 elements. The first line indicates that we received a “message”, the second indicates from which channel we received the message, in our example, it’s “news_bbc”, and the third indicates the message we have received from that particular channel. This is how the output is received at the subscriber end.

We have looked into two main commands used in Redis Pub/Sub, i.e. PUBLISH and SUBSCRIBE, in the next section we will discuss all the commands in the Pub/Sub

Redis Pub/Sub Commands

In this section, we will look at all the available commands in Redis Pub/Sub, go through each of them, explain the options in each command, and provide an example for each command.

SUBSCRIBE channel [channel …]

This command for the client to SUBSCRIBE to one or more channels at a time by giving the respective channel names. It takes one or more arguments where each argument is the channel name that it wants to subscribe to. We have seen an example of this in the above section.

PUBLISH channel message

This command is for posting messages from a particular channel. It takes the first argument as the channel name and the second argument as the message it wants to send to that channel. In the above section, we sent a message from the news_bbc channel through this command

UNSUBSCRIBE [channel [channel …]]

This command is for unsubscribing from one or more channels by providing the channel names. It takes one or more arguments where each argument is the channel name that it wants to unsubscribe to. Let’s try unsubscribing to the news_bbc channel.

UNSUBSCRIBE news_bbc

In the above image, we see that we have unsubscribed from the news_bbc channel. The output contains, the first stating that we performed an UNSUBSCRIBE option, the second line is the channel that we have unsubscribed and the third is the number of remaining channels we have subscribed to, in our example it is 0 because we subscribed to only one channel news_bbc

PSUBSCRIBE pattern [pattern …]

This command is for a client to subscribe and listen to different channels by providing one or more patterns. It takes one or more arguments and each argument is a pattern for a group of channels that it wants to subscribe to. Let’s try to subscribe to all the weather channels and also listen to them

# Left Pane 
PSUBSCRIBE weather:*

# Top Right Pane
PUBLISH weather:humidity 20%

# Bottom Right Pane
PUBLISH weather:temperature 34C

In the above Image, we have provided the pattern weather:* to subscribe to channels “weather:temperature” and “weather:humidity”. We see that on the right panes, we have published some messages through the “weather:temperature” and “weather:humidity” channels, and these messages we received to the client on the left pane.

PUBSUB NUMPAT

This command is run to return the number of unique patterns used by the client to subscribe to different channels. It is a count of patterns provided by the client in the PSUBSCRIBE command. Let’s try this out because we performed the PSUBSCRIBE command a while ago.

# Top Window
PSUBSCRIBE weather:*

# Bottom Window
PUBSUB NUMPAT

Running this command gives us an (integer) 1 response, stating we have 1 pattern, which is the “weather:*” which was provided when running the PUBSCRIBE command.

PUBSUB CHANNELS [pattern]

This command lists all the channels with the provided pattern. If the pattern is not provided, it lists all the channels active channels. Active channels are the ones having at least 1 subscriber. It excludes the clients that subscribed to a pattern, that is using the PSUBSCRIBE command. Let’s try subscribing to two channels and running the above command

# Left Window
SUBSCRIBE news_espn news_bbc

# Right Window
PUBSUB CHANNELS

In the above image, on the left pane, we have subscribed to two channels “news_bbc” and “news_espn”. After subscribing to them, then we run the command PUBSUB CHANNELS on the right pane. We see that in the result, we get the names of all the channels that were subscribed by the client i.e. the “news_bbc” and the “news_espn” channels.

PUBSUB NUMSUB [channel [channel …]]

Lists the number of SUBSCRIBERS/CLIENTS who subscribed to the provided channels. Even this command, excludes the clients who subscribed using PSUBSCRIBE. It takes one or more arguments and each argument is the channel name. The output is of the format: channel, count, channel, count, …, in the order in which the channels are provided to the command. Let’s try to count the number of subscribers to the news_bbc and news_espn channel

# Top Right Window
SUBSCRIBE news_bbc

# Bottom Right Window
SUBSCRIBE news_bbc

# Bottom Left Window
SUBSCRIBE news_espn

# Top Left Window
PUBSUB NUMSUB news_bbc news_espn

In the above image, a total of two subscribers have subscribed to the news_bbc channel and one to the news_espn channel. In the top left pane, when we run the PUBSUB NUMSUB news_bbc news_espn, command, we see that in the output it gives us the channel name and the respective subscriber count.

PUNSUBSCRIBE [pattern [pattern …]]

This command when run, unsubscribers the client from the given patterns. It takes one or more arguments and each argument is a pattern for a group of channels that it wants to unsubscribe. More than one pattern can be provided in the command. If the pattern option is kept blank then the client is unsubscribed from all the existing subscribed patterns. When the command is run successfully, it returns the message punsubscribe followed by the pattern provided in the command. Let’s try this to unsubscribe from all the news channels.

PUNSUBSCRIBE news_*

In the above image, we see that we have successfully unsubscribed from all the news channels. In the output we get the message of the operation we performed which is the pattern unsubscribe and followed by that is the pattern, which is news_* which we have provided when running the command.

Redis Pub/Sub vs Redis Streams

Redis Pub/Sub is a Publish and Subscribe messaging mechanism, on the other hand, Redis Streams is a Data Structure, that stores incoming messages or events. It can be viewed as time series data which is an append-only list. In Redis Pub/Sub, there is a Publisher who publishes messages to a channel and there is a Subscriber who subscribes to these channels to receive those messages. In the case of Redis Streams, there is a Stream Producer who adds messages/data to the stream where each stream entry message in the Redis Stream has a unique ID and there are Consumer Groups, who consume these messages using their unique IDs

Another major difference between the Redis Streams and Redis Pub/Sub is message retention. Let’s think of a situation in Redis PubSub, where a Publisher publishes a message to the channel. Suppose the client is not connected at the time, i.e. offline, then the subscriber will never receive this message. While in the Redis Streams, if the client is not connected and a message is sent, the subscriber will receive the message when they connect back again. This is because Redis Pub/Sub doesn’t retain notifications, that is, it doesn’t store messages. The data is delivered immediately to all connected subscribers. It’s like following a fire-and-forget principle. On the other hand, Streams store their messages in the memory, because they are a Redis Data Structure, hence they store the messages in their memory for later consumption by the consumer groups/subscribers.

Redis Pub/Sub is chosen when at-most-once delivery semantics is needed. It’s commonly employed in applications like real-time messaging, notifications, broadcasting, etc. Redis Streams are considered in situations where at least once or at most once delivery semantics is needed. Its applications include storing time-series data, streaming data, data preprocessing, in applications where data is too big to fit in the memory, data aggregation, etc.

Redis Pub/Sub Applications

Real-Time Notifications

Redis Pub/Subs can be leveraged to publish real-time notifications. Different applications can integrate Redis Pub/Sub in them so that their subscribers receive updates and notifications about what is happening in the application with very low latency. Though the Redis Pub/Sub cannot provide a guaranteed delivery (in case of the subscriber being offline / not connected), it is meant to provide at-most-once delivery, with low latency.

Real-Time Sensor Updates

Redis Pub/Sub can be leveraged to publish and receive sensor updates in real time. For example, people working in weather stations, need this data from different sensors, like humidity sensors, temperature sensors, and pressure sensors to get a better idea of the weather. And these people need this information in real-time, so to predict what can be the temperature, and humidity the next day or to analyze if it will rain or snow the next day. In these situations Redis Pub/Sub will come in handy, thus allowing the workers at the weather station to subscribe to the channels of different sensors that sense these environmental factors, so as to get their values to the subscribers of these channels in real-time with low latency.

Distributed Systems

Distributed Systems as the name suggests are a distribution of systems working together to run a common application. These systems often need to communicate with each other. Information needs to be passed from one system to another in these distributed systems, so the application keeps updated always and functions properly. Thus Redis Pub/Sub allows for scalable and flexible communication between the different systems in a Distributed System Space.

Chat Applications

Chat Applications are one of those classic use cases of the Pub/Sub mechanism. In chat applications, people would want to subscribe to chat rooms that have a particular Pub/Sub topic. When some user publishes a message to a chatroom. Then that message is automatically sent to all the subscribers who subscribed to that particular chat room. Redis Pub/Sub can be leveraged to the full extent in these applications where users can subscribe to one or more chatrooms to receive messages and even unsubscribe to stop receiving messages. Roblox employs Redis Pub/Sub for its real-time message delivery system, that is when a user wants to send a message to another user. Roblox does this on a very large scale.

Real-Time Health Analytics

Redis Pub/Sub comes in handy in situations where a company needs to analyze the real-time health analytics of its applications, like how many users are currently logged in, where are the users from, their geographical location, and how much time they spend in their application. This same thing was done by the company Machine Zone. Machine Zone a mobile gaming company developing free-to-play games, with the help of Redis Pub/Sub, they were able to extract information such as their user base across the globe, the throughput of the CDNs serving game assets, the crash rate, or the frame rate per second. All this information was collected at scale was sent to different charting applications to get a Real-Time Health Analytics for their mobile game applications.

Conclusion

Redis Pub/Sub is the message mechanism implemented in Redis allowing real-time communication, which contains a Publisher, who is the source of the data, a Channel, which is a connection between the Publisher and Subscriber and through which the data is sent, and finally, the Subscriber who subscribers to different channels of their interest. In this guide, we have looked at all the available commands in Redis Pub/Sub, and gone through each of them with an example. Even we discussed the differences between the Redis Pub/Sub and Redis Streams. Finally, we have even seen the potential applications of Redis Pub/Sub in the tech industry and even looked at two companies that leverage Redis Pub/Sub in their applications.

The following are some of the key takeaways from this Redis Pub/Sub guide:

• Redis Pub/Sub is possible even in the shards, i.e.in a Redis Cluster.
• Redis Pub/Sub is an implementation of the Pub/Sub mechanism in Redis
• It is relatively fast and efficient because it doesn’t store messages in the memory
• Redis Pub/Sub allows a client to subscribe to multiple channels following a particular pattern
• It finds its applications in the field of real-time applications, chatrooms, real-time message delivery systems, etc.

The media shown in this article is not owned by Analytics Vidhya and is used at the Author’s discretion. 

Ajay

I work as a Developer in the field of Data Science. I constantly spend time learning new things be it related to AI, DataSceine, and CyberSecurity. Deep learning and machine learning are two topics that I find particularly fascinating, and Python is my preferred language for programming. Cyber Security is another field that I'm touching upon recently. I have experience with large-scale data analysis, and I have a solid grasp of a variety of deep learning and machine learning approaches, including neural networks, regression models, and natural language processing. I'm eager to take on new challenges and make a meaningful contribution to the industry, so I'm constantly seeking for ways to enlarge and deepen my knowledge and skills in the subject.