Introduction to Memcached using Python

This article was published as a part of the Data Science Blogathon.

Introduction

Memcached is a highly-performant distributed caching system. It is an in-memory key-value data store, which makes it a type of NoSQL database. Memcached is used by tech giants like Facebook, Twitter, Instagram, and Netflix. In my previous article, I explained Redis which is another in-memory key-value data store that can be used for the purpose of caching. This article will explain how to connect and use Memcached in python.

Installing Memcached on Windows

Memcached is not officially supported on Windows. However, you can install it by setting up Windows Subsystem for Linux 2 and configuring it. Alternatively, you can run Memcached on a container using Docker, which I will be covering in this article. The first step is to install Docker on your Windows machine. You can download Docker Desktop from here. The installation process is fairly simple and direct. Now that we have Docker on our machine, enter the following command on a command prompt to fetch the Redis image from Docker Hub, which can be used to build and run a container.

docker pull memcached

Once this is done, the third step is to start a container using the Memcached image that we downloaded earlier.

Here, click on the ‘RUN’ button.

Please refer to the below picture for reference.

Congrats, you have now successfully started the Redis server on your machine.

Installing Pymemcache

To connect and use Memcached in python, we will be using a python module called pymemcache. It can be installed by running the following command in the command prompt.

pip install pymemcache

Now that we have everything ready, let’s get our hands dirty and dive into the programming part.

Using Pymemcache

Before performing any CRUD operations, we first need to connect to the Memcached server. Let’s go ahead and connect to our Memcached server.

from pymemcache.client import base
client = base.Client(('localhost', 11211))

Now that we have successfully connected to the Memcached server, let’s start performing simple CRUD operations.

To set a key-value pair, we can use either the “set” function or the “add” function that accepts the key and the value as the parameter. Please note that the key should always be either of string data type or bytes.

client.set('hello', 'world')

To get the value for a key stored in the Memcached server, we can use the “get” function that accepts the key name as the parameter. The response is always in bytes and hence, the response should be decoded to string by us manually which is fairly easy.

client.get('hello') # this always returns bytes
b'world'

To convert bytes to string, we can simply decode the bytes in ‘UTF-8’ format.

client.get('hello').decode("utf-8") # convert bytes to string
'world'

By default, the data stored does not expire. Since Memcached is an in-memory data store, the storage is limited and data might get flushed due to insufficient memory. We can set the expiry for a key-value pair using the “add” function by specifying the time in seconds. The expiration time should be carefully decided based on the type of application that you are building.

client.add(key="lorem", value="ipsum", expire=10) # expire after 10 seconds
client.get('lorem')
'ipsum'

If we try to access the key after 10 seconds, we will not get any response as the key would have been deleted from the database.

We can set multiple key-value pairs using the “set_multi” function which accepts the multiple key-value pairs as a dictionary dataset. It also accepts the “expire” parameter for the key-value pair to expire and makes space for new data in the Memcached server.

client.set_multi({"Ram": 22, "Vishnu": 21, "Sanjay": 17},expire=200)
client.get('Ram')
b'22'

Now let us try to increase the value of the key “Ram”. We can make use of the “incr” function to perform an addition operation to a numerical value in Memcached.

client.incr('Ram', value=5)
client.get('Ram')
b'27'

Similarly, we can make use of the “decr” function to perform a subtraction operation to a numerical value in Memcached.

client.decr('Ram', value=2)
client.get('Ram')
b'25'

All the values apart from numerical values in key-value pairs would be converted to string bytes before storing and so we would only get a string as a response and not the original data structure as such, which might be a problem.

client.add(key="languages", value=['python', 'C++', 'C'])
client.get('languages').decode('utf-8')
"['python', 'C++', 'C']"

Here, we can see that we stored a list in the Memcached server, but when we try to fetch this data, we get a string as a response and not a list. To make sure we store the data type as well, apart from just strings, we can make use of serialization and deserialization techniques using modules like JSON or pickle. I will be using JSON to explain how the serialization and deserialization techniques can be used to store and extract the data with the same data type.

Let us first try to store a nested object in the Memcached server and see what response we get so that it can be compared with the response we get after performing the serialization and deserialization using JSON.

personal_data = {
    'name': 'Prakash',
    'age': 34,
    'programming_languages': ['python', 'C#', 'java'],
    'address':{
        'flat_no': 177,
        'area': 'Velachery',
        'pincode': 600042
    }
}
client.set('prakask_personal_data', personal_data)
client.get('prakask_personal_data').decode("utf-8")
"{'name': 'Prakash', 'age': 34, 'programming_languages': ['python', 'C#', 'java'], 'address': {'flat_no': 177, 'area': 'Velachery', 'pincode': 600042}}"

If we try to access the name property, we would get an error as the response is a string and not a dictionary. Now let us use JSON to stringify the data and then store it in Memcached and extract and deserialize using the same JSON.

import json
personal_data = {
    'name': 'Prakash',
    'age': 34,
    'programming_languages': ['python', 'C#', 'java'],
    'address':{
        'flat_no': 177,
        'area': 'Velachery',
        'pincode': 600042
    }
}
client.set('prakask_personal_data', json.dumps(personal_data))
data = json.loads(client.get('prakask_personal_data').decode("utf-8"))
print(data)

{'name': 'Prakash',
 'age': 34,
 'programming_languages': ['python', 'C#', 'java'],
 'address': {'flat_no': 177, 'area': 'Velachery', 'pincode': 600042}}

Now, let’s try to access the area property present inside the address property and check if we are getting any errors or not.

data['address']['area']
'Velachery'

We can see that we get the correct response.

A caching system is just a way to speed up your application. Traditional SQL databases like MySQL, Oracle, and new NoSQL databases like MongoDB store data on the disk. Operations performed on disk are generally slower when compared to the operations performed in memory or RAM. Since technologies like Redis and Memcached are in-memory data stores, the operations are lightning-fast but the downside of these technologies is that RAM is limited and cannot be used to store all information. To make the application fast, technologies like Redis and Memcached are used alongside proper full-fledged databases like MySQL, MongoDB, etc. Let’s see how Memcached can be used alongside another database in applications.

Let’s say you set a key-value pair in the Memcached server and are trying to access it after a while. Assume the key has expired from the Memcached server. In such a case, we try to access the data from our traditional database, say MySQL. All the data is persistent and doesn’t expire, unlike Memcached.

client.add('analytics', 'vidhya', expire=15)
def fetch_from_mysql(key= ''): 
	data = ...  # fetch from MySQL server
	return data
data = client.get('analytics')
if not data
	# fetch from MySQL as the key is not found in the Memcached server
	data = fetch_from_mysql('analytics')
	print(data)
else: print(data)

This is a simple example to show how caching systems can be designed to work alongside full-fledged databases in applications.

Conclusion

In this article, we discussed and covered the following:

• What is Memcached?
• Installing docker on windows
• Running Memcached on a docker container
• Connecting to Memcached using Pymemcache
• Performing simple CRUD operations using Pymemcache
• Caching mechanism using Memcached in Applications

The functions discussed in this article are necessary for understanding and performing basic CRUD operations in Memcached, but there are a lot of different functions readily available. Caching mechanisms are present in almost all applications and we must understand and learn the basics of caching to build production-ready applications. Memcached is designed to work only as a caching server, while its competitor Redis is built to solve multiple use cases. Redis is single-threaded while Memcached is multi-threaded, which makes Memcached perform better than Redis when the data size is huge. To know more about Redis, read this article.

That’s it for this article. Hope you enjoyed reading this article and learned something new. Thanks for reading and happy learning!

The media shown in this article is not owned by Analytics Vidhya and is used at the Author’s discretion.

TK

I love exploring ML, DL, Machine Vision, Databases, and Full-stack Web Development. I've worked on multiple projects with different stacks and I also hold a patent in the domain of machine vision for manufacturing.