How to Aggregate Pipeline with MongoDB

This article was published as a part of the Data Science Blogathon

“Working smart and having utilities is always a great skill to have”

Whatever we do, if we know multiple tools for various tasks that we might have to perform then it’s an extremely useful set of knowledge. For data science and machine learning, we need data and without it, both domains cannot even exist to be applied. So, definitely knowing how to store, manipulate, ingest and import data is critical for any relative field practitioner.

Today, we will look at a very powerful functionality that is available to us when working with NoSQL data in the MongoDB database. These are Aggregate Pipelines. Such pipelines have a big contribution in exercising good MLOps practices, automating the process of data manipulation, and importing the data to feed into the network with the work of the pipeline.

Steps for MLOps are divided into 4 general categories:

1. Scoping: Defining your project, target users and a plan for how you are gonna carry a project.

2. Data: Define data and establish baseline, label and organize data, feature extraction, data validation and etc.

3. Modeling: Select and train model, perform error analysis on it.

4. Deployment: Deploy in production, monitor, and maintain the system.

So, in terms of these defined stages, the scope of the article would lie in the data category which makes the complete process of MLOps much more systematic, faster, and flexible.

Why don’t we start with the meaning of Aggregation?

Aggregation: According to formal definitions, it is a process of forming or calculating an element by the combination of several separate elements. So, in terms of MongoDB, we can add various functions on top of each other as separate elements and form a more refined and complicated function that does the job easily and with a single step.

In this article, I will show you how to use these aggregate pipelines in MongoDB compass and use these pipelines inside our python code. From my personal experience, this integration is extremely helpful. In case you don’t know about MongoDB compass and how we use it along with python, here is another article I wrote for just this purpose. Please have a look before moving further. Article Link

Let’s look at MongoDB Compass and make a simple pipeline.

1. Open the collection in the Database

In my case, I have taken an already present startup_log collection in my ‘local’ dataset in MongoDB local server. Click on startup_log to open the collection.

 

You can see the records in my startup_log collection. There is a total of 60 records and now we are gonna build a simple aggregate pipeline on top of it. To start click on Aggregations just beside the Documents tab below your collection name(local.startup_log).

2. Start with Aggregate Pipelines

Once you click on it, you will see such an interface, where all these left windows are where we write code for each element for aggregate, and on the right you will see the output that will be retrieved once the raw data performs the task mentioned in the window. Let us see this in action.

 

you can add as many stages as you want, just click the ‘ADD STAGE” button below the previous stage. The first row that you can see in the image is our complete raw data as records aligned horizontally.

Now, you can see a green button on each stage, this on-off button is used to active or inactive a certain stage in the pipeline. Really helpful for debugging if needed. A drop-down menu on the left of the button is where we will choose the functionality for each stage. Let’s see how.

 

Why not perform a very simple function of projection on the dataset and just extract the ‘pid’ attribute from each record.

Just to show how versatile they are, let me create another stage and create a new attribute of my own, which will have the value of the attribute (startTime) as a String (it is in Date time Format in raw data).

Once you are done with your pipelines, don’t FORGET to save the pipeline to use it later and not have to create one again and again.

Giving the pipeline a name and Saving it.

3. Access your saved Pipelines and exporting them for various purposes

Click on the icon, mentioned in the image below, and there you will see all the saved pipelines. So just hover over the name of the pipeline you wish to use again and there you will see a button for open. See it in action.

Now time to use them with Python, but before doing that we need to copy the pipeline / or export the pipeline in a string format that is suitable for python. here is how we do it.

Click the GREEN button to copy the pipeline and be ready to use it in Python.

4. Aggregate Pipeline in Python

Again, if in case you have no idea how to use MongoDB in python, please go through this Article Link. Here is the code to do the task.

import pymongo
database = pymongo.MongoClient().local.startup_log
data = database.aggregate(list( <>))

you can see in the above image that the ‘database’ variable is a collection pointing to ‘startup_log’.

But even after all this, one small step is left. You see the result from the aggregate is a PyMongo Cursor which helps in iterating over the values that are retrieved from the pipeline. So to save the resultant data in a DataFrame, simply wrap the cursor with Pandas.DataFrame.

import pandas as pd
dataset = pd.DataFrame(data)
data.head()

Now, here you have your filtered, processed data with just a single automated pipeline. Doesn’t this feel so powerful? Well, it is for me and completely loves using them, I hope you do too.

Gargeya Sharma

B.Tech in Computer science (4th year)
Specialized in Data Science and Deep learning
Data Scientist Intern at Upswing Cognitive Hospitality Solutions
For more information, check out my GitHub Home Page.

LinkedIn GitHub

Photo by Kaley Dykstra on Unsplash

Gargeya