A Classic Computer Vision Project – How to Add an Image Behind Objects in a Video

Overview

• Adding an image behind a moving object is a classic computer vision project
• Learn how to add a logo in a video using traditional computer vision techniques

Introduction

I was thrown a challenge by one of my colleagues – build a computer vision model that could insert any image in a video without distorting the moving object. This, as you imagine, was quite an intriguing project and I had a blast working on it.

Working with videos is notoriously difficult because of their dynamic nature. Unlike images, we don’t have static objects that we can easily identify and track. The complexity level goes up several levels – and that’s where our hold on image processing and computer vision techniques comes to the fore.

I decided to go with a logo in the background. The challenge, which I will elaborate on later, was to insert a logo in a way that wouldn’t impede the dynamic nature of the object in any given video. I used Python and OpenCV to build this computer vision system – and have shared my approach in this article.

We will be using the image processing concepts and OpenCV in this article. We have a collection of comprehensive free courses and an article on these topics to get you up to speed:

• A Comprehensive Learning Path for Deep Learning in 2020
• Getting Started with Neural Networks
• OpenCV Functions | OpenCV for Computer Vision

Table of Contents

1. Understanding the Problem Statement
2. Getting the Data for this Project
3. Setting the Blueprint for our Computer Vision Project
4. Implementing the Technique in Python – Let’s Add the Logo!

Understanding the Problem Statement

This is going to be quite an uncommon use case of computer vision. We will be embedding a logo in a video. Now you must be thinking – what’s the big deal in that? We can simply paste the logo on top of the video, right?

However, that logo might just hide some interesting action in the video. What if the logo impedes the moving object in front? That doesn’t make a lot of sense and makes the editing looks amateurish.

Therefore, we have to figure out how we can add the logo somewhere in the background such that it doesn’t block the main action going on in the video. Check out the video below – the left half is the original video and the right half has the logo appearing on the wall behind the dancer:

This is the idea we’ll be implementing in this article.

Getting the Data for this Project

I have taken this video from pexels.com, a website for free stock videos. As I mentioned earlier, our objective is to put a logo in the video such that it should appear behind a certain moving object. So, for the time being, we will use the logo of OpenCV itself. You can use any logo you want (perhaps your favorite sports team?).

You can download both the video and the logo from here.

Setting the Blueprint for our Computer Vision Project

Let’s first understand the approach before we implement this project. To perform this task, we will take the help of image masking. Let me show you some illustrations to understand the technique.

Let’s say we want to put a rectangle (fig 1) in an image (fig 2) in such a manner that the circle in the second image should appear on top of the rectangle:

So, the desired outcome should look like this:

However, it is not that straightforward. When we take the rectangle from Fig 1 and insert it in Fig 2, it will appear on top of the pink circle:

This is not what we want. The circle should have been in front of the rectangle. So, let’s understand how we can solve this problem.

These images are essentially arrays. The values of these arrays are the pixel values and every color has its own pixel value. So, we would somehow set the pixel values of the rectangle to 1 where it is supposed to be overlapping with the circle (in Fig 5), while leaving the rest of the pixel values of the rectangle as they are.

In Fig 6, the region enclosed by blue-dotted lines is the region where we would put the rectangle. Let’s denote this region by R. We would set all the pixel values of R to 1 as well. However, we would leave the pixel values of the entire pink circle unchanged:

Our next step is to multiply the pixel values of the rectangle with the pixel values of R. Since multiplying any number by 1 results in that number itself, so all those pixel values of R that are 1 will be replaced by the pixels of the rectangle. Similarly, the pixel values of the rectangle that are 1 will be replaced by the pixels of Fig 6. The final output will turn out to be something like this:

This is the technique we are going to use to embed the OpenCV logo behind the dancing guy in the video. Let’s do it!

Implementing the Technique in Python – Let’s Add the Logo!

You can use a Jupyter Notebook or any IDE of your choice and follow along. We will first import the necessary libraries.

Import Libraries

Note: The version of the OpenCV library used for this tutorial is 4.0.0.

Load Images

Next, we will specify the path to the working directory where the logo and video are kept. Please note that you are supposed to specify the “path” in the code snippet below:

So, we have loaded the logo image and the first frame of the video. Now let’s look at the shape of these images or arrays:

logo.shape, frame.shape

Output: ((240, 195, 3), (1080, 1920, 3))

Both the outputs are 3-dimensional. The first dimension is the height of the image, the second dimension is the width of the image and the third dimension is the number of channels in the image, i.e., blue, green, and red.

Now, let’s plot and see the logo and the first frame of the video:

plt.imshow(logo)
plt.show()

plt.imshow(cv2.cvtColor(frame,cv2.COLOR_BGR2RGB))
plt.show()

Technique to Create Image Mask

The code snippet above will load the frames from the video, pre-process it, and create HSV images and masks and finally insert the logo into the video. And there you have it!

End Notes

In this article, we covered a very interesting use case of computer vision and implemented it from scratch. In the process, we also learned about working with image arrays and how to create masks from these arrays.

This is something that would help you when you work on other computer vision tasks. Feel free to reach out to me if you have any doubts or feedback to share. I would be glad to help you.

Prateek joshi

Data Scientist at Analytics Vidhya with multidisciplinary academic background. Experienced in machine learning, NLP, graphs & networks. Passionate about learning and applying data science to solve real world problems.