Learn How To Do Real-Time Background Replacement using OpenCV and CVzone

This article was published as a part of the Data Science Blogathon

Introduction

OpenCV is an open-source computer vision library that provides privileges to play with different images and video streams and also helps in end-to-end projects like object detection, face detection, object tracking, etc.

CVzone is a computer vision package that makes us easy to run like face detection, hand tracking, pose estimation, etc., and also image processing and other AI functions. At the core, it uses OpenCV and MediaPipe libraries. Check here for more information.

Why Real-Time Background Removal is required?

For many reasons, the background of the video needs to be modified as there are so many other interruptions in the background or the background colour doesn’t suit the person due to which background or the color needs to be modified. So, we use the real-time background replacement technique to substitute the backgrounds and add replace them with the desired content.

Popular background removal techniques

Image clipping path – This technique is used if the subject of the image has sharp edges. All those elements that fall outside the path will be eliminated.

Image cut-out – Here we cut the required region or subject in a frame and remove the background.

Image masking – If the images have frills or fine edges we can use image masking techniques.

Erasing the background – Erasing the background of an image using any different tools

Many famous applications use a background removal technique and replace it with a custom one. Here we are going to implement something similar, but using OpenCV and CVzone.

Let’s start the implementation

Install the required modules.

-- pip install OpenCV-python

-- pip install cvzone

-- pip install mediapipe

First, let us check if our webcam is working fine.

import cv2

cap = cv2.VideoCapture(0)
cap.set(3, 640)
cap.set(4, 480)

while True:
success, img = cap.read()
cv2.imshow("Image", img)
if cv2.waitKey(1) & 0xFF == ord('q'):
break

The above code pops up a window if you have a webcam, Here the frame size is 640 X 480. So we need to take a note here because the background replacing images should be of the same size as the frame, that is 640 X 480.

Now create the folder inside the project directory here, I am creating a folder with the name ‘BackgroundImages’. You can download any images or any number of images and place them in this directory.

The project structure will look like the image given below:

Let us write a small piece of code in a separate python file to resize all the images in the folder ‘BackgroundImages’ to 640 X 480.

import cv2
import os

for root, subdirs, files in os.walk('D:/pycharmprojects/BackgroundRemover/BackgroundImages'):
    for f in files:
        if f.endswith('jpg'):
            # print(f)
            img = cv2.imread('D:/pycharmprojects/BackgroundRemover/BackgroundImages/' + f)
            img = cv2.resize(img, (640, 480))
            cv2.imwrite('D:/pycharmprojects/BackgroundRemover/BackgroundImages/'+f, img)
            print(*["Image", f, "is resized to 640 X 480"])

The above code will read the image (jpg) files in the specified folder and resize all the images to 640 X480 at once.

Now we are all set to implement the background replacement technique.

Import the required modules

import cv2
import cvzone
from cvzone.SelfiSegmentationModule import SelfiSegmentation
import os

Here in the above module, ‘SelfiSegmentation’ is used to remove the background of the frame and replace it with our images in the directory.

cap = cv2.VideoCapture(0)
cap.set(3, 640)
cap.set(4, 480)
# cap.set(cv2.CAP_PROP_FPS, 60)
segmentor = SelfiSegmentation()
fpsReader = cvzone.FPS()

# imgBG = cv2.imread("BackgroundImages/3.jpg")

listImg = os.listdir("BackgroundImages")
imgList = []
for imgPath in listImg:
    img = cv2.imread(f'BackgroundImages/{imgPath}')
    imgList.append(img)

indexImg = 0

In the above code, we take input from the webcam and also set the frame width to 640 X 480. Then we call SelfiSegmentation() and assign it to a variable called segmentor, and in order to display the frames per second(fps) in the output frames, we use cvzone.FPS() function.

Then we create a list of images present in the BackgroundImages folder and we loop through that list and read each and every image and append it to an empty list. The initial index is set to zero.

while True:
    success, img = cap.read()
    # imgOut = segmentor.removeBG(img, (255,0,255), threshold=0.83)
    imgOut = segmentor.removeBG(img, imgList[indexImg], threshold=0.8)

    imgStack = cvzone.stackImages([img, imgOut], 2,1)
    _, imgStack = fpsReader.update(imgStack)
    print(indexImg)
    cv2.imshow("image", imgStack)
    key = cv2.waitKey(1)
    if key == ord('a'):
        if indexImg>0:
            indexImg -=1
    elif key == ord('d'):
        if indexImg<len(imgList)-1:
            indexImg +=1
    elif key == ord('q'):
        break

Now the main part, inside a while loop reads the frames from the webcam, and then we use segmentor.removeBG() function to remove the background from the frames and replace it with our images in the directory. In the above code, you can see we have passed three parameters to segmentor.removeBG() function, that is image frame from webcam (img), then the list of images present in the directory along with an index of image (imgList[indexImg]) and finally the threshold. The threshold cuts everything if it’s set to 1, here we set it to 0.8, for better edges, play with different threshold values.  

Then we stack the images using cvzone.stackImages, here we will get the output of the background replaced image or frames. Then using a simple if statement we assign keys to change the background. For example, if we have 10 background images, as per the above code we can use key “a” or key “d” to change the background of the frames.

The entire code is given below.

import cv2
import cvzone
from cvzone.SelfiSegmentationModule import SelfiSegmentation
import os

cap = cv2.VideoCapture(0)
cap.set(3, 640)
cap.set(4, 480)
# cap.set(cv2.CAP_PROP_FPS, 60)

segmentor = SelfiSegmentation()
fpsReader = cvzone.FPS()

# imgBG = cv2.imread("BackgroundImages/3.jpg")

listImg = os.listdir("BackgroundImages")
imgList = []
for imgPath in listImg:
    img = cv2.imread(f'BackgroundImages/{imgPath}')
    imgList.append(img)

indexImg = 0

while True:
    success, img = cap.read()
    # imgOut = segmentor.removeBG(img, (255,0,255), threshold=0.83)
    imgOut = segmentor.removeBG(img, imgList[indexImg], threshold=0.8)

    imgStack = cvzone.stackImages([img, imgOut], 2,1)
    _, imgStack = fpsReader.update(imgStack)
    print(indexImg)
    cv2.imshow("image", imgStack)
    key = cv2.waitKey(1)
    if key == ord('a'):
        if indexImg>0:
            indexImg -=1
    elif key == ord('d'):
        if indexImg<len(imgList)-1:
            indexImg +=1
    elif key == ord('q'):
        break

The output screenshots are given below

The entire code is also available here:

https://github.com/BakingBrains/Real-Time_Background_remover 

Reference:

https://www.youtube.com/watch?v=k7cVPGpnels

My LinkedIn

Thank you

The media shown in this article are not owned by Analytics Vidhya and are used at the Author’s discretion.

Syed Abdul Gaffar

I thrive on the thrill of the challenge, tackling complex problems and crafting innovative AI solutions that make a difference. Whether it's optimizing or building sustainable AI ecosystems, I believe in harnessing the power of AI for the greater good. Let's brainstorm, collaborate, and change the world, one byte at a time.