Learn How to Build Face Detection System

So in this section, we’ll understand the problem statement that we have picked for this article.

The objective of this article is to build a model that can detect faces from the images. So by the end of this article, we’ll be able to build a model that can take images as shown here. As an input, detect the faces in that image that returns the bounding boxes for the locations of these faces present in the image.

So we’ll be training the model, and we know that in order to train the model, we need the training data. So let’s look at the dataset that we’ll be using in order to train the face detection model.

In this module, we’ll be using the wider face dataset, which is benchmark data for the face detection tasks. It contains more than 32000 images comprising of approximately 0.4 million faces and this data set contains images and faces which have a large variation in scale, occlusion pose and so on.

So scale basically refers to the image size, and based on the size of the image, the dataset is grouped into three scales: small, medium and large.

Similarly, occlusion also has three categories, which is no occlusion, partial occlusion and heavy occlusion. The wider dataset contains images from a range of 60 even categories like the parade, festival, football, tennis and many more.

So the diversity of the data set is huge, and these even categories are further divided into three sets based on the ease of detection which is easy, medium and hard.

The categories marked in the above image,  green belong to the easy set, those in red belong to the medium Set, and the blue ones belong to the hard Set. In this article, we’ll be working with images belonging to the easy Set.

Also, we’ll be using the detectoron2 library in order to build the face detection model, since it provides the state-of-the-art implementation for object detection tasks.

And by now, we know that in order to work with detectron2, our dataset must follow a specific format. So in the next section, we’ll first look at the format of the wider phase data set and then see how the dataset should be pre-processed in order to use it for the directron2 library.

Converting the Annotations of the WIDER FACE Dataset as per Detectron2

Till now we have understood the face detection problem and the dataset which we’ll be using to build the face detection model. In the above section, we discussed that we’ll be using detectron2 to build the model.

So, let’s quickly have an overview of detectron2. detectoron2 is a platform for object detection and segmentation tasks. It is created by the research team of Facebook AI. And detectron2 implements state-of-the-art architectures like faster R-CNN retina net and so on.

Now, let’s look at the format of the dataset we currently have. So, this is what the annotations of the wider face dataset look like. We have images containing faces, and corresponding to these images, we have the annotations.

The first thing here represents the file name then we have the number of faces present in a particular image. And since this image has a single face the value here is one. Finally, we have the bounding box coordinates for this image. These bounding boxes follow the following format. We have the x-min,y-min values followed by the height and width of the bounding box. Also, We have the values representing the different measures of variabilities that we have discussed in the above section, Including the blur expression, illumination, occlusion and so on.

So this is what the annotations of wider face data sets look like. Now here is another example from the dataset.

So first of all we have the image name, followed by the number of faces in the image that is 3 in this case, followed by the bounding box coordinates. The bounding boxes for each face are represented using a separate dictionary. For this image, since we have three faces, there will be three different dictionaries.

Now, the above image is the annotation from the wider phase dataset. And in order to use this dataset in detectron2, we must convert the annotations to this format as shown below.

So, we first have a dictionary for the bounding boxes. and instead of having x-min, y-min, width and height for detectron2. We should have the x-min, y-min, x-max and y-max values that we’ll be using here. The category id here represents the category of this image. Then we have to pass the file name, which will be the name of the above image, followed by the height, width and image id.

So this is the format that detectron2 expects as input and hence we’ll have to pre-process our data accordingly. So while creating the face detection model we’ll be pre-processing these input formats into the format required for detectron2.

So that’s all on the data preparation part. in the next section, we’ll discuss the steps that we’ll be following to build the face detection model, and then finally we will work on the implementation.

Steps to Solve the Face Detection Problem

In this section, we will look at the steps that we’ll be following, while building the face detection model using detectron2. So we’ll start with these steps:-

1. Install Dependencies
2. Loading and pre-processing the data
3. Creating annotations as per Detectron2
4. Register the dataset
5. Fine Tuning the model
6. Evaluating model performance

I  have already uploaded the dataset on the drive and in order to run the notebooks at your end, you should also upload the data set on your drive.

Dataset Link:- https://drive.google.com/drive/folders/1bbPG4pcMT_rMyqInAissTsIOMYJ3rnPj

So, here are the files which are available in the dataset. we have an easy.txt file, that includes the names of classes belonging to the easy set. Then we have the wider space split.zip which contains the annotations for images in the training and the validation set. WIDER_train contains the training images and WIDER_val contains the validation images.

Till now we have understood what is face detection, how we have to annotate the data set as per detectron2 and the steps that we’ll be following to build our face detection model.

Now as already discussed, there can be scenarios where we have only a single face in the image and there can be situations where the task is to detect multiple faces in the images.

So in this article, we’ll solve both of these tasks. We’ll first build a model that can detect a single face from the image, and then we’ll build another model which will be able to detect multiple objects from the image. So, in this article, our focus will be on building a model that can detect a single face. So to build a face detection system that can detect a single face from the image, a few steps that we are going to follow, which we have discussed in the last section.

1. Install Dependencies

We’ll first start with installing all the dependencies. First of all, we are installing the 5.1 version of the PYML library. This is a prerequisite for detectoron2. And if we have an older version of this library some of the functionalities of detectron2 might not work correctly.

Then we are installing the detectron2 library and we have seen this step earlier while we were implementing the RetinaNet model.

# install required version
!pip install pyyaml==5.1

# installing detectron2
!pip install detectron2==0.1.3 -f https://dl.fbaipublicfiles.com/detectron2/
                    wheels/cu101/torch1.5/index.html

2. Loading and pre-processing the data

The next step is to load the dataset and preprocess it. since the data set is present in google drive, we have to first mount the drive.

# mounting drive
from google.colab import drive
drive.mount('drive')

Next, we are extracting the files which are in the zip format. So we are extracting our training, validation, and annotations. Now we are reading the annotation files for both the training and the validation set.

# extracting files
!unzip '/content/drive/My Drive/Wider_dataset/WIDER_train.zip'
!unzip '/content/drive/My Drive/Wider_dataset/WIDER_val.zip'
!unzip '/content/drive/My Drive/Wider_dataset/wider_face_split.zip'

So to read the files, we’ll be using the pandas library and hence importing that here first. And then we are specifying the paths for the annotation files of the training and the validation set and then read them using the read_csv function of pandas.

# reading files
import pandas as pd

# specify path of the data
path_train = '/content/wider_face_split/wider_face_train_bbx_gt.txt'
path_val = '/content/wider_face_split/wider_face_val_bbx_gt.txt'

# reading data
train = pd.read_csv(path_train,header=None)
val = pd.read_csv(path_val,header=None)

Let’s quickly look at the CSV files. So here are the first 10 rows from the train CSV file that we have. Here we have the file names followed by the number of faces in that image, and then the bounding box coordinates and the same part. And you can see is repeated. So currently the format is not readable. So let’s convert this format to a more meaningful and readable form. Which should look something like this. We would want the format where we have names in a separate column then numbers of faces as a separate variable and then the bounding box coordinates. hence we’ll reformat the data set accordingly.

# pre-processing data
# this function accepts the dataframe and returns modified dataframe
def reformat(df):
  # fetch values of first column
  values = df[0].values
  # creating empty lists
  names=[]
  num_faces=[]
  bbox=[]
  # fetch values into corresponding lists
  for i in range(len(values)):
      
      # if an image
      if ".jpg" in values[i]:
        
        # no. of faces
        num=int(values[i+1])
        
        # append image name to list
        names.append(values[i])
        
        # append no. of faces to list
        num_faces.append(num)
        
        # create bbox list
        box=[]
        for j in range(i+2,i+2+num):
          box.append(values[j])
        
        # append bbox list to list
        bbox.append(box)
  
  return pd.DataFrame({'name':names,'num_faces':num_faces,'bbox':bbox})

we are going to convert the training and the validation data sets and store it as train and val.

# pre-processing the data 
train = reformat(train)
val = reformat(val)

So let’s look at the first few rows of these data sets. So we are first printing the head of the train. you can see that this data set is now in the required format.

# first 5 rows of the pre-processed data
train.head()

let us also look at the shape of the training data and the shape of the validation data. so we can see that in the training data we have 12 880 rows and in the validation, we have 3226 rows.

# shape of the training data
train.shape

The output is:- (12880, 3)

# shape of validation data
val.shape

The output is:-  (3226, 3)

Next, we’ll do some pre-processing on this data set so here we are adding the complete path before the file name and for training, we are adding the path wider_ train/images and for validation, the path will be wider_val/images.

# adding full path
train['name'] = train['name'].apply(lambda x: 'WIDER_train/images/'+x )
val['name']   = val['name'].apply(lambda x: 'WIDER_val/images/'+x )

After applying this the new dataset would look something like the below image. So we’ll have the complete path for all of these images.

# first 5 rows
train.head()

Next, we are converting the bounding box coordinates to floating points using the float_function of the numpy library. Hence we are first importing the numpy library here and then converting the bounding box coordinates for both training and the validation set and again printing the first five rows of the data set.

# converting bbox to floating point
import numpy as np
train['bbox'] = train['bbox'].apply
                        (lambda row:[ np.float_(annos.split()) for annos in row] )
val['bbox']   = val['bbox'].apply
                    (lambda row:[ np.float_(annos.split()) for annos in row] )

# first 5 rows
train.head()

So the above image, the output confirms that the bounding boxes are converted into floating points. Here so you can see this is bbox column value 27.0, 26.0, and so on.

Next, we are going to extract the class names from the names of the images. So remember that we discussed previously will only be using the images belonging to the easy set. Hence these class names will help us to get the images which are belonging to the easy set. So here we are extracting the class name from the name of the images and
storing them in a new variable named as a class for both the training as
well as the validation data sets.

# extracting class names
train['class']= train['name'].apply(lambda x:x.split("/")[2].split("--")[1])
val['class']  = val['name'].apply(lambda x:x.split("/")[2].split("--")[1])

# first 5 rows
train.head()

you can see that now we have a new column class that has the class against each of these images.

Next, we are going to read the easy.txt file which contains the names of the classes belonging to the easy set. So we are reading this file here using the read_csv function and printing the values that we get.

# reading class names
easy_df = pd.read_csv('drive/My Drive/Wider_dataset/easy.txt',header=None)
easy_labels = easy_df[0].values
# easy labels
easy_labels

Here is a list of the classes that belong to the easy set. so we have a total of 20 classes here.

Now we’ll select only those images which belong to any of these above-mentioned classes. So here we are running a for loop for only the easy classes and fetching the rows which have the easy categories.

# creating empty dataframes
train_df, val_df= pd.DataFrame(), pd.DataFrame()

# fetching rows of easy classes only
for i in easy_labels:
  train_df = pd.concat( [train_df, train[train['class']==i]] )
  val_df   = pd.concat( [val_df, val[val['class']==i]] )

We have taken a subset from the data set for both training and validations. As discussed earlier the aim of this article is to build a face detection model that will work for single faces and hence we are only selecting those images which have a single face so we are doing this for both our training and validation.

Now before we go ahead, let’s quickly check the train shape and the validation shape.

# shape of dataframe
train_df.shape, val_df.shape

So we have 1000 images in train and 274 images in the validation data. Next, We will see how to convert the annotations of a wider dataset to the annotations of Detectron2.

3. Creating annotations as per Detectron2

Now, we’ll convert the annotations of our dataset as per the requirements of the detectoron2 library. So here is what the annotations should look like. We’ll first have the file name and then the number of faces in that image followed by the bounding box coordinates which are of the format Xmin, Ymin, width, and height, and finally the class.

So we want the annotations to be in the below format. We’ll first see how to get this format for a single image and then we’ll write a generalized function that will convert the annotations format for all of these images.

# custom annotation format
idx=0
values = train_df.values[idx]
print(values)

Source:- Author