Decode Meta AI’s Tool for Animating Children’s Drawings

Boring drawings are a thing of the past! Get ready to unleash your child’s creativity like never before. In this blog, we reveal an exciting breakthrough method for animating your child’s drawings of the human figure. Say goodbye to static art and hello to dynamic animations! We’ll provide a concise summary of Meta AI’s research paper “A Method for Animating Children’s Drawings of the Human Figure” which introduces this groundbreaking technique. Get ready to discover how the Meta AI tool for animating children’s drawings can revolutionize the way we view children’s art and bring their imaginations to life. You won’t want to miss it!

1. Overview of the Research

Meta AI recently developed a framework to animate children’s drawings of the human figure. It’s the first kind of experiment that can automatically create the animation of children’s drawings of human figures. Given an image of a child drawing, it converts the human character into animation.

Insane right? You also have the option to record your own video and synchronize the movements of the animated characters with your own physical motions. You can try the demo here.

The team collected the dataset of 178,166 children’s drawings, along with the annotations. The annotations include the bounding box, segmentation mask, and joint location annotations.

Broadly, there are 2 primary outcomes of the research-

1. Developing a framework to animate the children’s drawings of human figures
2. Building the first kind of dataset of children’s drawings.

Apart from these, they have also open-sourced the entire code to further continue the development and research on it.

Also Read: Meta to Commercialize Generative AI by December

2. Understand the Framework

Now, let us understand the framework developed. The process of creating animations from children’s drawings of the human figure involves four main steps as mentioned below.

1. Human Figure Detection
2. Figure Segmentation
3. Pose estimation
4. Animation

We will discuss each step in detail now.

1. Human Figure Detection

The first step is to detect the human figure from the drawing i.e. to identify the human figure and its location from the image. It’s an object detection problem since we are interested in exactly knowing the position of the human figure from the image.

In order to detect the human figure, the researchers used the pre-trained model (Mask R-CNN with ResNet50+FPN backbone) trained on the MS COCO dataset. MS COCO dataset consists of real world images (330K) of different objects and categories (80 object categories). There is no customization made in the model architecture. The model is finetuned on the labeled dataset containing children’s drawings and bounding boxes around human figures.

Are you concerned about finetuning given that we possess pre-trained models? This will be revealed in a while. Now, we will compare the performance of the pre-trained model and fine-tuned model-

Finetuning the model on the custom dataset improved the performance of the detections. It also produces inaccurate detections sometimes. Now, we will look at some of them.

2. Figure Segmentation

The next step is to segment the human figure from the image. This is a very critical step in the process because the segmented character is used to create a 2d textured mesh. 2d textured mesh is commonly used in computer graphics to create realistic 3-dimensional objects. It is created by applying a 2D texture (an image) onto a 3D mesh (a collection of interconnected triangles that form the surface of the object).

In order to create a 2D textured mesh, it is necessary for the polygon to be closed. This means that the resulting segmentation mask must also be a closed polygon.

The researchers observed that Mask R CNN did not provide the results as expected. Hence, the traditional image segmentation approach is used to segment the figure. The reason is simple and highly accurate.

Here are the steps involved in segmentation-

1. Image Resizing: The first step is to crop the human figure from the drawing with the help of a bounding box and then resized it to the width of 400px keeping the aspect ratio constant.
2. Adaptive Thresholding: The cropped image is converted into grayscale and adaptive thresholding is done.
3. Morphological Closing: Morphological closing removes the noise and connects the foreground pixels from an image.
4. Dilation: Dilation ensures that the foreground pixels are connected in an image.
5. Flood filling: Fill the edges of the closed groups in an image with white pixels to ensure that the edges are connected.
6. Retain the Largest Polygon: Finally, retain the largest polygon based on area.

3. Pose estimation

The next step is to locate the key points from the human figure. Knowing the key points will help us to create the motion of a character as required. We will use pose estimation to do it.

The current pre-trained models available were inadequate because they were designed to recognize the posture of real people, while our situation involves human drawings, which differ significantly from real-life.

Hence, the team built their own pose detection model using resnet 50 as the backbone and keypoint head that predicts the heatmaps of each keypoint in a top-down manner.

4. Animation

In the last step, 2d textured mesh and character skeleton are created based on the segmentation mask and pose estimation. At last, the character rig undergoes animation by moving its joints and employing shape manipulation techniques based on the as-rigid-as-possible (ARAP) algorithm to adjust the character mesh into different poses.

Also Read: More Profound than Fire or Electricity: Google CEO Sundar Pichai on AI Developments

3. Results of Meta AI Tool for Animating Drawings

The researchers first evaluated the performance of the models individually in terms of [email protected]:0.95. But, this isn’t always the best way to evaluate the models for making animations.

Suppose we have an object detection model, and it yields an iou of 0.9. While this prediction may be reasonable, it would not be appropriate for an animation if the hands are not included in the detection.

Hence, the right way to evaluate the model performance is to check the percentage of model predictions that can successfully be used for animation.

End Notes

We hope this blog post has piqued your interest and inspired you to explore Meta AI tool for animating drawing technique further. Don’t hesitate to dive in and see for yourself the amazing results this breakthrough technology can bring. You can try the demo here. With Meta AI’s tech, your child’s drawings will never be boring again!

Thank you for reading, and we wish you all the best in your artistic endeavors. You can find this Meta research paper here!

Aravind Pai

Aravind Pai is passionate about building data-driven products for the sports domain. He strongly believes that Sports Analytics is a Game Changer.