Overview
- Facebook’s Artificial intelligence system can manipulate closed eyes in photos
- The researchers used a GAN network to learn about the shape, color, and other minute details on the eyes; it’s called an ExGAN
- The testing process went about as well as you can imagine; most people failed to tell the difference between real and manipulated images
Introduction
We’ve all taken not-so-great pictures and tried using various photo editing techniques to rehash them and make them presentable. With the advances in technology and hence software applications, most of these tasks are now a piece of cake.
But there are still certain things like ‘closed eyes’ in pictures which have eluded developers. How many times have you had to re-take pictures because someone invariably closed their eyes? It’s an all to common occurrence and yet there’s no “easy” solution for it. Sure there are apps that attempt to make things better but so far, there hasn’t been a convincing solution. Emphasis on “so far”, because Facebook’s AI research team has used machine learning to build a model that literally opens closed eyes in pictures.
These researchers have worked on a technique called “in- painting”, which basically does the job of filling in spaces in a picture according to what it thinks belongs there. In this instance, Facebook’s team used Exemplar Generative Adversarial Network (ExGAN). Basically the training data included faces of the people with the eyes open, so the ExGAN was able to learn which set of eyes belonged to which face. Additionally, the ExGAN also learned about the shape, color and other minute details about the eyes.
As explained in the research paper published by the team, below is a high level overview of how a GAN works:
- One part of the system learns to recognize faces. It learns the shape, color and other features of the eyes of a particular person
- The other part of the system repeatedly creates images based on feedback from the recognition part
When the model was tested and the results were shown to other people, quite a lot of them were unable to tell the difference between the original photo and the manipulated one. There were some obvious issues with the results. For example, if a lock of hair was coming over the eyes, the model struggled to understand how to replicate that and gave out some bizarre results. But it won’t take long for Facebook to fix that, that’s for sure.
Our take on this
This isn’t the first time in-painting has been used – Adobe uses it regularly for context-based image painting. But the face that it’s being used for specific eye-opening purposes definitely makes it a first in that regard.
Previously deep neural networks (DNN) have been used for a similar task of replacing closed eyes but the results were not very promising. DNNs are able to learn about closed eyes but they fail to take into account the other features of the face (for instance the shape, or tone) and hence the images produced are easily distinguishable from the original photos.
ExGAN on the other hand has presented really impressive results. From a machine learning perspective, this is an awesome use case.
Subscribe to AVBytes here to get regular data science, machine learning and AI updates in your inbox!
An avid reader and blogger who loves exploring the endless world of data science and artificial intelligence. Fascinated by the limitless applications of ML and AI; eager to learn and discover the depths of data science.
