Overview
- A developer has created an algorithm that adds an external element to a painting and makes it look undistinguishable from the original
- The algorithm is a convolutional neural network (CNN) that was trained on a dataset of 80,000 paintings
- Check out the examples in the article below, read up on the paper and download the python code
Introduction
If you have ever tried inserting images of objects into a photo, you would know that the most difficult task is to make the boundaries merge with the original photo. Even when you do manage it, the resulting photo clearly looks like it’s been photoshopped. But with deep learning, this is slowly being overcome.
We covered NVIDIA’s ‘FastPhotoStyle’ library that makes styling photos simpler and now a developer has come up with an algorithm that takes a painting, adds an external element to it, and harmonizes it to make it look almost undistinguishable from the original painting.
The developer trained a VGG convolutional neural network (CNN) on a dataset of 80,000 paintings which he collected from wikiart.org. The CNN estimates the stylization level of a painting and then adjusts the weights accordingly. Since they have not yet released any pre-trained model for this, you will have to adjust the weights manually if you use this on an image.
It is a two-pass algorithm, an improvisaatoin on the previously used single-pass approach. In order to make the object merge with the painting, VGG neural network is used.
Below is the breakdown of the algorithm behind this:
- The input image and style are processed with the VGG network to produce a set of activation values. These capture the statistics that represent the style of each image
- The style activations are then mapped to the input ones, using either the entire set, or nearest neighbors neural patches
- Once that is done, the output image is reconstructed through an optimization process, by preserve the content of the input image and simultaneously matching the visual appearance of the style image
This algorithm produces far more precise results than photo compositing or global stylization techniques and it achieves levels of edits that have so far been very difficult to achieve.
Check out some of the images below that utilized this model. The original painting is on the left, the naive composite in the middle and the final output is on the right.
You can read the paper in full here and also check out the GitHub respository to get the python codes.
Our take on this
This is one of the coolest things we have covered on AVBytes. Some of the stylized images did not turn out as well but most of them are deeply impressive. This is the first example I’ve seen of such an idea that can be packaged into a business idea (perhaps as a filter in tools like Photoshop). But the developer has mentioned that this is only intended for personal and not commercial use so he must have something in mind there!
Go ahead and try it out on your machines. Let us know your experience in the comments section below!
Subscribe to AVBytes here to get regular data science, machine learning and AI updates in your inbox!
Senior Editor at Analytics Vidhya.Data visualization practitioner who loves reading and delving deeper into the data science and machine learning arts. Always looking for new ways to improve processes using ML and AI.
