Overview
- Rosetta is Facebook’s own large-scale machine learning system to deal with text in images
- The system extracts text from more than a billion Facebook and Instagram images each day in real time, and recognises multiple languages
- The text extraction works in two steps: detection and recognition
Introduction
Facebook processes a ludicrous number of images per day. Even with the recent controversies it has had to face, the number of people using the platform has not diminished a whole lot. And the uploading and sharing of photos continues unabated.
Now Facebook faces a challenge every day. Quite a lot of these images have text in them (could be a meme, quote, street sign, menu, business card, etc.). How can the big tech giant make use of this text? How can they extract it and use it to improve the user experience?
Given the sheer amount of images Facebook has to process, using a traditional optical character recognition (OCR) software won’t cut it. The OCR might be able to recognize the characters, but it definitely won’t understand the context.
Step up Rosetta, Facebook’s own large-scale machine learning system.
Rosetta extracts text from more than a billion public Facebook and Instagram images (and even videos) on a daily basis. The text isn’t just limited to English, Rosetta is able to recognize multiple languages in real time. This text data is then fed to a text recognition model that has been trained on classifiers with the singular aim of understanding the context of the text in each image.
Text extraction is performed in two steps, independent of each other:
- Detection: The system detects rectangular regions that could potentially contain text. Facebook’s approach was based on using Faster R-CNN, a state-of-the-art object detection framework
- Recognition: Once the regions had been detected, a CNN was used to recognize and transcribe the world present in each region
The below image is a nice illustration of Rosetta’s architecture:
I strongly recommend reading the entire blog post on Facebook’s Code site. It is a marvellous explanation of how Rosetta works, and especially how the detection and recognition models were designed from scratch. Alternatively, you can watch the below video from KDD2018 which summarises the inner workings of Rosetta in under two and half minutes:
Our take on this
It’s always a pleasure to read Facebook and Google’s AI research posts. There’s so much knowledge to be gained with each breakthrough or service they write about. Most of us in the data science domain must have wondered how a behemoth like Facebook uses machine learning in real-world cases (except their news feed, of course) and bit by bit, the curtains are drawn back.
If you’re a NLP enthusiast, the text detection using Faster R-CNN approach sounds pretty intriguing, doesn’t it? Rosetta is already being heavily used by Facebook and Instagram.There’s a lot more work to be done since text comes in all forms and structures, and Facebook’s research team is just getting started.
Subscribe to AVBytes here to get regular data science, machine learning and AI updates in your inbox!
Senior Editor at Analytics Vidhya.Data visualization practitioner who loves reading and delving deeper into the data science and machine learning arts. Always looking for new ways to improve processes using ML and AI.
