Exploring MoViNets: Efficient Mobile Video Recognition

Introduction

Let us dive into the fascinating world of mobile video recognition with “MoViNets Unleashed”! This blog takes you on an exploration of how MoViNets are transforming video analysis on mobile devices, combining cutting-edge techniques like neural architecture search, stream buffering, and temporal ensembling. Discover how these innovative models, built on the robust architectures, are pushing the boundaries of what’s possible in real-time video processing, all while staying lightweight and efficient. Join us as we unravel the technology behind MoViNets and explore their potential to revolutionize mobile video applications, from streaming to surveillance, in the palm of your hand.

Learning Objectives

• Explore the MoViNet Search Space.
• Understand stream buffer technique for online-inference in real-time.
• Analyze the strengths, and challenges of MoViNet.
• Generate inference using pretrained MoViNet.
• Navigate through it’s applications in real-world scenarios.

This article was published as a part of the Data Science Blogathon.

What is MoViNet?

MoViNet, short for Mobile Video Network, is an advanced video recognition model specifically optimized for mobile and resource-constrained devices. It leverages cutting-edge techniques such as Neural Architecture Search (NAS), stream buffering, and temporal ensembling to deliver high accuracy and efficiency in real-time video processing. Designed to handle the unique challenges of video analysis on mobile platforms, MoViNet can process video streams efficiently while maintaining low memory usage, making it suitable for applications ranging from surveillance and healthcare monitoring to sports analytics and smart home systems.

Key Features of MoViNet

Let us now explore key features of MOViNet below:

MoViNet search space

The MoViNet search space is a structured approach to designing efficient video recognition models for mobile devices. It starts with a foundation based on MobileNetV3, expanding it into 3D to handle video inputs. By using Neural Architecture Search (NAS), the framework explores different architectural configurations, like kernel sizes, filter widths, and layer depths, to find the best balance between accuracy and efficiency. The goal is to capture the temporal aspects of video data without overwhelming the limited resources available on mobile hardware.

This search space enables the development of a range of models, each optimized for specific use cases. From lightweight models suited for low-power devices to more complex architectures designed for edge computing, the MoViNet framework allows for customization based on the needs of the application. The use of NAS ensures that each model is tailored to achieve the best possible performance within the constraints, making MoViNet a practical solution for mobile video recognition tasks.

Stream Buffers

Stream buffers are used in MoViNet models to reduce memory usage when processing long videos. Instead of evaluating the entire video at once, the video is split into smaller subclips. Stream buffers store the feature information from the edges of these subclips, allowing the model to keep track of information across the entire video without reprocessing overlapping frames. This method preserves long-term dependencies in the video while maintaining efficient memory usage. By using causal operations like CausalConv, the model processes video frames sequentially, making it suitable for real-time video streaming with reduced memory and computational requirements.

Temporal Ensembles

Temporal ensembles in MoViNets help restore the slight accuracy drop caused by using stream buffers. This is done by training two identical models independently, each processing the video at half the original frame rate, but with a one-frame offset between them. The predictions from both models are combined using an arithmetic mean before applying softmax. Despite each model having slightly lower accuracy on its own, the ensemble of the two models provides a more accurate prediction, effectively maintaining accuracy while keeping computational costs low.

Inferencing Pre-trained MoViNet

To harness the power of MoViNet, we need to go through a few key steps: importing necessary libraries, loading the pre-trained model, reading and processing video data, and finally, generating predictions. Let’s dive into each step in detail.

Step1: Import libraries

Before we begin, we need to import several essential Python libraries. These libraries provide the tools necessary for video processing and model inference.

• Numpy : A powerful Python library for numerical computing, providing support for arrays, matrices, and a wide range of mathematical functions to perform operations on these data structures efficiently.
• CV2 : Computer vision and machine learning software library that provides a comprehensive set of tools and functions for image and video analysis, including real-time processing capabilities.
• TensorFlow : Framework developed by Google, designed to facilitate the creation, training, and deployment of deep learning models and complex numerical computations.
• TensorFlow_Hub : A library for reusable machine learning modules, enabling easy sharing and integration of pre-trained models and components for various tasks such as image classification, text embedding, and more, within TensorFlow workflows.

import pathlib
import numpy as np
import cv2
import tensorflow as tf
import tensorflow_hub as hub

Step2: Load the pre-trained model from tensorflow_hub

Next, we need to load the MoViNet model from TensorFlow Hub. This step involves setting up the model architecture and loading the pre-trained weights.

hub_url = "https://www.kaggle.com/models/google/movinet/TensorFlow2/a0-base-kinetics-600-classification/3"

encoder = hub.KerasLayer( hub_url )

inputs = tf.keras.layers.Input(
    shape = [ None, None, None, 3 ],
    dtype = tf.float32,
    name = 'image')
    
outputs = encoder( dict( image= inputs ) )

model = tf.keras.Model( inputs, outputs, name='MoViNet' )

print( model.summary() )

Step3: Load the video

With the model ready, the next step is to prepare our video data. This involves reading the video file and processing it into a format suitable for the MoViNet model.

video_path = VIDEO_PATH # Path to video
vidcap = cv2.VideoCapture(video_path) # Create a VideoCapture object

if not vidcap.isOpened():
    print(f"Error: Could not open video {video_path}")
    exit()

video_data = []

# Read the sequence of frames(video) into a list
while True:
    success, image = vidcap.read()
    if not success:
        break
    image = cv2.resize(image, (172, 172))
    image_rgb = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)
    video_data.append(image_rgb)

# Release the video object
vidcap.release()

# Convert the list to a numpy array
video_data = np.array(video_data)

print(video_data.shape)

Step4: Preprocess the video (reshape), input it to model and generate output

Finally, we preprocess the video data and run it through the model to generate predictions. This step involves reshaping the data and interpreting the model’s output.

input_tensor= tf.expand_dims(video_data, axis= 0) # Expand dimension of input
print(input_tensor.shape) # Print the shape of input tensor

logits= model.predict(input_tensor) # Generate prediction from model

max_index= np.argmax( tf.nn.softmax(logits) ) # Apply softmax function on logits and find the index having maximum probability value

# Load index-to-label mapping into an array
labels_path = tf.keras.utils.get_file(
    fname='labels.txt',
    origin='https://raw.githubusercontent.com/tensorflow/models/f8af2291cced43fc9f1d9b41ddbf772ae7b0d7d2/official/projects/movinet/files/kinetics_600_labels.txt'
)
labels_path = pathlib.Path(labels_path)

lines = labels_path.read_text().splitlines()
KINETICS_600_LABELS = np.array([line.strip() for line in lines])

print( KINETICS_600_LABELS[max_index] ) # Print the output label

Strengths of MoViNets

• High Accuracy: MoViNets achieve state-of-the-art accuracy on various video recognition benchmarks, demonstrating their ability to effectively extract and learn spatiotemporal features.
• Memory Efficiency: Significant reduction in memory usage compared to traditional 3D convolutional networks. Making them suitable for deployment on resource-constrained devices like mobile phones.
• Online Inference: Supports online inference, enabling frame-by-frame predictions and efficient processing of streaming video, which is crucial for real-time applications.

Challenges and Limitations

• Generalization: MoViNets train with a specific frame rate and clip duration, which may limit their ability to generalize to videos with different temporal characteristics.
• Limited Search Space: The design of MoViNets’ search space is based on existing 2D models, which may limit the exploration of novel 3D architectures.
• Streaming Architecture: While stream buffers allow for online inference, they can cause a small accuracy drop compared to non-streaming models due to the limitations of causal operations.

Real-World Applications

• Surveillance and Security: Can enhance security by enabling real-time detection of objects, activities, and anomalies in surveillance systems, improving safety in public spaces, buildings, and at borders.
• Healthcare Monitoring: Can analyze medical images like X-rays and MRIs to assist healthcare professionals in diagnosing and tracking the spread of diseases like cancer.
• Sports Analytics: Sports organizations can use video classification to analyze player performance, tactics, and opponent strategies, leading to improved decision-making and player development.
• Video Conferencing: Can enable real-time video analysis in video conferencing applications, optimizing video quality, detecting participants’ emotions, and enhancing the overall conferencing experience.
• Smart Homes: Smart home systems use MoViNet to analyze security footage in real-time, detect suspicious activity, and alert homeowners.

Conclusion

MoViNets represent a significant breakthrough in efficient video recognition. They demonstrate that powerful video understanding is achievable even on resource-constrained devices like mobile phones. By leveraging stream buffers and causal operations, MoViNets enable real-time inference on streaming video. This capability opens up exciting possibilities for a wide range of applications, including augmented reality, self-driving cars, video conferencing, and mobile gaming.

Despite their impressive accuracy and efficiency, MoViNets have areas for improvement. Further research can focus on expanding their search space. Optimizing their performance across diverse hardware platforms is also crucial. Additionally, enhancing their generalization capabilities can unlock even greater potential in the field of video understanding.

Explore the code behind this article on GitHub!

Resources: MoViNets: Mobile Video Networks for Efficient Video Recognition   

Key Takeaways

• MoViNets offer state-of-the-art accuracy and efficiency, making them ideal for mobile video recognition tasks.
• With stream buffers and online inference capabilities, MoViNets enable real-time video analysis on resource-constrained devices.
• MoViNets significantly reduce memory usage compared to traditional 3D convolutional networks, ensuring smooth performance on mobile hardware.
• From surveillance and security to healthcare monitoring and smart homes, MoViNets have broad real-world applications.
• Continued research and development can expand MoViNet’s capabilities, enhancing their performance across diverse hardware platforms and use cases.

Frequently Asked Questions

The media shown in this article is not owned by Analytics Vidhya and is used at the Author’s discretion.

Boddu Sripavan

Over a year of working experience as an AI ML Engineer, I have developed state-of-the-art models for human body posture recognition, hand and mouth gesture recognition systems with +90% accuracies. I look forward to continue my work on data-driven machine learning.