Efficient Human Posture Estimation: A Step-by-Step Guide with Active Display Control

Introduction

Within the domain of computer vision, Human Posture Estimation stands as a captivating field with applications extending from increased reality and gaming to mechanical autonomy and healthcare. This article sheds light on the complexities of human posture estimation, its significance, fundamental advances, and striking applications.

Posture estimation, an intriguing field inside computer vision, includes recognizing key focuses on a person’s body to get it and analyze their pose. Our objective is to bring this innovation into the domain of yoga, permitting us to consequently recognize and classify yoga postures from pictures.

Learning Objective

• Gain a deep understanding of human pose estimation principles and their significance in computer vision.
• Comprehend how human pose estimation technology enhances yoga practice with personalized guidance and real-time feedback.
• Develop practical skills in implementing human pose estimation algorithms for yoga applications using Python and relevant libraries.

This article was published as a part of the Data Science Blogathon.

Understanding Human Pose Estimation

Human Pose Estimation is a computer vision task that involves representing the orientation of a person graphically. This technique, leveraging model-based approaches, identifies and classifies poses of human body parts and joints in images or videos. The key lies in capturing a set of coordinates defining joints like wrists, shoulders, and knees, which collectively describe a person’s pose.

Importance of Human Pose Estimation

The detection of people has evolved with machine learning algorithms, enabling computers to understand human body language through pose detection and tracking. This technology has become commercially viable, impacting various industries such as security, business intelligence, health and safety, and entertainment. Notably, in the era of the coronavirus pandemic, real-time pose detection aids in implementing social distancing measures.

Contrast Between 2D and 3D Human Posture Estimation

Two major methods exist are 2D Posture Estimation and 3D Posture Estimation. The previous gauges body joint areas in 2D space, whereas the last mentioned changes a 2D picture into a 3D protest by anticipating an extra Z-dimension. 3D pose estimation, though challenging, allows for accurate spatial positioning in representations.

Types of Human Pose Estimation Models

Human Pose Estimation models fall into three main types:

• Skeleton-based Model: Represents the skeletal structure, used for both 3D and 2D pose estimation.
• Contour-based Model: Focuses on 2D pose estimation, emphasizing the body’s appearance and shape.
• Volume-based Model: Employed for 3D pose estimation, utilizes 3D human body models and poses.

Bottom-Up vs. Top-Down Methods of Pose Estimation

Methods for human pose estimation are broadly classified into two approaches: bottom-up and top-down. Bottom-up evaluates each body joint individually, while top-down employs a body detector first and determines joints within discovered bounding boxes.

Understanding the workings of human pose estimation involves delving into the basic structure, model architecture overview, and various approaches for pose estimation. The process encompasses absolute pose estimation, relative pose estimation, and their combination.

Popular Libraries and Tools

Several open-source libraries facilitate human pose estimation:

• OpenPose: A multi-person system supporting 2D and 3D pose estimation.
• PoseDetection: Built on TensorFlow.js, offering real-time pose estimation models.
• DensePose: Maps human pixels from 2D RGB images to a 3D surface-based model.
• AlphaPose: A real-time multi-person pose estimation library using a top-down approach.
• HRNet (High-Resolution Net): Suitable for high-accuracy key point heatmap prediction.

Enhanced Human Pose Estimation: A Simple and Efficient Approach

Let us now begin with simple human pose estimation code by following certain steps.

Step 1: Setting the Stage

To kick off our journey, we need to set up our environment by installing the necessary libraries. OpenCV, NumPy, and MediaPipe are essential for our project. Execute the following command to install them:

!pip install opencv-python mediapipe

We have introduce MediaPipe in this article, an open-source framework developed by Google for building machine learning pipelines focused on computer vision tasks. MediaPipe simplifies the implementation of complex visual applications, offering pre-trained models for human pose estimation that can be integrated with minimal effort. Its cross-platform capability ensures consistent performance on mobile devices, web applications, and desktops, while its design for real-time processing allows for quick video input analysis.

Step 2: Import Necessary Library

import math
import cv2
import numpy as np
from time import time
import mediapipe as mp
import matplotlib.pyplot as plt
from IPython.display import HTML

•  `math`: Provides mathematical functions for calculations.
• `cv2`: OpenCV library for computer vision tasks like image manipulation and processing.
• `numpy as np`: NumPy library for numerical computing with support for arrays and matrices.
• `time`: Module for working with time, used here to measure execution time.
• `mediapipe as mp`: MediaPipe framework for building perception pipelines for various media types.
• `matplotlib.pyplot as plt`: Matplotlib library for creating plots and visualizations.
• `IPython.display import HTML`: IPython module for displaying HTML content within the notebook.

Step 3: Initialze MediaPipe Package

Set up MediaPipe’s Pose and Drawing utilities for pose detection and visualization.

# Initializing mediapipe pose class.
mp_pose = mp.solutions.pose

# Setting up the Pose function.
pose = mp_pose.Pose(static_image_mode=True, min_detection_confidence=0.3, model_complexity=2)

# Initializing mediapipe drawing class, useful for annotation.
mp_drawing = mp.solutions.drawing_utils 

• These lines initialize the necessary components from the MediaPipe framework for performing pose estimation tasks.
• mp_pose = mp.solutions.pose initializes the MediaPipe Pose class, enabling pose estimation functionality.
• pose = mp_pose.Pose(static_image_mode=True, min_detection_confidence=0.3, model_complexity=2) sets up the Pose function with specific parameters, such as static image mode, minimum detection confidence, and model complexity.
• mp_drawing = mp.solutions.drawing_utils initializes the MediaPipe drawing utilities class, which provides functions for annotating images with pose landmarks and connections, facilitating visualization of pose estimation results.

Step 4: Load and Display Image

Use OpenCV to load an image and Matplotlib to display it.

sample_img  = cv2.imread('/content/istockphoto-664637378-612x612.jpg')
plt.figure(figsize = [10,10])
plt.title("sample_Image")
plt.axis('off')
plt.imshow(sample_img[:,:,::-1]);plt.show()

• This code segment loads a sample image from a specified file path using the OpenCV library (cv2.imread()).
• It then uses Matplotlib to display the loaded image in a figure with a specified size (plt.figure(figsize=[10, 10])), title (plt.title(“Sample Image”)), and without axis ticks (plt.axis(‘off’)).
• The image is finally shown using plt.imshow() function, which takes care of displaying the image in the specified figure. The [:, :, ::-1] indexing is used to convert the image from BGR to RGB format, as Matplotlib expects RGB images for display.

Step5: Detect and Print Landmarks

Convert the image to RGB and use MediaPipe to detect pose landmarks. Print the first two detected landmarks (e.g., NOSE, LEFT_EYE_INNER).

Keypoint_Identification

keypoint_Landmark

# Perform pose detection after converting the image into RGB format.
results = pose.process(cv2.cvtColor(sample_img, cv2.COLOR_BGR2RGB))

# Check if any landmarks are found.
if results.pose_landmarks:
    
    # Iterate two times as we only want to display first two landmarks.
    for i in range(2):
        
        # Display the found normalized landmarks.
        print(f'{mp_pose.PoseLandmark(i).name}:\n{results.pose_landmarks.landmark[mp_pose.PoseLandmark(i).value]}') 

• This code segment performs pose detection on the sample image after converting it into RGB format using OpenCV’s cv2.cvtColor() function.
• It then checks if any pose landmarks are found in the image using the results.pose_landmarks attribute.
• If landmarks are found, it iterates over the first two landmarks and prints their names and coordinates.
• The landmark name is obtained using mp_pose.PoseLandmark(i).name, and the coordinates are accessed using results.pose_landmarks.landmark[mp_pose.PoseLandmark(i).value].

Output:

NOSE:
x: 0.7144814729690552
y: 0.3049055337905884
z: -0.1483774036169052
visibility: 0.9999918937683105
LEFT_EYE_INNER:
x: 0.7115224599838257
y: 0.2835153341293335
z: -0.13594578206539154
visibility: 0.9999727010726929

Step6: Draw Landmarks on Image

Create a copy of the image, draw detected landmarks using MediaPipe utilities, and display it.

# Create a copy of the sample image to draw landmarks on.
img_copy = sample_img.copy()

# Check if any landmarks are found.
if results.pose_landmarks:
    
    # Draw Pose landmarks on the sample image.
    mp_drawing.draw_landmarks(image=img_copy, landmark_list=results.pose_landmarks, connections=mp_pose.POSE_CONNECTIONS)
       
    # Specify a size of the figure.
    fig = plt.figure(figsize = [10, 10])

    # Display the output image with the landmarks drawn, also convert BGR to RGB for display. 
    plt.title("Output")
    plt.axis('off')
    plt.imshow(img_copy[:,:,::-1])
    plt.show()

• This code segment creates a copy of the sample image to preserve the original image while drawing landmarks on a separate image.
• It checks if any pose landmarks are found in the results.
• If landmarks are found, it draws the landmarks on the copied image using mp_drawing.draw_landmarks().
• The size of the figure for displaying the output image is specified using plt.figure(figsize=[10, 10]).
• Finally, it displays the output image with landmarks drawn using plt.imshow(). The [:,:,::-1] indexing is used to convert the image from BGR to RGB format for proper display with Matplotlib.

Step 7: 3D Pose Visualization

Use MediaPipe’s plot_landmarks() to visualize the detected landmarks in 3D.

# Plot Pose landmarks in 3D.
mp_drawing.plot_landmarks(results.pose_world_landmarks, mp_pose.POSE_CONNECTIONS)

• This code segment plots the pose landmarks in 3D space using MediaPipe’s plot_landmarks() function.
• It takes results.pose_world_landmarks as input, which represents the pose landmarks in world coordinates.
• mp_pose.POSE_CONNECTIONS specifies the connections between different landmarks, helping to visualize the skeletal structure.

Step 8: Custom Pose Detection Function

For custom pose detection we will use detectpose(). This function performs pose detection, displays results, and optionally returns landmarks.

def detectPose(image, pose, display=True):
    '''
    This function performs pose detection on an image.
    Args:
        image: The input image with a prominent person whose pose landmarks needs to be detected.
        pose: The pose setup function required to perform the pose detection.
        display: A boolean value that is if set to true the function displays the original input image, the resultant image, 
                 and the pose landmarks in 3D plot and returns nothing.
    Returns:
        output_image: The input image with the detected pose landmarks drawn.
        landmarks: A list of detected landmarks converted into their original scale.
    '''
    
    # Create a copy of the input image.
    output_image = image.copy()
    
    # Convert the image from BGR into RGB format.
    imageRGB = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)
    
    # Perform the Pose Detection.
    results = pose.process(imageRGB)
    
    # Retrieve the height and width of the input image.
    height, width, _ = image.shape
    
    # Initialize a list to store the detected landmarks.
    landmarks = []
    
    # Check if any landmarks are detected.
    if results.pose_landmarks:
    
        # Draw Pose landmarks on the output image.
        mp_drawing.draw_landmarks(image=output_image, landmark_list=results.pose_landmarks,
                                  connections=mp_pose.POSE_CONNECTIONS)
        
        # Iterate over the detected landmarks.
        for landmark in results.pose_landmarks.landmark:
            
            # Append the landmark into the list.
            landmarks.append((int(landmark.x * width), int(landmark.y * height),
                                  (landmark.z * width)))
    
    # Check if the original input image and the resultant image are specified to be displayed.
    if display:
    
        # Display the original input image and the resultant image.
        plt.figure(figsize=[22,22])
        plt.subplot(121);plt.imshow(image[:,:,::-1]);plt.title("Original Image");plt.axis('off');
        plt.subplot(122);plt.imshow(output_image[:,:,::-1]);plt.title("Output Image");plt.axis('off');
        
        # Also Plot the Pose landmarks in 3D.
        mp_drawing.plot_landmarks(results.pose_world_landmarks, mp_pose.POSE_CONNECTIONS)
        
    # Otherwise
    else:
        
        # Return the output image and the found landmarks.
        return output_image, landmarks

• This function detectPose() performs pose detection on an input image using MediaPipe’s Pose model.
• It takes three parameters: image (the input image), pose (the pose setup function), and display (a boolean indicating whether to display the results).
• It copies the input image to preserve the original and converts the image from BGR to RGB format, as required by MediaPipe.
• It detects poses on the converted image and draws the detected landmarks on the output image using mp_drawing.draw_landmarks().
• The function also retrieves the height and width of the input image and initializes an empty list to store the detected landmarks.
• If the display parameter is set to True, it displays the original input image, the output image with landmarks drawn, and plots the landmarks in 3D space using mp_drawing.plot_landmarks().
• If display is False, it returns the output image with landmarks drawn and the detected landmarks list.

Step 9: Sample Execution

Run pose detection on a new sample image using the detectPose() function.

# Read another sample image and perform pose detection on it.
image = cv2.imread('/content/HD-wallpaper-yoga-training-gym-pose-woman-yoga-exercises.jpg')
detectPose(image, pose, display=True)

• This code segment reads another sample image from the specified file path.
• It then calls the detectPose() function to perform pose detection on the image using the previously initialized pose setup.
• Setting the display parameter to True directs the function to show the original input image, the resultant image with drawn landmarks, and the 3D plot of landmarks.

Step 10: Pose Classification (Optional)

The next step involves defining a function to classify poses like Warrior, Tree, etc., based on joint angles.

Warrior-Pose, T-Pose, Tree-Pose, Unknown

def classifyPose(landmarks, output_image, display=False):
    '''
    This function classifies yoga poses depending upon the angles of various body joints.
    Args:
        landmarks: A list of detected landmarks of the person whose pose needs to be classified.
        output_image: A image of the person with the detected pose landmarks drawn.
        display: A boolean value that is if set to true the function displays the resultant image with the pose label 
        written on it and returns nothing.
    Returns:
        output_image: The image with the detected pose landmarks drawn and pose label written.
        label: The classified pose label of the person in the output_image.

    '''
    
    # Initialize the label of the pose. It is not known at this stage.
    label = 'Unknown Pose'

    # Specify the color (Red) with which the label will be written on the image.
    color = (0, 0, 255)
    
    # Calculate the required angles.
    #----------------------------------------------------------------------------------------------------------------
    
    # Get the angle between the left shoulder, elbow and wrist points. 
    left_elbow_angle = calculateAngle(landmarks[mp_pose.PoseLandmark.LEFT_SHOULDER.value],
                                      landmarks[mp_pose.PoseLandmark.LEFT_ELBOW.value],
                                      landmarks[mp_pose.PoseLandmark.LEFT_WRIST.value])
    
    # Get the angle between the right shoulder, elbow and wrist points. 
    right_elbow_angle = calculateAngle(landmarks[mp_pose.PoseLandmark.RIGHT_SHOULDER.value],
                                       landmarks[mp_pose.PoseLandmark.RIGHT_ELBOW.value],
                                       landmarks[mp_pose.PoseLandmark.RIGHT_WRIST.value])   
    
    # Get the angle between the left elbow, shoulder and hip points. 
    left_shoulder_angle = calculateAngle(landmarks[mp_pose.PoseLandmark.LEFT_ELBOW.value],
                                         landmarks[mp_pose.PoseLandmark.LEFT_SHOULDER.value],
                                         landmarks[mp_pose.PoseLandmark.LEFT_HIP.value])

    # Get the angle between the right hip, shoulder and elbow points. 
    right_shoulder_angle = calculateAngle(landmarks[mp_pose.PoseLandmark.RIGHT_HIP.value],
                                          landmarks[mp_pose.PoseLandmark.RIGHT_SHOULDER.value],
                                          landmarks[mp_pose.PoseLandmark.RIGHT_ELBOW.value])

    # Get the angle between the left hip, knee and ankle points. 
    left_knee_angle = calculateAngle(landmarks[mp_pose.PoseLandmark.LEFT_HIP.value],
                                     landmarks[mp_pose.PoseLandmark.LEFT_KNEE.value],
                                     landmarks[mp_pose.PoseLandmark.LEFT_ANKLE.value])

    # Get the angle between the right hip, knee and ankle points 
    right_knee_angle = calculateAngle(landmarks[mp_pose.PoseLandmark.RIGHT_HIP.value],
                                      landmarks[mp_pose.PoseLandmark.RIGHT_KNEE.value],
                                      landmarks[mp_pose.PoseLandmark.RIGHT_ANKLE.value])
    
    #----------------------------------------------------------------------------------------------------------------
    
    # Check if it is the warrior II pose or the T pose.
    # As for both of them, both arms should be straight and shoulders should be at the specific angle.
    #----------------------------------------------------------------------------------------------------------------
    
    # Check if the both arms are straight.
    if left_elbow_angle > 165 and left_elbow_angle < 195 and right_elbow_angle > 165 and right_elbow_angle < 195:

        # Check if shoulders are at the required angle.
        if left_shoulder_angle > 80 and left_shoulder_angle < 110 and right_shoulder_angle > 80 and right_shoulder_angle < 110:

    # Check if it is the warrior II pose.
    #----------------------------------------------------------------------------------------------------------------

            # Check if one leg is straight.
            if left_knee_angle > 165 and left_knee_angle < 195 or right_knee_angle > 165 and right_knee_angle < 195:

                # Check if the other leg is bended at the required angle.
                if left_knee_angle > 90 and left_knee_angle < 120 or right_knee_angle > 90 and right_knee_angle < 120:

                    # Specify the label of the pose that is Warrior II pose.
                    label = 'Warrior II Pose' 
                        
    #----------------------------------------------------------------------------------------------------------------
    
    # Check if it is the T pose.
    #----------------------------------------------------------------------------------------------------------------
    
            # Check if both legs are straight
            if left_knee_angle > 160 and left_knee_angle < 195 and right_knee_angle > 160 and right_knee_angle < 195:

                # Specify the label of the pose that is tree pose.
                label = 'T Pose'

    #----------------------------------------------------------------------------------------------------------------
    
    # Check if it is the tree pose.
    #----------------------------------------------------------------------------------------------------------------
    
    # Check if one leg is straight
    if left_knee_angle > 165 and left_knee_angle < 195 or right_knee_angle > 165 and right_knee_angle < 195:

        # Check if the other leg is bended at the required angle.
        if left_knee_angle > 315 and left_knee_angle < 335 or right_knee_angle > 25 and right_knee_angle < 45:

            # Specify the label of the pose that is tree pose.
            label = 'Tree Pose'
                
    #----------------------------------------------------------------------------------------------------------------
    
    # Check if the pose is classified successfully
    if label != 'Unknown Pose':
        
        # Update the color (to green) with which the label will be written on the image.
        color = (0,0,255)  
    
    # Write the label on the output image. 
    cv2.putText(output_image, label, (10, 30),cv2.FONT_HERSHEY_PLAIN, 2, color, 5)
    
    # Check if the resultant image is specified to be displayed.
    if display:
    
        # Display the resultant image.
        plt.figure(figsize=[10,10])
        plt.imshow(output_image[:,:,::-1]);plt.title("Output Image");plt.axis('off');
        
    else:
        
        # Return the output image and the classified label.
        return output_image, label

# Read a sample image and perform pose classification on it.
image = cv2.imread('/content/amp-1575527028-- triangle pose.jpg')
output_image, landmarks = detectPose(image, pose, display=False)
if landmarks:
    classifyPose(landmarks, output_image, display=True)

• This code segment reads a sample image from the specified file path.
• It then calls the detectPose() function to perform pose detection on the image using the previously initialized pose setup.
• If the display parameter is False, the function skips displaying the results.
• If the image contains detected landmarks, the function calls classifyPose() to classify the pose based on these landmarks and display the result.

# Read a sample image and perform pose classification on it.
image = cv2.imread('/content/warrior2.jpg')
output_image, landmarks = detectPose(image, pose, display=False)
if landmarks:
    classifyPose(landmarks, output_image, display=True)

• This code segment reads a sample image from the specified file path.
• It then calls the detectPose() function to perform pose detection on the image using the previously initialized pose setup.
• The display parameter is set to False, indicating that the function should not display the results.
• If landmarks are detected in the image, it calls the classifyPose() function to classify the pose based on the detected landmarks and display the result.

Applications of Human Pose Estimation

Human pose estimation finds applications in diverse domains:

Fitness and Wellness Industry

• Personalized Guidance: Pose detection applications guide users through yoga sessions, offering real-time feedback on their pose alignment.
• Progress Tracking: Systems monitor users’ progress, suggesting modifications or advancements tailored to individual skill levels.

Industry-Level Applications

• Corporate Wellness Programs: Companies can integrate yoga pose detection, enhancing employee health through wellness programs and stress reduction.

Healthcare

• Posture Correction: Pose detection aids in correcting posture during rehabilitation exercises, ensuring correct movement execution.
• Remote Monitoring: Healthcare professionals remotely monitor patients’ yoga sessions, offering virtual assistance and adjusting routines as needed.

Sports Training

• Flexibility and Strength Training: Pose detection in sports training programs benefit athletes requiring flexibility and strength, boosting overall performance.

Education

• Interactive Learning: Pose detection enhances the interactive and accessible learning of yoga for students in educational institutions.
• Skill Assessment: Teachers assess students’ yoga skills using technology, offering targeted guidance for improvement.

Entertainment and Gaming

• Immersive Experiences: VR or AR applications create immersive yoga experiences with virtual instructors guiding users through poses.
• Interactive Gaming: Pose detection in fitness games makes exercise enjoyable and motivating for users.

Ergonomics in Industry

• Desk Yoga Sessions: Integrating pose detection into workplace wellness programs offers short yoga sessions, improving posture and reducing stress for employees.
• Ergonomic Assessments: Employers use pose detection to assess ergonomic aspects of workstations, promoting better health among employees.

User Benefits

• Correct Form: Immediate feedback on the form reduces the risk of injuries, ensuring users gain maximum benefits from yoga practices.
• Convenience: Users can practice yoga at their convenience, guided by virtual instructors or applications, eliminating the need for physical classes.
• Motivation: Real-time progress tracking and feedback motivate for users to stay consistent with their yoga routines.

Conclusion

The integration of human pose detection with yoga poses transcends diverse sectors, revolutionizing wellness and fitness. From personalized guidance and progress tracking in the fitness industry to enhancing rehabilitation and physical therapy in healthcare, this technology offers a versatile range of applications. In sports training, it contributes to athletes’ flexibility and strength, while in education, it brings interactive and assessable yoga learning experiences.

The workplace benefits from desk yoga sessions and ergonomic assessments, promoting employee well-being. Users, guided by virtual instructors, enjoy correct form feedback, convenience, and motivation, fostering a healthier and more efficient approach to yoga practices. This transformative combination of antiquated practices with cutting-edge innovation clears the way for an all-encompassing well-being insurgency.

Key Takeaways

• Human Posture Estimation, a field inside computer vision, includes recognizing key focuses on a person’s body to get it and analyze their pose.
• Human posture estimation has assorted applications, extending from wellness and wellness to healthcare, sports preparation, instruction, amusement, and working environment ergonomics.
• Consolidating posture discovery innovation into Yoga Hone offers clients personalized direction, real-time input, advanced following, comfort, and inspiration, driving them to move forward with well-being and more proficient workouts.
• The integration of human pose detection with yoga practice represents a significant advancement in wellness technology, paving the way for a comprehensive well-being revolution.

Frequently Asked Questions

The media shown in this article is not owned by Analytics Vidhya and is used at the Author’s discretion.