MediaPipe Pose on two synchronized videos

[mcdonasd1212]

Have I written custom code (as opposed to using a stock example script provided in MediaPipe)

Yes

OS Platform and Distribution

Ubuntu 22.04.4 LTS

MediaPipe Tasks SDK version

0.10.11

Task name (e.g. Image classification, Gesture recognition etc.)

Pose

Programming Language and version (e.g. C++, Python, Java)

Python

Describe the actual behavior

When implementing 10.11 the code example is very slow. However with the legacy code, the example performance is acceptable.

Describe the expected behaviour

MediaPipe 10.11 to have similar performance as legacy for the example.

Standalone code/steps you may have used to try to get what you need

Here is an example the example code with acceptable performance. How can I implement 10.11?

import sys
import cv2
import mediapipe as mp
from PySide6.QtCore import QTimer, Qt
from PySide6.QtGui import QImage, QPixmap
from PySide6.QtWidgets import QApplication, QMainWindow, QWidget, QHBoxLayout, QVBoxLayout, QPushButton, QLabel, QCheckBox


class VideoPlayer(QWidget):
    def __init__(self, video_path, label_text, parent=None):
        super(VideoPlayer, self).__init__(parent)
        self.video_path = video_path
        self.cap = cv2.VideoCapture(self.video_path)
        if not self.cap.isOpened():
            print("Error: Could not open video.")
            return

        self.label_text = label_text

        self.label = QLabel()
        layout = QVBoxLayout(self)
        layout.addWidget(self.label)

        self.frame_timer = QTimer(self)
        self.frame_timer.timeout.connect(self.update_frame)

        self.frame_number = 0
        self.total_frames = int(self.cap.get(cv2.CAP_PROP_FRAME_COUNT))

        self.paused = True  # Initialize paused to True

        self.loop = True

        # Initialize MediaPipe Pose
        self.mp_pose = mp.solutions.pose.Pose(static_image_mode=False, min_detection_confidence=0.5,
                                                min_tracking_confidence=0.5)

    def update_frame(self):
        ret, frame = self.cap.read()
        if not ret:
            if self.loop:
                self.cap.set(cv2.CAP_PROP_POS_FRAMES, 0)
                self.frame_number = 0
            else:
                self.frame_timer.stop()
                return

        if not self.paused:
            self.frame_number += 1

        # Convert frame to RGB
        frame_rgb = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)

        # Detect pose
        results = self.mp_pose.process(frame_rgb)

        # Draw pose landmarks
        if results.pose_landmarks:
            mp.solutions.drawing_utils.draw_landmarks(frame_rgb, results.pose_landmarks, mp.solutions.pose.POSE_CONNECTIONS)

        # Add frame number and label text to the frame
        font = cv2.FONT_HERSHEY_SIMPLEX
        cv2.putText(frame_rgb, f"Frame: {self.frame_number}", (20, 40), font, 1, (255, 255, 255), 2, cv2.LINE_AA)
        cv2.putText(frame_rgb, self.label_text, (20, 80), font, 1, (255, 255, 255), 2, cv2.LINE_AA)

        # Convert the frame to QImage
        h, w, ch = frame_rgb.shape
        q_image = QImage(frame_rgb.data, w, h, ch * w, QImage.Format_RGB888)

        # Convert QImage to QPixmap
        pixmap = QPixmap.fromImage(q_image)
        self.label.setPixmap(pixmap)

    def play_pause(self):
        if self.paused:
            self.frame_timer.start(30)
        else:
            self.frame_timer.stop()
        self.paused = not self.paused

    def go_forward(self):
        if not self.paused:
            self.frame_timer.stop()
        self.frame_number = min(self.total_frames - 1, self.frame_number + 1)
        self.cap.set(cv2.CAP_PROP_POS_FRAMES, self.frame_number)
        self.update_frame()

    def go_back(self):
        if not self.paused:
            self.frame_timer.stop()
        self.frame_number = max(0, self.frame_number - 1)
        self.cap.set(cv2.CAP_PROP_POS_FRAMES, self.frame_number)
        self.update_frame()

    def stop(self):
        self.frame_timer.stop()
        self.cap.set(cv2.CAP_PROP_POS_FRAMES, 0)
        self.frame_number = 0
        self.paused = True
        self.update_frame()

    def toggle_loop(self, state):
        self.loop = state == Qt.Checked


class MainWindow(QMainWindow):
    def __init__(self):
        super(MainWindow, self).__init__()

        self.setWindowTitle("Video Synchronization")
        self.setGeometry(100, 100, 800, 400)

        central_widget = QWidget()
        self.setCentralWidget(central_widget)

        layout = QHBoxLayout(central_widget)

        video_left = VideoPlayer("video_left.mp4", "Face On")
        video_right = VideoPlayer("video_right.mp4", "DTL")

        control_widget = QWidget()
        control_layout = QVBoxLayout(control_widget)
        play_pause_button = QPushButton("Play / Pause")
        play_pause_button.clicked.connect(self.play_pause_all)
        forward_button = QPushButton("Forward")
        forward_button.clicked.connect(self.go_forward_all)
        back_button = QPushButton("Back")
        back_button.clicked.connect(self.go_back_all)
        stop_button = QPushButton("Stop")
        stop_button.clicked.connect(self.stop_all)
        loop_checkbox = QCheckBox("Loop")
        loop_checkbox.setChecked(True)
        loop_checkbox.stateChanged.connect(self.toggle_loop_all)

        control_layout.addWidget(play_pause_button)
        control_layout.addWidget(forward_button)
        control_layout.addWidget(back_button)
        control_layout.addWidget(stop_button)
        control_layout.addWidget(loop_checkbox)

        layout.addWidget(video_left)
        layout.addWidget(video_right)
        layout.addWidget(control_widget)

        self.show()

        self.video_players = [video_left, video_right]

    def play_pause_all(self):
        for player in self.video_players:
            player.play_pause()

    def go_forward_all(self):
        for player in self.video_players:
            player.go_forward()

    def go_back_all(self):
        for player in self.video_players:
            player.go_back()

    def stop_all(self):
        for player in self.video_players:
            player.stop()

    def toggle_loop_all(self, state):
        for player in self.video_players:
            player.toggle_loop(state)


if __name__ == "__main__":
    app = QApplication(sys.argv)
    window = MainWindow()
    sys.exit(app.exec())

### Other info / Complete Logs

_No response_

[kuaashish]

Hi @mcdonasd1212,

After reviewing the provided standalone code, it appears that you are currently using the outdated pose solution, which is no longer maintained, and we have ceased support for it. This functionality has been integrated into the new Pose Landmarker Task API, detailed here.

We encourage you to explore the features of our updated Pose Landmarker Task API and suggest replacing the legacy Pose with the new Pose Landmarker. The new solution offers improved performance and additional functionality compared to the legacy pose solution. You can find the guide for the new Pose Landmarker here, along with specific instructions for implementation in the Python platform provided here. Additionally, a corresponding example Colab notebook is available for reference here.

Please report any observed behavior, ensuring to check if similar issues persist in the upgraded task API. We value your feedback. Unfortunately, beyond this, there is limited action we can take to address the specific issue you are facing.

Thank you!!

[mcdonasd1212]

Sorry for the confusion, I am asking why the legacy pose solution works, the NEW POSE LANDMARKER DOES NOT. Any advice?

…

________________________________ From: kuaashish ***@***.***> Sent: Monday, May 6, 2024 5:53:07 AM To: google/mediapipe ***@***.***> Cc: mcdonasd1212 ***@***.***>; Mention ***@***.***> Subject: Re: [google/mediapipe] MediaPipe Pose on two synchronized videos (Issue #5376) Hi @mcdonasd1212<https://github.com/mcdonasd1212>, After reviewing the provided standalone code, it appears that you are currently using the outdated pose solution, which is no longer maintained, and we have ceased support for it. This functionality has been integrated into the new Pose Landmarker Task API, detailed here<https://developers.google.com/mediapipe/solutions/guide#legacy>. We encourage you to explore the features of our updated Pose Landmarker Task API and suggest replacing the legacy Pose with the new Pose Landmarker. The new solution offers improved performance and additional functionality compared to the legacy pose solution. You can find the guide for the new Pose Landmarker here<https://developers.google.com/mediapipe/solutions/vision/pose_landmarker>, along with specific instructions for implementation in the Python platform provided here<https://developers.google.com/mediapipe/solutions/vision/pose_landmarker/python>. Additionally, a corresponding example Colab notebook is available for reference here<https://colab.sandbox.google.com/github/googlesamples/mediapipe/blob/main/examples/pose_landmarker/python/%5BMediaPipe_Python_Tasks%5D_Pose_Landmarker.ipynb>. Please report any observed behavior, ensuring to check if similar issues persist in the upgraded task API. We value your feedback. Unfortunately, beyond this, there is limited action we can take to address the specific issue you are facing. Thank you!! — Reply to this email directly, view it on GitHub<#5376 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AUU6XBZWYIAU3GQPPX3A7PLZA5HIHAVCNFSM6AAAAABHHF4HFCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAOJVGU4TMNBXGE>. You are receiving this because you were mentioned.Message ID: ***@***.***>

[kuaashish]

Hi @mcdonasd1212,

Please verify with the latest version 0.10.13, which we recently released. If the issue persists, could you kindly provide the standalone code or steps you are following from our documentation, along with the version of Python being used, to reproduce the issue? This will help us identify and address any genuine bugs.

Thank you!!

[github-actions]

This issue has been marked stale because it has no recent activity since 7 days. It will be closed if no further activity occurs. Thank you.

[github-actions]

This issue was closed due to lack of activity after being marked stale for past 7 days.

[google-ml-butler]

Are you satisfied with the resolution of your issue?
Yes
No