Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

image_format=<ImageFormat.SRGB: 1> __init__(): incompatible constructor arguments. #5379

Open
aerinkim opened this issue May 6, 2024 · 2 comments
Open

image_format=<ImageFormat.SRGB: 1> __init__(): incompatible constructor arguments. #5379

aerinkim opened this issue May 6, 2024 · 2 comments
Assignees
@kuaashish
Labels
os:linux-non-arm Issues on linux distributions which run on x86-64 architecture. DOES NOT include ARM devices. platform:python MediaPipe Python issues stale stat:awaiting response Waiting for user response task:face landmarker Issues related to Face Landmarker: Identify facial features for visual effects and avatars. type:support General questions

Comments

@aerinkim
Copy link

aerinkim commented May 6, 2024

Have I written custom code (as opposed to using a stock example script provided in MediaPipe)

None

OS Platform and Distribution

Ubuntu 20.04.4 LTS

MediaPipe Tasks SDK version

mediapipe-0.10.11

Task name (e.g. Image classification, Gesture recognition etc.)

tasks.vision.FaceLandmarker

Programming Language and version (e.g. C++, Python, Java)

Python 3.8 (Conda)

Describe the actual behavior

Exception has occurred: TypeError init(): incompatible constructor arguments. The following argument types are supported: 1. mediapipe.python._framework_bindings.image.Image(image_format: mediapipe::ImageFormat_Format, data: numpy.ndarray[numpy.uint8]) 2. mediapipe.python._framework_bindings.image.Image(image_format: mediapipe::ImageFormat_Format, data: numpy.ndarray[numpy.uint16]) 3. mediapipe.python._framework_bindings.image.Image(image_format: mediapipe::ImageFormat_Format, data: numpy.ndarray[numpy.float32]) Invoked with: kwargs: image_format=<ImageFormat.SRGB: 1>,

Describe the expected behaviour

should not get exception.

Standalone code/steps you may have used to try to get what you need

import mediapipe as mp
from mediapipe.tasks import python
from mediapipe.tasks.python import vision

BaseOptions = mp.tasks.BaseOptions
FaceLandmarker = mp.tasks.vision.FaceLandmarker
FaceLandmarkerOptions = mp.tasks.vision.FaceLandmarkerOptions

base_options = python.BaseOptions(model_asset_path="/home/ubuntu/Projects/miraflow-ml-aerin/SyncNet288Experiment/weights/face_landmarker_v2_with_blendshapes.task")
options = FaceLandmarkerOptions(base_options=base_options,
                                      output_face_blendshapes=True,
                                      output_facial_transformation_matrixes=True,
                                      num_faces=1)
face_landmarks_detector = FaceLandmarker.create_from_options(options)


def get_mask_from_lower_face(image):
    """
    Generate and save a mask that covers the lower face, including lips and surrounding areas but excluding the nose.

    Args:
        image: numpy array of an image (H,W,C)

    Returns:
        A uint8 numpy array with the same height and width of the input image,
        containing a binary mask of the lower face
    """
    image = np.asarray(image, dtype=np.float32)

    # Initialize mask
    mask = np.zeros((image.shape[0], image.shape[1]), dtype=np.uint8)

    # detect face landmarks    mp_image = mp.Image(image_format=mp.ImageFormat.SRGB, data=image)
    detection = face_landmarks_detector.detect(mp_image)

I tried the image with numpy array - uint8, unit16, float32 but I keep getting this error.



### Other info / Complete Logs

```shell
Exception has occurred: TypeError
__init__(): incompatible constructor arguments. The following argument types are supported:
    1. mediapipe.python._framework_bindings.image.Image(image_format: mediapipe::ImageFormat_Format, data: numpy.ndarray[numpy.uint8])
    2. mediapipe.python._framework_bindings.image.Image(image_format: mediapipe::ImageFormat_Format, data: numpy.ndarray[numpy.uint16])
    3. mediapipe.python._framework_bindings.image.Image(image_format: mediapipe::ImageFormat_Format, data: numpy.ndarray[numpy.float32])

Invoked with: kwargs: image_format=<ImageFormat.SRGB: 1>, data=array([[[0.84313726, 0.9137255 , 0.9098039 ],
        [0.84313726, 0.9137255 , 0.9098039 ],
        [0.84705883, 0.91764706, 0.9137255 ],
        ...,
        [0.88235295, 0.95686275, 0.9764706 ],
        [0.88235295, 0.95686275, 0.9764706 ],
        [0.8862745 , 0.9647059 , 0.9843137 ]],

       [[0.8117647 , 0.88235295, 0.8862745 ],
        [0.8117647 , 0.8862745 , 0.8862745 ],
        [0.8156863 , 0.8901961 , 0.8901961 ],
        ...,
        [0.8627451 , 0.9411765 , 0.9607843 ],
        [0.8627451 , 0.9411765 , 0.9607843 ],
        [0.8627451 , 0.9411765 , 0.9607843 ]],

       [[0.8156863 , 0.8862745 , 0.8901961 ],
        [0.8156863 , 0.8862745 , 0.8901961 ],
        [0.8117647 , 0.88235295, 0.8862745 ],
        ...,
        [0.85490197, 0.93333334, 0.9529412 ],
        [0.85490197, 0.93333334, 0.9529412 ],
        [0.85882354, 0.9372549 , 0.95686275]],

       ...,

       [[0.8117647 , 0.88235295, 0.8862745 ],
        [0.8117647 , 0.88235295, 0.8862745 ],
        [0.8117647 , 0.88235295, 0.8862745 ],
        ...,
        [0.3254902 , 0.8862745 , 0.91764706],
        [0.3254902 , 0.8862745 , 0.91764706],
        [0.32156864, 0.8784314 , 0.90588236]],

       [[0.80784315, 0.8784314 , 0.88235295],
        [0.80784315, 0.8784314 , 0.88235295],
        [0.8117647 , 0.88235295, 0.8862745 ],
        ...,
        [0.3254902 , 0.8862745 , 0.91764706],
        [0.3254902 , 0.8862745 , 0.91764706],
        [0.32156864, 0.8784314 , 0.90588236]],

       [[0.8117647 , 0.88235295, 0.8862745 ],
        [0.8117647 , 0.88235295, 0.8862745 ],
        [0.8156863 , 0.8862745 , 0.8901961 ],
        ...,
        [0.31764707, 0.8784314 , 0.90588236],
        [0.31764707, 0.8784314 , 0.90588236],
        [0.32941177, 0.8862745 , 0.9137255 ]]], dtype=float32)
  File "/home/ubuntu/Projects/miraflow-ml-aerin/SyncNet288Experiment/utils_aerin.py", line 332, in get_mask_from_lower_face
    mp_image = mp.Image(image_format=mp.ImageFormat.SRGB, data=image)
  File "/home/ubuntu/Projects/miraflow-ml-aerin/SyncNet288Experiment/hq_train_speed_up_mask.py", line 269, in apply_mouth_mask
    mask = get_mask_from_lower_face(image)
  File "/home/ubuntu/Projects/miraflow-ml-aerin/SyncNet288Experiment/hq_train_speed_up_mask.py", line 320, in train
    x = apply_mouth_mask(x) #x:[B, 3+3, T, H, W]
  File "/home/ubuntu/Projects/miraflow-ml-aerin/SyncNet288Experiment/hq_train_speed_up_mask.py", line 596, in <module>
    train(device, model, disc, train_data_loader, test_data_loader, optimizer, disc_optimizer,
TypeError: __init__(): incompatible constructor arguments. The following argument types are supported:
    1. mediapipe.python._framework_bindings.image.Image(image_format: mediapipe::ImageFormat_Format, data: numpy.ndarray[numpy.uint8])
    2. mediapipe.python._framework_bindings.image.Image(image_format: mediapipe::ImageFormat_Format, data: numpy.ndarray[numpy.uint16])
    3. mediapipe.python._framework_bindings.image.Image(image_format: mediapipe::ImageFormat_Format, data: numpy.ndarray[numpy.float32])

Invoked with: kwargs: image_format=<ImageFormat.SRGB: 1>, data=array([[[0.84313726, 0.9137255 , 0.9098039 ],
        [0.84313726, 0.9137255 , 0.9098039 ],
        [0.84705883, 0.91764706, 0.9137255 ],
        ...,
        [0.88235295, 0.95686275, 0.9764706 ],
        [0.88235295, 0.95686275, 0.9764706 ],
        [0.8862745 , 0.9647059 , 0.9843137 ]],

       [[0.8117647 , 0.88235295, 0.8862745 ],
        [0.8117647 , 0.8862745 , 0.8862745 ],
        [0.8156863 , 0.8901961 , 0.8901961 ],
        ...,
        [0.8627451 , 0.9411765 , 0.9607843 ],
        [0.8627451 , 0.9411765 , 0.9607843 ],
        [0.8627451 , 0.9411765 , 0.9607843 ]],

       [[0.8156863 , 0.8862745 , 0.8901961 ],
        [0.8156863 , 0.8862745 , 0.8901961 ],
        [0.8117647 , 0.88235295, 0.8862745 ],
        ...,
        [0.85490197, 0.93333334, 0.9529412 ],
        [0.85490197, 0.93333334, 0.9529412 ],
        [0.85882354, 0.9372549 , 0.95686275]],

       ...,

       [[0.8117647 , 0.88235295, 0.8862745 ],
        [0.8117647 , 0.88235295, 0.8862745 ],
        [0.8117647 , 0.88235295, 0.8862745 ],
        ...,
        [0.3254902 , 0.8862745 , 0.91764706],
        [0.3254902 , 0.8862745 , 0.91764706],
        [0.32156864, 0.8784314 , 0.90588236]],

       [[0.80784315, 0.8784314 , 0.88235295],
        [0.80784315, 0.8784314 , 0.88235295],
        [0.8117647 , 0.88235295, 0.8862745 ],
        ...,
        [0.3254902 , 0.8862745 , 0.91764706],
        [0.3254902 , 0.8862745 , 0.91764706],
        [0.32156864, 0.8784314 , 0.90588236]],

       [[0.8117647 , 0.88235295, 0.8862745 ],
        [0.8117647 , 0.88235295, 0.8862745 ],
        [0.8156863 , 0.8862745 , 0.8901961 ],
        ...,
        [0.31764707, 0.8784314 , 0.90588236],
        [0.31764707, 0.8784314 , 0.90588236],
        [0.32941177, 0.8862745 , 0.9137255 ]]], dtype=float32)
@kuaashish kuaashish added os:linux-non-arm Issues on linux distributions which run on x86-64 architecture. DOES NOT include ARM devices. task:face landmarker Issues related to Face Landmarker: Identify facial features for visual effects and avatars. type:support General questions platform:python MediaPipe Python issues labels May 6, 2024
@kuaashish
Copy link
Collaborator

Hi @aerinkim,

We are aware of this behavior. Prior to creating an mp.Image, you should preprocess the frame by converting something like it to a numpy array using frame = np.array(frame). We have already confirmed with the team that there are no plans to extend support to numpy views.

For, more understanding you may refer to this issue #5251.

Thank you!!

@kuaashish kuaashish added the stat:awaiting response Waiting for user response label May 8, 2024
@github-actions GitHub Actions
Copy link

This issue has been marked stale because it has no recent activity since 7 days. It will be closed if no further activity occurs. Thank you.

@github-actions github-actions bot added the stale label May 16, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@kuaashish kuaashish

Labels
os:linux-non-arm Issues on linux distributions which run on x86-64 architecture. DOES NOT include ARM devices. platform:python MediaPipe Python issues stale stat:awaiting response Waiting for user response task:face landmarker Issues related to Face Landmarker: Identify facial features for visual effects and avatars. type:support General questions
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@aerinkim @kuaashish