Giters

sign-language-processing / pose

Library for viewing, augmenting, and handling .pose files

Home Page:https://pose-format.readthedocs.io/en/latest/

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Data Interpolation

yayayru opened this issue · comments

commented

Thanks for the very interesting the repo and video.

I have data on custom pose format (csv) with various frame rate. How to convert all different frame rates to one frame rate?

commented

Let's say that for each pose file you create a NumPyPoseBody object with the following:

  1. fps set to that video's framerate
  2. data an array of [frames, people, points, dimensions] (for example, 100 frames, 1 person, 137 points, and 2 dimensions X and Y)
  3. confidence an array of [frames, people, points] (which is the confidence from OpenPose or 0/1 for Holistic)

(Or better yet, if you have the data in OpenPose file format, you can use this utility)

Then, for each pose_body: NumPyPoseBody, you just do pose_body.interpolate(new_fps), where new_fps is the uniform fps you desire.

Does that answer your question?

commented

Thanks for the answer. I have got an error from custom data via MediaPipe

  • I think I have 25 points, not 137 points. Can the function take any number of points? And other reasons?
  • Can work without confidence?
import pose_format.numpy.pose_body as pb
def ex_pose_format(df_XY, np_confidences, fps):
    print("df_XY.shape", df_XY.shape)
    print("fps",  fps)
    np_XY = df_XY.to_numpy()
    np_X = np_XY[:,0::2]
    np_Y = np_XY[:,1::2]
    print("np_X.shape", np_X.shape)
    print("np_Y.shape", np_Y.shape)

    data = np.zeros((np_X.shape[0], 1, np_X.shape[1], 2))
    

    np_confidences = np_confidences.to_numpy()
    print("np_confidences.shape", np_confidences.shape)

    confidences = np.zeros((np_confidences.shape[0], 1, np_confidences.shape[1]))

    for i, n in enumerate(np_X):
        for j, _ in enumerate(n):
            data[i,0,j] = [np_X[i,j], np_Y[i,j]]
            confidences[i,0,j] = np_confidences[i,j]

    print("data.shape", data.shape)
    print("confidences.shape", confidences.shape)
    
    print("data[0]", data[0])  
    print("confidences[0]", confidences[0])

    p = pb.NumPyPoseBody(int(fps), data, np_confidences)

Output:

df_XY.shape (648, 50)
fps 25.0
np_X.shape (648, 25)
np_Y.shape (648, 25)
np_confidences.shape (648, 25)
data.shape (648, 1, 25, 2)
confidences.shape (648, 1, 25)
data[0] [[[0.50806 0.36881]
  [0.52154 0.33973]
  [0.528   0.34269]
  [0.53612 0.34509]
  [0.49732 0.33561]
  [0.48915 0.33593]
  [0.48107 0.33604]
  [0.54665 0.36755]
  [0.46729 0.35604]
  [0.52203 0.4121 ]
  [0.49046 0.40759]
  [0.59711 0.58093]
  [0.41701 0.57241]
  [0.62065 0.84448]
  [0.39682 0.82634]
  [0.63415 1.04434]
  [0.37019 1.0276 ]
  [0.64466 1.10731]
  [0.35865 1.08363]
  [0.62907 1.10806]
  [0.36939 1.08999]
  [0.61526 1.06249]
  [0.38095 1.04907]
  [0.56658 1.00712]
  [0.44615 1.00659]]]
confidences[0] [[0.95735 0.78171 1.      0.98936 0.73913 0.98416 0.766   0.91153 1.
  0.90826 0.7716  0.87509 1.      0.96537 0.79294 0.78755 0.86932 0.72665
  0.63127 0.7313  0.88502 0.75964 0.80586 0.97228 0.93328]]
Traceback (most recent call last):
  File "examples.py", line 173, in <module>
    ex_pose_format(df_x, confidences, fps)
  File "examples.py", line 160, in ex_pose_format
    p = pb.NumPyPoseBody(int(fps), data, np_confidences)
  File "/home/cv2020/miniconda3/envs/tf/lib/python3.7/site-packages/pose_format/numpy/pose_body.py", line 21, in __init__
    stacked_mask = np.stack([mask] * data.shape[-1], axis=3)
  File "<__array_function__ internals>", line 6, in stack
  File "/home/cv2020/.local/lib/python3.7/site-packages/numpy/core/shape_base.py", line 430, in stack
    axis = normalize_axis_index(axis, result_ndim)
numpy.AxisError: axis 3 is out of bounds for array of dimension 3
commented

Your data is of good shape.

Your confidence seems to be of:
confidences.shape (648, 25)

Try having it of shape (648, 1, 25)

commented

I have updated the code and output above to make it clear to you. Yes, i have (648, 1, 25)

commented

Biggggggggggggggggggggggg thank for the example in colab)))))))

import pose_format.numpy.pose_body as pb
def ex_pose_format(df_XY, df_conf, fps):
    print("fps", fps)

    np_XY = df_XY.to_numpy()
    np_conf = df_conf.to_numpy()

    np_X = np_XY[:,0::2]
    np_Y = np_XY[:,1::2]
    print("np_X.shape", np_X.shape)
    print("np_Y.shape", np_Y.shape) 

    #fps = 10
    data = np.zeros((np_XY.shape[0], 1, np_X.shape[1], 2))
    conf = np.zeros((np_conf.shape[0], 1, np_conf.shape[1]))
    print("data.shape", data.shape)
    print("conf.shape", conf.shape)

    for i, n in enumerate(np_X):
        for j, _ in enumerate(n):
            data[i,0,j] = [np_X[i,j], np_Y[i,j]]
            conf[i,0,j] = np_conf[i,j]    

    p = pb.NumPyPoseBody(int(fps), data, conf)

    print("pose_body_shape", p.data.shape)
    p = p.interpolate(new_fps=50)
    print("pose_body_shape", p.data.shape)
    pass

Output:

fps 25.0
np_X.shape (648, 25)
np_Y.shape (648, 25)
data.shape (648, 1, 25, 2)
conf.shape (648, 1, 25)
pose_body_shape (648, 1, 25, 2)
pose_body_shape (1296, 1, 25, 2)
commented

No problem :)

commented

Why is Confidence Parameter used? Can the class(NumPyPoseBody) and method(interpolate) work without confidence?

commented

you can set the confidence to np.ones
It is used so that if there are completely missing keypoints in OpenPose in various frames, they don't count for the interpolation.
The interpolation will fill them in afterwards.

commented

Biggggggggggggggggggggggg thank for the example in colab)))))))

import pose_format.numpy.pose_body as pb
def ex_pose_format(df_XY, df_conf, fps):
    print("fps", fps)

    np_XY = df_XY.to_numpy()
    np_conf = df_conf.to_numpy()

    np_X = np_XY[:,0::2]
    np_Y = np_XY[:,1::2]
    print("np_X.shape", np_X.shape)
    print("np_Y.shape", np_Y.shape) 

    #fps = 10
    data = np.zeros((np_XY.shape[0], 1, np_X.shape[1], 2))
    conf = np.zeros((np_conf.shape[0], 1, np_conf.shape[1]))
    print("data.shape", data.shape)
    print("conf.shape", conf.shape)

    for i, n in enumerate(np_X):
        for j, _ in enumerate(n):
            data[i,0,j] = [np_X[i,j], np_Y[i,j]]
            conf[i,0,j] = np_conf[i,j]    

    p = pb.NumPyPoseBody(int(fps), data, conf)

    print("pose_body_shape", p.data.shape)
    p = p.interpolate(new_fps=50)
    print("pose_body_shape", p.data.shape)
    pass

Output:

fps 25.0
np_X.shape (648, 25)
np_Y.shape (648, 25)
data.shape (648, 1, 25, 2)
conf.shape (648, 1, 25)
pose_body_shape (648, 1, 25, 2)
pose_body_shape (1296, 1, 25, 2)

Only works for body pose? And for the hand? Only works for body poses? And for the hand? Because pose_format.numpy.pose_body.NumPyPoseBody(int(fps), data, conf).interpolate(new_fps=to_fps) doesn't work for hand:(

commented

It does not actually differentiate between body and hands, it should work for everything...
A failing example is required to understand what issue you are having (if the hand only appears at 1 frame, it won't be interpolated. 2 or more will)

commented

Thanks for the quick response! I realized that only more than 2 frames, OK. I have another problem. There is no hand in the beginning and last frames of the example, and only in the middle there is a hand. But only the linear kind works, and cubes and quadratic do not work. What do you think?

commented

interpolation is defined between bounds, not out-of-bounds. If we tried to interpolate out-of-bounds, we would get extremely diverging points -
See here, imagine sampling the green line not between the two points
image

commented

Nice linear kind convert for hand from 24 to 50 fps
image