Mapping of 2d joints ?

[basicvisual]

Hi Denis,
I am trying to map the 2d joints, from another 2d detector. I note that transformed_pose2d is a 1x14x2 array . I was wondering what is the joint order that is expected. I have the following two standard detectors with joint order .I was wondering what is the joint order that is expected so that
I can map another 2d joint detector .

`  For StackedHourglass
  0  'RFoot'
  1  'RKnee'
  2  'RHip'
  3  'LHip'
  4  'LKnee'
  5  'LFoot',
  6  'Hip'
  7  'Spine'
  8  'Thorax'
  9  'Head'
  10  'RWrist'
  11  'RElbow'
  12  'Rshoulder'
  13  'LShoulder'
  14  'LElbow'
  15  'LWrist'
  
  For Posenet 
  'nose',             0
  'leftEye',          1
  'rightEye',         2
  'leftEar',          3
  'rightEar',         4
  'leftShoulder',     5
  'rightShoulder',    6
  'leftElbow',        7
  'rightElbow',       8
  'leftWrist',        9
  'rightWrist',      10 
  'leftHip',         11  
  'rightHip',        12
  'leftKnee', '      13
  rightKnee',        14
  'leftAnkle',       15
  'rightAnkle'       16
`

[Guo-king]

The image is inferred from the code.Maybe it will help you.

[Mritula Chandrasekaran]

Hi Guo_King,
This is the connection value given in draw.py
[0, 1], [1, 2], [2, 3], [0, 4], [4, 5], [5, 6], [0, 7], [7, 8],
[8, 9], [9, 10], [8, 11], [11, 12], [12, 13], [8, 14], [14, 15],
[15, 16]]

I don't think the above figure can be connected using these values,
i.e : 0 is connected to 4 and 4 is connected to 5; but in your above diagram 3 is connected to 4. Every coordinate is centered to hip. So I think the hip point should be 0.

Hope this helps!

Thanks
Mritula

[Guo-king]

hi Mritula
My answer above is about the 2D joints.
The code you have found in draw.py is for 3D joints. And I can share the map of 3D joints that I have inferred.

[Mritula Chandrasekaran]

Oh yes... thanks for confirming that.

[basicvisual]

@Guo-king @MritulaC , thank you for clearing , out the doubts.and visuals . Really appreciate that . From what it looks like , i think posenet cannot be exactly mapped to the 2D estimation done here. I was thinking to filter out the following from the 2D detections

leftEye', 1 'rightEye', 2 'leftEar', 3 'rightEar', 4

But that returns more like a 13x2 array instead of 14x2 , and since the lifting is basically assigning weights to the 14x2 array perhaps it wont work . That is what i understand . What do you think ?

[basicvisual]

The image is inferred from the code.Maybe it will help you.

I modified the keypoints from posenet , and for keypoint 1 , i just took the mean of left shoulder and right shoulder but unfortunately i got some weird results. I was trying to estimate , squat position but i´the 3d inference was totally of.
Thanks for the heads up

[Mritula Chandrasekaran]

hi Guys,
Do any of you face the below error?
in normalise_data
d2 = d2.reshape(d2.shape[0], -1, 2).transpose(0, 2, 1)
ValueError: cannot reshape array of size 0 into shape (0,newaxis,2)

I tried playing with Visible Part value in threshold still no use. Help would be greatly appreciated

[basicvisual]

Hi @Guo-king , i tried to have the following as 2D keypoint and extend it to the 3d pose lifting.https://github.com/rwightman/posenet-python/blob/1d8192833d816d78e0c7c58a9aa717cce73975cd/webcam_demo.py#L41 which returns a 10x17x2 array. As suggested by you the following changes were made to align it to have as lifting from deep and dump it to a json file (converting into 14x2 vector)

keypoint_coords *= output_scale #reshaping from 10x17x2 to 17x2
Followed by conversion into 14x2
keypoint_14 = [keypoint_coords_17_reshape[0], ((keypoint_coords_17_reshape[5]+keypoint_coords_17_reshape[6])/2),keypoint_coords_17_reshape[5], keypoint_coords_17_reshape[7],keypoint_coords_17_reshape[9], keypoint_coords_17_reshape[6],keypoint_coords_17_reshape[8], keypoint_coords_17_reshape[10], keypoint_coords_17_reshape[11], keypoint_coords_17_reshape[13], keypoint_coords_17_reshape[15],keypoint_coords_17_reshape[12], keypoint_coords_17_reshape[14], keypoint_coords_17_reshape[16]]
Which should be the input the 3d pose lifting . However when i do the 3d predictions , the image is totally off and upside down and no relation. Do you know what could be the problem . Basically , I am trying to ignore some keypoints on the equation .

Looking to know more

[Denis Tome']

In order to switch to a different 2D pose estimator, you cannot simply just remove and plug in a different one, since the 2D to 3D lifter has to learn the geometry from something it knows in 2D.
To successfully switch the 2D detector and get reasonable 3D poses, you need to re-train the second module where the 2D inputs are now coming from the new 2D pose estimator.

[Denis Tome']

Closing for inactivity.