`tfjs-models/face-landmarks-detection`: `keypoint` `291` missing in `MEDIAPIPE_FACE_MESH_KEYPOINTS_BY_CONTOUR.lips`

[Jack B. Du]

System information

• Have I written custom code (as opposed to using a stock example script provided in TensorFlow.js): No
• OS Platform and Distribution (e.g., Linux Ubuntu 16.04): macOS Sonoma 14.4
• Mobile device (e.g. iPhone 8, Pixel 2, Samsung Galaxy) if the issue happens on mobile device: n/a
• TensorFlow.js installed from (npm or script link): n/a
• TensorFlow.js version (use command below): n/a
• Browser version: Firefox Developer Edition 125.0b3 (64-bit)
• Tensorflow.js Converter Version: n/a

Describe the current behavior
keypoint 291 (a corner of the lips, see mesh_map) is not present in MEDIAPIPE_FACE_MESH_KEYPOINTS_BY_CONTOUR.lips

Describe the expected behavior
keypoint 291 should be included in MEDIAPIPE_FACE_MESH_KEYPOINTS_BY_CONTOUR.lips

Standalone code to reproduce the issue
Lips corner not rendered correctly in face-landmarks-detection MediaPipe demo

Other info / logs
I suspect the issue comes from constants.ts. See snippets below:

lips: connectionsToIndices(LIPS_CONNECTIONS),

function connectionsToIndices(connections: PairArray) {
  const indices = connections.map(connection => connection[0]);
  indices.push(connections[connections.length - 1][1]);
  return indices;
}

const LIPS_CONNECTIONS: PairArray = [
  [61, 146],  [146, 91],  [91, 181],  [181, 84],  [84, 17],   [17, 314],
  [314, 405], [405, 321], [321, 375], [375, 291], [61, 185],  [185, 40],
  [40, 39],   [39, 37],   [37, 0],    [0, 267],   [267, 269], [269, 270],
  [270, 409], [409, 291], [78, 95],   [95, 88],   [88, 178],  [178, 87],
  [87, 14],   [14, 317],  [317, 402], [402, 318], [318, 324], [324, 308],
  [78, 191],  [191, 80],  [80, 81],   [81, 82],   [82, 13],   [13, 312],
  [312, 311], [311, 310], [310, 415], [415, 308],
];

In particular, connections.map(connection => connection[0]); eliminates keypoint 291 from the indices.
This happens because 291 appears as the second element in both [375, 291] and [409, 291] whereas 61 (the other corner of the lips) appears as the first element in both [61, 146] and [61, 185].

Hope this helps!

[Linda Paiste]

There are two other keypoints which will be missing for the same reason.

Part: lips is missing indices: 291
Part: leftEye is complete.
Part: leftEyebrow is missing indices: 285
Part: leftIris is complete.
Part: rightEye is complete.
Part: rightEyebrow is missing indices: 55
Part: rightIris is complete.
Part: faceOval is complete.

It seems like the expectation is that the end of one pair will be the start of the next pair, so I ran a quick code to find instances where this is not the case. Not all of these "jumps" lead to missing indices. For the left/right eye the index which would potentially be left off (362/ 133) also appears at the end of the last connection so it gets included there. Same for index 308 in the lips.

Part: lips
Connection jump at index 10: 291 -> 61 between pairs [375,291] and [61,185].
Connection jump at index 20: 291 -> 78 between pairs [409,291] and [78,95].
Connection jump at index 30: 308 -> 78 between pairs [324,308] and [78,191].
Part: leftEye
Connection jump at index 8: 362 -> 263 between pairs [382,362] and [263,466].
Part: leftEyebrow
Connection jump at index 4: 285 -> 300 between pairs [295,285] and [300,293].
Part: leftIris
All okay
Part: rightEye
Connection jump at index 8: 133 -> 33 between pairs [155,133] and [33,246].
Part: rightEyebrow
Connection jump at index 4: 55 -> 70 between pairs [65,55] and [70,63].
Part: rightIris
All okay
Part: faceOval
All okay

Code I used for both checks:

const parts = {
  lips: LIPS_CONNECTIONS,
  leftEye: LEFT_EYE_CONNECTIONS,
  leftEyebrow: LEFT_EYEBROW_CONNECTIONS,
  leftIris: LEFT_IRIS_CONNECTIONS,
  rightEye: RIGHT_EYE_CONNECTIONS,
  rightEyebrow: RIGHT_EYEBROW_CONNECTIONS,
  rightIris: RIGHT_IRIS_CONNECTIONS,
  faceOval: FACE_OVAL_CONNECTIONS,
};

Object.entries(parts).forEach(([part, connections]) => {
  const allIndices = MEDIAPIPE_FACE_MESH_KEYPOINTS_BY_CONTOUR[part];
  const omissions = connections
    .flat()
    .filter((index) => !allIndices.includes(index));
  if (omissions.length > 0) {
    console.log(`Part: ${part} is missing indices: ${omissions.join(", ")}`);
  } else {
    console.log(`Part: ${part} is complete.`);
  }
});

Object.entries(parts).forEach(([part, connections]) => {
  const jumps = connections
    .map((current, i) => {
      const previous = connections[i - 1];
      if (previous && current[0] !== previous[1]) {
        return `Connection jump at index ${i}: ${previous[1]} -> ${current[0]} between pairs [${previous}] and [${current}].`;
      }
      return;
    })
    .filter(Boolean);
  console.log(`Part: ${part}`);
  if (jumps.length > 0) {
    console.log(jumps.join("\n"));
  } else {
    console.log("All okay");
  }
});

[gaikwadrahul8]

Hi, @jackbdu, @lindapaiste

We sincerely apologize for the delay in our response and thank you for bringing this issue to our attention, I tried face-landmarks-detection MediaPipe demo and it seems like there are total 03 missing keypoints so our relavant team will look into this issue and I believe they'll fix this issue soon

In the meantime, we welcome contributions from the community. If you have a solution to address these missing keypoints, please feel free to submit a pull request (PR). Our team will thoroughly review your PR and take appropriate action, including merging it if it meets our guidelines.

Thank you for your cooperation and patience.