Semi automatic annotation Skeleton w/ Mediapipe : TypeError Cannot read properties of undefined (reading 'attr')
H04K opened this issue · comments
Actions before raising this issue
- I searched the existing issues and did not find anything similar.
- I read/searched the docs
Steps to Reproduce
- Use the provided function.yaml and main.py files to set up a Nuclio function.
- Call the annotate function.
- Observe the JavaScript error in the CVAT UI
Expected Behavior
Semi automatic annotation for right hand using media pipe, similar to open pose skeleton annotation but with media pipe
Possible Solution
No response
Context
I'm trying to use media pipe and cvat for semi automatic annotation of hands, I'm using the mmpose code as reference.
Here is the function.yaml
metadata:
name: mediapipe
namespace: cvat
annotations:
name: Mediapipe
type: detector
framework: tensorflow
spec: |
[{
"name":"r_hand",
"type":"skeleton",
"svg":"<line x1=\"74.8981704711914\" y1=\"59.111854553222656\" x2=\"77.56928253173828\" y2=\"55.27212142944336\" data-type=\"edge\" data-node-from=\"19\" data-node-to=\"20\"></line>\n<line x1=\"70.72454833984375\" y1=\"64.12020111083984\" x2=\"74.8981704711914\" y2=\"59.111854553222656\" data-type=\"edge\" data-node-from=\"18\" data-node-to=\"19\"></line>\n<line x1=\"67.0517578125\" y1=\"68.79466247558594\" x2=\"70.72454833984375\" y2=\"64.12020111083984\" data-type=\"edge\" data-node-from=\"17\" data-node-to=\"18\"></line>\n<line x1=\"63.87980270385742\" y1=\"56.60768127441406\" x2=\"64.8814697265625\" y2=\"48.92821502685547\" data-type=\"edge\" data-node-from=\"15\" data-node-to=\"16\"></line>\n<line x1=\"62.71118927001953\" y1=\"62.617698669433594\" x2=\"63.87980270385742\" y2=\"56.60768127441406\" data-type=\"edge\" data-node-from=\"14\" data-node-to=\"15\"></line>\n<line x1=\"60.874794006347656\" y1=\"67.9599380493164\" x2=\"62.71118927001953\" y2=\"62.617698669433594\" data-type=\"edge\" data-node-from=\"13\" data-node-to=\"14\"></line>\n<line x1=\"54.196998596191406\" y1=\"52.767948150634766\" x2=\"54.36394500732422\" y2=\"40.914859771728516\" data-type=\"edge\" data-node-from=\"11\" data-node-to=\"12\"></line>\n<line x1=\"53.362274169921875\" y1=\"59.94657897949219\" x2=\"54.196998596191406\" y2=\"52.767948150634766\" data-type=\"edge\" data-node-from=\"10\" data-node-to=\"11\"></line>\n<line x1=\"53.86310958862305\" y1=\"67.45909881591797\" x2=\"53.362274169921875\" y2=\"59.94657897949219\" data-type=\"edge\" data-node-from=\"9\" data-node-to=\"10\"></line>\n<line x1=\"41.175296783447266\" y1=\"52.434059143066406\" x2=\"39.00501251220703\" y2=\"45.75625991821289\" data-type=\"edge\" data-node-from=\"7\" data-node-to=\"8\"></line>\n<line x1=\"43.3455810546875\" y1=\"60.28046798706055\" x2=\"41.175296783447266\" y2=\"52.434059143066406\" data-type=\"edge\" data-node-from=\"6\" data-node-to=\"7\"></line>\n<line x1=\"45.849754333496094\" y1=\"68.62771606445312\" x2=\"43.3455810546875\" y2=\"60.28046798706055\" data-type=\"edge\" data-node-from=\"5\" data-node-to=\"6\"></line>\n<line x1=\"67.0517578125\" y1=\"68.79466247558594\" x2=\"48.85476303100586\" y2=\"88.32721710205078\" data-type=\"edge\" data-node-from=\"17\" data-node-to=\"0\"></line>\n<line x1=\"60.874794006347656\" y1=\"67.9599380493164\" x2=\"67.0517578125\" y2=\"68.79466247558594\" data-type=\"edge\" data-node-from=\"13\" data-node-to=\"17\"></line>\n<line x1=\"53.86310958862305\" y1=\"67.45909881591797\" x2=\"60.874794006347656\" y2=\"67.9599380493164\" data-type=\"edge\" data-node-from=\"9\" data-node-to=\"13\"></line>\n<line x1=\"45.849754333496094\" y1=\"68.62771606445312\" x2=\"53.86310958862305\" y2=\"67.45909881591797\" data-type=\"edge\" data-node-from=\"5\" data-node-to=\"9\"></line>\n<line x1=\"48.85476303100586\" y1=\"88.32721710205078\" x2=\"45.849754333496094\" y2=\"68.62771606445312\" data-type=\"edge\" data-node-from=\"0\" data-node-to=\"5\"></line>\n<line x1=\"24.981639862060547\" y1=\"71.79966735839844\" x2=\"18.639402389526367\" y2=\"66.79132080078125\" data-type=\"edge\" data-node-from=\"3\" data-node-to=\"4\"></line>\n<line x1=\"29.155261993408203\" y1=\"78.31051635742188\" x2=\"24.981639862060547\" y2=\"71.79966735839844\" data-type=\"edge\" data-node-from=\"2\" data-node-to=\"3\"></line>\n<line x1=\"37.502506256103516\" y1=\"85.15525817871094\" x2=\"29.155261993408203\" y2=\"78.31051635742188\" data-type=\"edge\" data-node-from=\"1\" data-node-to=\"2\"></line>\n<line x1=\"48.85476303100586\" y1=\"88.32721710205078\" x2=\"37.502506256103516\" y2=\"85.15525817871094\" data-type=\"edge\" data-node-from=\"0\" data-node-to=\"1\"></line>\n<circle r=\"0.75\" cx=\"48.85476303100586\" cy=\"88.32721710205078\" data-type=\"element node\" data-element-id=\"0\" data-node-id=\"0\" data-label-id=\"0\" data-label-name=\"0\"></circle>\n<circle r=\"0.75\" cx=\"37.502506256103516\" cy=\"85.15525817871094\" data-type=\"element node\" data-element-id=\"1\" data-node-id=\"1\" data-label-id=\"1\" data-label-name=\"1\"></circle>\n<circle r=\"0.75\" cx=\"29.155261993408203\" cy=\"78.31051635742188\" data-type=\"element node\" data-element-id=\"2\" data-node-id=\"2\" data-label-id=\"2\" data-label-name=\"2\"></circle>\n<circle r=\"0.75\" cx=\"24.981639862060547\" cy=\"71.79966735839844\" data-type=\"element node\" data-element-id=\"3\" data-node-id=\"3\" data-label-id=\"3\" data-label-name=\"3\"></circle>\n<circle r=\"0.75\" cx=\"18.639402389526367\" cy=\"66.79132080078125\" data-type=\"element node\" data-element-id=\"4\" data-node-id=\"4\" data-label-id=\"4\" data-label-name=\"4\"></circle>\n<circle r=\"0.75\" cx=\"45.849754333496094\" cy=\"68.62771606445312\" data-type=\"element node\" data-element-id=\"5\" data-node-id=\"5\" data-label-id=\"5\" data-label-name=\"5\"></circle>\n<circle r=\"0.75\" cx=\"43.3455810546875\" cy=\"60.28046798706055\" data-type=\"element node\" data-element-id=\"6\" data-node-id=\"6\" data-label-id=\"6\" data-label-name=\"6\"></circle>\n<circle r=\"0.75\" cx=\"41.175296783447266\" cy=\"52.434059143066406\" data-type=\"element node\" data-element-id=\"7\" data-node-id=\"7\" data-label-id=\"7\" data-label-name=\"7\"></circle>\n<circle r=\"0.75\" cx=\"39.00501251220703\" cy=\"45.75625991821289\" data-type=\"element node\" data-element-id=\"8\" data-node-id=\"8\" data-label-id=\"8\" data-label-name=\"8\"></circle>\n<circle r=\"0.75\" cx=\"53.86310958862305\" cy=\"67.45909881591797\" data-type=\"element node\" data-element-id=\"9\" data-node-id=\"9\" data-label-id=\"9\" data-label-name=\"9\"></circle>\n<circle r=\"0.75\" cx=\"53.362274169921875\" cy=\"59.94657897949219\" data-type=\"element node\" data-element-id=\"10\" data-node-id=\"10\" data-label-id=\"10\" data-label-name=\"10\"></circle>\n<circle r=\"0.75\" cx=\"54.196998596191406\" cy=\"52.767948150634766\" data-type=\"element node\" data-element-id=\"11\" data-node-id=\"11\" data-label-id=\"11\" data-label-name=\"11\"></circle>\n<circle r=\"0.75\" cx=\"54.36394500732422\" cy=\"40.914859771728516\" data-type=\"element node\" data-element-id=\"12\" data-node-id=\"12\" data-label-id=\"12\" data-label-name=\"12\"></circle>\n<circle r=\"0.75\" cx=\"60.874794006347656\" cy=\"67.9599380493164\" data-type=\"element node\" data-element-id=\"13\" data-node-id=\"13\" data-label-id=\"13\" data-label-name=\"13\"></circle>\n<circle r=\"0.75\" cx=\"62.71118927001953\" cy=\"62.617698669433594\" data-type=\"element node\" data-element-id=\"14\" data-node-id=\"14\" data-label-id=\"14\" data-label-name=\"14\"></circle>\n<circle r=\"0.75\" cx=\"63.87980270385742\" cy=\"56.60768127441406\" data-type=\"element node\" data-element-id=\"15\" data-node-id=\"15\" data-label-id=\"15\" data-label-name=\"15\"></circle>\n<circle r=\"0.75\" cx=\"64.8814697265625\" cy=\"48.92821502685547\" data-type=\"element node\" data-element-id=\"16\" data-node-id=\"16\" data-label-id=\"16\" data-label-name=\"16\"></circle>\n<circle r=\"0.75\" cx=\"67.0517578125\" cy=\"68.79466247558594\" data-type=\"element node\" data-element-id=\"17\" data-node-id=\"17\" data-label-id=\"17\" data-label-name=\"17\"></circle>\n<circle r=\"0.75\" cx=\"70.72454833984375\" cy=\"64.12020111083984\" data-type=\"element node\" data-element-id=\"18\" data-node-id=\"18\" data-label-id=\"18\" data-label-name=\"18\" ></circle>\n<circle r=\"0.75\" cx=\"74.8981704711914\" cy=\"59.111854553222656\" data-type=\"element node\" data-element-id=\"19\" data-node-id=\"19\" data-label-id=\"19\" data-label-name=\"19\"></circle>\n<circle r=\"0.75\" cx=\"77.56928253173828\" cy=\"55.27212142944336\" data-type=\"element node\" data-element-id=\"20\" data-node-id=\"20\" data-label-id=\"20\" data-label-name=\"20\"></circle>",
"sublabels":[
{
"id":0,
"name":"0",
"type":"points"
},
{
"id":1,
"name":"1",
"type":"points"
},
{
"id":2,
"name":"2",
"type":"points"
},
{
"id":3,
"name":"3",
"type":"points"
},
{
"id":4,
"name":"4",
"type":"points"
},
{
"id":5,
"name":"5",
"type":"points"
},
{
"id":6,
"name":"6",
"type":"points"
},
{
"id":7,
"name":"7",
"type":"points"
},
{
"id":8,
"name":"8",
"type":"points"
},
{
"id":9,
"name":"9",
"type":"points"
},
{
"id":10,
"name":"10",
"type":"points"
},
{
"id":11,
"name":"11",
"type":"points"
},
{
"id":12,
"name":"12",
"type":"points"
},
{
"id":13,
"name":"13",
"type":"points"
},
{
"id":14,
"name":"14",
"type":"points"
},
{
"id":15,
"name":"15",
"type":"points"
},
{
"id":16,
"name":"16",
"type":"points"
},
{
"id":17,
"name":"17",
"type":"points"
},
{
"id":18,
"name":"18",
"type":"points"
},
{
"id":19,
"name":"19",
"type":"points"
},
{
"id":20,
"name":"20",
"type":"points"}
]
}
]
spec:
description: mediapipe
runtime: 'python:3.8'
handler: main:handler
eventTimeout: 30s
build:
image: cvat.tensortflow.mediapipe.base
baseImage: ubuntu:20.04
directives:
preCopy:
- kind: RUN
value: |-
apt update \
&& apt install -y --no-install-recommends \
git \
ca-certificates \
python-is-python3 \
python3 \
python3-pip \
&& rm -rf /var/lib/apt/lists/*
- kind: RUN
value: pip3 install mediapipe
- kind: RUN
value: pip3 install opencv-python-headless
- kind: RUN
value: pip3 install numpy
- kind: RUN
value: pip3 install matplotlib
- kind: RUN
value: pip3 install pyyaml
- kind: RUN
value: pip3 install pillow
- kind: WORKDIR
value: /opt/nuclio
triggers:
myHttpTrigger:
maxWorkers: 2
kind: 'http'
workerAvailabilityTimeoutMilliseconds: 10000
attributes:
maxRequestBodySize: 33554432 # 32MB
platform:
attributes:
restartPolicy:
name: always
maximumRetryCount: 3
mountMode: volume
resources:
limits:
cpu: 2
memory: 350M
and here is the main.py
import cv2
import mediapipe as mp
import json
import base64
import numpy as np
from PIL import Image
import io
import yaml
import cv2
import mediapipe as mp
import json
import base64
import numpy as np
from PIL import Image
import io
import yaml
def transform_data(data):
transformed_data = []
for item in data:
if item['type'] == 'points':
elements = []
for point in item['points']:
elements.append({
'label': point['name'],
'type': 'points',
'outside': 0,
'points': [float(point['x']), float(point['y'])],
'confidence': '0.8'
})
transformed_data.append({
'confidence': '0.90301526',
'label': item['name'],
'type': 'skeleton',
'elements': elements
})
return transformed_data
def init_context(context):
context.logger.info("Init detector...")
mp_holistic = mp.solutions.holistic
holistic = mp_holistic.Holistic()
context.logger.info("Init labels...")
with open("/opt/nuclio/function.yaml", "rb") as function_file:
functionconfig = yaml.safe_load(function_file)
labels_spec = functionconfig["metadata"]["annotations"]["spec"]
labels = json.loads(labels_spec)
context.user_data.labels = labels
context.user_data.holistic = holistic
context.logger.info("Function initialized")
def convert_landmarks_to_cvat_format(landmarks, name):
if landmarks is None:
return None
# Load the CVAT format from the YAML file
with open('function.yaml', 'r') as f:
cvat_format = yaml.safe_load(f)
# Convert the spec string to a dictionary
spec = yaml.safe_load(cvat_format['metadata']['annotations']['spec'])
# Find the correct skeleton in the CVAT format
skeleton = next((s for s in spec if s['name'] == name), None)
# Convert the landmarks to the CVAT format
points = []
if skeleton is not None:
for i, landmark in enumerate(landmarks.landmark):
if i < len(skeleton['sublabels']):
point = {
'id': skeleton['sublabels'][i]['id'],
'name': str(i),
'x': landmark.x,
'y': landmark.y,
'visibility': '0.8',
}
points.append(point)
return points
def handler(context, event):
context.logger.info("Run MediaPipe Holistic model")
# Load the image from the event data
data = event.body
buf = io.BytesIO(base64.b64decode(data["image"]))
image = Image.open(buf).convert("RGB")
image = np.array(image)
results = context.user_data.holistic.process(image)
# Extract the pose landmarks
right_hand_landmarks = results.right_hand_landmarks
# Convert the landmarks to the format expected by CVAT
right_hand_data = convert_landmarks_to_cvat_format(right_hand_landmarks, 'r_hand')
# Combine the data for all the parts into a list of dictionaries
data_list = []
if right_hand_data is not None:
data_list.append({
'name': 'r_hand',
'type': 'points',
'points': right_hand_data
})
# Transform the data
transformed_data = transform_data(data_list)
print(transformed_data)
# Return the data as a JSON object using context.Response
return context.Response(body=json.dumps(transformed_data), headers={}, content_type="application/json", status_code=200)
Calling the annotate works great here the log from the nuclio function
[{'confidence': '0.90301526', 'label': 'r_hand', 'type': 'skeleton', 'elements': [{'label': '0', 'type': 'points', 'outside': 0, 'points': [0.4395136833190918, 0.5782752633094788], 'confidence': '0.0'}, {'label': '1', 'type': 'points', 'outside': 0, 'points': [0.47249719500541687, 0.5689968466758728], 'confidence': '0.0'}, {'label': '2', 'type': 'points', 'outside': 0, 'points': [0.49751439690589905, 0.5525428056716919], 'confidence': '0.0'}, {'label': '3', 'type': 'points', 'outside': 0, 'points': [0.5127729773521423, 0.5342648029327393], 'confidence': '0.0'}, {'label': '4', 'type': 'points', 'outside': 0, 'points': [0.5175373554229736, 0.510733962059021], 'confidence': '0.0'}, {'label': '5', 'type': 'points', 'outside': 0, 'points': [0.49938878417015076, 0.49907180666923523], 'confidence': '0.0'}, {'label': '6', 'type': 'points', 'outside': 0, 'points': [0.5178049206733704, 0.4843536615371704], 'confidence': '0.0'}, {'label': '7', 'type': 'points', 'outside': 0, 'points': [0.5204618573188782, 0.5074937343597412], 'confidence': '0.0'}, {'label': '8', 'type': 'points', 'outside': 0, 'points': [0.5153696537017822, 0.528052031993866], 'confidence': '0.0'}, {'label': '9', 'type': 'points', 'outside': 0, 'points': [0.4885108172893524, 0.4938916265964508], 'confidence': '0.0'}, {'label': '10', 'type': 'points', 'outside': 0, 'points': [0.5073314309120178, 0.48257797956466675], 'confidence': '0.0'}, {'label': '11', 'type': 'points', 'outside': 0, 'points': [0.5106727480888367, 0.5072055459022522], 'confidence': '0.0'}, {'label': '12', 'type': 'points', 'outside': 0, 'points': [0.5052586197853088, 0.5285157561302185], 'confidence': '0.0'}, {'label': '13', 'type': 'points', 'outside': 0, 'points': [0.4755867123603821, 0.4925291836261749], 'confidence': '0.0'}, {'label': '14', 'type': 'points', 'outside': 0, 'points': [0.49422499537467957, 0.48339420557022095], 'confidence': '0.0'}, {'label': '15', 'type': 'points', 'outside': 0, 'points': [0.49988627433776855, 0.5063366293907166], 'confidence': '0.0'}, {'label': '16', 'type': 'points', 'outside': 0, 'points': [0.4961303174495697, 0.5273551940917969], 'confidence': '0.0'}, {'label': '17', 'type': 'points', 'outside': 0, 'points': [0.4613873362541199, 0.49610546231269836], 'confidence': '0.0'}, {'label': '18', 'type': 'points', 'outside': 0, 'points': [0.48074111342430115, 0.4904457926750183], 'confidence': '0.0'}, {'label': '19', 'type': 'points', 'outside': 0, 'points': [0.4883764982223511, 0.5075832605361938], 'confidence': '0.0'}, {'label': '20', 'type': 'points', 'outside': 0, 'points': [0.4879501163959503, 0.5252431035041809], 'confidence': '0.0'}]}]
But I get this screen exactly when the inference is done
TypeError
Cannot read properties of undefined (reading 'attr')
TypeError: Cannot read properties of undefined (reading 'attr')
at kF.addSkeleton (https://cvat.retorik.ai/assets/cvat-ui.121b31fdc29d9ddb5d35.min.js:2:9517117)
at kF.addObjects (https://cvat.retorik.ai/assets/cvat-ui.121b31fdc29d9ddb5d35.min.js:2:9490407)
at kF.setupObjects (https://cvat.retorik.ai/assets/cvat-ui.121b31fdc29d9ddb5d35.min.js:2:9433518)
at kF.notify (https://cvat.retorik.ai/assets/cvat-ui.121b31fdc29d9ddb5d35.min.js:2:9469495)
at NN.notify (https://cvat.retorik.ai/assets/cvat-ui.121b31fdc29d9ddb5d35.min.js:2:8442100)
at NN.setup (https://cvat.retorik.ai/assets/cvat-ui.121b31fdc29d9ddb5d35.min.js:2:8503675)
at TF.setup (https://cvat.retorik.ai/assets/cvat-ui.121b31fdc29d9ddb5d35.min.js:2:9540573)
at WBe.updateCanvas (https://cvat.retorik.ai/assets/cvat-ui.121b31fdc29d9ddb5d35.min.js:2:11799318)
at WBe.componentDidUpdate (https://cvat.retorik.ai/assets/cvat-ui.121b31fdc29d9ddb5d35.min.js:2:11795844)
at ui (https://cvat.retorik.ai/assets/cvat-ui.121b31fdc29d9ddb5d35.min.js:2:5429432)
The results from the inference/Nuclio function are the same as mmpose in terms of logic and format.
Environment
- Browser is Chrome on Mac OS Ventura 13.6
- CVAT version
Server: 2.12.0
Core: 15.0.2
Canvas: 2.20.0
UI: 1.63.6
- Nuclio function conf is in function.yaml
I do not think that relative coordinates are correct:
'points': [0.4395136833190918, 0.5782752633094788]
The code on client expects pixel coordinates.
Hello, thank you for the answer just got back working on it
So, I attempted the solution given in #622 (comment) to convert the relative coordinates into pixel coordinates
Here is the new data
[{'confidence': '0.90301526', 'label': 'r_hand', 'type': 'skeleton', 'elements': [{'label': '0', 'type': 'points', 'outside': 0, 'points': [228.1690216064453, 244.72214698791504], 'confidence': '0.8'}, {'label': '1', 'type': 'points', 'outside': 0, 'points': [244.52947616577148, 256.14699840545654], 'confidence': '0.8'}, {'label': '2', 'type': 'points', 'outside': 0, 'points': [255.15756607055664, 270.76623916625977], 'confidence': '0.8'}, {'label': '3', 'type': 'points', 'outside': 0, 'points': [259.13318634033203, 283.39852809906006], 'confidence': '0.8'}, {'label': '4', 'type': 'points', 'outside': 0, 'points': [262.84093856811523, 292.98545837402344], 'confidence': '0.8'}, {'label': '5', 'type': 'points', 'outside': 0, 'points': [236.5384864807129, 296.3512659072876], 'confidence': '0.8'}, {'label': '6', 'type': 'points', 'outside': 0, 'points': [238.6057472229004, 315.60779571533203], 'confidence': '0.8'}, {'label': '7', 'type': 'points', 'outside': 0, 'points': [238.4959602355957, 327.4267387390137], 'confidence': '0.8'}......
Out of curiosity I used matplotlib to plot the extracted coordinate (plot limits are matching image input size)
(looks exactly like the source picture)
Still no success whatsoever, the exact same error persists right after inference
Cannot read properties of undefined (reading 'attr')
Check the label specification in your function.yml.
On your instance I tried to add a label from a model and annotate it manually. It didn't work.
At least you have to remove data-label-id from the svg specification.
Ok made some adjustemenst and removed data-label-id and pretty much got it to work, hand is upside down but nothing too painful to fix thanks for your inputs and help :)