sports_counting_by_pose_estimation

It is interesting to use pose estimation for sports counting. This project serves as a demo for this goal, which contains a whole pipeline of model conversion and ncnn Android deployment.

Sit up counting is supported now. It is easy to add other sports support.

Here is a demo result for Sit up counting.

model conversion and preparation

Google provides the pretrained movenet model in tensorflow format. What I did to convert it to ncnn format is recorded as following.

Download the tflite model from tensorflow model hub
- the Lightning version of movenet is used in this project because it is intended for latency-critical applications.

Convert tfllite model to ONNX model using tensorflow-onnx

python -m tf2onnx.convert --tflite movenet.tflite --output movenet.onnx

Optimize the ONNX model using onnx-simplifier

python -m onnxsim movenet.onnx sim_movenet.onnx

Edit the optimized ONNX model
- This step is necessary because some of the post processing operators are not supported in ncnn.
- What we need to do are:
  - Remove the pre-processing and post-processing operators.
    - They will be implemented in raw C++
    - tensorflow use NHWC format while ONNX use NCHW format. So the Transpose operators in pre-processing and post-processing operators should also be removed.
  - rename the model input and output.
- Then we get modified_sim_movenet.onnx
  
  ⭐⭐ onnx-modifier is highly recommended for efficient and intuitive ONNX editing! ⭐⭐

Convert the ONNX model to ncnn format

# onnx2ncnn is provided in ncnn
onnx2ncnn modified_sim_movenet.onnx movenet.param movenet.bin

Optimize the converted ncnn model

NOTE: the layer name in the converted ncnn .param is long and exceed the max length ncnn supports (255), so we need to truncate them.

Otherwise the load_model error at layer 2, parameter file has inconsistent content will be encountered.

The following Python script can be used

src = open("movenet_l.param", "r")  # the original param file with long layer name
dst = open("movenet.param", "w")

for i, src_line in enumerate(src):
    dst_line = []
    src_items = src_line.split()
    for item in src_items:
        if len(item) >= 255:
            item = item[-255:]
        dst_line.append(item)
    dst.write(" ".join(dst_line) + "\n")

Then we can optimize the ncnn model with

# # ncnnoptimize is provided in ncnn
ncnnoptimize movenet.param movenet.bin movenet_opt.param movenet_opt.bin 0

Then the ncnn model is ready! What I got is saved here

ncnn Android deployment

clone this repo and follow the instructions to build and run the Android demo.

step1

https://github.com/Tencent/ncnn/releases

Download ncnn-YYYYMMDD-android-vulkan.zip or build ncnn for android yourself
Extract ncnn-YYYYMMDD-android-vulkan.zip into app/src/main/jni and change the ncnn_DIR path to yours in app/src/main/jni/CMakeLists.txt

step2

https://github.com/nihui/opencv-mobile

Download opencv-mobile-XYZ-android.zip
Extract opencv-mobile-XYZ-android.zip into app/src/main/jni and change the OpenCV_DIR path to yours in app/src/main/jni/CMakeLists.txt

step3

Open this project with Android Studio, build it and enjoy!

The prebuilt android apk file can be downloaded here

how counting works

Take the supported Sit up for example, The angle between joints are calculated and serves as clues for counting.

/* The map from joint index to joint:
* 0 : neck; 1 & 2 : eyes; 3 & 4 : ears
* 5 & 6 : shoulders; 7 & 8 : elbows; 9 & 10 : hands
* 11 & 12 : hips; 13 & 14 : knees;
* 15 & 16 : feet
*/
float neck_hip_foot_angle = angle(points[0], points[11], points[15]);
if (!count_lock && neck_hip_foot_angle < 120) {
    count_lock = true;
    count_number += 1;
}
if (count_lock && neck_hip_foot_angle > 150) {
    count_lock = false;
}

The count_lock is a bool flag to avoid duplicated counting.

reference

movenet official blog: https://blog.tensorflow.org/2021/05/next-generation-pose-detection-with-movenet-and-tensorflowjs.html
An earlier version of ncnn movenet: https://github.com/FeiGeChuanShu/ncnn_Android_MoveNet
The sit-up video is picked here: https://www.wikihow.com/Do-Sit-Ups

ChanglinZhou / sports_counting_by_pose_estimation