train_sequential.py is missing the PoseExtractor, or requires a different preprocess.py

[Kyle Harrison]

Hello,

Just wondering if I'm missing something or if train_sequential.py is currently a template?

dataset = pd.read_csv('data/data.csv', index_col=None)
If data.csv is generated by preprocess.py it is a csv list of all image paths. Reshaping a data.csv with:

X_test = X_test.reshape(X_test.shape[0], pose_vec_dim, window)
How is it reshaped unless data.csv is constructed in a different format like an image being transformed and its feature vector being recorded.

[Goh Jia Yi, Jesa]

From my understanding, I believe that you're right. I managed to preprocess my images with trt_pose after modifying and merging codes from transformer.py and preprocess.py. The output was a .csv file with 37 columns (36 x, 1 y variable). However, I was unable to run train_sequential.py right away because of the window variable.

I don't truly understand what the window variable is supposed to do. The default train_sequential.py file has window = 3 while the train_legday.py example under /examples/yogai has window = 5. Without adjusting the window variable, the following error pops up.

>>> X_test = X_test.reshape(X_test.shape[0], pose_vec_dim, window)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ValueError: cannot reshape array of size 124920 into shape (3470,36,3)
>>> X_train = X_train.reshape(X_train.shape[0], pose_vec_dim, window)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ValueError: cannot reshape array of size 499608 into shape (13878,36,3)

In the sample file squats_deadlifts_stand5.csv, there are 141 columns (140 x, 1 y variable). The model used in this example produces 28 points (unlike trt_pose above which produces 36 points) and I figured that the model must have no. of x variables = no. of points * window (140 = 28 * 5). Thus, when I adjusted to window=1 in my file, I was able to train my model using train_sequential.py (since 36 = 36 * 1 for my case). I was wondering what the window variable does and will there be any repercussions by adjusting my window variable to 1. I would be glad to share my code that I used to preprocess my images if anyone wants to see them.

[Goh Jia Yi, Jesa]

Hi, just an update which may help you. In iva.py there is a variable WRITE2CSV and from there, you can tweak the code to process training videos and save the data into csv file. From there, you can run train_sequential.py and the model will be able to detect your custom actions.

[Laurent Weingart]

Hello @gohjiayi,

I'm coming today facing the same issue as you did at the time.
I would like to try and use train_sequential.py to train my own action recognition model, but as noticed a year ago, it is not exactly ready for the task as is.

It would seem that you managed to find a solution. Would you mind detailing it a bit more, to help me successfully prepare the dataset (or the csv file) and train the model ?

Thank you for your esteemed help
Cheers

[Goh Jia Yi, Jesa]

Sharing the codes I've used to generate the csv data on a Google Drive link here. Note that I've used a 36 point pose estimation library instead so you might want to change to 28. It'll be fantastic if someone can use this to create a PR and document it down for future references. 😄

[nomaad]

Hello @gohjiayi,
Thanks a lot for your share! I tried to use it, but I have missing files, such as "trt_pose.coo, trt_pose.models, torch2trt". Can I ask, where did you get them?

[Goh Jia Yi, Jesa]

@aidynabirov You'll have to install the trt_pose dependency as indicated in the README. Not too sure what device you're working on but I've previously used a NVIDIA Jetson Nano.

[nomaad]

@gohjiayi, what was your torch version (and other versions of dependent libraries) for installing trt_pose? I'm working on my laptop (GeForce GTX 1050). Thanks in advance!

[Goh Jia Yi, Jesa]

@aidynabirov I can't really recall given that it was quite some time back and I'm no longer using my Jetson Nano.. I can't advise you on this one!

[nomaad]

@gohjiayi, thanks a lot for your advice! I installed all dependencies, and at last, it asked me for "human_pose.json". I downloaded this JSON file from "https://github.com/NVIDIA-AI-IOT/trt_pose/blob/master/tasks/human_pose/human_pose.json". Then I got this error: "json.decoder.JSONDecodeError: Expecting value: line 7 column 1 (char 6)". Have you met this kind of error? Sorry for bothering you again, it's just you're the only person, who managed to preprocess and train custom data. By the way, I am doing it on my laptop, not Jetson Nano

[Goh Jia Yi, Jesa]

@aidynabirov I would actually recommend you to go onto StackOverflow. It definitely looks like a generic error that the json can't be read rather than a project specific issue. You might want to print out logs as you try to debug.

I'm pretty out of touch with this topic so I can't provide much advice on codes. It will be better if you open a new issue so that people currently working on the same project can contribute.