Questions about training

Question

Questions about training

0xMarsRover opened this issue 3 years ago · comments

Hi Akshitac,

Sorry to open a new issue since I get more questions about training.

Question 1: For each epoch during training, the model is trained with seen data and then evaluate the model (from this epoch) with test set. After that, the accuracy is obtained after evaluation in this epoch. After 30 epochs, you pick the best accuracy as the final result. Am I right?

Question 2: How about the argparse parameter: --syn_num? I can see that the default is set to 600 (means generating 600 visual representations for each action class?). Is there any experiment to indicate that this parameter can significantly influence in the model performance? Any suggestions for that?

Thanks in advance.

Kind regards.
Kaiqiang

Mars Rover · Answer 1 · Thu Aug 19 2021 06:33:13 GMT+0800 (China Standard Time)

In addition, I tried a different class embedding (dimension is 2048), but got the error RuntimeError: CUDA error: device-side assert triggered'. I set argparse as: --nz 2048 --attSize 2048

Also, this error comes from optimizerE.step() in train_action.py. Is this error about tensor size mismatch? I have no idea right now. Thanks

Mars Rover · Answer 2 · Thu Aug 19 2021 16:54:31 GMT+0800 (China Standard Time)

In addition, I tried a different class embedding (dimension is 2048), but got the error RuntimeError: CUDA error: device-side assert triggered'. I set argparse as: --nz 2048 --attSize 2048

Also, this error comes from optimizerE.step() in train_action.py. Is this error about tensor size mismatch? I have no idea right now. Thanks

For the issue above, there are more details as follows.
'/pytorch/aten/src/ATen/native/cuda/Loss.cu:111: operator(): block: [1267,0,0], thread: [31,0,0] Assertion input_val >= zero && input_val <= one failed.' `

File "/content/kg_gnn_gan/train_tfvaegan.py", line 274, in <module>', ' vae_loss_seen = loss_fn(recon_x, input_resv, means, log_var)', ' File "/content/kg_gnn_gan/train_tfvaegan.py", line 72, in loss_fn', ' BCE = torch.nn.functional.binary_cross_entropy(recon_x + 1e-12, x.detach(), size_average=False)', ' File "/usr/local/lib/python3.7/dist-packages/torch/nn/functional.py", line 2893, in binary_cross_entropy', ' return torch._C._nn.binary_cross_entropy(input, target, weight, reduction_enum)', 'RuntimeError: CUDA error: device-side assert triggered']

It seems like the issue is about the calculation of loss function.

Mars Rover · Answer 3 · Fri Aug 20 2021 01:10:59 GMT+0800 (China Standard Time)

In addition, I tried a different class embedding (dimension is 2048), but got the error RuntimeError: CUDA error: device-side assert triggered'. I set argparse as: --nz 2048 --attSize 2048

Also, this error comes from optimizerE.step() in train_action.py. Is this error about tensor size mismatch? I have no idea right now. Thanks

Some updates for the issue above.

I tested another semantic embedding, which has 1024 dimensions. It works.

But this failed -> the case with using 2048 dimensional vector as semantic embedding.

Notes: When I apply different semantic embeddings, I set proper parameters according to the size of embedding in the script. Also, I do not change other codes or settings (only changing semantic embedding).

Looking forward to hearing from you. Thanks.

Kind regards.
Kai

MaLong · Answer 4 · Tue Aug 31 2021 16:16:27 GMT+0800 (China Standard Time)

Hi Akshitac,

Sorry to open a new issue since I get more questions about training.

Question 1: For each epoch during training, the model is trained with seen data and then evaluate the model (from this epoch) with test set. After that, the accuracy is obtained after evaluation in this epoch. After 30 epochs, you pick the best accuracy as the final result. Am I right?

Question 2: How about the argparse parameter: --syn_num? I can see that the default is set to 600 (means generating 600 visual representations for each action class?). Is there any experiment to indicate that this parameter can significantly influence in the model performance? Any suggestions for that?

Thanks in advance.

Kind regards.
Kaiqiang

Based on my understanding of this paper: Your understanding is correct. For the issue of syn_num parameter, please refer to FREE: Feature Refinement for Generalized Zero-Shot Learning。

Mars Rover · Answer 5 · Tue Aug 31 2021 16:29:36 GMT+0800 (China Standard Time)

Hi Akshitac,

Sorry to open a new issue since I get more questions about training.

Question 1: For each epoch during training, the model is trained with seen data and then evaluate the model (from this epoch) with test set. After that, the accuracy is obtained after evaluation in this epoch. After 30 epochs, you pick the best accuracy as the final result. Am I right?

Question 2: How about the argparse parameter: --syn_num? I can see that the default is set to 600 (means generating 600 visual representations for each action class?). Is there any experiment to indicate that this parameter can significantly influence in the model performance? Any suggestions for that?

Thanks in advance.

Kind regards.

Kaiqiang

Based on my understanding of this paper: Your understanding is correct. For the issue of syn_num parameter, please refer to FREE: Feature Refinement for Generalized Zero-Shot Learning。

Thank you so much. Nice paper recommendation.

MaLong · Answer 6 · Tue Aug 31 2021 16:34:20 GMT+0800 (China Standard Time)

Hi Akshitac,

Sorry to open a new issue since I get more questions about training.

Question 1: For each epoch during training, the model is trained with seen data and then evaluate the model (from this epoch) with test set. After that, the accuracy is obtained after evaluation in this epoch. After 30 epochs, you pick the best accuracy as the final result. Am I right?

Question 2: How about the argparse parameter: --syn_num? I can see that the default is set to 600 (means generating 600 visual representations for each action class?). Is there any experiment to indicate that this parameter can significantly influence in the model performance? Any suggestions for that?

Thanks in advance.

Kind regards.

Kaiqiang

Based on my understanding of this paper: Your understanding is correct. For the issue of syn_num parameter, please refer to FREE: Feature Refinement for Generalized Zero-Shot Learning。

Thank you so much. Nice paper recommendation.

You bet! Do you know this question？ #24 (comment)

Mars Rover · Answer 7 · Tue Aug 31 2021 16:37:30 GMT+0800 (China Standard Time)

Hi Akshitac,

Sorry to open a new issue since I get more questions about training.

Question 1: For each epoch during training, the model is trained with seen data and then evaluate the model (from this epoch) with test set. After that, the accuracy is obtained after evaluation in this epoch. After 30 epochs, you pick the best accuracy as the final result. Am I right?

Question 2: How about the argparse parameter: --syn_num? I can see that the default is set to 600 (means generating 600 visual representations for each action class?). Is there any experiment to indicate that this parameter can significantly influence in the model performance? Any suggestions for that?

Thanks in advance.

Kind regards.

Kaiqiang

Based on my understanding of this paper: Your understanding is correct. For the issue of syn_num parameter, please refer to FREE: Feature Refinement for Generalized Zero-Shot Learning。

Thank you so much. Nice paper recommendation.

You bet! Do you know this question？ #24 (comment)

Sorry, I did not test on image ZSL. I just focus on action recognition. I guess the issue may be caused by hyper-parameter settings.

MaLong · Answer 8 · Tue Aug 31 2021 16:39:02 GMT+0800 (China Standard Time)

Hi Akshitac,

Sorry to open a new issue since I get more questions about training.

Question 1: For each epoch during training, the model is trained with seen data and then evaluate the model (from this epoch) with test set. After that, the accuracy is obtained after evaluation in this epoch. After 30 epochs, you pick the best accuracy as the final result. Am I right?

Question 2: How about the argparse parameter: --syn_num? I can see that the default is set to 600 (means generating 600 visual representations for each action class?). Is there any experiment to indicate that this parameter can significantly influence in the model performance? Any suggestions for that?

Thanks in advance.

Kind regards.

Kaiqiang

Based on my understanding of this paper: Your understanding is correct. For the issue of syn_num parameter, please refer to FREE: Feature Refinement for Generalized Zero-Shot Learning。

Thank you so much. Nice paper recommendation.

You bet! Do you know this question？ #24 (comment)

Sorry, I did not test on image ZSL. I just focus on action recognition. I guess the issue may be caused by hyper-parameter settings.

Oh, it doesn't matter.

Akshita Gupta · Answer 9 · Wed Sep 01 2021 03:07:29 GMT+0800 (China Standard Time)

Question 1: For each epoch during training, the model is trained with seen data and then evaluate the model (from this epoch) with test set. After that, the accuracy is obtained after evaluation in this epoch. After 30 epochs, you pick the best accuracy as the final result. Am I right?

Ans - Yes, your understanding is correct.

Question 2: How about the argparse parameter: --syn_num? I can see that the default is set to 600 (means generating 600 visual representations for each action class?). Is there any experiment to indicate that this parameter can significantly influence in the model performance? Any suggestions for that?

Ans - syn_num parameters controls how many unseen samples per class you want to synthesis from the generator. Yes this parameter influences the model performance because further after synthesis you also train a classifier. The classifier training input data is a combination of these synthesised unseen class samples along with the real samples.

Mars Rover · Answer 10 · Wed Sep 01 2021 06:16:17 GMT+0800 (China Standard Time)

Question 1: For each epoch during training, the model is trained with seen data and then evaluate the model (from this epoch) with test set. After that, the accuracy is obtained after evaluation in this epoch. After 30 epochs, you pick the best accuracy as the final result. Am I right?

Ans - Yes, your understanding is correct.

Question 2: How about the argparse parameter: --syn_num? I can see that the default is set to 600 (means generating 600 visual representations for each action class?). Is there any experiment to indicate that this parameter can significantly influence in the model performance? Any suggestions for that?

Ans - syn_num parameters controls how many unseen samples per class you want to synthesis from the generator. Yes this parameter influences the model performance because further after synthesis you also train a classifier. The classifier training input data is a combination of these synthesised unseen class samples along with the real samples.

Thanks for your clarification. Much appreciated.