how to reproduce the scores in paper?
ivy94419 opened this issue · comments
I have train a model and the parameters are set similar with yours:
20 epochs
~80,000 train set
~4000 val set
but I only got 19.3 bleu-4 in epoch 8, but the paper has achieved 24.3 bleu-4
When continue training, the scores decrease as follows:
Epoch bleu4
-------------
15 18.28
16 17.71
17 17.79
18 17.72
19 17.43
20 17.35
I am a beginner in deep learning, and how should I adjust parameters or other things to get higher scores?
Are you using the same code from the notebook? I ask since most couldn't even get it to run
@mukeshmithrakumar Yes I run the same code, and I used Python 3.6, Tensorflow 1.4, it can run on both Windows10 and ubuntu, I only change some trivial code for adaptation.
Hi @ivy94419 I will test the code and will get back to you
Hi @ivy94419 @mukeshmithrakumar I am able to train the model (python 2.7 tensorflow = 1 4)but could not evaluate them.
Any help regarding it would be helpful
Hi @rijuldhir, I wanted this to be a benchmark and it turned out to be a lot of hustle to get it to predict something so moved on to a different model. Will let you know if I come back to this but I highly doubt anytime soon. You will have better luck with asking ivy
@rijuldhir what errors have you faced during evaluating?
When i run evaluate_model.ipynb
I get the following error
InvalidArgumentError (see above for traceback): Assign requires shapes of both tensors to match. lhs shape= [1500] rhs shape= [1024]
[[Node: save/Assign_7 = Assign[T=DT_FLOAT, _class=["loc:@initial_lstm/b_c"], use_locking=true, validate_shape=true, _device="/job:localhost/replica:0/task:0/device:GPU:0"](initial_lstm/b_c, save/RestoreV2_7/_29)]]
[[Node: save/RestoreV2_19/_14 = _SendT=DT_FLOAT, client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device_incarnation=1, tensor_name="edge_62_save/RestoreV2_19", _device="/job:localhost/replica:0/task:0/device:CPU:0"]]
@ivy94419 In case you know it shapes are not equal?
@rijuldhir Sorry for late reply, try to change this to 1024
Thanks for the reply @ivy94419 .
I actually got it while checking the code.
Did you got the BLEU scores as given in the paper as I got the max of BLEU-4 19.2 only?
Do I need to make any changes to the code?
@rijuldhir I only achieved 19.3 BLEU-4 as mentioned above ...
@ivy94419 I have been trying to change vgg network from vgg19 to vgg16 in the code but I am getting some errors.
Any chance you know what's the problem?
do you get the same score in the paper？thanks