How long the training should take?

Question

How long the training should take?

Joshua-Yu opened this issue 5 years ago · comments

Fanghua(Joshua) Yu commented 5 years ago

Hi David,

I have been training the model for 24 hours by now(using default settings) and it's been running 7.6+ millions steps. The global_step per second is about 75. I am running it in a Docker CentOS container with 4GB RAM and 2 CPU cores (i7 7500 2.7G).

Do you think it is the expected behavior? How many steps it would do?

Cheers

Joshua

David Mack · Answer 1 · Sun Mar 03 2019 23:13:33 GMT+0800 (China Standard Time)

Hello! It should only need a few minutes of training. What’s your test accuracy at?

…

On Sun, Mar 3, 2019 at 5:08 AM Fanghua(Joshua) Yu ***@***.***> wrote: Hi David, I have been training the model for 24 hours by now(using default settings) and it's been running 7.6+ millions steps. The global_step per second is about 75. I am running it in a Docker CentOS container with 4GB RAM and 2 CPU cores (i7 7500 2.7G). Do you think it is the expected behavior? How many steps it would do? Cheers Joshua — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#1>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AOUFImOCQRFkblbMlrDhEXWIZayTADYsks5vS8lXgaJpZM4ba-uQ> .

Fanghua(Joshua) Yu · Answer 2 · Mon Mar 04 2019 07:31:21 GMT+0800 (China Standard Time)

I'd think so too. I didn't change any settings from the repository, and used this to run:

$ python3.6 -m macgraph.train

The last lines on screen before I terminated the execution were:
... ...
INFO:tensorflow:global_step/sec: 81.7011
INFO:tensorflow:loss = 0.0, step = 10511101 (1.224 sec)
INFO:tensorflow:loss = 0.0, step = 10511101 (1.224 sec)

Where do I set up the accuracy?

Cheers

Joshua

David Mack · Answer 3 · Mon Mar 04 2019 07:49:53 GMT+0800 (China Standard Time)

Loss = 0.0 implies that the accuracy is probably really high. You can see the accuracy by running `pipenv run tensorboard —logdir ./model` or similar Enjoy :) On March 3, 2019 at 3:31:22 PM, Fanghua(Joshua) Yu (notifications@github.com) wrote: I'd think so too. I didn't change any settings from the repository, and used this to run: $ python3.6 -m macgraph.train The last lines on screen before I terminated the execution were: ... ... INFO:tensorflow:global_step/sec: 81.7011 INFO:tensorflow:loss = 0.0, step = 10511101 (1.224 sec) INFO:tensorflow:loss = 0.0, step = 10511101 (1.224 sec) Where do I set up the accuracy? Cheers Joshua — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#1 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AOUFIm8Q1KxISCPMdvrXG_bOaABQun_Iks5vTFtJgaJpZM4ba-uQ> .