Running the tests gives a segmentation fault.

Question

Running the tests gives a segmentation fault.

shivamvats opened this issue 8 years ago · comments

@rizar I followed the instructions in README to run the tests. However, I am getting seg faults for all the tests I tried. My guess is that code is not getting the correct data. Are there additional steps I need to take to properly set the run time paths?

Shivam Vats commented 8 years ago

Both.

Dzmitry Bahdanau · Answer 1 · Mon Dec 19 2016 21:49:16 GMT+0800 (China Standard Time)

By "tests", do you mean the content of "tests" folder, or do you mean the actual experiments? The former was included in the repository by mistake and should be deleted.

Shivam Vats · Answer 2 · Tue Dec 20 2016 10:01:54 GMT+0800 (China Standard Time)

I am sorry for the confusion. I mean the actual experiments billion words and ted used in the paper.

Dzmitry Bahdanau · Answer 3 · Tue Dec 20 2016 10:11:33 GMT+0800 (China Standard Time)

OK, I see. Can you please provide a bit more information about the exact commands that you ran?

…

On Mon, 19 Dec 2016 at 21:01 Shivam Vats ***@***.***> wrote: I am sorry for the confusion. I mean the actual experiments billion words and ted used in the paper. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#1 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAn8Yi2riOFcxLm6T9Q6e6njSzz-MKeFks5rJzcSgaJpZM4LQih3> .

Shivam Vats · Answer 4 · Tue Dec 20 2016 10:30:18 GMT+0800 (China Standard Time)

source env.sh.
cd exp/billion_words
$LVSR/bin/run.py train autoencoder3 configs/autoencoder3.yaml
cd ../ted
./create_dataset.sh (I first set the path to MOSES)`
$LVSR/bin/run.py train ted12 config/ted12.yaml

Error:

Using gpu device 0: GeForce GTX 960M (CNMeM is enabled with initial size: 90.0% of memory, cuDNN not available)    
Segmentation fault

Dzmitry Bahdanau · Answer 5 · Wed Dec 21 2016 00:18:04 GMT+0800 (China Standard Time)

Hold on, did the step 3 work for you? You should've downloaded the data first. I will add the instruction on how to download the data today.

Dzmitry Bahdanau · Answer 6 · Wed Dec 21 2016 01:02:20 GMT+0800 (China Standard Time)

https://github.com/rizar/actor-critic-public/blob/master/exp/billion_words/README.md is now updated

Shivam Vats · Answer 7 · Thu Dec 22 2016 02:27:44 GMT+0800 (China Standard Time)

I did download the data (I put it in lvsr/datasets though).

I still get the same error after putting the data in FUEL_DATA_PATH. I am surprised to see a seg fault with Python. Are you using C wrappers in your code? From fuel's docs, it seems that it needs data as a .hdf5 file. Does it need to be converted to fuel's format?

Also, in the configs for ted the data filename is set as TED/de-en/ted.h5 (I would expect it to be ted.h5).

Dzmitry Bahdanau · Answer 8 · Thu Dec 22 2016 02:34:04 GMT+0800 (China Standard Time)

No, their is no C code in our implementation. But libhdf5 could trigger one. But let's focus on one experiment. Do you have a segfault when you are trying to run on TED or on billion_words?

…

On Wed, 21 Dec 2016 at 13:27 Shivam Vats ***@***.***> wrote: I did download the data (I put it in lvsr/datasets though). I still get the same error after putting the data in FUEL_DATA_PATH. I am surprised to see a seg fault with Python. Are you using C wrappers in your code? From fuel's docs, it seems that it needs data as a .hdf5 file. Does it need to be converted to fuel's format? Also, in the configs for ted the data filename is set as TED/de-en/ted.h5 (I would expect it to be ted.h5). — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#1 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAn8Yof13Ky_9ziHFe0I6XT0m9OYGTMWks5rKW-ggaJpZM4LQih3> .

Dzmitry Bahdanau · Answer 9 · Thu Dec 22 2016 02:37:34 GMT+0800 (China Standard Time)

OK, let's focus on billion words then. In order to help you, I would need to know at which line of lvsr/main.py the segmentation fault happens.

…

On Wed, 21 Dec 2016 at 13:34 Shivam Vats ***@***.***> wrote: Both. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#1 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAn8YoyIGF1WO2OazlRzwl3ZfnqvJHJUks5rKXFDgaJpZM4LQih3> .

Shivam Vats · Answer 10 · Thu Dec 22 2016 03:47:52 GMT+0800 (China Standard Time)

I figured out the issue. Importing fst is leading to a seg fault. I tried reinstalling pyfst and openfst but it doesn't work. The code should work without fst, right?

Dzmitry Bahdanau · Answer 11 · Thu Dec 22 2016 03:51:18 GMT+0800 (China Standard Time)

Yes, this an obsolete import. Feel free to remove it. I will remove it later as well. Thanks for the heads-up!

…

On Wed, 21 Dec 2016 at 14:47 Shivam Vats ***@***.***> wrote: I figured out the issue. Importing fst is leading to a seg fault. I tried reinstalling pyfst and openfst but it doesn't work. The code should work without fst, right? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#1 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAn8YtAcvcARoC5oFt02ky7S03NNC9aTks5rKYJogaJpZM4LQih3> .

Shivam Vats · Answer 12 · Thu Dec 22 2016 03:56:00 GMT+0800 (China Standard Time)

Thanks a lot for your prompt help! :)
Would it be possible for you to give me a rough estimate of the training time? I am using a GTX 960, with 4GB memory.

Dzmitry Bahdanau · Answer 13 · Thu Dec 22 2016 04:00:46 GMT+0800 (China Standard Time)

I am afraid this amount of memory might not be sufficient. On a K80 the pretraining (first actor, then critic) takes about 36 hours, and then you can run the main stage for as long as you want...

…

On Wed, 21 Dec 2016 at 14:56 Shivam Vats ***@***.***> wrote: Thanks a lot for your prompt help! :) Would it be possible for you to give me a rough estimate of the training time? I am using a GTX 960, with 4GB memory. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#1 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAn8YqIz_JcTVDlrZPr9RpHxkfghjYhxks5rKYRQgaJpZM4LQih3> .