Training on top of an existing model
opened this issue · comments
Hi there,
I am trying to train a new clstm model containing +1000 lines, the training process would take days.
My technique would be to train a couple of hours a day, and continue training the next day, as such.
I created an arabic-8000.clstm model for testing, and added to the script:
load=arabic-8000.clstm
start=8000
But the problem is that clstmocrtrain starts from 0 all over again.
Waiting for your reply
Please paste here the full commands you used to:
- Train the model
- Load the model
Which branch/tag are you using? The option to load a model is only supported in the master branch.
@amitdo The train model
set -x
-a
sort -R manifest.txt > /tmp/manifest2.txt
sed 1,100d /tmp/manifest2.txt > train.txt
sed 100q /tmp/manifest2.txt > test.txt
report_every=1000
save_every=1000
maxtrain=50000
target_height=48
dewarp=center
display_every=1000
test_every=1000
hidden=100
lrate=1e-4
save_name=arabic
clstmocrtrain train.txt test.txt
-------------------------------------------------------------
And the Load model is:
set -x
-a
sort -R manifest.txt > /tmp/manifest2.txt
sed 1,100d /tmp/manifest2.txt > train.txt
sed 100q /tmp/manifest2.txt > test.txt
report_every=1000
save_every=1000
maxtrain=50000
target_height=48
dewarp=center
display_every=1000
test_every=1000
hidden=100
lrate=1e-4
save_name=arabic
load=arabic-8000.clstm
start=8000
clstmocrtrain train.txt test.txt
@Christophered Your "load"/"train" steps are the same script?
Also, you can enclose multi-line code in triple backticks (```) in markdown, like so:
#!/bin/bash
set -x
-a
sort -R manifest.txt > /tmp/manifest2.txt
sed 1,100d /tmp/manifest2.txt > train.txt
sed 100q /tmp/manifest2.txt > test.txt
report_every=1000
save_every=1000
maxtrain=50000
target_height=48
dewarp=center
display_every=1000
test_every=1000
hidden=100
lrate=1e-4
save_name=arabic
load=arabic-8000.clstm
start=8000
clstmocrtrain train.txt test.txt@kba loading script is similar to the training script except for the last 3 lines
load=arabic-8000.clstm
start=8000
#129 (comment)
@amitdo what do you mean? I have the default main clstm installed.
How can I train on top of an existing model, or stop and continue training later?
I was using the seperat-derive "legacy" clstm version, it doesn't have save/load options