with low precision and recall when training roberta and xlnet on stage2

[xiuzhilu]

I train roberta and xlnet with PIE Synthetically a1(train:8865347 dev:3000), The parameters of the first stage model are as you provided them in Git(https://github.com/grammarly/gector/blob/master/docs/training_parameters.md). The training parameter and I get the result as following:
xlnet training parameter:
TRAIN_SET=train/train_seq_tag.txt
DEV_SET=dev/dev_seq_tag.txt
MODEL_PATH=model/train/stage1_xlnet # 变化
VOCAB_PATH=data/output_vocabulary
transformer_model=xlnet # 变化
special_tokens_fix=0 #变化
n_epoch=20
cold_steps_count=2
accumulation_size=4
updates_per_epoch=10000
tn_prob=0
tp_prob=1
tune_bert=1
skip_correct=1
skip_complex=0
max_len=50
batch_size=64
cold_lr=1e-3
lr=1e-5
predictor_dropout=0.0
lowercase_tokens=0
pieces_per_token=5
label_smoothing=0.0
#patience=0
patience=3
PYTHONIOENCODING=utf-8 python3 train.py --train_set $TRAIN_SET --dev_set $DEV_SET
--model_dir $MODEL_PATH --vocab_path $VOCAB_PATH --transformer_model $transformer_model
--special_tokens_fix $special_tokens_fix --n_epoch $n_epoch --cold_steps_count $cold_steps_count
--accumulation_size $accumulation_size --updates_per_epoch $updates_per_epoch --tn_prob $tn_prob
--tp_prob $tp_prob --tune_bert $tune_bert
--skip_correct $skip_correct
--skip_complex $skip_complex
--max_len $max_len
--batch_size $batch_size
--cold_lr $cold_lr
--lr $lr
--predictor_dropout $predictor_dropout
--lowercase_tokens $lowercase_tokens
--pieces_per_token $pieces_per_token
--label_smoothing $label_smoothing
--patience $patience
xlnet results:

Precision	Recall	F0.5
80.47%	74.47%	79.20%

roberta training parameter:
TRAIN_SET=train/train_seq_tag.txt
DEV_SET=dev/dev_seq_tag.txt
MODEL_PATH=model/train/stage1_roberta # 变化
VOCAB_PATH=data/output_vocabulary
transformer_model=roberta # 变化
special_tokens_fix=1 #变化
n_epoch=20
cold_steps_count=2
accumulation_size=4
updates_per_epoch=10000
tn_prob=0
tp_prob=1
tune_bert=1
skip_correct=1
skip_complex=0
max_len=50
batch_size=64
cold_lr=1e-3
lr=1e-5
predictor_dropout=0.0
lowercase_tokens=0
pieces_per_token=5
label_smoothing=0.0
#patience=0
patience=3

python3 train.py --train_set $TRAIN_SET --dev_set $DEV_SET
--model_dir $MODEL_PATH --vocab_path $VOCAB_PATH --transformer_model $transformer_model
--special_tokens_fix $special_tokens_fix --n_epoch $n_epoch --cold_steps_count $cold_steps_count
--accumulation_size $accumulation_size --updates_per_epoch $updates_per_epoch --tn_prob $tn_prob
--tp_prob $tp_prob --tune_bert $tune_bert
--skip_correct $skip_correct
--skip_complex $skip_complex
--max_len $max_len
--batch_size $batch_size
--cold_lr $cold_lr
--predictor_dropout $predictor_dropout
--lowercase_tokens $lowercase_tokens
--pieces_per_token $pieces_per_token
--label_smoothing $label_smoothing
--patience $patience
roberta:

Precision	Recall	F0.5
89.52%	66.57%	83.75%

I train xlnet and roberta on stage2 with NUCLE, Lang-8,FCE,Write & Improve + LOCNESS,all total train dataset is 1157038, dev dataset is 6573. I get the low precision and recall when I test the stage2 trained model.
xlnet:

Precision	Recall	F0.5
48.73%	20.06%	37.9%

roberta:

Precision	Recall	F0.5
55.09%	9.77%	28.58%

I use the 8-GPUS when train stage1 and stage2 model

the parameter train xlnet and roberta on stage2 as following:
xlnet training parameter:
TRAIN_SET=train/stage2_train/all.train.seq_tag
DEV_SET=dev/stage2/stage2dev.seq_tag
MODEL_PATH=model/train/stage2_xlnet_all # 变化
VOCAB_PATH=data/output_vocabulary
transformer_model=xlnet # 变化
special_tokens_fix=0 #变化
n_epoch=9 #变化
cold_steps_count=2
accumulation_size=2

accumulation_size=1

updates_per_epoch=0
tn_prob=0
tp_prob=1
pretrain_folder=model/train/stage1_xlnet #变化
pretrain=best
tune_bert=1
skip_correct=1
skip_complex=0
max_len=50
batch_size=64
cold_lr=1e-3
lr=1e-5
predictor_dropout=0.0
lowercase_tokens=0
pieces_per_token=5
label_smoothing=0.0
#patience=0
patience=3

python3 train.py --train_set $TRAIN_SET --dev_set $DEV_SET
--model_dir $MODEL_PATH --vocab_path $VOCAB_PATH --transformer_model $transformer_model
--special_tokens_fix $special_tokens_fix --n_epoch $n_epoch --cold_steps_count $cold_steps_count
--accumulation_size $accumulation_size --updates_per_epoch $updates_per_epoch --tn_prob $tn_prob
--tp_prob $tp_prob --tune_bert $tune_bert
--skip_correct $skip_correct
--skip_complex $skip_complex
--max_len $max_len
--batch_size $batch_size
--cold_lr $cold_lr
--lr $lr
--predictor_dropout $predictor_dropout
--lowercase_tokens $lowercase_tokens
--pieces_per_token $pieces_per_token
--label_smoothing $label_smoothing
--patience $patience

roberta training parameter:
TRAIN_SET=train/stage2_train/all.train.seq_tag
DEV_SET=dev/stage2/stage2dev.seq_tag
MODEL_PATH=model/train/stage2_roberta_all # 变化
VOCAB_PATH=data/output_vocabulary
transformer_model=roberta # 变化
special_tokens_fix=1 #变化
n_epoch=10 #变化
cold_steps_count=2
accumulation_size=2
updates_per_epoch=0
tn_prob=0
tp_prob=1
pretrain_folder=model/train/stage1_roberta #变化
pretrain=best
tune_bert=1
skip_correct=1
skip_complex=0
max_len=50
batch_size=64
cold_lr=1e-3
lr=1e-5
predictor_dropout=0.0
lowercase_tokens=0
pieces_per_token=5
label_smoothing=0.0
#patience=0
patience=3

python3 train.py --train_set $TRAIN_SET --dev_set $DEV_SET
--model_dir $MODEL_PATH --vocab_path $VOCAB_PATH --transformer_model $transformer_model
--special_tokens_fix $special_tokens_fix --n_epoch $n_epoch --cold_steps_count $cold_steps_count
--accumulation_size $accumulation_size --updates_per_epoch $updates_per_epoch --tn_prob $tn_prob
--tp_prob $tp_prob --tune_bert $tune_bert
--skip_correct $skip_correct
--skip_complex $skip_complex
--max_len $max_len
--batch_size $batch_size
--cold_lr $cold_lr
--lr $lr
--predictor_dropout $predictor_dropout
--lowercase_tokens $lowercase_tokens
--pieces_per_token $pieces_per_token
--label_smoothing $label_smoothing
--patience $patience
could you tell me what went wrong? @skurzhanskyi @komelianchuk

[Alex Skurzhanskyi]

Not sure, tbh. This would require reviewing the whole training process.