Giters

kudkudak / common-sense-prediction

Common sense prediction using DL.

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Explore regularizers for glove

kudkudak opened this issue · comments

commented

Assume that performance on farthest bucket is a good predictor of test performance. Performance of farthest bucket is not ATM computed within script (will add before starting this task)

Try to see if any regularizer for glove improves scores. Idea:

  • Embedding dropout
  • Learning rate schedule with very large LR at the beginning

Compare to numbers in https://www.overleaf.com/12326026ybwcgtxbxpkr#/46877048/. Generate your own data with python scripts/data/split_intrinsic_ACL.py

Example command (glove embedding, l2=1e-5, random split)

python scripts/train_factorized.py root $SCRATCH/l2lwe/results/factorized/3_01_root_conceptnet_my_random_100k_l2a=1e-5_glove  --data_dir=$DATA_DIR/LiACL/conceptnet_my_random_100k --l2_a=1e-5 --embedding_file=embeddings/LiACL/embeddings_glove200_norm.txt