shovelingpig / deepest_task

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Deepest Task

Question3 Kaggle - CommonLit Readability Prize

Submission

submission

Run

  1. Copy this kaggle notebook.
  2. Click "File > Import Notebook" and import deepest_question3_kaggle.ipynb file.
  3. Click "Run All" button in kaggle notebook or jupyter notebook.
  4. Check the training results.

Solution

  • RoBERTa: (1) robustly optimized BERT approach. (2) training the model longer, with bigger batches over more data. (3) removing the next sentence prediction objective. (4) training on longer sequences. (5) dynamically changing the masking pattern applied to the training data.
  • Sentence Summary: This model computes context vector as the weighted average of last hidden states.
  • Pre-training on CLRP: This model is pre-trained on CommonLit Readability Dataset on MLM task.
  • LP-FT: Naive full fine-tuning can distort the pretrained feature extractor. The test error of fine-tuning is high when we initialize with a fixed or random head. This is because while fine-tuning learns the head, the lower layers of the neural network change simultaneously and distort the pretrained features.

Loss Graph

loss

References

About


Languages

Language:Jupyter Notebook 100.0%