dwzhu-pku / PoSE

Positional Skip-wise Training for Efficient Context Window Extension of LLMs to Extremely Length (ICLR 2024)

https://arxiv.org/abs/2309.10400

dwzhu-pku/PoSE Issues

“input_ids” is the same as "labels" in train_preprocess_function_pose function ?
Closed a month ago2
Comparative experiments on PI directly on 2k length
Closed a month ago8
建议把数据上传到国内的魔塔社区一份
Closed a month ago3
How to reproduce the ppl behaviours?
Closed 2 months ago1
LM Evaluate
Closed 2 months ago5
To create a model for **textual similarity tasks** involving **JSON-structured data
Closed 3 months ago1
A question about data preproecess
Closed 4 months ago1
Models do well on PASS-KEY retrieval but very badly on longchat topic retrieval
Closed 5 months ago6
What is the performance of PoSE without using interpolations like linear / NTK / YaRN?
Closed 6 months ago3
Example Training data
Closed 7 months ago2
训练过程
Closed 8 months ago7