Hyperparameter for training SEED Tokenizer

[Cheolhyun Mun]

Hi!
Thank you for the wonderful work.

I wonder if you can provide detailed information on training SEED Tokenizer.
I cannot find the hyperparameter for training SEED Tokenizer in your paper.

Also, I have another question.
In the paper, SEED Tokenizer training is divided into two stages. Does that mean the Q-former is pre-trained in stage 1 and then the Q-former, codebook, decoder, and MLP are trained in stage 2?

Thank you.

[Yuying Ge]

We have released the training code of SEED-LLaMa, including SEED tokenizer, Multimodal LLM pretraining and instruction tuning. Our Multimodal LLM training codebase supports 1. large-scale multi-node training with deepspeed 2. highly-efficient multiple training datapipes.