Bad result with vqgan

[shizhediao]

Hi, I am using VQGAN on the MSCOCO training dataset (also tried adding Visual Genome to construct a 1 Million dataset), but got a bad result. The pixels are wired.

Here are my settings, thanks!

transformer_dim = 512 rotary_emb = False image_fmap_size = 32 self.transformer = Transformer( dim = transformer_dim, causal = True, seq_len = seq_len, depth = self.config_visual_decoder.num_hidden_layers, heads = 8, dim_head = 64, reversible = False, attn_dropout = 0.0, ff_dropout = 0.0, attn_types = 'full', image_fmap_size = image_fmap_size, sparse_attn = False, stable = False, sandwich_norm = False, shift_tokens = False, rotary_emb = rotary_emb, # shared_attn_ids = None, # shared_ff_ids = None, # optimize_for_inference = False, )

I looked up several previous issues and reports and notice that people usually get loss < 4.5 while my loss is around 5.4.

I use a large batch size (more than 3000) while others use a far smaller batch size (like 16), does that matter?
Thanks

[Romain Beaumont]

yes if you can use such a large batch size, it either means that you have hundred of gpus or that your model is too small
you should increase the depth parameter, and probably train for longer. How long did you train?

[shizhediao]

Thanks for your quick reply!
I have tried three settings, all of them are based on a 6layer Transformer.

1. MSCOCO training data: 400,000 image-text-pairs
2. MSCOCO + VG + some private data: around 6M image-text-pairs
3. MSCOCO + VG + CC15M: around 16M image-text-pairs

To speed up the training, I use many A100 GPUs with only 15 epochs. The first setting could be finished in several hours.

The loss is from 7.0 -> 6.5 -> 5.4, and stuck at 5.4 from epoch5.

[shizhediao]

What could I do if I want to both speed up with large batch size and use only 6layer Transformer?
Maybe larger learning rate and train longer?
It seems that even with lr=1e-3, the loss will get stuck at 5.0

[Romain Beaumont]

Try the default lr and depth 16
What distributor do you use ? Deepspeed? Horovod?

Increasing depth usually gets much better results

[shizhediao]

Try the default lr and depth 16 What distributor do you use ? Deepspeed? Horovod?

Increasing depth usually gets much better results

Neither, I am using NCCL constructed by a company.
I suddenly found there is an issue said ""adamw" optimizer + weight decay = poor generations" #170
I am using adamw + weight decay0.01, does that matter?