Learn Vision Transformer
This is a repo I used to learn Vision transformer in deep learning. I also tested ideas such as kinetic/H1 regularization via hooks and layerscales. Distributed training was also implemented. Compact convolutional transformer (cct) was also tested.
Also, summary writer is used to log the training statistics, including the accuracy, norm of output in each layer, and cosine similarities.
Many experiments are here. Code not cleaned.