Due to GPU and data constraints
Metric: Classfication Accuracy Loss: CrossEntropy Optimizer: Adam
Original ViT: ViT( image_size = 256, patch_size = 16, num_classes = 10, dim = 1024, depth = 6, heads = 16, mlp_dim = 2048, dropout = 0.1, emb_dropout = 0.1 )
Model | Accuracy |
---|---|
Original ViT | 75,66 small,64 small p8, 66 small p8 mean |
Adapted BigBird | 73.85, 61 small p8 mean |
Random Attention | 68 small , 65 small p8. 58 small p8 mean |
Random Attention + Global | - , -, 60 small p8 mean |
Random Attention + Window | - , -, 61.7 small p8 mean |
Windowed Attention | - |
Windowed Attention + Global | - |
Global Attention | - |
Combinations of the above
Model | Accuracy |
---|---|
Original ViT | 53.33 |
Adapted BigBird | 64.58 |
Random Attention | 59.37 |
Random Attention + Global | 62.29 |
Random Attention + Window | 63.39 |
Windowed Attention | 66.25 |
Windowed Attention+ Global | 65.66 |
Global Attention | 62.39 |