Replace SW-MSA with PixelShuffle

As sliding window in Swin-Transformer tries to get correlation attention between different window partitions, we intuitively replace sliding window with pixel shuffle, which is also named as Space2Depth and Depth2Space. As pixel shuffle has nothing to do with window shift, the cyclic shift is no-more required, thus may give a natural support for non-square window size. What's more, as pixel shuffle splice image in Space2Depth format, one window in W-MSA is connected to all other windows in SW-MSA(though only one pixel is connected for each window), we argue that this may give a better connection for transformer and may gain counterpart of better accuracy.

The main change is:

def window_packing(x, window_size):
    B, H, W, C = x.shape
    x= x.contiguous().view(B, window_size, H // window_size, window_size, W // window_size, C)
    windows = x.permute(0, 2, 4, 1, 3, 5).contiguous().view(-1, window_size, window_size, C)
    return windows

def window_unpacking(windows, window_size, H, W):
    B = int(windows.shape[0] / (H * W / window_size / window_size))
    x = windows.view(B, H // window_size, W // window_size, window_size, window_size, -1)
    x = x.permute(0, 3, 1, 4, 2, 5).contiguous().view(B, H, W, -1)
    return x

Due to limited GPU resources, we only conducted an simple experiment with Swin-Tiny for image-classification on ImageNet-1K (which takes about 19 days, 0_0), the result is reported as follows:

name	pretrain	resolution	acc@1	acc@5	#params	FLOPs	FPS	1K model
Swin-T	ImageNet-1K	224x224	81.4(+0.2)	95.6(+0.1)	28M	4.5G	755	Google Drive

If anyone interested in full experiments for complex architecture and other image tasks, you're welcome to download the code and report the result. To run the code, everything is same as Swin-Tranformer.

Ref:

we adopt pixel shuffle code from: PackNet-SFM: https://github.com/TRI-ML/packnet-sfm

Swin Transformer

By Ze Liu*, Yutong Lin*, Yue Cao*, Han Hu*, Yixuan Wei, Zheng Zhang, Stephen Lin and Baining Guo.

This repo is the official implementation of "Swin Transformer: Hierarchical Vision Transformer using Shifted Windows". It currently includes code and models for the following tasks:

Image Classification: Included in this repo. See get_started.md for a quick start.

Object Detection and Instance Segmentation: See Swin Transformer for Object Detection.

Semantic Segmentation: See Swin Transformer for Semantic Segmentation.

Self-Supervised Learning: See Transformer-SSL.

Updates

05/12/2021

Used as a backbone for Self-Supervised Learning: Transformer-SSL

Using Swin-Transformer as the backbone for self-supervised learning enables us to evaluate the transferring performance of the learnt representations on down-stream tasks, which is missing in previous works due to the use of ViT/DeiT, which has not been well tamed for down-stream tasks.

04/12/2021

Initial commits:

Pretrained models on ImageNet-1K (Swin-T-IN1K, Swin-S-IN1K, Swin-B-IN1K) and ImageNet-22K (Swin-B-IN22K, Swin-L-IN22K) are provided.
The supported code and models for ImageNet-1K image classification, COCO object detection and ADE20K semantic segmentation are provided.
The cuda kernel implementation for the local relation layer is provided in branch LR-Net.

Introduction

Swin Transformer (the name Swin stands for Shifted window) is initially described in arxiv, which capably serves as a general-purpose backbone for computer vision. It is basically a hierarchical Transformer whose representation is computed with shifted windows. The shifted windowing scheme brings greater efficiency by limiting self-attention computation to non-overlapping local windows while also allowing for cross-window connection.

Swin Transformer achieves strong performance on COCO object detection (58.7 box AP and 51.1 mask AP on test-dev) and ADE20K semantic segmentation (53.5 mIoU on val), surpassing previous models by a large margin.

Main Results on ImageNet with Pretrained Models

ImageNet-1K and ImageNet-22K Pretrained Models

name	pretrain	resolution	acc@1	acc@5	#params	FLOPs	FPS	22K model	1K model
Swin-T	ImageNet-1K	224x224	81.2	95.5	28M	4.5G	755	-	github/baidu
Swin-S	ImageNet-1K	224x224	83.2	96.2	50M	8.7G	437	-	github/baidu
Swin-B	ImageNet-1K	224x224	83.5	96.5	88M	15.4G	278	-	github/baidu
Swin-B	ImageNet-1K	384x384	84.5	97.0	88M	47.1G	85	-	github/baidu
Swin-B	ImageNet-22K	224x224	85.2	97.5	88M	15.4G	278	github/baidu	github/baidu
Swin-B	ImageNet-22K	384x384	86.4	98.0	88M	47.1G	85	github/baidu	github/baidu
Swin-L	ImageNet-22K	224x224	86.3	97.9	197M	34.5G	141	github/baidu	github/baidu
Swin-L	ImageNet-22K	384x384	87.3	98.2	197M	103.9G	42	github/baidu	github/baidu

Note: access code for baidu is swin.

Main Results on Downstream Tasks

COCO Object Detection (2017 val)

Backbone	Method	pretrain	Lr Schd	box mAP	mask mAP	#params	FLOPs
Swin-T	Mask R-CNN	ImageNet-1K	3x	46.0	41.6	48M	267G
Swin-S	Mask R-CNN	ImageNet-1K	3x	48.5	43.3	69M	359G
Swin-T	Cascade Mask R-CNN	ImageNet-1K	3x	50.4	43.7	86M	745G
Swin-S	Cascade Mask R-CNN	ImageNet-1K	3x	51.9	45.0	107M	838G
Swin-B	Cascade Mask R-CNN	ImageNet-1K	3x	51.9	45.0	145M	982G
Swin-T	RepPoints V2	ImageNet-1K	3x	50.0	-	45M	283G
Swin-T	Mask RepPoints V2	ImageNet-1K	3x	50.3	43.6	47M	292G
Swin-B	HTC++	ImageNet-22K	6x	56.4	49.1	160M	1043G
Swin-L	HTC++	ImageNet-22K	3x	57.1	49.5	284M	1470G
Swin-L	HTC++^*	ImageNet-22K	3x	58.0	50.4	284M	-

Note: ^* indicates multi-scale testing.

ADE20K Semantic Segmentation (val)

Backbone	Method	pretrain	Crop Size	Lr Schd	mIoU	mIoU (ms+flip)	#params	FLOPs
Swin-T	UPerNet	ImageNet-1K	512x512	160K	44.51	45.81	60M	945G
Swin-S	UperNet	ImageNet-1K	512x512	160K	47.64	49.47	81M	1038G
Swin-B	UperNet	ImageNet-1K	512x512	160K	48.13	49.72	121M	1188G
Swin-B	UPerNet	ImageNet-22K	640x640	160K	50.04	51.66	121M	1841G
Swin-L	UperNet	ImageNet-22K	640x640	160K	52.05	53.53	234M	3230G

Citing Swin Transformer

@article{liu2021Swin,
  title={Swin Transformer: Hierarchical Vision Transformer using Shifted Windows},
  author={Liu, Ze and Lin, Yutong and Cao, Yue and Hu, Han and Wei, Yixuan and Zhang, Zheng and Lin, Stephen and Guo, Baining},
  journal={arXiv preprint arXiv:2103.14030},
  year={2021}
}

Getting Started

For Image Classification, please see get_started.md for detailed instructions.
For Object Detection and Instance Segmentation, please see Swin Transformer for Object Detection.
For Semantic Segmentation, please see Swin Transformer for Semantic Segmentation.
For Self-Supervised Learning, please see Transformer-SSL.

Third-party Usage and Experiments

In this pargraph, we cross link third-party repositories which use Swin and report results. You can let us know by raising an issue

(Note please report accuracy numbers and provide trained models in your new repository to facilitate others to get sense of correctness and model behavior)

[04/14/2021] Swin for RetinaNet in Detectron: https://github.com/xiaohu2015/SwinT_detectron2.

[04/16/2021] Included in a famous model zoo: https://github.com/rwightman/pytorch-image-models.

[04/20/2021] Swin-Transformer classifier inference using TorchServe: https://github.com/kamalkraj/Swin-Transformer-Serve

Contributing

This project welcomes contributions and suggestions. Most contributions require you to agree to a Contributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us the rights to use your contribution. For details, visit https://cla.opensource.microsoft.com.

When you submit a pull request, a CLA bot will automatically determine whether you need to provide a CLA and decorate the PR appropriately (e.g., status check, comment). Simply follow the instructions provided by the bot. You will only need to do this once across all repos using our CLA.

This project has adopted the Microsoft Open Source Code of Conduct. For more information see the Code of Conduct FAQ or contact opencode@microsoft.com with any additional questions or comments.

Trademarks

This project may contain trademarks or logos for projects, products, or services. Authorized use of Microsoft trademarks or logos is subject to and must follow Microsoft's Trademark & Brand Guidelines. Use of Microsoft trademarks or logos in modified versions of this project must not cause confusion or imply Microsoft sponsorship. Any use of third-party trademarks or logos are subject to those third-party's policies.

FangGet / Swin-Transformer