Instance-based Vision Transformer for Subtyping of Papillary Renal Cell Carcinoma in Histopathological Image
The whole framework is composed by two parts.
- (A) Nuclei segmentation & classification
- (B) Instance-based Vision Transformer
The instance-based Vision Transformer (i-ViT) for learning robust representations of histopathological images for the pRCC subtyping task by extracting finer features from instance patches (by cropping around segmented nuclei and assigning predicted grades). The i-ViT takes top-K instances as input and aggregates them for capturing both the cellular and cell-layer level patterns by a position-embedding layer, a grade-embedding layer, and a multi-head multi-layer self-attention module.
Link to MICCAI 2021 paper.
Our framework is composed by two parts, Please set up two environments based on:
Download the nuclei segmentation and classification dataset from this link.
Download the papillary RCC subtyping dataset from this link.
To use the micronet model trained on nuclei segmentation and classification dataset and used in our paper, download it following Segmentation Model File.
- Download the papillary RCC subtyping dataset and unzip it to
./dataset. - Run the nuclei segmentation and classification inference to get the predicted mask of each tissue image. (Or download the predicted results from the same link of pRCC dataset)
- Train the I-ViT model to follow the Running Scripts
If any part of this code is used, please give appropriate citation to our paper.
- Zeyu Gao (betpotti@gmail.com)
- Bangyang Hong (hby4732@stu.xjtu.edu.cn)
- Chen Li (cli@xjtu.edu.cn)
BioMedical Semantic Understanding Group, Xi'an Jiaotong University
This project is licensed under the MIT License - see the LICENSE file for details
We thank the implementation of nuclei segmentation and classification framework HoVerNet and the VIT implementation for image classification.
The datasets used are in whole or part based upon data generated by the TCGA Research Network.