Capturing Finer-grained Long-range Dependency at Higher Resolution: An Empirical Investigation of MLPs in Medical Image Dense Prediction
Dense prediction is a fundamental requirement for many medical vision tasks such as medical image restoration, registration, and segmentation. The most popular vision models, Convolutional Neural Networks (CNNs), have reached bottlenecks due to the intrinsic locality of convolution operations. Recently, transformers have been widely adopted for dense prediction for their ability to capture long-range dependency. However, due to the high computational complexity and memory consumption of self-attention operations, transformers are typically used at downsampled resolutions after patch embedding, thus disabling the modeling of long-range dependency at high resolutions (e.g., full/half of the image resolution). Such usage cannot effectively leverage the tissue-level textural information that is only available at high resolutions. This textural information is crucial for medical dense prediction as it can differentiate the subtle human anatomy /pathology in medical images. In this study, we hypothesize that Multi-Layer Perceptrons (MLPs) are superior alternatives to transformers in medical image dense prediction, as they can capture finer-grained long-range dependency at higher resolution under equal computation/memory constraints. To validate this hypothesis, we conducted a comprehensive empirical investigation of MLPs in medical image dense prediction. Specifically, we built a hierarchical MLP framework that uses MLPs to capture fine-grained long-range dependency beginning from the full image resolution, and then evaluated this framework with various MLP blocks on a wide range of medical image dense prediction tasks including image restoration, registration, and segmentation.
For more details, please refer to our paper:
- Mingyuan Meng, Yuxin Xue, Dagan Feng, Lei Bi, and Jinman Kim, "Full-resolution MLPs Empower Medical Dense Prediction," Under Review. [arXiv]