Oncology FM Evaluation Framework by kaiko.ai
Installation •
How To Use •
Quick Start •
Documentation •
Datasets •
Benchmarks
Contribute •
Acknowledgements
eva
is an evaluation framework for oncology foundation models (FMs) by kaiko.ai.
Check out the documentation for more information.
- Easy and reliable benchmark of Oncology FMs
- Supports path-level classification, slide-level classification and semantic segmentation downstream tasks
- Automatic embedding inference and evaluation of a downstream task
- Native support of popular medical datasets and models
- Produce statistics over multiple evaluation fits and multiple metrics
Simple installation from PyPI:
# to install the core version only
pip install kaiko-eva
# to install the expanded `vision` version
pip install 'kaiko-eva[vision]'
# to install everything
pip install 'kaiko-eva[all]'
To install the latest version of the main
branch:
pip install "kaiko-eva[all] @ git+https://github.com/kaiko-ai/eva.git"
You can verify that the installation was successful by executing:
eva --version
eva
can be used directly from the terminal as a CLI tool as follows:
eva {fit,predict,predict_fit} --config url/or/path/to/the/config.yaml
eva
uses jsonargparse to
make it easily configurable by automatically generating command line interfaces (CLIs),
which allows to call any Python object from the command line. Moreover, the configuration structure is always in sync with the code. Thus, eva
can be used either directly from Python or as a CLI tool (recommended).
For more information, please refer to the documentation.
Learn about Configs
The following interfaces are identical:
Python interface | Configuration file |
---|---|
# main.py
# execute with: `python main.py`
from torch import nn
from eva import core
from eva.vision import datasets, transforms
# initialize trainer
trainer = core.Trainer(max_steps=100)
# initialize model
model = core.HeadModule(
backbone=nn.Flatten(),
head=nn.Linear(150528, 4),
criterion=nn.CrossEntropyLoss(),
)
# initialize data
data = core.DataModule(
datasets=core.DatasetsSchema(
train=datasets.BACH(
root="data/bach",
split="train",
download=True,
transforms=transforms.ResizeAndCrop(),
),
),
dataloaders=core.DataloadersSchema(
train=core.DataLoader(batch_size=32),
),
)
# perform fit
pipeline = core.Interface()
pipeline.fit(trainer, model=model, data=data) |
# main.yaml
# execute with: `eva fit --config main.yaml`
---
trainer:
class_path: eva.Trainer
init_args:
max_steps: 100
model:
class_path: eva.HeadModule
init_args:
backbone: torch.nn.Flatten
head:
class_path: torch.nn.Linear
init_args:
in_features: 150528
out_features: 4
criterion: torch.nn.CrossEntropyLoss
data:
class_path: eva.DataModule
init_args:
datasets:
train:
class_path: eva.vision.datasets.BACH
init_args:
root: ./data/bach
split: train
download: true
transforms: eva.vision.transforms.ResizeAndCrop
dataloaders:
train:
batch_size: 32 |
The .yaml
file defines the functionality of eva
by parsing and translating its content to Python objects directly.
Native supported configs can be found at the
configs directory
of the repo, which can be both locally stored or remote.
We define two types of evaluations: online and offline. While online fit uses the backbone (FM) to perform forward passes during the fitting process, offline fit first generates embeddings with the backbone and then fits the model using these embeddings as input, resulting in a faster evaluation.
Here are some examples to get you started:
-
Perform a downstream offline classification evaluation of
DINO ViT-S/16
on theBACH
dataset with linear probing by first inferring the embeddings and then performing 5 sequential fits:export DOWNLOAD_DATA=true eva predict_fit --config https://raw.githubusercontent.com/kaiko-ai/eva/main/configs/vision/dino_vit/offline/bach.yaml
-
Perform a downstream online segmentation evaluation of
DINO ViT-S/16
on theMoNuSAC
dataset with theConvDecoderMS
decoder:export DOWNLOAD_DATA=true eva fit --config https://raw.githubusercontent.com/kaiko-ai/eva/main/configs/vision/dino_vit/online/monusac.yaml
For more examples, take a look at the configs and tutorials.
Note
All the datasets that support automatic download in the repo have by default the option to automatically download set to false.
For automatic download you have to manually set the environmental variable DOWNLOAD_DATA=true
or in the configuration file download=true
.
In this section you will find model benchmarks which were generated with eva
.
Model | BACH | CRC | MHIST | PCam | Camelyon16 | PANDA | CoNSeP | MoNuSAC |
---|---|---|---|---|---|---|---|---|
ViT-S/16 (random) [1] | 0.411 | 0.613 | 0.5 | 0.752 | 0.551 | 0.347 | 0.489 | 0.394 |
ViT-S/16 (ImageNet) [1] | 0.675 | 0.936 | 0.827 | 0.861 | 0.751 | 0.676 | 0.54 | 0.512 |
DINO(p=16) [2] | 0.77 | 0.936 | 0.751 | 0.905 | 0.869 | 0.737 | 0.625 | 0.549 |
Phikon [3] | 0.715 | 0.942 | 0.766 | 0.925 | 0.879 | 0.784 | 0.68 | 0.554 |
UNI [4] | 0.797 | 0.95 | 0.835 | 0.939 | 0.933 | 0.774 | 0.67 | 0.575 |
ViT-S/16 (kaiko.ai) [5] | 0.8 | 0.949 | 0.831 | 0.902 | 0.897 | 0.77 | 0.622 | 0.573 |
ViT-S/8 (kaiko.ai) [5] | 0.825 | 0.948 | 0.826 | 0.887 | 0.879 | 0.741 | 0.677 | 0.617 |
ViT-B/16 (kaiko.ai) [5] | 0.846 | 0.959 | 0.839 | 0.906 | 0.891 | 0.753 | 0.647 | 0.572 |
ViT-B/8 (kaiko.ai) [5] | 0.867 | 0.952 | 0.814 | 0.921 | 0.939 | 0.761 | 0.706 | 0.661 |
ViT-L/14 (kaiko.ai) [5] | 0.862 | 0.935 | 0.822 | 0.907 | 0.941 | 0.769 | 0.686 | 0.599 |
Table I: Linear probing evaluation of FMs on patch-level downstream datasets.
We report balanced accuracy
for classification tasks and generalized Dice score for semgetnation tasks, averaged over 5 runs. Results are
reported on the "test" split if available and otherwise on the "validation" split.
References:
- "Emerging properties in self-supervised vision transformers”, arXiv
- "Benchmarking self-supervised learning on diverse pathology datasets”, arXiv
- "Scaling self-supervised learning for histopathology with masked image modeling”, medRxiv
- "A General-Purpose Self-Supervised Model for Computational Pathology”, arXiv
- "Towards Training Large-Scale Pathology Foundation Models: from TCGA to Hospital Scale”, arXiv
eva
is an open source project and welcomes contributions of all kinds. Please checkout the developer
and contributing guide for help on how to do so.
All contributors must follow the code of conduct.
Our codebase is built using multiple opensource contributions
If you find this repository useful, please consider giving a star ⭐ and adding the following citation:
@inproceedings{kaiko.ai2024eva,
title={eva: Evaluation framework for pathology foundation models},
author={kaiko.ai and Ioannis Gatopoulos and Nicolas K{\"a}nzig and Roman Moser and Sebastian Ot{\'a}lora},
booktitle={Medical Imaging with Deep Learning},
year={2024},
url={https://openreview.net/forum?id=FNBQOPj18N}
}