kaiko-ai / eva

Evaluation framework for oncology foundation models (FMs)

Home Page:https://kaiko-ai.github.io/eva/

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool




Oncology FM Evaluation Framework by kaiko.ai

PyPI docs license
paper

InstallationHow To UseQuick StartDocumentationDatasetsBenchmarks
ContributeAcknowledgements


eva is an evaluation framework for oncology foundation models (FMs) by kaiko.ai. Check out the documentation for more information.

Highlights:

  • Easy and reliable benchmark of Oncology FMs
  • Supports path-level classification, slide-level classification and semantic segmentation downstream tasks
  • Automatic embedding inference and evaluation of a downstream task
  • Native support of popular medical datasets and models
  • Produce statistics over multiple evaluation fits and multiple metrics

Installation

Simple installation from PyPI:

# to install the core version only
pip install kaiko-eva

# to install the expanded `vision` version
pip install 'kaiko-eva[vision]'

# to install everything
pip install 'kaiko-eva[all]'

To install the latest version of the main branch:

pip install "kaiko-eva[all] @ git+https://github.com/kaiko-ai/eva.git"

You can verify that the installation was successful by executing:

eva --version

How To Use

eva can be used directly from the terminal as a CLI tool as follows:

eva {fit,predict,predict_fit} --config url/or/path/to/the/config.yaml 

eva uses jsonargparse to make it easily configurable by automatically generating command line interfaces (CLIs), which allows to call any Python object from the command line. Moreover, the configuration structure is always in sync with the code. Thus, eva can be used either directly from Python or as a CLI tool (recommended).

For more information, please refer to the documentation.

Learn about Configs

The following interfaces are identical:

Python interface Configuration file
# main.py
# execute with: `python main.py`

from torch import nn

from eva import core
from eva.vision import datasets, transforms

# initialize trainer
trainer = core.Trainer(max_steps=100)

# initialize model
model = core.HeadModule(
  backbone=nn.Flatten(),
  head=nn.Linear(150528, 4),
  criterion=nn.CrossEntropyLoss(),
)

# initialize data
data = core.DataModule(
  datasets=core.DatasetsSchema(
    train=datasets.BACH(
      root="data/bach",
      split="train",
      download=True,
      transforms=transforms.ResizeAndCrop(),
    ),
  ),
  dataloaders=core.DataloadersSchema(
    train=core.DataLoader(batch_size=32),
  ),
)

# perform fit
pipeline = core.Interface()
pipeline.fit(trainer, model=model, data=data)
# main.yaml
# execute with: `eva fit --config main.yaml`

---
trainer:
  class_path: eva.Trainer
  init_args:
    max_steps: 100
model:
  class_path: eva.HeadModule
  init_args:
    backbone: torch.nn.Flatten
    head:
      class_path: torch.nn.Linear
      init_args:
        in_features: 150528
        out_features: 4
    criterion: torch.nn.CrossEntropyLoss
data:
  class_path: eva.DataModule
  init_args:
    datasets:
      train:
        class_path: eva.vision.datasets.BACH
        init_args:
          root: ./data/bach
          split: train
          download: true
          transforms: eva.vision.transforms.ResizeAndCrop
    dataloaders:
      train:
        batch_size: 32

The .yaml file defines the functionality of eva by parsing and translating its content to Python objects directly. Native supported configs can be found at the configs directory of the repo, which can be both locally stored or remote.

Quick Start

We define two types of evaluations: online and offline. While online fit uses the backbone (FM) to perform forward passes during the fitting process, offline fit first generates embeddings with the backbone and then fits the model using these embeddings as input, resulting in a faster evaluation.

Here are some examples to get you started:

  • Perform a downstream offline classification evaluation of DINO ViT-S/16 on the BACH dataset with linear probing by first inferring the embeddings and then performing 5 sequential fits:

    export DOWNLOAD_DATA=true
    eva predict_fit --config https://raw.githubusercontent.com/kaiko-ai/eva/main/configs/vision/dino_vit/offline/bach.yaml
  • Perform a downstream online segmentation evaluation of DINO ViT-S/16 on the MoNuSAC dataset with the ConvDecoderMS decoder:

    export DOWNLOAD_DATA=true
    eva fit --config https://raw.githubusercontent.com/kaiko-ai/eva/main/configs/vision/dino_vit/online/monusac.yaml

For more examples, take a look at the configs and tutorials.

Note

All the datasets that support automatic download in the repo have by default the option to automatically download set to false. For automatic download you have to manually set the environmental variable DOWNLOAD_DATA=true or in the configuration file download=true.

Leaderboards

In this section you will find model benchmarks which were generated with eva.

Table I: WSI and microscopy image tasks


Model BACH CRC MHIST PCam Camelyon16 PANDA CoNSeP MoNuSAC
ViT-S/16 (random) [1] 0.411 0.613 0.5 0.752 0.551 0.347 0.489 0.394
ViT-S/16 (ImageNet) [1] 0.675 0.936 0.827 0.861 0.751 0.676 0.54 0.512
DINO(p=16) [2] 0.77 0.936 0.751 0.905 0.869 0.737 0.625 0.549
Phikon [3] 0.715 0.942 0.766 0.925 0.879 0.784 0.68 0.554
UNI [4] 0.797 0.95 0.835 0.939 0.933 0.774 0.67 0.575
ViT-S/16 (kaiko.ai) [5] 0.8 0.949 0.831 0.902 0.897 0.77 0.622 0.573
ViT-S/8 (kaiko.ai) [5] 0.825 0.948 0.826 0.887 0.879 0.741 0.677 0.617
ViT-B/16 (kaiko.ai) [5] 0.846 0.959 0.839 0.906 0.891 0.753 0.647 0.572
ViT-B/8 (kaiko.ai) [5] 0.867 0.952 0.814 0.921 0.939 0.761 0.706 0.661
ViT-L/14 (kaiko.ai) [5] 0.862 0.935 0.822 0.907 0.941 0.769 0.686 0.599

Table I: Linear probing evaluation of FMs on patch-level downstream datasets.
We report balanced accuracy for classification tasks and generalized Dice score for semgetnation tasks, averaged over 5 runs. Results are reported on the "test" split if available and otherwise on the "validation" split.


References:

  1. "Emerging properties in self-supervised vision transformers”, arXiv
  2. "Benchmarking self-supervised learning on diverse pathology datasets”, arXiv
  3. "Scaling self-supervised learning for histopathology with masked image modeling”, medRxiv
  4. "A General-Purpose Self-Supervised Model for Computational Pathology”, arXiv
  5. "Towards Training Large-Scale Pathology Foundation Models: from TCGA to Hospital Scale”, arXiv

Contributing

eva is an open source project and welcomes contributions of all kinds. Please checkout the developer and contributing guide for help on how to do so.

All contributors must follow the code of conduct.

Acknowledgements

Our codebase is built using multiple opensource contributions

python pytorch lightning
black isort Ruff Checked with pyright
pdm-managed Nox Built with Material for MkDocs

Citation

If you find this repository useful, please consider giving a star ⭐ and adding the following citation:

@inproceedings{kaiko.ai2024eva,
    title={eva: Evaluation framework for pathology foundation models},
    author={kaiko.ai and Ioannis Gatopoulos and Nicolas K{\"a}nzig and Roman Moser and Sebastian Ot{\'a}lora},
    booktitle={Medical Imaging with Deep Learning},
    year={2024},
    url={https://openreview.net/forum?id=FNBQOPj18N}
}

About

Evaluation framework for oncology foundation models (FMs)

https://kaiko-ai.github.io/eva/

License:Apache License 2.0


Languages

Language:Python 100.0%Language:Starlark 0.0%