Giters

deepchecks / deepchecks

Deepchecks: Tests for Continuous Validation of ML Models & Data. Deepchecks is a holistic open-source solution for all of your AI & ML validation needs, enabling to thoroughly test your data and models from research to production.

Home Page:https://docs.deepchecks.com/stable

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

[BUG] Outdated Examples - COCOData does not exist

LifeBoey opened this issue · comments

commented

Describe the bug
The example in https://docs.deepchecks.com/0.8/checks_gallery/vision/train_test_validation/plot_image_dataset_drift.html does not work due to the COCOData class not existing in 0.17.x .

To Reproduce
Steps to reproduce the behavior:

  1. Follow the code written in https://docs.deepchecks.com/0.8/checks_gallery/vision/train_test_validation/plot_image_dataset_drift.html
  2. Most of the code can be adapted, for example importing coco_torch instead of coco from deepchecks.vision.datasets.detection.
  3. Make the class DriftedCOCO(COCOData) and be unable to, due to it not existing.

Expected behavior
The code should have run with no issue, or at least for there to be a suitable replacement within the repository.
However, neither VisionData or CocoDataset works as a substitute.

Screenshots
Substituting VisionData
image

Substituting CocoDataset
image

Environment (please complete the following information):

  • OS: Windows 10
  • Python Version: 3.8.0
  • Deepchecks Version: 0.17.3

Additional Information:
Even if there is no perfect substitute, I'd still like to know if there is a way to convert the torch DataLoader class into Deepchecks' VisionData class that seems to be used as the dataset input for its drift detection checks. I.e. from

train_dataloader = load_dataset(train=True, object_type='DataLoader')

to the object that was passed in for the initial check:

train_ds = load_dataset(train=True, object_type='VisionData')

In general, not just for the COCO Dataset.

Much appreciated!

commented

Hi @LifeBoey, indeed we did a major refactor in the vision package, so the old code shouldn't work with the current stable version. In the new version, deepchecks VisionData objects are not defined (sublassed) for each use case. Rather, you initialize them using some data iterator - it can be a torch DataLoader, a tensorflow Dataset or any custom iterator that adheres to the specified format. You can read more about creating your VisionData object here - https://docs.deepchecks.com/stable/vision/usage_guides/visiondata_object.html. You can read our quickstart for Object Detection tasks here - https://docs.deepchecks.com/stable/vision/auto_tutorials/quickstarts/plot_detection_tutorial.html

Specifically for running DatasetDrift detection, you can read the new guide here - https://docs.deepchecks.com/stable/vision/auto_checks/train_test_validation/plot_image_dataset_drift.html