deepchecks / deepchecks

Deepchecks: Tests for Continuous Validation of ML Models & Data. Deepchecks is a holistic open-source solution for all of your AI & ML validation needs, enabling to thoroughly test your data and models from research to production.

Home Page:https://docs.deepchecks.com/stable

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

[BUG] Some checks from TrainTestValidationSuite don't show up in wandb

michaelwang1994-olo opened this issue · comments

Describe the bug
I ran a TrainTestValidationSuite on my data and when I do `.to_wandb(project='...'), I am missing the following checks in my wandb run charts:

  1. NewCategoryTrainTest
  2. StringMismatchComparison
  3. DatasetsSizeComparison --> This shows up under "Train Test Validation Suite/None"

See screenshot for all the Train Test Validation Suite checks that do show up in the run.

To Reproduce
Steps to reproduce the behavior:

train_ds = Dataset(
        df_train,
        label=label,
        index_name=index_name,
        datetime_name=datetime_name,
        cat_features=cat_features
    )
test_ds = Dataset(
    df_test,
    label=label,
    index_name=index_name,
    datetime_name=datetime_name,
    cat_features=cat_features
)
validation_suite = train_test_validation(
    timeout=timeout
)
validation_results = validation_suite.run(train_ds, test_ds)
validation_results.to_wandb(project=wandb.run.project)

Expected behavior
I would have expected all the checks from the suite to show up and be correctly named.

Screenshots
image

Environment (please complete the following information):

  • OS: MacOS Ventura 13.4
  • Python Version: 3.11.3
  • Deepchecks Version: 0.17.3

Additional context

Thanks for reporting @michaelwang1994-olo!