Confusion matrics and calibration plots don't visualize in kubeflow pipeline

[Jake Schmidt]

System information

• Have I written custom code (as opposed to using a stock example script
  provided in TensorFlow Model Analysis): Yes
• OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Ubuntu
• TensorFlow Model Analysis installed from (source or binary): binary (PyPI)
• TensorFlow Model Analysis version (use command below): 0.22.2
• Python version: 3.6.9
• Jupyter Notebook version: 6.0.3
• Exact command to reproduce:

from tensorflow_model_analysis import EvalConfig
from tensorflow_model_analysis.metrics import default_multi_class_classification_specs
from google.protobuf.json_format import ParseDict

class = ['class_1', 'class_2', ...]

eval_config = {
    'model_specs': [
        {
            'name': 'rig_state',
            'model_type': 'tf_keras',
            'signature_name': 'serve_raw',
            'label_key': ...,
            'example_weight_key': 'sample_weight'
        }
    ],
    'metrics_specs': [
        {
            'metrics': [
                {
                    'class_name': 'MultiClassConfusionMatrixPlot',
                    'config': '"thresholds": [0.5]'
                },
                {'class_name': 'ExampleCount'},
                {'class_name': 'WeightedExampleCount'},
                {'class_name': 'SparseCategoricalAccuracy'},
            ],
        },
        {
            'binarize': {'class_ids': {'values': list(range(len(classes)))}},
            'metrics': [
                {'class_name': 'AUC'},
                {'class_name': 'CalibrationPlot'},
                {'class_name': 'BinaryAccuracy'},
                {'class_name': 'MeanPrediction'}
            ]
        }
    ],
    'slicing_specs': [...]
}
eval_config: EvalConfig = ParseDict(eval_config, EvalConfig())

Describe the problem

When using this EvalConfig in tfma.run_model_analysis everything runs successfully. Inspecting the result, there is plot data for the confusion matrix, yet when trying to plot the result using tfma.view.render_slicing_metrics, the confusion matrix won't show.

This happens in a kubeflow pipeline.

[Mike Dreves]

Can you provide the exact call you are making. The plots are visualized using tfma.view.render_plot.

[Jake Schmidt]

"The plots are visualized using tfma.view.render_plot."

Ahh, I didn't know this. It seems to work now. However, the multiclass confusion matrix is virtually useless when it just displays integer class IDs (#78). Is there a way to add a list of class name strings in my EvalConfig or something?

[Mike Dreves]

Added comment in #78. The short answer is class labels are not yet supported. We are trying to avoid having to have this as part of the config and make it automatic.