tensorflow / model-analysis

Describe the problem

It seems like the config does not rely on the __init__ method; and relies on the get_config and from_config method of the (custom) metrics class.

This distinction is important as TFX's Evaluator serialize and deserialize the metric and __init__ is not used during the final metrics evaluation. Not sure if the issue should be raised with TFX or TFMA.

model-analysis/tensorflow_model_analysis/proto/config.proto

Lines 407 to 414 in 4ea7d16

    
           // Optional JSON encoded config settings associated with the class. 
        
           // 
        
           // The config settings will be passed as **kwarg values to the __init__ method 
        
           // for the class. For ease of use the leading and trailing '{' and '}' 
        
           // brackets may be omitted. 
        
           // 
        
           // Example: '"name": "my_metric", "thresholds": [0.5]' 
        
           string config = 3;

Link to TFX Issue: tensorflow/tfx#4022

	// Optional JSON encoded config settings associated with the class.
	//
	// The config settings will be passed as **kwarg values to the __init__ method
	// for the class. For ease of use the leading and trailing '{' and '}'
	// brackets may be omitted.
	//
	// Example: '"name": "my_metric", "thresholds": [0.5]'
	string config = 3;

Incorrect Documentation for MetricConfig

Describe the problem