Support for statistics of discrete numerical data

Question

Support for statistics of discrete numerical data

BeHalcyon opened this issue a year ago · comments

Halcyon commented a year ago

Hi. The architecture of UniqueTFDV benefits me greatly. I have the following needs:

Statistics of unique values of discrete numerical types, while the existing functions only support calculating unique values of string type data.
Calculate the frequency distribution of discrete numerical data.
Calculate the replication rate of several feature values in a set of data.

Where should I modify the codes?

Niraj Singh · Answer 1 · Fri Feb 17 2023 17:23:26 GMT+0800 (China Standard Time)

@BeHalcyon,

You can follow Custom Data Validation which guides on implementation of custom data validation.

Thank you!

Halcyon · Answer 2 · Fri Feb 17 2023 22:40:28 GMT+0800 (China Standard Time)

@BeHalcyon,

You can follow Custom Data Validation which guides on implementation of custom data validation.

Thank you!

Sorry, the question is about how to extend statistics, not validation. As I understand, custom data validation is used for extending the monitoring indicators.

caveness · Answer 3 · Fri Feb 17 2023 23:20:42 GMT+0800 (China Standard Time)

Hi - If you set is_categorical to true in the schema for a given numeric feature, and pass that schema when you generate statistics with TFDV, TFDV will calculate top-k and unique stats for that feature.

Niraj Singh · Answer 4 · Thu Mar 30 2023 14:38:59 GMT+0800 (China Standard Time)

Closing this due to inactivity. Please take a look into the answers provided above, feel free to reopen and post your comments(if you still have queries on this). Thank you!