Enable Created Timestamp Zero Value Ingestion
jjcollinge opened this issue · comments
Is your feature request related to a problem? Please describe.
By default Prometheus does not support initializing 0 value counters and will start from the first reported value for counters. Due to the aggregation and batching this can make it impossible to graph growth of a metric correctly from 0. Instead the first value may be reported as the maximum in the initial batch. Therefore, upstream Prometheus has added the feature flag --enable-feature=created-timestamp-zero-ingestion to allow the desired behavior (see: https://github.com/prometheus/prometheus/blob/main/docs/feature_flags.md#created-timestamps-zero-injection).
Describe the solution you'd like
It'd be very useful if we can leverage this feature in cortex by having an optional configuration property that allows us to ingest metrics in the same way.
Describe alternatives you've considered
- Explicitly setting every metric to 0 in the application, however, this becomes impossible for unbounded label sets.
- Intercept and set every metric value to 0 in the collector until after the first export has been completed but this is flakey.
Hey, thanks for opening the issue. I understand there is no change required on Cortex side.
It is a feature in Prometheus scraper, which means as long as you enable it at your client side application and Prometheus, Prometheus should be able to remote write the data to Cortex.
Let's close this issue since there is nothing to do on Cortex side.
Oh, good spot. Sorry just assumed this was in the ingester. Let me try that and I'll update this thread.