Multiprocess registry metrics cause "Error on ingesting samples with different value but same timestamp" after Prometheus upgrade >2.52.0

[ColinBe95]

Hi everyone,
with the upgrade to Prometheus>2.52.0 an error log has been introduced for duplicated samples, i.e.:

msg="Error on ingesting samples with different value but same timestamp" num_dropped=6

A similar issue was raised here, however this issue is not related to the python client.

The metrics in question were all defined like this:

from prometheus_client import CollectorRegistry, Counter, Summary
from prometheus_client.multiprocess import MultiProcessCollector

multiprocess_registry = CollectorRegistry()
multi_collector = MultiProcessCollector(multiprocess_registry)

CNT_MYMETRIC = Counter(
    "cnt_mymetric",
    "Counts increment of my metric",
    registry=multiprocess_registry,
)

and simply used like:

CNT_MYMETRIC.inc()

Now the problem is that we started seeing this error log. However, as I understand Multiprocess metrics, this is expected behaviour. See e.g. docs on Multiprocess Mode:

"Registering metrics to a registry later used by a MultiProcessCollector may cause duplicate metrics to be exported"

And I am not sure about what the best way to approach this issue is. One solution I could think of is adding an extra label for the process id, e.g.:

CNT_MYMETRIC = Counter(
    "cnt_mymetric",
    "Counts increment of my metric",
    registry=multiprocess_registry,
    labelnames=["PID"]
)

and increase the value by:

import os
CNT_MYMETRIC.labels(os.getpid()).inc()

But then that would not be a Multiprocess Metric anymore. So I am wondering now if there is a way to use multiprocess metrics with the latest Prometheus version, without getting a ton of these error logs or surpressing them.

Thank you very much!

[Chris Marchbanks]

You should be able to just delete registry=multiprocess_registry, and it will start working, let me know if it doesn't!

When you use multi process mode all of metrics stop using registries in the same way. I do wonder if we should just ignore them or something like that. There could be some very advanced cases where it is useful to still use registries, but it seems more confusing than worthwhile.

[ColinBe95]

Hey @csmarchbanks,
thank you so much for the reply. I now have simply:

from prometheus_client import CollectorRegistry, Counter, Summary
from prometheus_client.multiprocess import MultiProcessCollector

multiprocess_registry = CollectorRegistry()
multi_collector = MultiProcessCollector(multiprocess_registry)

CNT_MYMETRIC = Counter(
    "cnt_mymetric",
    "Counts increment of my metric",
)

i.e. the same code, just without assigning the multiprocess_registry directly to the Counter and it works.

So if I unerstand your comment correctly, it is enough to set up

multiprocess_registry = CollectorRegistry()
multi_collector = MultiProcessCollector(multiprocess_registry)

and all metrics will automatically use Multiprocess Mode? Assigning the registry=multiprocess_registry, will duplicate this metric?

I was indeed a little confused here, but I am happy to see it working now :)

[Chris Marchbanks]

Correct. I agree it is confusing. I try to call it out in https://prometheus.github.io/client_python/multiprocess/, but I think some warning from the code would also be nice to have.

[ColinBe95]

I agree that a logged warning could have made sense. Thank you very much! I think we can close this issue :)