expose histogram through prometheus.
emaxerrno opened this issue · comments
we already have an HDR histogram per core per server. Just expose it so we can push it up to prometheus. see
seastar::metrics::make_histogram( "smf_prc::<name>", "", [this]{ return prometheus::histogram(...)))
The hdr histogram is a bit inconvenient to work with for prometheus export. Prometheus expects histograms with a fixed set of buckets but hdr histogram works hard to hide things like empty buckets (which can be dynamic based on recorded values). Fortunately the C library isn't feature complete and it doesn't do dynamic reconfigurations which I think would make things more complicated.
I think I've about figured out a decent hack to get things working, but it would also be worthwhile keeping an eye out for other histogram approaches.
yeah, I've always wanted to have the simplest HDR Histogram only.
I know there are multi-threaded versions, etc. But I just care about the zig-zag encoding of the values and known memory use for dynamic ranges.
If the C version is too complex or gets too complex, we can port over the zig-zag encoding + simple print to our library anyway. I almost did this, but figured to use something existing.
So this sounds like a good plan to me.
Do you have a reference to a zig-zag histogram (which, unless I'm being unclear in my problem description, is not the same thing as zig-zag encoding you find in binary serialization).
so to me, you basically have pointer to a buffer 185KB. and you encode values w/ that particular encoding. with effective getters/setters/iterators.
That's about it. I think a minimal hdr_histogram can be ported - but I haven't looked deeply enough.
Seastar already has its own histogram abstraction which is nearly 1-1 with the Prometheus abstraction. Both of which expect a histogram structured in a way (as I described above) which is awkward with hdr, but not impossible.
Unless I'm missing some other path to publishing the histogram data to Prometheus, I'm honestly not sure how the binary encoding of the buffer is relevant to this topic.
We are talking about 2 different things i just realized. What do you mean by this:
I think I've about figured out a decent hack to get things working, but it would also be worthwhile keeping an eye out for other histogram approaches.
as in what is the hack