Allow selecting which go metrics you care about to emit
mahmoudm opened this issue · comments
Currently, you can either have or not have the GoMetrics wholesale. This puts daemons in a tough spot where the cost of each additional metric is pretty high, but some of the GoMetrics are more useful than others. Can we allow selection of which GoMetrics we want vs not want?
I would propose implementing this as a configurable metric blacklist (or whitelist?).
This would allow, for example, a service to blacklist server.request.size as a metric, in which case that metric would never be emitted. The functionality would apply for any metric name, whether it is a "built-in" Go metric or a custom user metric.
I believe the main thing to decide on is whether this should be a whitelist or a blacklist. I propose that it should be a blacklist -- by default, we should be emitting everything, and it's up to the service to opt out of fields they don't want. We could make it such that the defaults use a specific blacklist (or create defined blacklist presets).
My current plan is also to make it such that this is only configurable in code -- that is, any metric blacklisting should be performed by the product team (and not exposed as configuration).
would it only be on the metric level or can you blacklist the break-out of the metric as well? i.e p99 vs p95
Just commented on #127 about this.
If we decide to take the per-metric approach, would probably make sense to have an API that allows either blacklisting the entire metric or blacklisting specific values for that metric (such as p95).
I've opened PRs that implement the approaches proposed in these issues so that there's a more concrete implementation that can be examined. Let me know what you think of the approaches.
I think the main alternative would be to have a single blacklist that allows blacklisting a metric wholesale, or specific values of a metric. This would be 1 data structure, but would not allow the configuration to specify the equivalent of "exclude all p95 for a given metric type" (the exclusion would need to be done per-metric)