Percent delta from first should be the default
the8472 opened this issue · comments
Currently both the "graphs" page and the "30 day history" embedded in individual benchmarks display the "Raw" metrics.
Most of the time changes are so small that those views are ~useless without lots of zooming. And even then one has to zoom in on one of several lines.
Defaulting to "Delta from First" would make those views more useful, especially the embedded views since those don't have the dropdown.
Regarding the embedded graphs:
The idea was to show both a high-level picture of how did the perf. change (left side), and also changes relative to the previous commit (to find out noise spikes).
If I understand this issue correctly, you would like to change the left graph to be "percent from first", to make it easier to gauge the change from the start of the observed period?
For the graphs displayed above, it would look like this:
The right side is essentially a derivative of the raw graph, and the percent from first is essentially a zoomed in version of the raw graph. So this change makes sense to me.
If I understand this issue correctly, you would like to change the left graph to be "percent from first", to make it easier to gauge the change from the start of the observed period?
It's more about the zoom level. The absolute graphs start at a zero base which means the graph covers a huge value range which makes most changes tiny in comparison.
Better default zoom/axis offsets would help too, but since we already have percent-from-first in other places I think just reusing that would be simpler.
#1878 changed it for the benchmark detail here. I would keep the absolute value for the main graph page though.
Why though? Personally I never found those particularly useful. They're essentially flat.
Unless you're comparing full vs. incremental times or something like that I suppose.
A few reasons:
- It allows us to see the absolute values of e.g. instruction counts, which I find useful.
- This page is probably more often viewed by non-experts (w.r.t. the benchmark detail), so I would show the absolute numbers, since it makes the graphs easier to interpret, in my opinion.
- The fact that the profiles are mostly flat is fine to me, if they are mostly flat, it means that nothing much has changed in the past N days. People tend to overlook axis labels in my experience, and if we showed this:
I think that it might look too much is going on (when it reality, it can be just 0.1% etc.)
That being said, now that I went through the whole page with the percent from first mode, it doesn't look that bad/unintuitive.
I still think absolute is better to see how much is incr faster than non-incr.
Hrm, ok. The embedded case is more important anyway since that one doesn't have the dropdown.
Also with this:
I think that it would be far too easy for someone to miss the memo that it is a relative chart, and interpret this as incr-unchanged being slower than full.