[Question] If set PSI's initial value to 0 or its ssim value will be better than frame's time?
MikeLien opened this issue · comments
Hi,
Recently I work on some performance measurement tasks and leverage your code for SI and PSI calculation on Hasal project.
But we got a little confused why the PSI Initial Value is first frame's time instead of 0 or its ssim value as SI do.
If there has any reason to use frame's time as its initial value?
Pinging @pahammad who handled the SSIM version of PSI.
Visual completion % is what both SI and PSI are integrating. SI uses histogram difference to track the % progress of visual completion, while PSI uses SSIM difference to track visual completion % progress. In practice, to handle some corner cases, we assume the visual completion % to be zero before the first visual change occurs. This is true for both SI and PSI. If the visual completion % is zero, simply following the integral equation (for both SI and PSI), the integral upto first visual change is simply the time elapsed from beginning of the video until the first visual change (measured in milliseconds). The reason why the code for PSI looks that way is to align with the non-uniform sampling scheme used in VisualMetrics repository. It's not a bug, and the math works out if you write down the calculations frame by frame, while keeping in mind the expectation of zero visual completion before first visual change occurs.
There's a somewhat related thread that goes into an alternative way of writing SI / PSI integrals by breaking them into two parts; it may give you better clarity as well. See: paulirish/speedline#28
Thanks for the explanation and now I know the reason.
But I still have some questions about this solution.
- How do you define the start point of data, is it the timestamp of test script start?
- How do I indicate a PSI value is better or worse if I measure one website two times and get different time for firstPaint?
The code for SI and PSI in VisualMetrics repository work with a video as input. The video is usually from a WPT test run. The start point of the data is the first frame of the video, as defined by WPT's testing & video capture mechanism. If you measure a website twice, you will have two video files; just pass these videos one after another into the PSI code and you will get the corresponding PSI values. Same method works for SI. It is very common for all web performance metrics to vary from one test run to another even when you are testing the same exact website - the variation can even be non-Gaussian. This variation is mostly due to channel variations between the origin location and the client (testing) device. Hope this helps.
That's really helpful to me, thanks for the introduction about calculation flow. Eventually I found our framework doesn't adjust time stamp and align the offset thus result in a greater variation of firstPaint.
Accordingly, I think this issue of question can be closed. Sorry to post issue here because I don't have authority to tag question label.