Potential Issues in QoE Calculation

[Jun Fu]

As described in the competition website, the QoE of video i and the QoE is the sum of all watched chunks {j}:

However, on line 115 in the file run.py, the calculation of QoE is implemented as:

QoE += alpha * VIDEO_BIT_RATE[bit_rate] / 1000. - beta * rebuf / 1000. - gamma * abs(smooth) / 1000.

The above formulation faces the following risks.

First, for the first term, i.e., the quality of the chunk, it is inaccurately calculated in the following situations:

• When the sleep time is bigger than 0, the first term should be zero. However, as shown in line 115, the first item is not zero since bit_rate is set to 0 or other values.
• When we download an unwatched chunk, we should not add its quality score into the QoE of video i.

Second, for the third term, i.e., the smooth penalty, it is also a bit computationally inaccurate, as shown in the line from 100 to 103 in the file controller.py:

 bitrate_cnt = min(math.ceil(self.players[0].get_play_chunk()), self.players[0].get_chunk_sum())
 for i in range(1, bitrate_cnt):
        video_qualities.append(self.players[0].get_video_quality(i-1))
        smooth += abs(VIDEO_BIT_RATE[self.players[0].get_video_quality(i)] - VIDEO_BIT_RATE[self.players[0].get_video_quality(i-1)])

since there is the following formulation:

self.players[0].get_chunk_sum()>= self.players[0].get_play_chunk() >= self.user_models[0].get_ret_duration()

, I think the following way to calculate the bitrate_cnt will be more reasonable:

bitrate_cnt = min(math.ceil(self.user_models[0].get_ret_duration()/1000.0), self.players[0].get_chunk_sum())

Please correct me if my understanding is wrong. Look forward to your response :-).

[liy1shu]

Thanks very much for your issue!

About the first issue, we've checked the code and corrected the logic. You can check it in run.py.
About the second, we've designed the video's play timeline (divided by 1000.0 to calculate play chunk ) to stop at the retention duration. So our formula should be equal to yours. But before there is some issues in the video play simulator which may cause play timeline to exceed the actual watch duration. We've fixed it now.

Thanks again for your issue, it's really a great help ! If there are any other problems or if our new version still haven't solved your issue, please feel free to contact us again!

[Jun Fu]

Thanks a lot!