Fix number of executors for calculation CoreComputeTime
beriaanirudh opened this issue · comments
Currently, while calculating CoreComputeTime ( i.e. the "time-of-app" X "total-num-of-cores") , we include cores from all executors irrespective of how long an executor lived in the spark-app time-frame. This could give a blown-up number for CoreComputeTime available because executors could go down, and new executors could come up all the time.
On the other hand, finding the exact CoreComputeTime (i.e. summation across executors for ("time-of-executor" X "num-core-per-executor") ) is also not a good number, since it does not tell whether the machines had gone down along with executors or not (which is often 'not'). So the compute wasted on host-without-executors would not be accounted, if we only look at the executor up-time.
So a better number would be the "maximum-concurrent-executors-that-ran" X "time-of-app", which would give us a better estimate of the compute available.
Merged to master 2ddf779