support register shuffle task in map partition mode
RexXiong opened this issue · comments
Abstract
Make each partition location data can be grouped by (partitionId, attemptId) as partition result group is the necessary step to support map partition, because with each map attempt task a new set of results would be generated. Then we can support register map partition task. The image below will demonstrate this
Implements
- encode partitionId[Int] = (attemptId 8bit, rawPartitionId 24 bit)
- The epoch of each group will be managed by their own partition id as before and map/reduce partition use the same way.
- support register shuffle task in map partition mode(first time will offer and reserve all slots for attemptId=0 task)