Feature: Stripe among multiple machines in EC2
aparrapo opened this issue · comments
The last stage of the 6DoF rendering pipeline is to stripe the binaries into a single binary file. However, it is currently implemented to only be done on a single computer. This means that, regardless of the size of your cluster, the final stage will take the same amount of time, since it is all performed on a single machine. The striping, however, can be parallelized in a mergesort-like fashion.
The current behavior and new feature
The striping is performed by ConvertToBinary (source/mesh_stream/ConvertToBinary.cpp), which produces a single binary file along with a digest JSON that describes the contents of the binary. This is effectively the same as how a .TAR file is created. To stripe across multiple machines, you will need to figure out how to properly merge these binaries together, which should be nothing more than concatenating them, and merge the JSON digest, which will involve updating the offsets of where different pieces are located in this final binary.
Expected use cases
End users will benefit primarily from a great reduction (likely upwards of 5-10%) in rendering time.