Support parallel fetch and packaging
ikeydoherty opened this issue · comments
In the new ypkg2 we have some levels of parallelism already to drastically speed up packaging, such as the stripping, construction of dbginfo, and identification of shared libs (in progress) done in a multiprocessing pool.
To be more awesome, we should support:
Parallel fetch
If enabled, fetch multiple sources in parallel. Probably means using curl directly here with a pool. Should also do the verification in parallel. Don't do parallel extracts, this will cause headaches and drive up memory requirements for XZ packages.
- Not yet implemented
Parallel package
Not enabled by default, but if enabled would produce the resulting .eopkg's in parallel, i.e. several subpackages and/or main being compressed at the same time.
Note that although it would be nice to build emul32 and native at the same time, it's not really feasible without a clever merge strategy, and in reality we do need 32-bit to install first in order to
allow us to override those files with the native variant. Also, emul32 isn't too common for it to be an issue in our repos.
- Not yet implemented
Don't care about this now