Resque alike workerpool with redis(?) to queue up repo update jobs instead of lock

[Chao Li]

currently updating files in repo cannot scale with multiple users with high update traffic, we'd like to implement a workerpool and redis to back jobs exec, this should support us with the growth of use in next 6 months

[Jake Verbaten]

Have you tried allocating multiple temp folder and writing to them in parallel 🤣

[Chao Li]

as discussed today, yes :) I would explore this option

for tmp dir: the ability to serve update requests might be limited to disk IO, for redis there's no such limitation

• one single repository on disk
• user update payload queued in redis
• workerpool serialized per user requests with multiple file updates
• one worker write file at a time

multi tmp dir might have benefit of idempotency