Download chunks concurrently when in background download mode
gorbak25 opened this issue · comments
What is the version of your Overlaybd
No response
What would you like to be added?
I would like overlaybd to download chunks concurrently when in background download mode, instead of waiting for one chunk to download before starting on the next.
Why is this needed for Overlaybd?
OverlayBD lazy pulling works amazingly if the latency between the backing OCI registry and overlaybd-tcmu is less than 2ms. Unfortunately, this isn't the case if the latency is high, for example 200-300ms when talking to quay.io. Downloading each 256Kb chunk is near instant, but it takes 200ms for a roundtrip to the registry, and containers boot slowly.
Increasing the refillSize to 10 Mb helps. Lazy pulling from quay.io works better, but still the link is nowhere near saturated, and boot times are high. Probably raising the chunk size to 100 Mb would help, but at such a high chunk size all benefits of lazy pulling are negated.
When background download is enabled the layer download speed of overlaybd should be indistinguishable from just pulling the layers ahead of time using a tool like skopeo or crane. Currently, that's not the case as downloading chunks is synchronous. Overlaybd waits for one chunk to finish downloading before starting on the next. I want to solve this by having a thread pool downloading chunks concurrently, similar to prefetch.
Are you willing to submit PRs to contribute to this feature?
- Yes, I am willing to implement it.
@gorbak25 glad to meet you! We (another overlybd user) also trying to saturate our networking by pre-pull all the blobs as fast as we can. I think it's doable from the client side (ctr rpull --download-blobs), current download is not fast, but we internally is experimenting replacing the containerd client.Pull with our own fast puller.
There are other benefits for repull from client side is it give better observability and we can behave differently in the pull failed due to networking or any other failure so that networking availability will not affect application runtime stability.
@shuaichang In my case I'm not using containerd or the overlaybd containerd snapshoter, I've integrated directly with overlaybd-tcmu which is used by the overlaybd containerd snapshotter and run qemu VM's using it. By looking at the code I think this is a fairly easy fix in overlaybd for background download to be as performant as pre downloaded blobs. Previously I've used skopeo to download all blobs ahead of time and recently implemented optional lazy pulling. I have a good setup to compare the difference after I make a PR.
The current implementation tries to avoid downloading too fast in background, in the hope that it would not influence foreground work.
Indeed the speed may be too slow for networks with high bandwidth-delay product. Welcome to make a PR to improve this case!