Sequential mode is not sequential enough
alervd opened this issue · comments
Please provide the following information
libtorrent version (or branch): 1.2.x and 2.x
platform/architecture: Windows
compiler and compiler version: It does not matter.
please describe what symptom you see, what you would expect to see instead and
how to reproduce it.
- Find a FAT32/ExFAT drive.
- Try to download a reasonably big file to it. For example, 16GB.
- Windows system becomes unresponsive in some of its parts. It can't browse this disk in Windows Expolorer, it can't run Notepad, Paint. And the program which uses Libtorrent can hang too (e.g. when trying to open containing folder from qBittorrent menu).
Why is this happening? These file systems does not support sparse files. So when Libtorrent writes in some huge offset, the disk "hangs" writing zeroes to reach this offset.
What is a possible workaround? One can try to use sequential mode. To force Libtorrent to NOT write at big offsets (comparing to the current file size).
I've tried it on a single-file torrent of 16GB in size - it does not work. In both qBittorrent and my app Libtorrent downloaded 0.18% of 16 GB file in sequential mode, but for some unknown reason it written some bytes at 11GB offset. Libtorrent 2.x works even worse - it writes at 16GB position instead of 11GB. In sequential mode at the beginning of the download process.
What I want instead - it must not try to write in far offsets in sequential mode. It must download and write sequentially always. And not try to download and write far-away pieces.
The library does not have a mechanism for writing files in a strict sequence.
Since all modern file systems support sparse files
Details #4616
- search for similar questions before writing
- Close this issue
@master255 , #4616 has nothing in common with this issue. I've read the whole that topic and it was just a waste of time.
And please do not offer any technical advises - you just have no competence.
Okay :)
when libtorrent detects a slow peer. say it has timed out or generally is slow at responding to requests, it's "snubbed".
a snubbed peer:
- will only have a single request in flight (i.e. no pipelining) to avoid locking blocks/pieces that could otherwise be requested by someone else.
- will never be used to request time critical piece from
- does not use "slow-start" to ramp up the number of outstanding block requests
- inverses the piece picking logic (i.e. picks the most common pieces, rather than the rarest). The motivation for this is similar to (1), to avoid a bad peer stopping good peers from downloading the most important pieces.
I believe this last part is what's causing this issue. If you have a single snubbed peer and you're downloading in sequential mode, that snubbed peer will be downloading sequentially in reverse order.
I believe people generally use sequential mode as a poor-man's streaming, which benefits from (4). You lower the risk of having holes in the sequentially downloaded part.
To address this, we could introduce a new setting as "sequential, I really mean it" or just tweak the existing sequential to also apply to snubbed peers. I think tweaking would break the streaming use case.
Back in the day when set-top boxes wanted to run uTorrent on (very slow) flash drives formatted with FAT32, we had a more sophisticated approach to supporting filesystems without sparse files:
- For each file, set piece priority to
dont-downloadfor any piece more than 32 MiB away from the beginning of the file. - Track the highest piece written to each file.
- Continuously update the piece priorities to enable downloading of pieces within 32 MiB of the highest piece written to disk.
@arvidn Thanks for the reply. No, I don't think it's caused by a snubbed peer. That torrent I've tested has ~150 seeding peers. And actually (according to my app report), the most of data was written at the beginning of the file, as it should to. But, some bytes (I don't know their size) were written at 11GB offset due to an unknown reason. It's not a big deal when a disk supports sparse mode, because actually it appears to download sequentially (few bytes at the end does not matter). But it becomes a huge issue when we work with a disk with no sparse support.
* Track the highest piece _written_ to each file.
Is it not enough to track downloaded pieces? It seems it would have no difference which pieces we track - downloaded or written ones.
So, is any fix planned? Is it planned for 1.2.x versions? Any ETA of 1.2.x release with the fix?
I would like to know this to understand if I have to implement that sophisticated approach on my own. :)
. No, I don't think it's caused by a snubbed peer.
What are you observing that rules out a snubbed peer?
That torrent I've tested has ~150 seeding peers. And actually (according to my app report), the most of data was written at the beginning of the file, as it should to. But, some bytes (I don't know their size) were written at 11GB offset due to an unknown reason.
All of that seems consistent with a snubbed peer as far as I can tell.
on the other hand, it looks like even snubbed peers request sequentially:
https://github.com/arvidn/libtorrent/blob/RC_2_0/src/peer_connection.cpp#L908
if you connect to a peer that's super-seeding, or for some other reason only have pieces towards the end of the file, you would end up requesting those, even when in sequential download mode.
@arvidn OK, lets forget about snubbed peers :) Maybe I misunderstood something.
Still, any ETA? Yes, I would like something like strict sequential mode in addition to sequential mode.
Do you set sequential mode before adding the torrent?
When a torrent is not in sequential mode, there's an "intial" piece picker mode, where random pieces are picked (not rarest first). If you allow for any pieces to be picked before you set sequential mode, that might also explain why pieces towards the end of the file are picked.
Still, any ETA?
As far as I can tell, there's no problem to be fixed.
There are a number of reasons why late pieces would be picked. I'm sure there are a few more cases than the ones I've outlined above. The most reliable way to avoid it is to use piece priorities
Well, I add a torrent in a paused mode, then set it up (which includes setting sequential mode) and then call torrent_handle::resume on it. Is it not OK?
yes
OK, I'll try enabling it in add_torrent_params.
I've got the same issues (even after enabling lt::torrent_flags::sequential_download in lt::add_torrent_params).
So, if it's not an issue that LT writes to the end of single-torrent file even when sequential mode enabled, then would you consider a possibility to add this strict sequential mode feature you've told about earlier? Of course, we can do it manually, but maybe it's a useful feature which LT can even switch on automatically when it detects drive with no sparse files support.
The latest versions of uTorrent does not have this issue...
And, still, this is so strange... Today... Different peers for sure on that torrent. Still, the same 11GB of 16GB position LT tries to write on. Why 11 GB - a mystery.
The torrent I'm trying to download (just for the testing purposes): magnet:?xt=urn:btih:E2777688312631AC663E6B8CD839E53C36177DFC&tr=http%3A%2F%2Fbt2.t-ru.org%2Fann%3Fmagnet&dn=%D0%9F%D0%B5%D1%80%D0%B5%D0%B2%D0%BE%D0%B4%D1%87%D0%B8%D0%BA%20%2F%20The%20Covenant%20(%D0%93%D0%B0%D0%B9%20%D0%A0%D0%B8%D1%87%D0%B8%20%2F%20Guy%20Ritchie)%20%5B2023%2C%20%D0%92%D0%B5%D0%BB%D0%B8%D0%BA%D0%BE%D0%B1%D1%80%D0%B8%D1%82%D0%B0%D0%BD%D0%B8%D1%8F%2C%20%D0%98%D1%81%D0%BF%D0%B0%D0%BD%D0%B8%D1%8F%2C%20%D0%A1%D0%A8%D0%90%2C%20%D0%B1%D0%BE%D0%B5%D0%B2%D0%B8%D0%BA%2C%20%D1%82%D1%80%D0%B8%D0%BB%D0%BB%D0%B5%D1%80%2C%20%D0%B2%D0%BE%D0%B5%D0%BD%D0%BD%D1%8B%D0%B9%2C%20%D0%B8%D1%81%D1%82%D0%BE%D1%80%D0%B8%D1%8F%2C%20%D0%B4%D1%80%D0%B0%D0%BC%D0%B0%2C%20BDRip%201080p%5D%20Dub%20(%D0%9C%D0%BE
The latest versions of uTorrent does not have this issue...
If you give it a swam where only the last pieces are available, I'm confident it would write at the end.
What I was trying to convey is that just because you happen to have a swarm where it's not the case, relying on it is risky, because you may end up with a swarm that causes this problem, either by mistake or by a malicious attack. To reliably avoid it, you have to use piece priorities.
Still, if you do in fact only have seeds in the swarm, I can't explain why it would write to the end. Perhaps you can enable picker logging and see why a high piece is picked. or maybe set a conditional breakpoint in the picker for high piece indices.
@arvidn Sorry, I have no will to track this problem down so deep. I'm pretty sure you'll be able to easily reproduce it using the magnet link above if you want to.
"If you give it a swam where only the last pieces are available" - then, in strict sequential mode, Libtorrent must just wait for the first pieces to become available :)
then, in strict sequential mode, Libtorrent must just wait for the first pieces to become available :)
And until that, it should just write zeroes to disk :) Once it's done - it can start download those far-away pieces. Well, this is some extended mode then, it's optional :) KISS principle tells that there is no point (in strict sequential mode) to download far-away pieces in case we have no pieces at the beginning.
Just thoughts.
Sequential mode PLUS the sophisticated approach described above works like a charm :) I allow it to download first 512MB, but now it does not even try to write at these 500MB and really downloads sequentially.
(a less size of allowed "window" gives lower downloading speeds...)
The same issue (except that neither app, nor OS hangs) exists under Android when downloading to SD cards.
So, for now, these are affected (it seems the most popular):
- exFAT. This fs is typically used to transfer files between Windows and macOS.
- Android SD cards.
- FAT32 is affected too, but I don't know if anyone still uses it.
@arvidn, that sophisticated approach has the only one known flaw for now. Let's suppose we have a torrent with two files: A and B.
And lets suppose that A and B shares one common piece. In case the user wants to just download B, we end up with the same issue.
Solution? We need to allocate disk space for A before we'll be downloading that common piece.
Questions:
- Is there a way to force libtorrent to allocate space on disk for this skipped file?
- If no, it seems I'll have to do it on my own. Are there any side effects in this case?
There is another more complex solution (to use custom storage), but I don't want to switch to it for now.
if you set the priority of file A to 0, that partial piece will be saved in the partfile, so I wouldn't expect that to be a problem on FAT32
if you set the priority of file A to 0, that partial piece will be saved in the partfile, so I wouldn't expect that to be a problem on FAT32
Well, this is not what I see in practice.
Btw, you're free to close this ticked - I've implemented workaround and it's working fine....
if you set the priority of file A to 0, that partial piece will be saved in the partfile, so I wouldn't expect that to be a problem on FAT32
It seems you're too lazy to just reproduce it. Think whatever you like :)
are you saying the part file doesn't work?
Well, I never knew about part files until today. Because I don't use prioritize_files.
So, I've tried it. For me, it has no effect at all. No part file has appeared. Instead, LT wrote that partial piece to the end of skipped file (which has caused the issue described in this topic).