mktorrent seems to be able to create torrent files with more bencoded items than clients can handle

Question

mktorrent seems to be able to create torrent files with more bencoded items than clients can handle

colindean opened this issue 5 years ago · comments

I created using mktorrent a very large torrent file from approximately 5 TB of videos, some ~200 files ranging from a few GB each up to around 75 GB. The torrent file itself is nearly 100 MB. The folks to whom I've distributed the torrent file report that they've not been able to get their clients to read the torrent file.

biglybt:

qbittorrent:

I'm having trouble getting it to work in Transmission. My only Transmission instance is the web interface and uploading it just… crashes the web interface. Latest versions, all.

I see two resolutions for this on the mktorrent side:

Emit a warning when some threshold of number of bencoded items or depth of dictionary is met
Do nothing and let others find this bug report, and realize that they need to break the single large torrent file into a few smaller files!

Colin Dean · Answer 1 · Thu Nov 28 2019 05:21:36 GMT+0800 (China Standard Time)

Related: https://github.com/qbittorrent/qBittorrent/pull/8455/files#diff-9aac8235f4555bbff82d790aa3d186d0R107

Colin Dean · Answer 2 · Sat Dec 28 2019 03:14:07 GMT+0800 (China Standard Time)

This was fixed in pmktorrent v1.3: xxkfqz/pmktorrent@12b357f

ruthtern · Answer 3 · Thu Feb 20 2020 18:02:56 GMT+0800 (China Standard Time)

The torrent will be smaller if the piece size is larger
If clients have a limit - the 2MB limit in rutorrent is well-known - that's their problem
1TB is probably pushing the limit for most clients. 5TB is asking for trouble

Francisco Pombal · Answer 4 · Tue Jul 14 2020 23:43:54 GMT+0800 (China Standard Time)

This is tracked in qBittorrent here:

qbittorrent/qBittorrent#10913
qbittorrent/qBittorrent#12029

libtorrent 2.0 will make the torrent loading limits even more flexible and configurable, btw.

ruthtern · Answer 5 · Wed Jul 22 2020 21:23:59 GMT+0800 (China Standard Time)

There will always be limits, if only to avoid resource exhaustion
Even so, it has always been best practice to choose a larger piece size for larger torrent data sizes. A .torrent file stores a 20-byte hash for each piece. The size of this part of the torrent file is (20 * number of pieces). A .torrent for serving 5TB of data does not need to be 100MB. This is caused by setting the piece size to 1MB. If the piece size is 64MB, the torrent will have about 80000 pieces and will be about 1.6MB

A non-technical point: bittorrent is a sharing system. Very few people are going to dedicate terabytes of disk space to seed a single torrent