Abnormally slow IO performance on CIFS network share
constantins2001 opened this issue · comments
Versions
libtorrent version (or branch): 2.0.9-r2
platform/architecture: ARM64
What is the problem?
IO on mounted CIFS (SMB) network share are abnormally low.
- Downloading to the network share directly results in ~5.4 MB/s speeds.
- Downloading to local SSD results in speeds in excess of 26 MB/s
- Automatically moving to network share via qBittorrent results in about 5 MB/s speeds.
- Moving the file manually from local SSD to network share via the mv command results in speeds of ~35.7 MB/s.
This indicates qBittorrent has drastically slower speeds when moving downloads/downloading to network shares compared to a manual move operation.
Steps to reproduce
- Mount CIFS share
mount.cifs -o user=<XXX>,pass=<XXX>,cache=loose,actimeo=120,fsc //<URL>/<share> <mountpoint> - Set qBittorent download/move location to
<mountpoint> - Download torrent
- Observe performance impact
Additional context
I'll be happy to provide a full testing environment, just contact me directly.
Related to qbittorrent/qBittorrent#19407
2.0px branch unusable, bad. don't bother with it! it will never be good
@nagyimre1980 Why do you think that?
@constantins2001 Do you have a comparison with version 1.2 of the library?
How about #6561 qbittorrent/qBittorrent#18983 qbittorrent/qBittorrent#16043 and many more performance issues that I didn't comment on, so it's hard for me to track them down. Is that not enough to consider this branch bad?
@stalkerok Are these problems compared to the old version 1.2 of the library? Or are they general problems?
These are problems only 2.0.x
@arvidn Too many changes at one time (1.2->2). Is it possible to roll back the change of disk write type? (memory mapped files)
This can increase disk write speed and improve the popularity of the library 2.0 version.
@stalkerok You need to test 1.2 torrents and 2.0 torrents on Url-seeded torrent files. This will rule out problems with other torrent mechanisms. This is pure write speed. Use this ready-made torrent file.
511.65-notebook-win10-win11-64bit-international-dch-whql.zip
@constantins2001 Do you have a comparison with version 1.2 of the library?
Is it possible to roll back the change of disk write type? (memory mapped files)
I reverted back to version 2 due to the following issue with 1.2.19:
qbittorrent/qBittorrent#19407 (comment)
Better slowish write speeds than an unstable experience, but still I'd love to use the full potential of my platform...
https://user-images.githubusercontent.com/63958081/259389632-14476ece-989c-4394-b232-38e9da440111.png
RC_2_0 is NOT ABLE to download/upload 100MB/s from HDD.
Disk I/O type (POSIX-compliant) saves the situation a little, but these are far from the same results.
Also unable to match RC_1_2 is this new disk I/O #7013
So RC_1_2 remains the best.
@stalkerok Problem with the upload or download speed?
- Automatically moving to network share via qBittorrent results in about 5 MB/s speeds.
same issue on 1.2.19, only downloads directly to the share are fast
Looks like Arvin has abandoned library development. I don't know why. Maybe he has a creative crisis because of the wrong transition to version 2.+.
Or maybe he's just tired of torrents.
It must be hard to develop a program and not see the results of its work.
Using 2.x with NFS share. Upload/download rates above 100 MBit/s are no problem. I think it's working fine with the right OS and/or kernel version.
Using 2.x with NFS share
NFS and SMB seem to be very different in many regards
Update:
I moved to a non network share environment with 2x 8 TB HDDs (HGST HUH721008AL) in RAID 0 and now easily reach the limit of my network connection (1 Gbit/s).
The issues many others seem to experience with HDDs don't show for me.
Update: I moved to a non network share environment with 2x 8 TB HDDs (HGST HUH721008AL) in RAID 0 and now easily reach the limit of my network connection (1 Gbit/s). The issues many others seem to experience with HDDs don't show for me.
Yeah, it's a bit silly but that's expected. Remember each time you're doing this you're incurring a network roundtrip which can be multiple ms at the very least. Writing sequentially is also a very different from writing "random" blocks, especially with that latency involved. I saw you turned your dirty to nothing which was obviously a mistake on your part.
But yeah, mmap is never going to be as performant as the classic C linux api which 1.2 used. The future remains io_uring here for linux. The backends are interchangeable, so 2.0 can recover from this (and be baller if the networking is moved as well). Just takes time.
you're incurring a network roundtrip which can be multiple ms
Shouldn't matter, in combination with the write cache the 1 Gbit/s connection should easily be saturated (see OP).
Writing sequentially is also a very different from writing "random" blocks, especially with that latency involved.
Due to the same reasons as above I also don't think this is the issue. The move operation is just as slow as the download using random blocks (see OP).
I saw you turned your dirty to nothing which was obviously a mistake on your part.
Only on the old libtorrent, it crashed my system with the default settings.
I think once it filled the write cache the system switched to sync operations and in combination with the network share that somehow locked up the whole system. The config I posted above was the only way for me to prevent the system from locking up. I also observed that behavior with Transmission (which prompted my switch to qBittorrent in the first place).
Using 2.x with NFS share. Upload/download rates above 100 MBit/s are no problem. I think it's working fine with the right OS and/or kernel version.
@paxter Can you share your configuration with us?
libtorrent 2.0 defaults to using memory mapped files for disk I/O. This turned out to be sub-optimal in a lot of common cases (perhaps most cases).
My top priority is to implement a new disk I/O subsystem in master of libtorrent, which is multi-threaded, cached and uses pread/pwrite to be thread-safe (and windows equivalents). Once this is ready I will cut libtorrent 2.1 which includes this.
In the meantime, you may want to try to posix disk I/O subsystem available in libtorrent 2.0. It's not multi-threaded nor cached, so it will likely not perform optimally, but for a network share, it might be better than mmap.
@arvidn It would be better to do DHT development (switch to tcp and broadcast traffic through other nodes (as in libp2p)). But if you'd rather work with the disk subsystem, then ok.
It would be better to do DHT development
patches are welcome
It would be better to do DHT development
patches are welcome
I'm developing more important programs. Higher level. And it would be easier for me to do it if you improved the DHT.
It would be better to do DHT development
patches are welcome
Okay. I realize it's very complicated. So just help me on my tickets. I think that should be enough.
Using 2.x with NFS share. Upload/download rates above 100 MBit/s are no problem. I think it's working fine with the right OS and/or kernel version.
@paxter Can you share your configuration with us?
Which configuration? It's a default nfs share on bsd. Nothing special.
@arvidn, why can't the old disk I/O subsystem from 1.2 be used in 2.1, which works fine, if not by default, then at least as configurable parameters? You can't guarantee that the new disk subsystem will be better and there will be no problems with it.
@arvidn, why can't the old disk I/O subsystem from 1.2 be used in 2.1, which works fine, if not by default, then at least as configurable parameters? You can't guarantee that the new disk subsystem will be better and there will be no problems with it.
the p stands for position, which allows you to avoid the file pointer position and read and write to certain segments (as long as append isn't used). This is a general POSIX backend that was standardised in 2001 (not nearly as fast or efficient as io_uring or IoRing) - that should still be better if fully pipelined than the older one despite being 22 year-old technology. The risk here is the local cache, that's going to dump into the page cache, then finally spooled to disk.
People are having problems with the classic backend with multiple files not being committed to the page cache when cross-seeding (same data, different torrents). I'm unclear/concerned the local cache is going to still hamper these efforts, and the locking isn't going to be as tight as one would expect.
the real struggle for NVMe boxes right now (on 1.2) is the fact the network is single threaded. There's been custom TLS crypto and similar shoved in there that just kills the entire session if it's toggled on. io_uring "fixes" this as well by allowing you to do all that crypto for the next set of work between submissions (which could be threaded or not).
Anyway, it sounds like 2.1 is the next 1.2, and 2.0 will hopefully be forgotten in short order (it may even take calling it 3.0, but if there's a large problem when that's done it won't be possible to recover from this).
Versions
libtorrent version (or branch): 2.0.9-r2 platform/architecture: ARM64
What is the problem?
IO on mounted CIFS (SMB) network share are abnormally low.
- Downloading to the network share directly results in ~5.4 MB/s speeds.
- Downloading to local SSD results in speeds in excess of 26 MB/s
- Automatically moving to network share via qBittorrent results in about 5 MB/s speeds.
- Moving the file manually from local SSD to network share via the mv command results in speeds of ~35.7 MB/s.
This indicates qBittorrent has drastically slower speeds when moving downloads/downloading to network shares compared to a manual move operation.
Steps to reproduce
- Mount CIFS share
mount.cifs -o user=<XXX>,pass=<XXX>,cache=loose,actimeo=120,fsc //<URL>/<share> <mountpoint>- Set qBittorent download/move location to
<mountpoint>- Download torrent
- Observe performance impact
Additional context
I'll be happy to provide a full testing environment, just contact me directly.
Related to qbittorrent/qBittorrent#19407
This is driving me crazy as well. Literally tried everything, but the speeds are terrible. Wish I could just go around the NAS, but I can’t.