[Bug] Abnormally low bandwidth

[DruggleY]

Verify steps

• Is this something you can debug and fix? Send a pull request! Bug fixes and documentation fixes are welcome.
• I have searched on the issue tracker for a related issue.

Version

2.5.1

What OS are you seeing the problem on?

Linux

Description

I have three devices under the same LAN, and the communication bandwidth between them is close to 10Gbps. When I use one of the servers as a gateway to communicate based on tun2socks, the bandwidth has been greatly attenuated. The details are as follows:

CPU: Intel(R) Xeon(R) Silver 4109T CPU @ 2.00GHz, 32c
Mem: 128GB
Device IP: 192.168.3.101, 192.168.3.103, 192.168.3.104

Raw Speed

From 101 to 104

$ iperf3 -c 192.168.3.104 -R -P 10
...
[ ID] Interval           Transfer     Bitrate         Retr
...
[SUM]   0.00-10.00  sec  11.0 GBytes  9.41 Gbits/sec   55             sender
[SUM]   0.00-10.00  sec  11.0 GBytes  9.41 Gbits/sec                  receiver

From 104 To 103

$ iperf3 -c 192.168.3.103 -R -P 10
...
[ ID] Interval           Transfer     Bitrate         Retr
...
[SUM]   0.00-10.00  sec  10.6 GBytes  9.08 Gbits/sec  625             sender
[SUM]   0.00-10.00  sec  10.6 GBytes  9.07 Gbits/sec                  receiver

From 101 To 103

$ iperf3 -c 192.168.3.103 -R -P 10
...
[ ID] Interval           Transfer     Bitrate         Retr
...
[SUM]   0.00-10.00  sec  10.9 GBytes  9.40 Gbits/sec  279             sender
[SUM]   0.00-10.00  sec  10.9 GBytes  9.39 Gbits/sec                  receiver

With Tun2Socks

I run tun2socks on the 104 device, the relevant commands are as follows

$ ./tun2socks-linux-amd64 -version
tun2socks-2.5.1
linux/amd64, go1.20.4, 7ab86fd
$ ./tun2socks-linux-amd64 -device tun0 -proxy direct:// -interface bond0 -mtu 9000
INFO[0000] [DIALER] bind to interface: bond0
INFO[0000] [STACK] tun://tun0 <-> direct://

Then I enabled the network card and configured the routing in another window

$ ip link set tun0 up
$ route add -host 192.168.3.103 dev tun0

At this point, when I re-run iperf3 from 104 to 103

$ iperf3 -c 192.168.3.103 -R -P 10
...
[ ID] Interval           Transfer     Bitrate         Retr
...
[SUM]   0.00-10.00  sec  10.3 GBytes  8.84 Gbits/sec  829             sender
[SUM]   0.00-10.00  sec  10.3 GBytes  8.83 Gbits/sec                  receiver

Everything looks fine. However, when I configure the gateway of 101 to 104, run iperf3

$ route add -host 192.168.3.103 gw 192.168.3.104

$ iperf3 -c 192.168.3.104 -R -P 10
...
[ ID] Interval           Transfer     Bitrate
...
[SUM]   0.00-3.73   sec  0.00 Bytes  0.00 bits/sec                  sender
[SUM]   0.00-3.73   sec  4.08 GBytes  9.40 Gbits/sec                  receiver

$ iperf3 -c 192.168.3.103 -R -P 10
...
[ ID] Interval           Transfer     Bitrate         Retr
...
[SUM]   0.00-10.00  sec  2.31 GBytes  1.98 Gbits/sec  2239             sender
[SUM]   0.00-10.00  sec  2.27 GBytes  1.95 Gbits/sec                  receiver

The transmission speed between 101 and 103 has dropped significantly.

I want to know is this speed change normal? Is it due to some system configuration or something else?

Attached:
sysctl.conf in 101,104:

fs.file-max = 2097152
fs.nr_open=2097152
net.nf_conntrack_max = 2097152
net.ipv4.ip_local_port_range = 1024 65535
net.ipv4.ip_forward = 1
net.ipv4.tcp_rmem = 4096 32768 262142
net.ipv4.tcp_wmem = 4096 32768 262142
net.core.netdev_max_backlog = 8096
net.core.rmem_default = 262144
net.core.wmem_default = 262144
net.core.rmem_max = 2097152
net.core.wmem_max = 2097152

CLI or Config

No response

Logs

No response

How to Reproduce

No response

[Jason Lyu]

Hmm, that’s weird. I’ve never seen that happen on my local network. 🤔

[DruggleY]

So do you have any suggestions for debugging this issue?😥

[Jason Lyu]

So do you have any suggestions for debugging this issue?😥

Not sure how to debug this. Maybe you could try tcpdump and see what happens.

[DruggleY]

I found that I performed the same test in a bandwidth environment of 1Gbps, and the bandwidth attenuation was only 850Mbps/943Mbps, that is, about 10% of the bandwidth was attenuated. However, under the 10Gbps network environment, the attenuation is close to 80%. Have you done similar tests in a 10Gbps network environment?😧

[Jason Lyu]

The max throughput is related to your CPU performance. In my environment, I can archive 2.5-8Gbps on a 10G cable network depending on the connection numbers.

The number of total TCP connections affects the maximum bandwidth. You can refer this wiki https://github.com/xjasonlyu/tun2socks/wiki/Benchmarks for details.

[DruggleY]

I should have also used 10 connections during the test, and the CPU usage was up to 760%. If this does not reach the upper limit based on the number of cores, is it possible that this is related to the CPU clock speed, that is, a CPU with a higher clock speed may be required to further increase its bandwidth.

[Jason Lyu]

Yeah, I think it is possible too.

[github-actions]

This issue is stale because it has been open 60 days with no activity. Remove stale label or comment or this will be closed in 7 days