RX Speed drops to 200mbits when PHY is at 2.5GBit/s

[xervox]

Hi got issues with latest dkms version on ODROID H2+ running Debian 10.4:

Tested on both installed NICs to 10G switch. When I negotiate 1Gbit/s PHY, RX/TX is maxing out 1 Gbit/s, but when PHY is at 2,5 Gbit/s RX is only doing 200Mbit/s.

Cable is short CAT7 and was also tested with 6a no difference.

I observed similar behaviour for other 10G network cards before, when running lower speeds than 10G (5G/2.5G), so I assume it is a driver issue. But here 2.5Gbit/s PHY is the max default speed and I can't switch between other modes than 1G.

Kernel Module:
filename:       /lib/modules/4.19.0-9-amd64/updates/dkms/r8125.ko
version:        9.003.05-NAPI
license:        GPL
description:    Realtek RTL8125 2.5Gigabit Ethernet driver

IPerf3 Test  RX  Direction for onboard PHY.
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  23.9 MBytes   200 Mbits/sec  446   31.1 KBytes
[  5]   1.00-2.00   sec  22.2 MBytes   186 Mbits/sec  415   32.5 KBytes

Reverse mode, TX Direction for onboard PHY.
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec   274 MBytes  2.30 Gbits/sec
[  5]   1.00-2.00   sec   281 MBytes  2.35 Gbits/sec

ethtool -S enp3s0
NIC statistics:
     tx_packets: 6273329
     rx_packets: 1973763
     tx_errors: 1
     rx_errors: 2012
     rx_missed: 37402
     align_errors: 0
     tx_single_collisions: 0
     tx_multi_collisions: 0
     unicast: 1882159
     broadcast: 90696
     multicast: 908
     tx_aborted: 0
     tx_underrun: 0

Settings for enp3s0:
        Supported ports: [ TP ]
        Supported link modes:   10baseT/Half 10baseT/Full
                                100baseT/Half 100baseT/Full
                                1000baseT/Full
                                2500baseX/Full
        Supported pause frame use: Symmetric Receive-only
        Supports auto-negotiation: Yes
        Supported FEC modes: Not reported
        Advertised link modes:  10baseT/Half 10baseT/Full
                                100baseT/Half 100baseT/Full
                                1000baseT/Full
                                2500baseX/Full
        Advertised pause frame use: No
        Advertised auto-negotiation: Yes
        Advertised FEC modes: Not reported
        Speed: 2500Mb/s
        Duplex: Full
        Port: Twisted Pair
        PHYAD: 0
        Transceiver: internal
        Auto-negotiation: on
        MDI-X: Unknown
        Supports Wake-on: pumbg
        Wake-on: g
        Current message level: 0x00000033 (51)
                               drv probe ifdown ifup
        Link detected: yes

thx a lot!

[Deokgyu Yang]

Hi.

Thanks for your detailed report.
You're right, I think that even the Realtek driver for RTL8125B is not yet matured. We should wait the new driver. :)

I couldn't find any method to adjust link speed to more than 1 Gbits so far.. How about posting this issue to the developer community like Stackoverflow?

[Deokgyu Yang]

Hi again,

The new driver version pulled out from Realtek finally. Please try the new version. :)
I will close this issue.

[xervox]

Hi awesometic,

the issue still remains ~306Mbit/s RX now when synced to 2.5Gbit/s at 10GBase-T Port.

Linux server 4.19.0-12-amd64 #1 SMP Debian 4.19.152-1 (2020-10-18) x86_64 GNU/Linux

modinfo r8125
filename:       /lib/modules/4.19.0-12-amd64/updates/dkms/r8125.ko
version:        9.004.01-NAPI
license:        GPL
description:    Realtek RTL8125 2.5Gigabit Ethernet driver

ethtool -S enp2s0
NIC statistics:
     tx_packets: 79496186
     rx_packets: 84657407
     tx_errors: 10
     rx_errors: 500
     rx_missed: 49180

Please investigate!

[Lam]

Maybe it's a switch issue? Or an issue with that archaeological excavation kernel?

I have:

• Odroid H2+ just like you
• a native 2,5 Gbps switch (QNAP QSW-1105-5T, I'm a home user, don't laugh at me, but this is exactly what this NIC is supposed to be connected to, right?)
• every other client is 1 Gbps

I tested two gigabit clients running iperf3 to two server iper3f's on the H2+. Each got ~900 Mbps when running simultaneously. AKA, my Odroid H2+ and its on-board RTL-8125 does 1,8 Gbps no questions asked. I imagine it would do more if I had either more clients, or another 2,5 Gbps NIC on the LAN.

Furthermore, I also run r8125 9.003.05 because 9.005.01 is utterly broken (says link down when connected to a 1Gbps ONT).

Also, you're listing ancient kernels. So, my Odroid H2+ runs RHEL8 equivalent (was CentOS 8 but CentOS died, so now it's AlmaLinux 8), their kernel is 4.18+patches, pretty close to your 4.19, right? That thing limited my bandwidth to around 300 Mbps. For RHEL/CentOS/AlmaLinux, ELrepo's kernel-ml solved this problem instantly (it was 5.8 when I solved the problem, now I'm running 5.11).

Disclaimer: I haven't tested iperf3 on 4.18.whatever - back then I was just using speedtest.net to check if my Odroid can be a router. Because my 1 Gbps FTTH uplink is PPPoE, there could have been other optimizations making the difference (maybe my issues were caused by poor PPPoE performance and not poor Ethernet performance - which I hope you're checking with iperf3 on a LAN). But it's still worth checking!

[Deokgyu Yang]

Hi.

I cannot sure, but this might be helpful for you. It might you already know if you're in the Odroid forum. :)

That "low speed" problem on 1 GbE mode was discussed on the Odroid forum.
Some people get to fix it by editing ASPM options: https://forum.odroid.com/viewtopic.php?p=320336#p320336

And this ASPM feature also can be turned off by the Makefile on the source tree: https://github.com/awesometic/realtek-r8125-dkms/blob/master/src/Makefile#L38
This isn't tested on my side.

And, as @Lamieur mentioned, the new drivers 9.004.01 and 9.005.01 have poor performance or have excessive CPU consumption. For now, I suggest you switch your driver to 9.003.05 to get the best performance with the r8125 chipset unless you need something special features of the new drivers.