Cannot disable EEE on Raspberry Pi 5
b4dpxl opened this issue · comments
Describe the bug
My Raspberry Pi5, running bookworm, is experiencing frequent eth0 droppages. Example:
[Mar23 13:30] macb 1f00100000.ethernet eth0: Link is Down
[ +38.912436] macb 1f00100000.ethernet eth0: Link is Up - 1Gbps/Full - flow control off
[Mar24 00:23] macb 1f00100000.ethernet eth0: Link is Down
[ +3.072515] macb 1f00100000.ethernet eth0: Link is Up - 1Gbps/Full - flow control off
[Mar24 02:17] macb 1f00100000.ethernet eth0: Link is Down
[ +3.072523] macb 1f00100000.ethernet eth0: Link is Up - 1Gbps/Full - flow control off
[Mar24 14:31] macb 1f00100000.ethernet eth0: Link is Down
[Mar24 14:32] macb 1f00100000.ethernet eth0: Link is Up - 1Gbps/Full - flow control off
[Mar24 16:54] macb 1f00100000.ethernet eth0: Link is Down
[ +3.068473] macb 1f00100000.ethernet eth0: Link is Up - 1Gbps/Full - flow control off
[Mar24 18:58] macb 1f00100000.ethernet eth0: Link is Down
[ +3.072540] macb 1f00100000.ethernet eth0: Link is Up - 1Gbps/Full - flow control off
I've changed switch ports, cables, etc. This was also an issue on RPi 4, which could be resolved by disabling EEE, either with ethtool or dtparam. I want to see if this also solves the problem on Pi5, but neither of these options work on RPi5, and trying to run ethtool reports "Operation not supported".
See https://forums.raspberrypi.com/viewtopic.php?t=360924
Can EEE be disabled on RPi5 via another method, or is there an ETA when disabling it will be supported on Pi5?
Steps to reproduce the behaviour
run ethtool --set-eee eth0 eee off.
Get the response: netlink error: Operation not supported
Device (s)
Raspberry Pi 5
System
# cat /etc/rpi-issue
Raspberry Pi reference 2023-12-11
Generated using pi-gen, https://github.com/RPi-Distro/pi-gen, 2acf7afcba7d11500313a7b93bb55a2aae20b2d6, stage2
# vcgencmd version
2023/11/20 19:40:17
Copyright (c) 2012 Broadcom
version 361205c6 (release) (embedded)
# uname -a
Linux rocinante 6.1.0-rpi7-rpi-2712 #1 SMP PREEMPT Debian 1:6.1.63-1+rpt1 (2023-11-24) aarch64 GNU/Linux
Logs
No response
Additional context
No response
If your problem is the link always negotiating, you can just use these two commands here:
sudo ethtool -s eth0 autoneg off
sudo ethtool -s eth0 speed 1000
Obviously set the speed as the one you want, for 1Gbps you will put 1000, like I've already done in the example.
Also ensure to run the command as a root user / using sudo since you cannot do these changes with a normal user.
I don't believe it's a negotiation issue; I'd already checked playing with ethtool settings before hand. It still happened with 1gb autoneg off, but was fine when forced to 100mb with/without autoneg. This mirrors what other people had previously observed on Pi4's.
I was able to borrow a different switch (Netgear, same as my current one, but slightly newer version) and so far it's been fine.
This also mirrors what people observed on Pi4, where it seems to be an incompatibility with EEE between the Pi and some switch models.
Again, I can't confirm this as it's impossible to turn off EEE (AFAIK) on the Pi 5
Interesting to know @b4dpxl, that seems to be related more on the switch side, I know not being able to disable EEE can be seen as a bad thing but I also see no reason of why it would hurt not having that... the problem comes with non-compliant network equipment...
If you got the model of such switch let us know, would be nice to have a list around so people having connection issues know the reason...
Also EEE is mainly hardware related and I am not really sure if the chip used in Pi4/5 even allow such feature.
PS. Everything is fine here with a Cisco Catalyst switch, in 100 and 1000 modes.
In my case the problematic switch is a Netgear GS108Ev3, but a newer version Netgear GS108PEv3 is fine. I suspect most Pi users aren't running Cisco network equipment though 🤨
Well, as I understand it, the Pi4 and Pi5 both share the ethernet chip, and the Pi4 definitely allows you to disable EEE. It's referenced here in the overlays document and here is the pull request which was merged to allow it to be disabled via dtparam.
Clearly this was a known issue, otherwise this wouldn't have been added. As per the pull:
The EEE implementation on BCM2711 doesn't work well with all switches.
IMO it's not fair to place the blame on the switch when there are multiple reported switches at issue, and one common factor (the Pi). Did you actually look at any of the links I provided before?
In my case the problematic switch is a Netgear GS108Ev3, but a newer version Netgear GS108PEv3 is fine. I suspect most Pi users aren't running Cisco network equipment though 🤨
Well, as I understand it, the Pi4 and Pi5 both share the ethernet chip, and the Pi4 definitely allows you to disable EEE. It's referenced here in the overlays document and here is the pull request which was merged to allow it to be disabled via dtparam.
Clearly this was a known issue, otherwise this wouldn't have been added. As per the pull:
The EEE implementation on BCM2711 doesn't work well with all switches.
IMO it's not fair to place the blame on the switch when there are multiple reported switches at issue, and one common factor (the Pi). Did you actually look at any of the links I provided before?
Yes, I've looked into the links you shared but not really deeply since I had not much time, anyways since you mentioned that it can be disabled on Pi4, I will later test out on a 4 and see the results I get and if there's something possible in the 5...
Just tested on a 4 and I can see the NIC shutting down and powering back on... Will check later what could cause the problem on the 5.
Turns out I've found difference between the Pi4 and the Pi5, if you run "lspci" you can see that on the Pi5 the ethernet controller is connected through PCI unlike the 4...
I think this is causing some issues to ethtool at this moment, will continue to look around.
PS. I think that the PCI ethernet controller is practically the new RP1 chip.
Also @b4dpxl if you're interested into the RP1 chip you can find some infos here: https://datasheets.raspberrypi.com/rp1/rp1-peripherals.pdf
Sadly there's nothing about EEE so I guess we can do nothing at this moment : /
After looking around the current implementation of Ethernet on RP1 doesn't support EEE at all since the implementation of it is included in IEEE 802.3az, meanwhile on the RP1 docs it is just IEEE 802.3...
Not sure if this will be changed in the future but I doubt anything will come soon, even because EEE is a good feature to always have enabled.
I have seen no mention of EEE or 802.3az in the documentation for the Ethernet MAC in RP1.
Large file transfer between mbp and rpi5[fe80::e02a:17b4:c527:84e3%en9] using direct cat6 cable link between the two:
scp -6 [fe80::e02a:17b4:c527:84e3%en9]:/mnt/SSD/tmp/BIG.tar . BIG.tar 84% 11GB 397.4KB/s - stalled -client_loop: send disconnect: Broken pipe scp: Connection closed
Sustained data transfer rate was between 80-100 Mbytes/s until getting stalled.
Doing the same between rpi5 and rpi4 failed at 11% file transfer, data transfer rate prior to stall was 40MBytes/s. Rpi5 also loses it's ipv6 network address binding even when idling. It's either is an issue within rpi5 ethernet stack or my rpi5 ethernet is bad. I don't have another rpi5 at hand, someone else may try reproducing this problem.
Large file transfer between mbp and rpi5[fe80::e02a:17b4:c527:84e3%en9] using direct cat6 cable link between the two:
scp -6 [fe80::e02a:17b4:c527:84e3%en9]:/mnt/SSD/tmp/BIG.tar . BIG.tar 84% 11GB 397.4KB/s - stalled -client_loop: send disconnect: Broken pipe scp: Connection closedSustained data transfer rate was between 80-100 Mbytes/s until getting stalled.
Doing the same between rpi5 and rpi4 failed at 11% file transfer, data transfer rate prior to stall was 40MBytes/s. Rpi5 also loses it's ipv6 network address binding even when idling. It's either is an issue within rpi5 ethernet stack or my rpi5 ethernet is bad. I don't have another rpi5 at hand, someone else may try reproducing this problem.
I cannot replicate this since I am not able to disconnect my Pi5 but, have you tried using something like a USB -> Ethernet adapter? Would be nice to know if the problem is the hardware or something that is broken in general in the kernel with networking... I think that this still is a RP1 related problem tho.
Large file transfer between mbp and rpi5[fe80::e02a:17b4:c527:84e3%en9] using direct cat6 cable link between the two:
scp -6 [fe80::e02a:17b4:c527:84e3%en9]:/mnt/SSD/tmp/BIG.tar . BIG.tar 84% 11GB 397.4KB/s - stalled -client_loop: send disconnect: Broken pipe scp: Connection closed
Sustained data transfer rate was between 80-100 Mbytes/s until getting stalled.
Doing the same between rpi5 and rpi4 failed at 11% file transfer, data transfer rate prior to stall was 40MBytes/s. Rpi5 also loses it's ipv6 network address binding even when idling. It's either is an issue within rpi5 ethernet stack or my rpi5 ethernet is bad. I don't have another rpi5 at hand, someone else may try reproducing this problem.I cannot replicate this since I am not able to disconnect my Pi5 but, have you tried using something like a USB -> Ethernet adapter? Would be nice to know if the problem is the hardware or something that is broken in general in the kernel with networking... I think that this still is a RP1 related problem tho.
I managed to find an old USB-A wifi adapter. The following tests rule out my PI5's hardware being an issue. In these tests two consecutive downloads were performed with PI5 and then the same repeated with one of my PI4 using the the same usb wifi adapter but using a much bigger BIG.tar!
MBP downloading BIG.tar from RPI5 over usb ethernet (wifi router turned off)
MBP downloading BIG.tar from RPI5 over usb ethernet (wifi router turned off)
scp -6 [fe80::cd59:4551:bd97:3e80%en8]:/mnt/SSD/tmp/BIG.tar .
BIG.tar 100% 12GB 105.8MB/s 02:00
scp -6 [fe80::cd59:4551:bd97:3e80%en8]:/mnt/SSD/tmp/BIG.tar .
BIG.tar 19% 2542MB 430.8KB/s - stalled -client_loop: send disconnect: Broken pipe
*** Connection stalled in just over 2 mins of data transfer
MBP downloading BIG.tar from RPI4 over usb ethernet (wifi router turned off)
scp -6 [fe80::3d8e:457f:16cd:bc97%en8]:/mnt/SSD/tmp/BIG.tar .
BIG.tar 100% 54GB 41.5MB/s 22:00
scp -6 [fe80::3d8e:457f:16cd:bc97%en8]:/mnt/SSD/tmp/BIG.tar .
BIG.tar 100% 54GB 42.8MB/s 21:20
*** Connection stable over 43 mins of data transfer
In an earlier test I had used PI5's native Ethernet connected to the wifi router's Ethernet port for the same download test, which worked fine. The only thing different was PI's eth0 got allocated an IPV4 address by the router. An issue within Bookworm's IPV6 stack?
Large file transfer between mbp and rpi5[fe80::e02a:17b4:c527:84e3%en9] using direct cat6 cable link between the two:
scp -6 [fe80::e02a:17b4:c527:84e3%en9]:/mnt/SSD/tmp/BIG.tar . BIG.tar 84% 11GB 397.4KB/s - stalled -client_loop: send disconnect: Broken pipe scp: Connection closed
Sustained data transfer rate was between 80-100 Mbytes/s until getting stalled.
Doing the same between rpi5 and rpi4 failed at 11% file transfer, data transfer rate prior to stall was 40MBytes/s. Rpi5 also loses it's ipv6 network address binding even when idling. It's either is an issue within rpi5 ethernet stack or my rpi5 ethernet is bad. I don't have another rpi5 at hand, someone else may try reproducing this problem.I cannot replicate this since I am not able to disconnect my Pi5 but, have you tried using something like a USB -> Ethernet adapter? Would be nice to know if the problem is the hardware or something that is broken in general in the kernel with networking... I think that this still is a RP1 related problem tho.
I managed to find an old USB-A wifi adapter. The following tests rule out my PI5's hardware being an issue. In these tests two consecutive downloads were performed with PI5 and then the same repeated with one of my PI4 using the the same usb wifi adapter but using a much bigger BIG.tar!
MBP downloading BIG.tar from RPI5 over usb ethernet (wifi router turned off)
MBP downloading BIG.tar from RPI5 over usb ethernet (wifi router turned off) scp -6 [fe80::cd59:4551:bd97:3e80%en8]:/mnt/SSD/tmp/BIG.tar . BIG.tar 100% 12GB 105.8MB/s 02:00 scp -6 [fe80::cd59:4551:bd97:3e80%en8]:/mnt/SSD/tmp/BIG.tar . BIG.tar 19% 2542MB 430.8KB/s - stalled -client_loop: send disconnect: Broken pipe*** Connection stalled in just over 2 mins of data transfer
MBP downloading BIG.tar from RPI4 over usb ethernet (wifi router turned off)
scp -6 [fe80::3d8e:457f:16cd:bc97%en8]:/mnt/SSD/tmp/BIG.tar . BIG.tar 100% 54GB 41.5MB/s 22:00 scp -6 [fe80::3d8e:457f:16cd:bc97%en8]:/mnt/SSD/tmp/BIG.tar . BIG.tar 100% 54GB 42.8MB/s 21:20*** Connection stable over 43 mins of data transfer
In an earlier test I had used PI5's native Ethernet connected to the wifi router's Ethernet port for the same download test, which worked fine. The only thing different was PI's eth0 got allocated an IPV4 address by the router. An issue within Bookworm's IPV6 stack?
I just tried copying over a 10GB file between my laptop (Debian) using SCP too and the Pi5 without having any issue... Also I am not sure if there are issues within Bookworm IPv6 stack. I got the idea that the RP1 stack is still not fully working well...
Either ways, could you run the command:
uname -aSo we can at least understand what kernel you are running on, the output in my case is this one:
Linux raspberrypi5 6.6.22-v8-16k+ #1747 SMP PREEMPT Wed Mar 27 17:06:56 GMT 2024 aarch64 GNU/Linux
Just finished transferring 100GBs... I guess the problem is not happening here at all:
alessandro@alessandro-laptop:~$ scp alessandro@raspberrypi5.local:/home/alessandro/test ./
alessandro@raspberrypi5.local's password:
test 100% 100GB 93.6MB/s 18:14
Large file transfer between mbp and rpi5[fe80::e02a:17b4:c527:84e3%en9] using direct cat6 cable link between the two:
scp -6 [fe80::e02a:17b4:c527:84e3%en9]:/mnt/SSD/tmp/BIG.tar . BIG.tar 84% 11GB 397.4KB/s - stalled -client_loop: send disconnect: Broken pipe scp: Connection closed
Sustained data transfer rate was between 80-100 Mbytes/s until getting stalled.
Doing the same between rpi5 and rpi4 failed at 11% file transfer, data transfer rate prior to stall was 40MBytes/s. Rpi5 also loses it's ipv6 network address binding even when idling. It's either is an issue within rpi5 ethernet stack or my rpi5 ethernet is bad. I don't have another rpi5 at hand, someone else may try reproducing this problem.I cannot replicate this since I am not able to disconnect my Pi5 but, have you tried using something like a USB -> Ethernet adapter? Would be nice to know if the problem is the hardware or something that is broken in general in the kernel with networking... I think that this still is a RP1 related problem tho.
I managed to find an old USB-A wifi adapter. The following tests rule out my PI5's hardware being an issue. In these tests two consecutive downloads were performed with PI5 and then the same repeated with one of my PI4 using the the same usb wifi adapter but using a much bigger BIG.tar!
MBP downloading BIG.tar from RPI5 over usb ethernet (wifi router turned off)
MBP downloading BIG.tar from RPI5 over usb ethernet (wifi router turned off) scp -6 [fe80::cd59:4551:bd97:3e80%en8]:/mnt/SSD/tmp/BIG.tar . BIG.tar 100% 12GB 105.8MB/s 02:00 scp -6 [fe80::cd59:4551:bd97:3e80%en8]:/mnt/SSD/tmp/BIG.tar . BIG.tar 19% 2542MB 430.8KB/s - stalled -client_loop: send disconnect: Broken pipe*** Connection stalled in just over 2 mins of data transfer
MBP downloading BIG.tar from RPI4 over usb ethernet (wifi router turned off)
scp -6 [fe80::3d8e:457f:16cd:bc97%en8]:/mnt/SSD/tmp/BIG.tar . BIG.tar 100% 54GB 41.5MB/s 22:00 scp -6 [fe80::3d8e:457f:16cd:bc97%en8]:/mnt/SSD/tmp/BIG.tar . BIG.tar 100% 54GB 42.8MB/s 21:20*** Connection stable over 43 mins of data transfer
In an earlier test I had used PI5's native Ethernet connected to the wifi router's Ethernet port for the same download test, which worked fine. The only thing different was PI's eth0 got allocated an IPV4 address by the router. An issue within Bookworm's IPV6 stack?I just tried copying over a 10GB file between my laptop (Debian) using SCP too and the Pi5 without having any issue... Also I am not sure if there are issues within Bookworm IPv6 stack. I got the idea that the RP1 stack is still not fully working well...
Either ways, could you run the command:
uname -aSo we can at least understand what kernel you are running on, the output in my case is this one:
Linux raspberrypi5 6.6.22-v8-16k+ #1747 SMP PREEMPT Wed Mar 27 17:06:56 GMT 2024 aarch64 GNU/LinuxJust finished transferring 100GBs... I guess the problem is not happening here at all:
alessandro@alessandro-laptop:~$ scp alessandro@raspberrypi5.local:/home/alessandro/test ./ alessandro@raspberrypi5.local's password: test 100% 100GB 93.6MB/s 18:14
That's good to hear. Problem occurs at my end only when using direct cable between PI5 and any other host, for some reason nmtui does not allow changing or even adding IPV4 for link-local connections or I would tried with IPV4 stack for comparison. PI5's uname:
uname -a
Linux rpi5 6.6.28+rpt-rpi-2712 #1 SMP PREEMPT Debian 1:6.6.28-1+rpt1 (2024-04-22) aarch64 GNU/Linux
Perhaps repeating a file transfer with IPV6 src and dst addresses may reproduce the problem I am seeing.
Large file transfer between mbp and rpi5[fe80::e02a:17b4:c527:84e3%en9] using direct cat6 cable link between the two:
scp -6 [fe80::e02a:17b4:c527:84e3%en9]:/mnt/SSD/tmp/BIG.tar . BIG.tar 84% 11GB 397.4KB/s - stalled -client_loop: send disconnect: Broken pipe scp: Connection closed
Sustained data transfer rate was between 80-100 Mbytes/s until getting stalled.
Doing the same between rpi5 and rpi4 failed at 11% file transfer, data transfer rate prior to stall was 40MBytes/s. Rpi5 also loses it's ipv6 network address binding even when idling. It's either is an issue within rpi5 ethernet stack or my rpi5 ethernet is bad. I don't have another rpi5 at hand, someone else may try reproducing this problem.I cannot replicate this since I am not able to disconnect my Pi5 but, have you tried using something like a USB -> Ethernet adapter? Would be nice to know if the problem is the hardware or something that is broken in general in the kernel with networking... I think that this still is a RP1 related problem tho.
I managed to find an old USB-A wifi adapter. The following tests rule out my PI5's hardware being an issue. In these tests two consecutive downloads were performed with PI5 and then the same repeated with one of my PI4 using the the same usb wifi adapter but using a much bigger BIG.tar!
MBP downloading BIG.tar from RPI5 over usb ethernet (wifi router turned off)
MBP downloading BIG.tar from RPI5 over usb ethernet (wifi router turned off) scp -6 [fe80::cd59:4551:bd97:3e80%en8]:/mnt/SSD/tmp/BIG.tar . BIG.tar 100% 12GB 105.8MB/s 02:00 scp -6 [fe80::cd59:4551:bd97:3e80%en8]:/mnt/SSD/tmp/BIG.tar . BIG.tar 19% 2542MB 430.8KB/s - stalled -client_loop: send disconnect: Broken pipe*** Connection stalled in just over 2 mins of data transfer
MBP downloading BIG.tar from RPI4 over usb ethernet (wifi router turned off)
scp -6 [fe80::3d8e:457f:16cd:bc97%en8]:/mnt/SSD/tmp/BIG.tar . BIG.tar 100% 54GB 41.5MB/s 22:00 scp -6 [fe80::3d8e:457f:16cd:bc97%en8]:/mnt/SSD/tmp/BIG.tar . BIG.tar 100% 54GB 42.8MB/s 21:20*** Connection stable over 43 mins of data transfer
In an earlier test I had used PI5's native Ethernet connected to the wifi router's Ethernet port for the same download test, which worked fine. The only thing different was PI's eth0 got allocated an IPV4 address by the router. An issue within Bookworm's IPV6 stack?I just tried copying over a 10GB file between my laptop (Debian) using SCP too and the Pi5 without having any issue... Also I am not sure if there are issues within Bookworm IPv6 stack. I got the idea that the RP1 stack is still not fully working well...
Either ways, could you run the command:uname -aSo we can at least understand what kernel you are running on, the output in my case is this one:
Linux raspberrypi5 6.6.22-v8-16k+ #1747 SMP PREEMPT Wed Mar 27 17:06:56 GMT 2024 aarch64 GNU/Linux
Just finished transferring 100GBs... I guess the problem is not happening here at all:alessandro@alessandro-laptop:~$ scp alessandro@raspberrypi5.local:/home/alessandro/test ./ alessandro@raspberrypi5.local's password: test 100% 100GB 93.6MB/s 18:14That's good to hear. Problem occurs at my end only when using direct cable between PI5 and any other host, for some reason
nmtuidoes not allow changing or even adding IPV4 for link-local connections or I would tried with IPV4 stack for comparison. PI5'suname:uname -a Linux rpi5 6.6.28+rpt-rpi-2712 #1 SMP PREEMPT Debian 1:6.6.28-1+rpt1 (2024-04-22) aarch64 GNU/LinuxPerhaps repeating a file transfer with IPV6 src and dst addresses may reproduce the problem I am seeing.
I was running on kernel 6.22 so I will update my system to the latest kernel 6.30 and see if anything changes (I don't remember seeing any change on eth related stuff tho)
Just tested again and I confirm to have no issues with IPv4/6 at all when sharing my network using Debian.
Are you sure your network cable is functional, also if you run "ifconfig" do you see any Tx/Rx error on the eth interface? That would mean something is going really wrong, here's my output:
eno1: flags=4099<UP,BROADCAST,MULTICAST> mtu 1500
ether 08:bf:b8:d0:3c:d0 txqueuelen 1000 (Ethernet)
RX packets 19174297 bytes 26668819810 (26.6 GB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 2227528 bytes 160733910 (160.7 MB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
As you can see I've got no errors at all and the transfer went well (I stopped it at 26GB).
I could try on a MBP with M1 but I doubt the problem relies on MacOS.
Large file transfer between mbp and rpi5[fe80::e02a:17b4:c527:84e3%en9] using direct cat6 cable link between the two:
scp -6 [fe80::e02a:17b4:c527:84e3%en9]:/mnt/SSD/tmp/BIG.tar . BIG.tar 84% 11GB 397.4KB/s - stalled -client_loop: send disconnect: Broken pipe scp: Connection closed
Sustained data transfer rate was between 80-100 Mbytes/s until getting stalled.
Doing the same between rpi5 and rpi4 failed at 11% file transfer, data transfer rate prior to stall was 40MBytes/s. Rpi5 also loses it's ipv6 network address binding even when idling. It's either is an issue within rpi5 ethernet stack or my rpi5 ethernet is bad. I don't have another rpi5 at hand, someone else may try reproducing this problem.I cannot replicate this since I am not able to disconnect my Pi5 but, have you tried using something like a USB -> Ethernet adapter? Would be nice to know if the problem is the hardware or something that is broken in general in the kernel with networking... I think that this still is a RP1 related problem tho.
I managed to find an old USB-A wifi adapter. The following tests rule out my PI5's hardware being an issue. In these tests two consecutive downloads were performed with PI5 and then the same repeated with one of my PI4 using the the same usb wifi adapter but using a much bigger BIG.tar!
MBP downloading BIG.tar from RPI5 over usb ethernet (wifi router turned off)
MBP downloading BIG.tar from RPI5 over usb ethernet (wifi router turned off) scp -6 [fe80::cd59:4551:bd97:3e80%en8]:/mnt/SSD/tmp/BIG.tar . BIG.tar 100% 12GB 105.8MB/s 02:00 scp -6 [fe80::cd59:4551:bd97:3e80%en8]:/mnt/SSD/tmp/BIG.tar . BIG.tar 19% 2542MB 430.8KB/s - stalled -client_loop: send disconnect: Broken pipe*** Connection stalled in just over 2 mins of data transfer
MBP downloading BIG.tar from RPI4 over usb ethernet (wifi router turned off)
scp -6 [fe80::3d8e:457f:16cd:bc97%en8]:/mnt/SSD/tmp/BIG.tar . BIG.tar 100% 54GB 41.5MB/s 22:00 scp -6 [fe80::3d8e:457f:16cd:bc97%en8]:/mnt/SSD/tmp/BIG.tar . BIG.tar 100% 54GB 42.8MB/s 21:20*** Connection stable over 43 mins of data transfer
In an earlier test I had used PI5's native Ethernet connected to the wifi router's Ethernet port for the same download test, which worked fine. The only thing different was PI's eth0 got allocated an IPV4 address by the router. An issue within Bookworm's IPV6 stack?I just tried copying over a 10GB file between my laptop (Debian) using SCP too and the Pi5 without having any issue... Also I am not sure if there are issues within Bookworm IPv6 stack. I got the idea that the RP1 stack is still not fully working well...
Either ways, could you run the command:uname -aSo we can at least understand what kernel you are running on, the output in my case is this one:
Linux raspberrypi5 6.6.22-v8-16k+ #1747 SMP PREEMPT Wed Mar 27 17:06:56 GMT 2024 aarch64 GNU/Linux
Just finished transferring 100GBs... I guess the problem is not happening here at all:alessandro@alessandro-laptop:~$ scp alessandro@raspberrypi5.local:/home/alessandro/test ./ alessandro@raspberrypi5.local's password: test 100% 100GB 93.6MB/s 18:14That's good to hear. Problem occurs at my end only when using direct cable between PI5 and any other host, for some reason
nmtuidoes not allow changing or even adding IPV4 for link-local connections or I would tried with IPV4 stack for comparison. PI5'suname:uname -a Linux rpi5 6.6.28+rpt-rpi-2712 #1 SMP PREEMPT Debian 1:6.6.28-1+rpt1 (2024-04-22) aarch64 GNU/LinuxPerhaps repeating a file transfer with IPV6 src and dst addresses may reproduce the problem I am seeing.
I was running on kernel 6.22 so I will update my system to the latest kernel 6.30 and see if anything changes (I don't remember seeing any change on eth related stuff tho)
Just tested again and I confirm to have no issues with IPv4/6 at all when sharing my network using Debian.
Are you sure your network cable is functional, also if you run "ifconfig" do you see any Tx/Rx error on the eth interface? That would mean something is going really wrong, here's my output:
eno1: flags=4099<UP,BROADCAST,MULTICAST> mtu 1500 ether 08:bf:b8:d0:3c:d0 txqueuelen 1000 (Ethernet) RX packets 19174297 bytes 26668819810 (26.6 GB) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 2227528 bytes 160733910 (160.7 MB) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0As you can see I've got no errors at all and the transfer went well (I stopped it at 26GB).
I could try on a MBP with M1 but I doubt the problem relies on MacOS.
Great suggestions. I have tried multiple cables of various lengths and the tests are with a CAT6 cert cable. Also it matters not whether it's mbp of a Debian. This time I tested between PI5 and PI4.
ifconfig -a outputs for PI5 and PI4 before the start of file transfer
RPI5
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet6 fe80::e02a:17b4:c527:84e3 prefixlen 64 scopeid 0x20<link>
ether 2c:cf:67:08:68:e9 txqueuelen 1000 (Ethernet)
RX packets 65 bytes 11160 (10.8 KiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 109 bytes 20235 (19.7 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
device interrupt 112
RPI4
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 169.254.59.177 netmask 255.255.0.0 broadcast 169.254.255.255
inet6 fe80::6617:79bc:5bd6:5117 prefixlen 64 scopeid 0x20<link>
ether dc:a6:32:b7:b3:29 txqueuelen 1000 (Ethernet)
RX packets 5465 bytes 889281 (868.4 KiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 3156 bytes 355236 (346.9 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
SCP RPI5 -> RPI4 (Bullseye Linux rpi4c 6.1.21-v8+ #1642 SMP PREEMPT Mon Apr 3 17:24:16 BST 2023 aarch64 GNU/Linux)
scp -6 BIG.tar [fe80::6617:79bc:5bd6:5117%eth0]:/mnt/SSD
BIG.tar 100% 12GB 13.1MB/s 16:18
Now this is what really happened during this SCP session:
File transfer stalled at 50% after 2mins approx, PI5's eth0 lost its IPV6 address binding, a few mins later SCP file transfer restarts PI5's eth0 IPV6 binding appears restored, it then stalls again at 96% binding lost, file restarts again after few mins binding restored. Seems PI5's eth0 is regularly losing and regaining its IPV6 address binding. I will add few PI5's iconfig outputs to show these binding being lost-restored-lost phenomenon.
RPI5
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet6 fe80::e02a:17b4:c527:84e3 prefixlen 64 scopeid 0x20<link>
ether 2c:cf:67:08:68:e9 txqueuelen 1000 (Ethernet)
RX packets 65 bytes 11160 (10.8 KiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 109 bytes 20235 (19.7 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
device interrupt 112
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
ether 2c:cf:67:08:68:e9 txqueuelen 1000 (Ethernet)
RX packets 791970 bytes 86985777 (82.9 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 9362558 bytes 13809932029 (12.8 GiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
device interrupt 112
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet6 fe80::e02a:17b4:c527:84e3 prefixlen 64 scopeid 0x20<link>
ether 2c:cf:67:08:68:e9 txqueuelen 1000 (Ethernet)
RX packets 823384 bytes 90279654 (86.0 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 9667899 bytes 14259386311 (13.2 GiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
device interrupt 112
I then did a bash script to grep PI5's eth0 inet6 address every 15 secs, here are the outputs:
link-local-test.sh
Mon 20 May 12:10:44 AEST 2024 inet6 up
Mon 20 May 12:10:59 AEST 2024 User down!!
Mon 20 May 12:11:14 AEST 2024 User down!!
Mon 20 May 12:11:29 AEST 2024 User down!!
Mon 20 May 12:11:44 AEST 2024 User down!!
Mon 20 May 12:11:59 AEST 2024 User down!!
Mon 20 May 12:12:14 AEST 2024 User down!!
Mon 20 May 12:12:29 AEST 2024 User down!!
Mon 20 May 12:12:44 AEST 2024 User down!!
Mon 20 May 12:12:59 AEST 2024 User down!!
Mon 20 May 12:13:14 AEST 2024 User down!!
Mon 20 May 12:13:29 AEST 2024 User down!!
Mon 20 May 12:13:44 AEST 2024 User down!!
Mon 20 May 12:13:59 AEST 2024 User down!!
Mon 20 May 12:14:14 AEST 2024 User down!!
Mon 20 May 12:14:29 AEST 2024 User down!!
Mon 20 May 12:14:44 AEST 2024 User down!!
Mon 20 May 12:14:59 AEST 2024 User down!!
Mon 20 May 12:15:14 AEST 2024 User down!!
Mon 20 May 12:15:29 AEST 2024 User down!!
Mon 20 May 12:15:44 AEST 2024 User down!!
Mon 20 May 12:15:59 AEST 2024 inet6 up
Mon 20 May 12:16:14 AEST 2024 inet6 up
Mon 20 May 12:16:29 AEST 2024 inet6 up
Mon 20 May 12:16:44 AEST 2024 inet6 up
Mon 20 May 12:16:59 AEST 2024 inet6 up
Mon 20 May 12:17:14 AEST 2024 inet6 up
Mon 20 May 12:17:29 AEST 2024 inet6 up
Mon 20 May 12:17:44 AEST 2024 inet6 up
Mon 20 May 12:17:59 AEST 2024 inet6 up
Mon 20 May 12:18:14 AEST 2024 inet6 up
Mon 20 May 12:18:29 AEST 2024 inet6 up
Mon 20 May 12:18:44 AEST 2024 inet6 up
Mon 20 May 12:18:59 AEST 2024 User down!!
Mon 20 May 12:19:14 AEST 2024 User down!!
Mon 20 May 12:19:29 AEST 2024 User down!!
Mon 20 May 12:19:44 AEST 2024 User down!!
Mon 20 May 12:19:59 AEST 2024 User down!!
Mon 20 May 12:20:14 AEST 2024 User down!!
Mon 20 May 12:20:29 AEST 2024 User down!!
Mon 20 May 12:20:44 AEST 2024 User down!!
Mon 20 May 12:20:59 AEST 2024 User down!!
Mon 20 May 12:21:14 AEST 2024 User down!!
Mon 20 May 12:21:29 AEST 2024 User down!!
Mon 20 May 12:21:44 AEST 2024 User down!!
Mon 20 May 12:21:59 AEST 2024 User down!!
Mon 20 May 12:22:14 AEST 2024 User down!!
Mon 20 May 12:22:29 AEST 2024 User down!!
Mon 20 May 12:22:44 AEST 2024 User down!!
Mon 20 May 12:22:59 AEST 2024 User down!!
Mon 20 May 12:23:14 AEST 2024 User down!!
Mon 20 May 12:23:29 AEST 2024 User down!!
Mon 20 May 12:23:44 AEST 2024 User down!!
Mon 20 May 12:23:59 AEST 2024 inet6 up
Mon 20 May 12:24:14 AEST 2024 inet6 up
...
...
PI5's eth0 IPV6 binding regularly flaps between 3 mins up and 5 mins down