Raspberry Pi 5 USB 3.0 Write Speed Issue
schoolpost opened this issue · comments
I have conducted a number of tests that demonstrate the Raspberry Pi 5 has a noticeably slower write speed vs that of a Pi 4 when writing to external disks ( SSD's ) via USB 3.0
it also falls well short of the practical speeds that should be achievable via USB 3.0, even accounting for protocol / transfer overheads, 400-450MB/s
My original investigation and findings are detailed here: https://forums.raspberrypi.com/viewtopic.php?t=367370
Pi 5 speeds:
pi@raspberrypi:~ $ sudo ./raw_disk /dev/sda2
Written bytes: 41943040000
Time: 152975439398ns
Speed: 0.274182 GB/s
Pi 4 speeds:
pi@raspberrypi:~ $ sudo ./raw_disk /dev/sda2
Written bytes: 41943040000
Time: 127376266995ns
Speed: 0.329285 GB/s
The benchmark application is compiled from source found here: https://stackoverflow.com/questions/71038719/how-to-reach-10gb-s-writes-in-c-c-to-nvme-disks-under-linux
Have you set the cpufreq governor to performance
prior to running your benchmark? If you use Raspberry Pi OS the default is ondemand
without io_is_busy=1
Quick tests using this SSD:
root@rpi5:/mnt/sda1# sbc-bench.sh -S
* 111.8GB "Samsung SSD 750 EVO 120GB" SSD as /dev/sda [SATA 3.1, 6.0 Gb/s (current: 6.0 Gb/s)]: behind ASMedia SATA 6Gb/s bridge (174c:55aa), 3% worn out, Driver=uas, 5Gbps (capable of 12Mbps, 480Mbps, 5Gbps), drive temp: 26°C
* 29.7GB "SanDisk SN32G" UHS SDR104 SD card as /dev/mmcblk0: date 08/2023, manfid/oemid: 0x000003/0x5344, hw/fw rev: 0x8/0x0
Testing methodology: iozone -e -I -a -s 1000M -r 16384k -i 0 -i 1 -i 2
(results KB/sec):
random random
reclen write rewrite read reread read write
1) 16384 268674 268322 388941 389261 387650 268278
2) 16384 268748 268652 389988 390306 388778 268697
3) 16384 323152 323042 390132 390299 389076 323047
-
- were with OS defaults
-
- was after
echo 1 >/sys/devices/system/cpu/cpufreq/ondemand/io_is_busy
- was after
-
- switching ASPM from OS defaults (
powersave
) todefault
:echo default > /sys/module/pcie_aspm/parameters/policy
- switching ASPM from OS defaults (
No idea (so far) how switching ASPM from powersave
to default
has an impact on idle consumption. BTW: an additional test setting ASPM to performance
ended up with same numbers than default
.
As reference numbers from my Pi 4 made directly before with same SSD, same cable, same install on same SD card: 1st time ASPM set to powersave
(OS default), 2nd to performance
:
random random
reclen write rewrite read reread read write
1) 16384 326208 319854 357959 358105 356869 319998
2) 16384 328924 323672 364055 364183 362978 323817
the practical speeds that should be achievable via USB 3.0, even accounting for protocol / transfer overheads, 400-450MB/s
Can you show any such numbers achieved over an USB3 SuperSpeed (5 GBps) connection? Me not, the best I achieved were 325-330 MB/sec at 16M blocksize with an EVO840 that shows way higher sequential transfer speeds on a SATA controller.
Forgot that I personally already measured 415/420 MB/s with modern ARM SoCs and USB3 SuperSpeed.
Retested with that EVO840 on two different USB3-to-SATA bridges and same picture: with ASPM set to powersave
write performance suffers while when set to default
this looks ok-ish to me:
random random
reclen write rewrite read reread read write
ASM1153 16384 322309 321589 389908 390133 389848 321371
JMS567 16384 313090 312219 372423 372628 372313 312648
And an additional test with a btrfs raid1 out of the two EVOs shows also the expected difference wrt ASPM:
random random
reclen write rewrite read reread read write
powersave 16384 287698 287499 370139 370178 369596 282961
performance 16384 303702 304217 387700 387697 386282 300246
root@rpi5:/mnt/btrfs-raid1# sbc-bench.sh -S
* 111.8GB "Samsung SSD 750 EVO 120GB" SSD as /dev/sda [SATA 3.1, 6.0 Gb/s (current: 6.0 Gb/s)]: behind ASMedia SATA 6Gb/s bridge (174c:55aa), 3% worn out, Driver=uas, 5Gbps (capable of 12Mbps, 480Mbps, 5Gbps), drive temp: 25°C
* 111.8GB "Samsung SSD 840 EVO 120GB" SSD as /dev/sdb [SATA 3.1, 6.0 Gb/s (current: 6.0 Gb/s)]: behind JMicron JMS567 SATA 6Gb/s bridge (152d:3562), 3% worn out, Driver=uas, 5Gbps (capable of 12Mbps, 480Mbps, 5Gbps), drive temp: 26°C
* 29.7GB "SanDisk SN32G" UHS SDR104 SD card as /dev/mmcblk0: date 08/2023, manfid/oemid: 0x000003/0x5344, hw/fw rev: 0x8/0x0
root@rpi5:/mnt/btrfs-raid1# grep btrfs /etc/mtab
/dev/sda /mnt/btrfs-raid1 btrfs rw,relatime,space_cache=v2,subvolid=5,subvol=/ 0 0
root@rpi5:/mnt/btrfs-raid1# btrfs device usage /mnt/btrfs-raid1/
/dev/sda, ID: 1
Device size: 111.79GiB
Device slack: 8.00KiB
Data,RAID1: 3.00GiB
Metadata,RAID1: 1.00GiB
System,RAID1: 32.00MiB
Unallocated: 107.76GiB
/dev/sdb, ID: 2
Device size: 111.79GiB
Device slack: 8.00KiB
Data,RAID1: 3.00GiB
Metadata,RAID1: 1.00GiB
System,RAID1: 32.00MiB
Unallocated: 107.76GiB