igc driver not working on Pi5
lasselj opened this issue · comments
Describe the bug
It appears that with an Intel i225 and i226 NIC in the Pi5 both of which use the igc driver it is not possible to read any address in the ioremapped address space. I should mention that the Intel i210 which uses the igb driver works without a hitch (the architecture of the i210 and the i225/226 is remarkably similar).
In igc_probe the call to pci_resource_start and pci_resource_len return the correct addresses of the memory also observed in lspci. The call to ioremap returns a valid address, however all calls to igc_rd32 always without exception return 0x0. Similarly a direct call to readl in the remapped space returns 0 no matter the register.
For fun I have also tried to map the pci resource using the pci_iomap call used in the igb driver but it doesn't make a difference.
Loading the driver fails with error -13 (IGC_ERR_SWFW_SYNC) which is essentially the first time the driver needs to read a set bit in a read call to acquire the software/firmware semaphore in igc_get_hw_semaphore_i225.
The same device (225/226 with igc) works fine on a CM4.
I've set the dtparam pciex1 and I've tried using the dtoverlay pcie_32bit_dma but it does not make a difference and the latter is not required on the CM4.
#help :-)
Steps to reproduce the behaviour
(with a 225/226 nic in the PCIe slot)
modprobe igc
Device (s)
Raspberry Pi 5
System
Generated using pi-gen, https://github.com/RPi-Distro/pi-gen, f19ee211ddafcae300827f953d143de92a5c6624, stage4
2024/04/20 11:53:30
Copyright (c) 2012 Broadcom
version d1744d21 (release) (embedded)
Linux pi5-pcie 6.6.28-v8-16k+ #2 SMP PREEMPT Sun Apr 28 08:30:38 BST 2024 aarch64 GNU/Linux
Logs
Probably not worth a lot, but for the purposes of giving insight into the call stack:
[12587.426063] Intel(R) 2.5G Ethernet Linux Driver
[12587.426069] Copyright(c) 2018 Intel Corporation.
[12587.426171] igc 0000:01:00.0: PCIe PTM not supported by PCIe bus/controller
[12587.426244] igc_probe (1): 0x1b00000000-0x100000
[12587.426246] igc_probe (2): 0xffffc00083000000
[12587.426247] igc_probe (3): 0x0-0x0
[12587.426250] igc_get_invariants_base(1): 0
[12587.426251] igc_get_invariants_base(2): 0
[12587.426252] igc_get_invariants_base(3): 0
[12587.426253] igc_rd32 (1): 0x10! (0x83000000 8 1)
[12587.426256] igc_rd32 (2): 0x0 (4)
[12587.426257] igc_get_invariants_base(4): 0
[12587.426258] igc_rd32 (1): 0x10! (0x83000000 8 1)
[12587.426260] igc_rd32 (2): 0x0 (4)
[12587.426261] igc_rd32 (1): 0x8! (0x83000000 8 1)
[12587.426263] igc_rd32 (2): 0x0 (4)
[12587.426265] igc_rd32 (1): 0x5820! (0x83000000 8 1)
[12587.426267] igc_rd32 (2): 0x0 (4)
[12587.426268] igc_phy_hw_reset(1): 0
[12587.426269] igc_acquire_phy_base(1): 2i
[12587.426270] igc_get_hw_semaphore_i225(1): 123
[12587.426272] igc_rd32 (1): 0x5b50! (0x83000000 8 1)
[12587.426274] igc_rd32 (2): 0x0 (4)
[12587.426275] igc_rd32 (1): 0x5b50! (0x83000000 8 1)
[12587.426277] igc_rd32 (2): 0x0 (4)
[12587.426278] igc_get_hw_semaphore_i225(2): 0
[12587.426279] igc_get_hw_semaphore_i225(5): 65
[12587.426280] igc_rd32 (1): 0x5b50! (0x83000000 8 1)
[12587.426282] igc_rd32 (2): 0x0 (4)
[12587.426283] igc_rd32 (1): 0x5b50! (0x83000000 8 1)
[12587.426285] igc_rd32 (2): 0x0 (4)
[12587.426336] igc_rd32 (1): 0x5b50! (0x83000000 8 1)
[12587.426342] igc_rd32 (2): 0x0 (4)
...
[12587.430035] igc_rd32 (2): 0x0 (4)
[12587.430086] igc_rd32 (1): 0x5b50! (0x83000000 8 1)
[12587.430096] igc_rd32 (2): 0x0 (4)
[12587.430097] igc_rd32 (1): 0x5b50! (0x83000000 8 1)
[12587.430099] igc_rd32 (2): 0x0 (4)
[12587.430150] igc_rd32 (1): 0x5b50! (0x83000000 8 1)
[12587.430156] igc_rd32 (2): 0x0 (4)
[12587.430157] igc_get_hw_semaphore_i225(6): 123
[12587.430158] igc_acquire_swfw_sync_i225(1): -13
[12587.430159] igc_phy_hw_reset(2): -13
[12587.430160] igc_init_phy_params_base(1): -13
[12587.430161] igc_get_invariants_base(5): -13
[12587.430162] err = ei->get_invariants(hw): -13
[12587.430173] igc: probe of 0000:01:00.0 failed with error -13
lspci -vvvvvvvvvvv :
0000:01:00.0 Ethernet controller: Intel Corporation Ethernet Controller I226-LM (rev 04)
Subsystem: Intel Corporation Ethernet Controller I226-LM
Control: I/O- Mem+ BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- SERR- <PERR- INTx-
Interrupt: pin A routed to IRQ 37
Region 0: Memory at 1b00000000 (32-bit, non-prefetchable) [virtual] [size=1M]
Region 3: Memory at 1b00200000 (32-bit, non-prefetchable) [size=16K]
Expansion ROM at 1b00100000 [virtual] [disabled] [size=1M]
Capabilities: [40] Power Management version 3
Flags: PMEClk- DSI+ D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=1 PME-
Capabilities: [50] MSI: Enable- Count=1/1 Maskable+ 64bit+
Address: 0000000000000000 Data: 0000
Masking: 00000000 Pending: 00000000
Capabilities: [70] MSI-X: Enable- Count=5 Masked-
Vector table: BAR=3 offset=00000000
PBA: BAR=3 offset=00002000
Capabilities: [a0] Express (v2) Endpoint, MSI 00
DevCap: MaxPayload 512 bytes, PhantFunc 0, Latency L0s <512ns, L1 <64us
ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 0W
DevCtl: CorrErr+ NonFatalErr+ FatalErr+ UnsupReq+
RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+ FLReset-
MaxPayload 512 bytes, MaxReadReq 512 bytes
DevSta: CorrErr- NonFatalErr- FatalErr- UnsupReq- AuxPwr+ TransPend-
LnkCap: Port #0, Speed 5GT/s, Width x1, ASPM L1, Exit Latency L1 <4us
ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
LnkCtl: ASPM L1 Enabled; RCB 64 bytes, Disabled- CommClk+
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 5GT/s, Width x1
TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
DevCap2: Completion Timeout: Range ABCD, TimeoutDis+ NROPrPrP- LTR+
10BitTagComp- 10BitTagReq- OBFF Not Supported, ExtFmt- EETLPPrefix-
EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
FRS- TPHComp- ExtTPHComp-
AtomicOpsCap: 32bit- 64bit- 128bitCAS-
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis- LTR+ 10BitTagReq- OBFF Disabled,
AtomicOpsCtl: ReqEn-
LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-
Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
Compliance Preset/De-emphasis: -6dB de-emphasis, 0dB preshoot
LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete- EqualizationPhase1-
EqualizationPhase2- EqualizationPhase3- LinkEqualizationRequest-
Retimer- 2Retimers- CrosslinkRes: unsupported
Capabilities: [100 v2] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr-
CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+
AERCap: First Error Pointer: 00, ECRCGenCap+ ECRCGenEn- ECRCChkCap+ ECRCChkEn-
MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap-
HeaderLog: 00000000 00000000 00000000 00000000
Capabilities: [140 v1] Device Serial Number f0-b2-b9-ff-ff-11-b3-b8
Capabilities: [1c0 v1] Latency Tolerance Reporting
Max snoop latency: 0ns
Max no snoop latency: 0ns
Capabilities: [1f0 v1] Precision Time Measurement
PTMCap: Requester:+ Responder:- Root:-
PTMClockGranularity: 4ns
PTMControl: Enabled:- RootSelected:-
PTMEffectiveGranularity: Unknown
Capabilities: [1e0 v1] L1 PM Substates
L1SubCap: PCI-PM_L1.2+ PCI-PM_L1.1+ ASPM_L1.2+ ASPM_L1.1+ L1_PM_Substates+
PortCommonModeRestoreTime=55us PortTPowerOnTime=70us
L1SubCtl1: PCI-PM_L1.2- PCI-PM_L1.1- ASPM_L1.2- ASPM_L1.1-
T_CommonMode=55us LTR1.2_Threshold=131072ns
L1SubCtl2: T_PwrOn=70us
Kernel modules: igc
Additional context
WhatsApp.Video.2024-04-27.at.14.03.42.mp4
@popcornmix @ghollingworth one doesn't like to tag people to draw people into one's problems, but it feels like there could be an underlying problem here. Any ideas? Again, to summarise: i225/226 ok on CM4 (igc driver), i210 ok on Pi5 (igb driver), i225/226 not ok on Pi5. @ahmadexp captured PCIe debug pex files and I can forward if that assists. I am happy to send as many i225/226 NICs to Cambridge as it takes. I could wake up people at Intel too, but I really don't think this is their gig....
...either way, people who don't cooperate with the accurate time mafia can easily find themselves sent back to the Jurassic age... :-)
Not sure you are pinging the right people. I've had very little involvement with pcie.
It's possible @P33M may know more about this.
If you have analyser logs then please upload them.
Although, you should probably find out why igc_acquire_swfw_sync_i225 is failing with -EACCES.
@P33M Thank you for taking the time.
As I indicated above, it fails with -IGC_ERR_SWFW_SYNC (also -13) and it does so because from igc_probe it's the first time the driver requires something other than 0 from the rd32 calls to continue:
https://elixir.bootlin.com/linux/v6.6.28/source/drivers/net/ethernet/intel/igc/igc_i225.c#L81
(from igb / i210, but identical in every way and shareable)
Every rd32 call returns 0 in the ioremap'ed space.
Find please analyser logs here: Analyser Logs.zip
You need this to review: https://www.teledynelecroy.com/protocolanalyzer/pci-express/resources/analysis-software
The Pi 5 trace is self-explanatory - it looks like the memory space just reads back as zero. Possibly some ROM routine hasn't completed?
The CM4 trace appears to be missing downstream traffic until about 4.6s in. Without the requester info I can't tell if all the 1-dword completions full of 0 it's returning are related to the same semaphore register. The last wakeup where traffic from the RC is also recorded is at ~4.68s which also corresponds to the semaphore check succeeding.
If I assume the link has been up since before the trace started, that's a very long initialisation time - CM4 might just be slower and get lucky with timeouts. The Pi 5 trace doesn't cover the same length of time between config space init and driver polling - it's <1s of traffic.
To reduce the analyser churn to a minimum when capturing snapshot traffic, what I do in Recording Options is filter out:
- SKP OS
- UpdateFC DLLP
And turn off ASPM in /boot/firmware/cmdline.txt by adding pcie_aspm=off.
If I assume the link has been up since before the trace started, that's a very long initialisation time - CM4 might just be slower and get lucky with timeouts. The Pi 5 trace doesn't cover the same length of time between config space init and driver polling - it's <1s of traffic.
Yeah, I will mention that I have tried to make the timeout value 1, 2 and 3 orders of magnitude higher here:
https://elixir.bootlin.com/linux/v6.6.28/source/drivers/net/ethernet/intel/igc/igc_i225.c#L43
For some reason I don't quite understand the hw->nvm.word_size has dramatically different values on the Pi5 and the CM4.
There's a smattering of other reports (on PC) where the same failure mode crops up randomly.
https://www.mail-archive.com/debian-bugs-dist@lists.debian.org/msg1856970.html
https://bbs.archlinux.org/viewtopic.php?id=272719
The pattern is "doesn't work" -> some unrelated change happens -> "oh it works now".
I have a slot breakout board that lets me control power sequencing (sort of) - manually holding off perst# with respect to power sequencing, and delaying 12V/3v3 vs 3v3aux doesn't change things, so it's not a waiting-for-slow-firmware issue.
After diff'ing the config space on CM4 vs Pi 5 the only difference is what BAR0-5 get set to. It works on Pi 5 if I offset the 32-bit outbound address space to something other than zero.
The ROM/firmware isn't picking up on the fact that the BAR got written first with 0xffffffff then with 0 as the actual base address. And apparently doesn't care about the "MEM space enable" bit in the device control register until it sees a nonzero value in BAR0.