IONVMeController.cpp Kernel panic with CtlnaAHCIPort.kext in Big Sur 11.4

[keatliang2005]

Describe the bug

• Seems like system would kernel panic if cold boot (first boot, rest like 30 minute),
• Sleep wake starts will result in same panic
• sometime panic at verbose boot stage, or after login to desktop then it freeze. like the NVME disappear from system.

I suspect there is compatibility issue with bios and SSD firmware combo.

It work well before, until i update the NVME SSD firmware to latest version S7780102
https://media.kingston.com/support/downloads/SKC2500_S7780102_RN.pdf

but the site seems don't have a way to re-flash older version of SSD firmware

The workaround is not to put into sleep, and hard-reboot once is panic (from first cold boot). then it seems stable.

I had tried

• NVMEFix, with similar result. Panic often.
• disabled ASPM on BIOS
• disabled nvme SSD ASPM via property injection

Expected behavior

• First boot without panic
• Sleep wake on default settings without panic

Configuration

• Bios version 0067 (latest)
• HaC Mini version [e.g: 2.0]: 3.0.1
• OSX version [e.g: 10.14.6]: 11.4
• NUC model [e.g: NUC8i7HNK]: NUC8i7HVK
• Boot SSD model and protocol [e.g: Samsung EVO 860 1TB, SATA]: Kingston KC2500
• Second SSD model and protocol (if installed) [e.g: HP EX920 1TB, NVMe]: none
• First DIMM model [e.g: Patriot DDR4-2666 16GB]: Kingston KHX2666C15S4/16G
• Second DIMM model (if installed): none
• Display port(s) used [e.g: USB-C, mini DP]: mDP
• Display resolution(s) [e.g: 4K, 5K]: 2K
• Any TB3 devices: none
• Any USB devices: none
• Any Wi-Fi/Bluetooth devices [e.g:DW1820A, BCM94360CS2, Intel 8265]: Fenvi AC1200(BCM94352Z)

Panic log

panic(cpu 0 caller 0xffffff80134ce860): nvme: "Fatal error occurred. CSTS=0x1 US[1]=0x0 US[0]=0x103 VID=0x2646 DID=0x2262
. FW Revision=S7780102\n"@/System/Volumes/Data/SWE/macOS/BuildRoots/e90674e518/Library/Caches/com.apple.xbs/Sources/IONVMeFamily/IONVMeFamily-557.120.1/Common/IONVMeController.cpp:5499
Backtrace (CPU 0), Frame : Return Address
0xffffffa0b44cb960 : 0xffffff8010c8e0dd mach_kernel : _handle_debugger_trap + 0x3fd
0xffffffa0b44cb9b0 : 0xffffff8010dd4f33 mach_kernel : _kdp_i386_trap + 0x143
0xffffffa0b44cb9f0 : 0xffffff8010dc552a mach_kernel : _kernel_trap + 0x55a
0xffffffa0b44cba40 : 0xffffff8010c32a2f mach_kernel : _return_from_trap + 0xff
0xffffffa0b44cba60 : 0xffffff8010c8d8fd mach_kernel : _DebuggerTrapWithState + 0xad
0xffffffa0b44cbb80 : 0xffffff8010c8dbf3 mach_kernel : _panic_trap_to_debugger + 0x273
0xffffffa0b44cbbf0 : 0xffffff801149d81a mach_kernel : _panic + 0x54
0xffffffa0b44cbc60 : 0xffffff80134ce860 com.apple.iokit.IONVMeFamily : __ZN16IONVMeController14CommandTimeoutEP16AppleNVMeRequest.cold.1
0xffffffa0b44cbc80 : 0xffffff80134b345b com.apple.iokit.IONVMeFamily : __ZN16IONVMeController13FatalHandlingEv + 0x1af
0xffffffa0b44cbde0 : 0xffffff80113f4465 mach_kernel : __ZN18IOTimerEventSource15timeoutSignaledEPvS0_ + 0xa5
0xffffffa0b44cbe50 : 0xffffff80113f4366 mach_kernel : __ZN18IOTimerEventSource17timeoutAndReleaseEPvS0_ + 0xc6
0xffffffa0b44cbe80 : 0xffffff8010cd46c5 mach_kernel : _thread_call_delayed_timer + 0x4a5
0xffffffa0b44cbef0 : 0xffffff8010cd5634 mach_kernel : _thread_call_delayed_timer + 0x1414
0xffffffa0b44cbfa0 : 0xffffff8010c3213e mach_kernel : _call_continuation + 0x2e
      Kernel Extensions in backtrace:
         com.apple.iokit.IONVMeFamily(2.1)[38BC0D0C-B305-30F6-8A8D-6804830F891A]@0xffffff80134ac000->0xffffff80134d5fff
            dependency: com.apple.driver.AppleEFINVRAM(2.1)[423BF733-A622-3BC9-8F0F-920A2927CD35]@0xffffff80120fe000->0xffffff8012107fff
            dependency: com.apple.driver.AppleMobileFileIntegrity(1.0.5)[366C7BD4-9502-3D1C-ABBE-DE9749913D0C]@0xffffff80122c0000->0xffffff80122d5fff
            dependency: com.apple.iokit.IOPCIFamily(2.9)[99A70889-A31C-3B25-8E88-ADD3F317E4E4]@0xffffff801376a000->0xffffff8013792fff
            dependency: com.apple.iokit.IOReportFamily(47)[367CCAE7-DAB4-3424-B191-44044619D5CB]@0xffffff80137a1000->0xffffff80137a3fff
            dependency: com.apple.iokit.IOStorageFamily(2.1)[A0D72FE9-649B-316A-8B5C-934E295FF0B5]@0xffffff801386d000->0xffffff801387efff

Process name corresponding to current thread: kernel_task
Boot args: alcid=11 agdpmod=ignore -disablegfxfirmware shikigva=32 shiki-id=Mac-BE088AF8C5EB4FA2 igfxmetal=1 keepsyms=1 debug=0x146 watchdog=0 -liludbgall -v

Mac OS version:
20F71

Kernel version:
Darwin Kernel Version 20.5.0: Sat May  8 05:10:33 PDT 2021; root:xnu-7195.121.3~9/RELEASE_X86_64
Kernel UUID: 52A1E876-863E-38E3-AC80-09BBAB13B752
KernelCache slide: 0x0000000010a00000
KernelCache base:  0xffffff8010c00000
Kernel slide:      0x0000000010a10000
Kernel text base:  0xffffff8010c10000
__HIB  text base: 0xffffff8010b00000
System model name: Macmini8,1 (Mac-7BA5B2DFE22DDD8C)
System shutdown begun: NO
Panic diags file available: YES (0x0)
Hibernation exit count: 0

System uptime in nanoseconds: 185477492187
Last Sleep:           absolute           base_tsc          base_nano
  Uptime  : 0x0000002b2f51f54e
  Sleep   : 0x0000000000000000 0x0000000000000000 0x0000000000000000
  Wake    : 0x0000000000000000 0x0000000dac28ac90 0x0000000000000000

[keatliang2005]

After tinkering for days and countless of Kernel panics

• Flashing bios various bios version from 0059-0067
• Jumper Recovery

None of them resolve the kernel panics.

until i decided to look into config.plist one by one and look for possible culprit.
none of the quirks combination works

disable ASPM on drive also does not fix. i even start to think the updated firmware of the SSD drive broke the ASPM on the drive
acidanthera/bugtracker#830 (comment)

until

this prompt me looking into CtlnaAHCIPort.kext

disable loading of CtlnaAHCIPort.kext, it seems might have some conflict with original IOVNME class, the issue seems resolved.

Cold boot no longer panics, sleep wake works (Default ASPM L1 without NVMEFix).

My advise is if not using any SATA drive just disable the kext as in cause additional instability. maybe macOS 11.3+ something has changed in IONVME class is conflict with CtlnaAHCIPort.kext, oddly this happen more frequent as daily after i updated the SSD firmware.

Maybe someone is interest to look into debugging for CtlnaAHCIPort.kext if someone happen to use NVME + SATA drive.