Kernel Panic: Kernel Data Abort (OpenZFS 2.1.0, M1 Ultra)
dmzimmerman opened this issue · comments
On my M1 Ultra with 128GB RAM, I've been getting panics like the one below.
The interesting thing about them is that what's in the backtrace is com.apple.filesystems.hfs.kext - but I'm pretty sure it's a ZFS panic, and it's just manifesting this way because I have com.apple.mimic turned on.
panic(cpu 3 caller 0xfffffe001b9a9ab0): Kernel data abort. at pc 0xfffffe001cd4ef54, lr 0xfffffe001cd4ef3c (saved state: 0xfffffe7fe1fcf6c0)
x0: 0x0000000000000000 x1: 0x0000000000300000 x2: 0x0000000000000000 x3: 0xfffffe14cd08aca0
x4: 0x0000000000000000 x5: 0x0000000000300000 x6: 0x0000000000000000 x7: 0xfffffe7fe1fcf868
x8: 0x0000000000000000 x9: 0x91ffe945c10c0090 x10: 0x0000000000000000 x11: 0x0000000000000000
x12: 0x0000000000000001 x13: 0x0000000000000014 x14: 0x0000000000044b7b x15: 0x00000000ffffffff
x16: 0x8bbd7e001b419780 x17: 0xfffffe001e4089d8 x18: 0x0000000000000000 x19: 0xfffffe24cc62c400
x20: 0xfffffe2999925700 x21: 0x0000000000000000 x22: 0x0000000000300000 x23: 0x0000000000000000
x24: 0x0000000000000000 x25: 0xfffffe7fe1fcfb20 x26: 0xfffffe001eca9920 x27: 0x0000000000000000
x28: 0x0000000003018001 fp: 0xfffffe7fe1fcfa40 lr: 0xfffffe001cd4ef3c sp: 0xfffffe7fe1fcfa10
pc: 0xfffffe001cd4ef54 cpsr: 0x60401208 esr: 0x96000045 far: 0x0000000000000000
Debugger message: panic
Memory ID: 0x6
OS release type: User
OS version: 21G72
Kernel version: Darwin Kernel Version 21.6.0: Sat Jun 18 17:07:22 PDT 2022; root:xnu-8020.140.41~1/RELEASE_ARM64_T6000
Fileset Kernelcache UUID: 5CFF5B82BEB8FAE1094CE5733AD4234D
Kernel UUID: 43BBF43C-8008-3830-8A00-C8706889EEA1
iBoot version: iBoot-7459.141.1
secure boot?: YES
Paniclog version: 13
KernelCache slide: 0x0000000013870000
KernelCache base: 0xfffffe001a874000
Kernel slide: 0x0000000014034000
Kernel text base: 0xfffffe001b038000
Kernel text exec slide: 0x000000001411c000
Kernel text exec base: 0xfffffe001b120000
mach_absolute_time: 0x15a419e072
Epoch Time: sec usec
Boot : 0x62e101c3 0x0001eef4
Sleep : 0x00000000 0x00000000
Wake : 0x00000000 0x00000000
Calendar: 0x62e110da 0x00058855
Zone info:
Zone map: 0xfffffe100040c000 - 0xfffffe300040c000
. VM : 0xfffffe100040c000 - 0xfffffe14cd0d8000
. RO : 0xfffffe14cd0d8000 - 0xfffffe1666a70000
. GEN0 : 0xfffffe1666a70000 - 0xfffffe1b3373c000
. GEN1 : 0xfffffe1b3373c000 - 0xfffffe2000408000
. GEN2 : 0xfffffe2000408000 - 0xfffffe24cd0d4000
. GEN3 : 0xfffffe24cd0d4000 - 0xfffffe2999da0000
. DATA : 0xfffffe2999da0000 - 0xfffffe300040c000
Metadata: 0xfffffe8fb1dec000 - 0xfffffe8fb9dec000
Bitmaps : 0xfffffe8fb9dec000 - 0xfffffe8fd9dec000
CORE 0 PVH locks held: None
CORE 1 PVH locks held: None
CORE 2 PVH locks held: None
CORE 3 PVH locks held: None
CORE 4 PVH locks held: None
CORE 5 PVH locks held: None
CORE 6 PVH locks held: None
CORE 7 PVH locks held: None
CORE 8 PVH locks held: None
CORE 9 PVH locks held: None
CORE 10 PVH locks held: None
CORE 11 PVH locks held: None
CORE 12 PVH locks held: None
CORE 13 PVH locks held: None
CORE 14 PVH locks held: None
CORE 15 PVH locks held: None
CORE 16 PVH locks held: None
CORE 17 PVH locks held: None
CORE 18 PVH locks held: None
CORE 19 PVH locks held: None
CORE 0: PC=0xfffffe001b120be8, LR=0xfffffe001b2ac6fc, FP=0xfffffe7fe278ef30
CORE 1: PC=0x000000019b60fecc, LR=0x000000019e27db24, FP=0x000000016d21a420
CORE 2: PC=0xfffffe001b1af0c0, LR=0xfffffe001b1af0bc, FP=0xfffffe7fe18aff00
CORE 3 is the one that panicked. Check the full backtrace for details.
CORE 4: PC=0xfffffe001b1af0c0, LR=0xfffffe001b1af0bc, FP=0xfffffe7fe272ff00
CORE 5: PC=0xfffffe001b1af0c0, LR=0xfffffe001b1af0bc, FP=0xfffffe7fe20cff00
CORE 6: PC=0xfffffe001b1af0bc, LR=0xfffffe001b1af0bc, FP=0xfffffe7fe288ff00
CORE 7: PC=0xfffffe001b1af0bc, LR=0xfffffe001b1af0bc, FP=0xfffffe7fe294ff00
CORE 8: PC=0xfffffe001b1af0bc, LR=0xfffffe001b1af0bc, FP=0xfffffe87c7d6ff00
CORE 9: PC=0xfffffe001b1af0bc, LR=0xfffffe001b1af0bc, FP=0xfffffe87c868ff00
CORE 10: PC=0xfffffe001b19d2f0, LR=0xfffffe001dcf972c, FP=0xfffffe87c738efb0
CORE 11: PC=0xfffffe001dcd81cc, LR=0xfffffe001dcd812c, FP=0xfffffe7fe178ef70
CORE 12: PC=0xfffffe001b1af0bc, LR=0xfffffe001b1af0bc, FP=0xfffffe7fe20eff00
CORE 13: PC=0xfffffe001b1af0bc, LR=0xfffffe001b1af0bc, FP=0xfffffe87c694ff00
CORE 14: PC=0xfffffe001b1af0bc, LR=0xfffffe001b1af0bc, FP=0xfffffe7fe29aff00
CORE 15: PC=0xfffffe001b1af0bc, LR=0xfffffe001b1af0bc, FP=0xfffffe87c6acff00
CORE 16: PC=0xfffffe001b1af0bc, LR=0xfffffe001b1af0bc, FP=0xfffffe7fe1e0ff00
CORE 17: PC=0xfffffe001b1af0bc, LR=0xfffffe001b1af0bc, FP=0xfffffe8fafc0ff00
CORE 18: PC=0xfffffe001b1af0bc, LR=0xfffffe001b1af0bc, FP=0xfffffe7fe2aaff00
CORE 19: PC=0xfffffe001b1af0bc, LR=0xfffffe001b1af0bc, FP=0xfffffe8fb0baff00
Compressor Info: 0% of compressed pages limit (OK) and 0% of segments limit (OK) with 0 swapfiles and OK swap space
Panicked task 0xfffffe1b33760678: 0 pages, 1375 threads: pid 0: kernel_task
Panicked thread: 0xfffffe1fffaeb0c0, backtrace: 0xfffffe7fe1fced80, tid: 2043
lr: 0xfffffe001b179268 fp: 0xfffffe7fe1fcedf0
lr: 0xfffffe001b178f30 fp: 0xfffffe7fe1fcee60
lr: 0xfffffe001b2bf180 fp: 0xfffffe7fe1fcee80
lr: 0xfffffe001b2b0f10 fp: 0xfffffe7fe1fceef0
lr: 0xfffffe001b2aeaf4 fp: 0xfffffe7fe1fcefb0
lr: 0xfffffe001b1277f8 fp: 0xfffffe7fe1fcefc0
lr: 0xfffffe001b178bb4 fp: 0xfffffe7fe1fcf360
lr: 0xfffffe001b178bb4 fp: 0xfffffe7fe1fcf3d0
lr: 0xfffffe001b9a10e0 fp: 0xfffffe7fe1fcf3f0
lr: 0xfffffe001b9a9ab0 fp: 0xfffffe7fe1fcf570
lr: 0xfffffe001b2b0d10 fp: 0xfffffe7fe1fcf5e0
lr: 0xfffffe001b2aedf8 fp: 0xfffffe7fe1fcf6a0
lr: 0xfffffe001b1277f8 fp: 0xfffffe7fe1fcf6b0
lr: 0xfffffe001cd4ef3c fp: 0xfffffe7fe1fcfa40
lr: 0xfffffe001cd7edc0 fp: 0xfffffe7fe1fcfaa0
lr: 0xfffffe001cd3e1dc fp: 0xfffffe7fe1fcfca0
lr: 0xfffffe001cd41e18 fp: 0xfffffe7fe1fcfd10
lr: 0xfffffe001cd41c2c fp: 0xfffffe7fe1fcfd40
lr: 0xfffffe001b3e1f80 fp: 0xfffffe7fe1fcfde0
lr: 0xfffffe001b3e1478 fp: 0xfffffe7fe1fcfe50
lr: 0xfffffe001b3e2ef8 fp: 0xfffffe7fe1fcfeb0
lr: 0xfffffe001b3e2c88 fp: 0xfffffe7fe1fcff20
lr: 0xfffffe001b130e78 fp: 0x0000000000000000
Kernel Extensions in backtrace:
com.apple.filesystems.hfs.kext(583.100.10)[B4C4BBD9-E1DB-3999-96C5-07C2DD3FDED1]@0xfffffe001cd38a40->0xfffffe001cd87683
dependency: com.apple.filesystems.hfs.encodings.kext(1)[CB32F375-A103-33CB-AA27-05258BE85789]@0xfffffe001cd87690->0xfffffe001cd885bf
last started kext at 3118929074: com.apple.filesystems.smbfs 4.0 (addr 0xfffffe001b01cb50, size 65122)
loaded kexts:
org.openzfsonosx.zfs 2.1.0
com.apple.filesystems.smbfs 4.0
com.apple.driver.AppleTopCaseHIDEventDriver 5450.8
com.apple.filesystems.autofs 3.0
com.apple.driver.AppleBiometricServices 1
com.apple.driver.CoreKDL 1
com.apple.AppleEthernetAquantiaAqtionFirmware 1.0.36
com.apple.driver.AppleAHCIPort 351.100.4
com.apple.driver.usb.AppleUSBHostBillboardDevice 1.0
com.apple.driver.DiskImages.ReadWriteDiskImage 493.0.0
com.apple.driver.DiskImages.UDIFDiskImage 493.0.0
com.apple.driver.DiskImages.RAMBackingStore 493.0.0
com.apple.driver.DiskImages.FileBackingStore 493.0.0
com.apple.iokit.SCSITaskUserClient 456.140.3
com.apple.driver.BCMWLANFirmware4387.Hashstore 1
com.apple.driver.AppleAOPVoiceTrigger 140.1
com.apple.driver.SEPHibernation 1
com.apple.driver.AppleUSBDeviceNCM 5.0.0
com.apple.driver.AppleFileSystemDriver 3.0.1
com.apple.driver.AppleThunderboltIP 4.0.3
com.apple.nke.l2tp 1.9
com.apple.filesystems.tmpfs 1
com.apple.filesystems.lifs 1
com.apple.filesystems.apfs 1934.141.2
com.apple.IOTextEncryptionFamily 1.0.0
com.apple.filesystems.hfs.kext 583.100.10
com.apple.security.BootPolicy 1
com.apple.BootCache 40
com.apple.AppleFSCompression.AppleFSCompressionTypeZlib 1.0.0
com.apple.AppleFSCompression.AppleFSCompressionTypeDataless 1.0.0d1
com.apple.AppleEmbeddedSimpleSPINORFlasher 1
com.apple.driver.AppleSmartIO2 1
com.apple.driver.ApplePMP 1
com.apple.driver.AppleSN012776Amp 550.3
com.apple.driver.AppleCS42L84Audio 550.3
com.apple.driver.AppleT6000SOCTuner 1
com.apple.driver.AppleT6002CLPC 1
com.apple.driver.AppleSmartBatteryManager 161.0.0
com.apple.driver.ApplePMPFirmware 1
com.apple.driver.AppleMCDP29XXUpdateSupport 1
com.apple.driver.AppleS5L8920XPWM 1.0.0d1
com.apple.driver.AppleJPEGDriver 4.8.1
com.apple.driver.AudioDMAController-T600x 160.2
com.apple.driver.AppleT6001PMGR 1
com.apple.driver.AppleS5L8960XNCO 1
com.apple.driver.AppleS8000AES 1
com.apple.driver.AppleSamsungSerial 1.0.0d1
com.apple.driver.AppleSerialShim 1
com.apple.driver.AppleProResHW 177.0.0
com.apple.AGXG13X 190.22
com.apple.driver.AppleAVE2 560.5.0
com.apple.driver.AppleAVD 566
com.apple.driver.AppleSPIMC 1
com.apple.driver.AppleSDXC 3.2.1
com.apple.driver.usb.AppleSynopsysUSB40XHCI 1
com.apple.driver.AppleMobileDispT600X-DCP 140.0
com.apple.driver.AppleSPMIPMU 1.0.1
com.apple.driver.AppleEventLogHandler 1
com.apple.driver.AppleS8000DWI 1.0.0d1
com.apple.driver.AppleInterruptControllerV2 1.0.0d1
com.apple.driver.AppleT8110DART 1
com.apple.driver.AppleBluetoothModule 1
com.apple.driver.AppleBCMWLANBusInterfacePCIe 1
com.apple.driver.AppleS5L8940XI2C 1.0.0d2
com.apple.driver.AppleT6000DART 1
com.apple.driver.AppleT6000 1
com.apple.driver.AppleM68Buttons 1.0.0d1
com.apple.iokit.IOUserEthernet 1.0.1
com.apple.driver.usb.AppleUSBUserHCI 1
com.apple.iokit.IOKitRegistryCompatibility 1
com.apple.iokit.EndpointSecurity 1
com.apple.driver.AppleDiskImages2 126.141.2
com.apple.AppleSystemPolicy 2.0.0
com.apple.nke.applicationfirewall 402
com.apple.kec.InvalidateHmac 1
com.apple.kec.AppleEncryptedArchive 1
com.apple.driver.driverkit.serial 6.0.0
com.apple.driver.AppleActuatorDriver 5460.1
com.apple.driver.AppleMultitouchDriver 5460.1
com.apple.driver.AppleInputDeviceSupport 5460.1
com.apple.driver.AppleHSBluetoothDriver 5450.8
com.apple.driver.IOBluetoothHIDDriver 9.0.0
com.apple.iokit.IOAVBFamily 1040.6
com.apple.plugin.IOgPTPPlugin 1040.3
com.apple.iokit.IOEthernetAVBController 1.1.0
com.apple.iokit.IOAHCIBlockStorage 333.140.2
com.apple.kext.triggers 1.0
com.apple.driver.AppleUSBMergeNub 900.4.2
com.apple.driver.usb.cdc.ecm 5.0.0
com.apple.driver.usb.cdc.acm 5.0.0
com.apple.driver.usb.serial 6.0.0
com.apple.driver.usb.cdc.ncm 5.0.0
com.apple.driver.AppleMesaSEPDriver 100.99
com.apple.iokit.IOBiometricFamily 1
com.apple.iokit.IOAHCIFamily 297
com.apple.driver.AppleEthernetAquantiaAqtion 1.0.64
com.apple.driver.usb.IOUSBHostHIDDevice 1.2
com.apple.driver.AppleThunderboltPCIUpAdapter 4.1.1
com.apple.driver.AppleThunderboltDPOutAdapter 8.5.1
com.apple.driver.DiskImages.KernelBacked 493.0.0
com.apple.driver.usb.cdc 5.0.0
com.apple.driver.AppleUSBAudio 416.2
com.apple.iokit.IOAudioFamily 340.2
com.apple.vecLib.kext 1.2.0
com.apple.driver.AppleXsanScheme 3
com.apple.driver.AppleAOPAudio 150.2
com.apple.driver.AppleSEPHDCPManager 1.0.1
com.apple.driver.AppleTrustedAccessory 1
com.apple.iokit.AppleSEPGenericTransfer 1
com.apple.driver.usb.AppleEmbeddedUSBXHCIPCI 1
com.apple.driver.usb.AppleUSBXHCIPCI 1.2
com.apple.driver.AppleEmbeddedUSB 1
com.apple.driver.AppleConvergedIPCOLYBTControl 1
com.apple.driver.AppleConvergedPCI 1
com.apple.driver.AppleBluetoothDebug 1
com.apple.driver.usb.networking 5.0.0
com.apple.nke.ppp 1.9
com.apple.driver.AppleBSDKextStarter 3
com.apple.filesystems.hfs.encodings.kext 1
com.apple.driver.AppleThunderboltPCIDownAdapter 4.1.1
com.apple.driver.AppleThunderboltUSBDownAdapter 1.0.4
com.apple.driver.AppleThunderboltDPInAdapter 8.5.1
com.apple.driver.AppleThunderboltDPAdapterFamily 8.5.1
com.apple.driver.AppleDiagnosticDataAccessReadOnly 1.0.0
com.apple.driver.AppleBTM 1.0.1
com.apple.driver.AppleI2CEthernetAquantia 1.0.0
com.apple.driver.AppleDCPDPTXProxy 1.0.0
com.apple.driver.DCPDPFamilyProxy 1
com.apple.driver.AppleHPM 3.4.4
com.apple.driver.AppleCSEmbeddedAudio 550.3
com.apple.driver.AppleEmbeddedAudio 550.3
com.apple.iokit.AppleARMIISAudio 140.2
com.apple.driver.ApplePassthroughPPM 3.0
com.apple.driver.usb.AppleUSBHostPacketFilter 1.0
com.apple.AGXFirmwareKextG13XRTBuddy 190.22
com.apple.AGXFirmwareKextRTBuddy64 190.22
com.apple.iokit.IONVMeFamily 2.1.0
com.apple.driver.AppleNANDConfigAccess 1.0.0
com.apple.driver.AppleStockholmControl 1.0.0
com.apple.driver.DCPAVFamilyProxy 1
com.apple.driver.AppleMCA2-T600x 640.5
com.apple.driver.AppleEmbeddedAudioLibs 140.1
com.apple.driver.ApplePMGR 1
com.apple.driver.AppleSPU 1
com.apple.iokit.IOGPUFamily 35.29
com.apple.driver.AppleT6000TypeCPhy 1
com.apple.driver.AppleT8103TypeCPhy 1
com.apple.driver.AppleUSBXDCIARM 1.0
com.apple.driver.AppleUSBXDCI 1.0
com.apple.iokit.IOUSBDeviceFamily 2.0.0
com.apple.driver.usb.AppleSynopsysUSBXHCI 1
com.apple.driver.usb.AppleUSBXHCI 1.2
com.apple.driver.AppleEmbeddedUSBHost 1
com.apple.driver.usb.AppleUSBHub 1.2
com.apple.driver.usb.AppleUSBHostCompositeDevice 1.2
com.apple.iokit.IOMobileGraphicsFamily-DCP 343.0.0
com.apple.driver.AppleDCP 1
com.apple.driver.AppleFirmwareKit 1
com.apple.driver.AppleFirmwareUpdateKext 1
com.apple.iokit.IOMobileGraphicsFamily 343.0.0
com.apple.driver.AppleDialogPMU 1.0.1
com.apple.driver.AppleSPMI 1.0.1
com.apple.driver.AppleSART 1
com.apple.driver.AppleARMWatchdogTimer 1
com.apple.driver.AppleDisplayCrossbar 1.0.0
com.apple.iokit.IODisplayPortFamily 1.0.0
com.apple.driver.AppleTypeCPhy 1
com.apple.driver.AppleThunderboltNHI 7.2.81
com.apple.driver.AppleT6000PCIeC 1
com.apple.iokit.IOThunderboltFamily 9.3.3
com.apple.driver.ApplePIODMA 1
com.apple.driver.AppleT600xPCIe 1
com.apple.driver.AppleMultiFunctionManager 1
com.apple.driver.AppleBluetoothDebugService 1
com.apple.driver.AppleBCMWLANCore 1.0.0
com.apple.iokit.IO80211Family 1200.12.2b2
com.apple.driver.IOImageLoader 1.0.0
com.apple.driver.AppleOLYHAL 1
com.apple.driver.corecapture 1.0.4
com.apple.driver.AppleEthernetAquantiaAqtionPortMonitor 1.0.0
com.apple.driver.AppleEmbeddedPCIE 1
com.apple.driver.AppleGPIOICController 1.0.2
com.apple.driver.AppleFireStormErrorHandler 1
com.apple.driver.AppleMobileApNonce 1
com.apple.iokit.IOTimeSyncFamily 1040.3
com.apple.driver.DiskImages 493.0.0
com.apple.iokit.IOGraphicsFamily 597
com.apple.iokit.IOBluetoothSerialManager 9.0.0
com.apple.iokit.IOBluetoothHostControllerUSBTransport 9.0.0
com.apple.iokit.IOBluetoothHostControllerUARTTransport 9.0.0
com.apple.iokit.IOBluetoothHostControllerTransport 9.0.0
com.apple.driver.IOBluetoothHostControllerPCIeTransport 9.0.0
com.apple.iokit.IOBluetoothFamily 9.0.0
com.apple.driver.FairPlayIOKit 68.16.0
com.apple.iokit.CSRBluetoothHostControllerUSBTransport 9.0.0
com.apple.iokit.BroadcomBluetoothHostControllerUSBTransport 9.0.0
com.apple.driver.AppleSSE 1.0
com.apple.driver.AppleSEPKeyStore 2
com.apple.driver.AppleUSBTDM 533.120.2
com.apple.iokit.IOUSBMassStorageDriver 210.120.3
com.apple.iokit.IOPCIFamily 2.9
com.apple.iokit.IOSCSIBlockCommandsDevice 456.140.3
com.apple.iokit.IOSCSIArchitectureModelFamily 456.140.3
com.apple.driver.AppleIPAppender 1.0
com.apple.driver.AppleFDEKeyStore 28.30
com.apple.driver.AppleEffaceableStorage 1.0
com.apple.driver.AppleCredentialManager 1.0
com.apple.driver.KernelRelayHost 1
com.apple.iokit.IOUSBHostFamily 1.2
com.apple.driver.AppleUSBHostMergeProperties 1.2
com.apple.driver.usb.AppleUSBCommon 1.0
com.apple.driver.AppleSMC 3.1.9
com.apple.driver.RTBuddy 1.0.0
com.apple.driver.AppleEmbeddedTempSensor 1.0.0
com.apple.driver.AppleARMPMU 1.0
com.apple.iokit.IOAccessoryManager 1.0.0
com.apple.driver.AppleOnboardSerial 1.0
com.apple.iokit.IOSkywalkFamily 1.0
com.apple.driver.mDNSOffloadUserClient 1.0.1b8
com.apple.iokit.IONetworkingFamily 3.4
com.apple.iokit.IOSerialFamily 11
com.apple.driver.AppleSEPManager 1.0.1
com.apple.driver.AppleA7IOP 1.0.2
com.apple.driver.IOSlaveProcessor 1
com.apple.driver.AppleBiometricSensor 2
com.apple.iokit.IOHIDFamily 2.0.0
com.apple.iokit.CoreAnalyticsFamily 1
com.apple.driver.AppleANELoadBalancer 5.68.2
com.apple.driver.AppleH11ANEInterface 5.68.0
com.apple.AUC 1.0
com.apple.iokit.IOAVFamily 1.0.0
com.apple.iokit.IOHDCPFamily 1.0.0
com.apple.iokit.IOCECFamily 1
com.apple.iokit.IOAudio2Family 1.0
com.apple.driver.AppleIISController 140.1
com.apple.driver.AppleAudioClockLibs 140.1
com.apple.driver.AppleM2ScalerCSCDriver 265.0.0
com.apple.iokit.IOSurface 302.14
com.apple.driver.IODARTFamily 1
com.apple.security.quarantine 4
com.apple.security.sandbox 300.0
com.apple.kext.AppleMatch 1.0.0d1
com.apple.driver.AppleMobileFileIntegrity 1.0.5
com.apple.security.AppleImage4 4.2.0
com.apple.kext.CoreTrust 1
com.apple.iokit.IOCryptoAcceleratorFamily 1.0.1
com.apple.driver.AppleARMPlatform 1.0.2
com.apple.iokit.IOStorageFamily 2.1
com.apple.iokit.IOSlowAdaptiveClockingFamily 1.0.0
com.apple.iokit.IOReportFamily 47
com.apple.kec.pthread 1
com.apple.kec.Libm 1
com.apple.kec.corecrypto 12.0
** Stackshot Succeeded ** Bytes Traced 342779 (Uncompressed 1006768) **
Another one:
panic(cpu 2 caller 0xfffffe00159a9ab0): Kernel data abort. at pc 0xfffffe0016d4ef54, lr 0xfffffe0016d4ef3c (saved state: 0xfffffe686beef510)
x0: 0x0000000000000000 x1: 0x0000000000300000 x2: 0x0000000000000000 x3: 0xfffffe1000542ca0
x4: 0x0000000000000000 x5: 0x0000000000300000 x6: 0x0000000000000000 x7: 0xfffffe686beef6b8
x8: 0x0000000000000000 x9: 0x2f7cb76c5cdb00c5 x10: 0x0000000000000000 x11: 0x0000000000000000
x12: 0x0000000000000001 x13: 0x0000000000000014 x14: 0x000000000003726c x15: 0x00000000ffffffff
x16: 0x4c9c7e0015419780 x17: 0xfffffe00184089d8 x18: 0x0000000000000000 x19: 0xfffffe299942ac00
x20: 0xfffffe2001226a80 x21: 0x0000000000000000 x22: 0x0000000000300000 x23: 0x0000000000000000
x24: 0x0000000000000000 x25: 0xfffffe686beef970 x26: 0xfffffe0018ca9920 x27: 0x0000000000000000
x28: 0x0000000003018001 fp: 0xfffffe686beef890 lr: 0xfffffe0016d4ef3c sp: 0xfffffe686beef860
pc: 0xfffffe0016d4ef54 cpsr: 0x60401208 esr: 0x96000046 far: 0x0000000000000000
Debugger message: panic
Memory ID: 0x6
OS release type: User
OS version: 21G72
Kernel version: Darwin Kernel Version 21.6.0: Sat Jun 18 17:07:22 PDT 2022; root:xnu-8020.140.41~1/RELEASE_ARM64_T6000
Fileset Kernelcache UUID: 5CFF5B82BEB8FAE1094CE5733AD4234D
Kernel UUID: 43BBF43C-8008-3830-8A00-C8706889EEA1
iBoot version: iBoot-7459.141.1
secure boot?: YES
Paniclog version: 13
KernelCache slide: 0x000000000d870000
KernelCache base: 0xfffffe0014874000
Kernel slide: 0x000000000e034000
Kernel text base: 0xfffffe0015038000
Kernel text exec slide: 0x000000000e11c000
Kernel text exec base: 0xfffffe0015120000
mach_absolute_time: 0x2fec9d56c1
Epoch Time: sec usec
Boot : 0x62e23e3d 0x000d8bf1
Sleep : 0x00000000 0x00000000
Wake : 0x00000000 0x00000000
Calendar: 0x62e25fb4 0x00041d68
Zone info:
Zone map: 0xfffffe1000524000 - 0xfffffe3000524000
. VM : 0xfffffe1000524000 - 0xfffffe14cd1f0000
. RO : 0xfffffe14cd1f0000 - 0xfffffe1666b88000
. GEN0 : 0xfffffe1666b88000 - 0xfffffe1b33854000
. GEN1 : 0xfffffe1b33854000 - 0xfffffe2000520000
. GEN2 : 0xfffffe2000520000 - 0xfffffe24cd1ec000
. GEN3 : 0xfffffe24cd1ec000 - 0xfffffe2999eb8000
. DATA : 0xfffffe2999eb8000 - 0xfffffe3000524000
Metadata: 0xfffffe3000534000 - 0xfffffe3008534000
Bitmaps : 0xfffffe3008534000 - 0xfffffe3028534000
CORE 0 PVH locks held: None
CORE 1 PVH locks held: None
CORE 2 PVH locks held: None
CORE 3 PVH locks held: None
CORE 4 PVH locks held: None
CORE 5 PVH locks held: None
CORE 6 PVH locks held: None
CORE 7 PVH locks held: None
CORE 8 PVH locks held: None
CORE 9 PVH locks held: None
CORE 10 PVH locks held: None
CORE 11 PVH locks held: None
CORE 12 PVH locks held: None
CORE 13 PVH locks held: None
CORE 14 PVH locks held: None
CORE 15 PVH locks held: None
CORE 16 PVH locks held: None
CORE 17 PVH locks held: None
CORE 18 PVH locks held: None
CORE 19 PVH locks held: None
CORE 0: PC=0xfffffe00151dfac4, LR=0xfffffe00146d87d0, FP=0xfffffe608b64df10
CORE 1: PC=0xfffffe00151af0bc, LR=0xfffffe00151af0bc, FP=0xfffffe686db2ff00
CORE 2 is the one that panicked. Check the full backtrace for details.
CORE 3: PC=0xfffffe00151af0bc, LR=0xfffffe00151af0bc, FP=0xfffffe687292ff00
CORE 4: PC=0xfffffe00151af0bc, LR=0xfffffe00151af0bc, FP=0xfffffe686b9cff00
CORE 5: PC=0xfffffe00151af0bc, LR=0xfffffe00151af0bc, FP=0xfffffe686bacff00
CORE 6: PC=0xfffffe00152b3df8, LR=0xfffffe00152b3df4, FP=0xfffffe686ce2fe90
CORE 7: PC=0xfffffe00151af0bc, LR=0xfffffe00151af0bc, FP=0xfffffe686bbcff00
CORE 8: PC=0xfffffe00151af0bc, LR=0xfffffe00151af0bc, FP=0xfffffe608b6cff00
CORE 9: PC=0xfffffe00151af0bc, LR=0xfffffe00151af0bc, FP=0xfffffe686becff00
CORE 10: PC=0xfffffe00151af0bc, LR=0xfffffe00151af0bc, FP=0xfffffe686c2aff00
CORE 11: PC=0xfffffe00151af0bc, LR=0xfffffe00151af0bc, FP=0xfffffe608b76ff00
CORE 12: PC=0xfffffe00151af0bc, LR=0xfffffe00151af0bc, FP=0xfffffe686c0eff00
CORE 13: PC=0xfffffe00151af0bc, LR=0xfffffe00151af0bc, FP=0xfffffe686bf0ff00
CORE 14: PC=0xfffffe00151af0bc, LR=0xfffffe00151af0bc, FP=0xfffffe686cf8ff00
CORE 15: PC=0xfffffe00151af0bc, LR=0xfffffe00151af0bc, FP=0xfffffe686c18ff00
CORE 16: PC=0xfffffe00152b3df8, LR=0xfffffe00152b3df4, FP=0xfffffe686cecfe90
CORE 17: PC=0xfffffe00151af0bc, LR=0xfffffe00151af0bc, FP=0xfffffe686ec2ff00
CORE 18: PC=0xfffffe00151af0bc, LR=0xfffffe00151af0bc, FP=0xfffffe60883aff00
CORE 19: PC=0xfffffe00151af0bc, LR=0xfffffe00151af0bc, FP=0xfffffe686ce0ff00
Compressor Info: 0% of compressed pages limit (OK) and 0% of segments limit (OK) with 0 swapfiles and OK swap space
Panicked task 0xfffffe1ffeb8b470: 101 pages, 1 threads: pid 3097: rsync
Panicked thread: 0xfffffe24cb7fa8a0, backtrace: 0xfffffe686beeebd0, tid: 68762
lr: 0xfffffe0015179268 fp: 0xfffffe686beeec40
lr: 0xfffffe0015178f30 fp: 0xfffffe686beeecb0
lr: 0xfffffe00152bf180 fp: 0xfffffe686beeecd0
lr: 0xfffffe00152b0f10 fp: 0xfffffe686beeed40
lr: 0xfffffe00152aeaf4 fp: 0xfffffe686beeee00
lr: 0xfffffe00151277f8 fp: 0xfffffe686beeee10
lr: 0xfffffe0015178bb4 fp: 0xfffffe686beef1b0
lr: 0xfffffe0015178bb4 fp: 0xfffffe686beef220
lr: 0xfffffe00159a10e0 fp: 0xfffffe686beef240
lr: 0xfffffe00159a9ab0 fp: 0xfffffe686beef3c0
lr: 0xfffffe00152b0d10 fp: 0xfffffe686beef430
lr: 0xfffffe00152aedf8 fp: 0xfffffe686beef4f0
lr: 0xfffffe00151277f8 fp: 0xfffffe686beef500
lr: 0xfffffe0016d4ef3c fp: 0xfffffe686beef890
lr: 0xfffffe0016d7edc0 fp: 0xfffffe686beef8f0
lr: 0xfffffe0016d3e1dc fp: 0xfffffe686beefaf0
lr: 0xfffffe0016d41e18 fp: 0xfffffe686beefb60
lr: 0xfffffe0016d41c2c fp: 0xfffffe686beefb90
lr: 0xfffffe001540eb64 fp: 0xfffffe686beefc20
lr: 0xfffffe001540ea18 fp: 0xfffffe686beefc80
lr: 0xfffffe001568c038 fp: 0xfffffe686beefce0
lr: 0xfffffe001568cf60 fp: 0xfffffe686beefdb0
lr: 0xfffffe00157e2614 fp: 0xfffffe686beefe50
lr: 0xfffffe00152aebc8 fp: 0xfffffe686beeff10
lr: 0xfffffe00151277f8 fp: 0xfffffe686beeff20
Kernel Extensions in backtrace:
com.apple.filesystems.hfs.kext(583.100.10)[B4C4BBD9-E1DB-3999-96C5-07C2DD3FDED1]@0xfffffe0016d38a40->0xfffffe0016d87683
dependency: com.apple.filesystems.hfs.encodings.kext(1)[CB32F375-A103-33CB-AA27-05258BE85789]@0xfffffe0016d87690->0xfffffe0016d885bf
last started kext at 2084133148: com.apple.filesystems.smbfs 4.0 (addr 0xfffffe001501cb50, size 65122)
loaded kexts:
org.openzfsonosx.zfs 2.1.0
com.apple.filesystems.smbfs 4.0
com.apple.driver.AppleTopCaseHIDEventDriver 5450.8
com.apple.filesystems.autofs 3.0
com.apple.AppleEthernetAquantiaAqtionFirmware 1.0.36
com.apple.driver.AppleBiometricServices 1
com.apple.driver.CoreKDL 1
com.apple.driver.AppleAHCIPort 351.100.4
com.apple.driver.usb.AppleUSBHostBillboardDevice 1.0
com.apple.driver.DiskImages.ReadWriteDiskImage 493.0.0
com.apple.driver.DiskImages.UDIFDiskImage 493.0.0
com.apple.driver.DiskImages.RAMBackingStore 493.0.0
com.apple.driver.DiskImages.FileBackingStore 493.0.0
com.apple.iokit.SCSITaskUserClient 456.140.3
com.apple.driver.BCMWLANFirmware4387.Hashstore 1
com.apple.driver.SEPHibernation 1
com.apple.driver.AppleUSBDeviceNCM 5.0.0
com.apple.driver.AppleThunderboltIP 4.0.3
com.apple.driver.AppleFileSystemDriver 3.0.1
com.apple.nke.l2tp 1.9
com.apple.filesystems.tmpfs 1
com.apple.filesystems.lifs 1
com.apple.filesystems.apfs 1934.141.2
com.apple.IOTextEncryptionFamily 1.0.0
com.apple.filesystems.hfs.kext 583.100.10
com.apple.security.BootPolicy 1
com.apple.BootCache 40
com.apple.AppleFSCompression.AppleFSCompressionTypeZlib 1.0.0
com.apple.AppleFSCompression.AppleFSCompressionTypeDataless 1.0.0d1
com.apple.driver.AppleSmartIO2 1
com.apple.driver.ApplePMP 1
com.apple.AppleEmbeddedSimpleSPINORFlasher 1
com.apple.driver.AppleSN012776Amp 550.3
com.apple.driver.AppleCS42L84Audio 550.3
com.apple.driver.AppleT6000SOCTuner 1
com.apple.driver.AppleT6002CLPC 1
com.apple.driver.AppleSmartBatteryManager 161.0.0
com.apple.driver.AppleAOPVoiceTrigger 140.1
com.apple.driver.ApplePMPFirmware 1
com.apple.driver.AppleMCDP29XXUpdateSupport 1
com.apple.driver.AppleS5L8920XPWM 1.0.0d1
com.apple.driver.AppleJPEGDriver 4.8.1
com.apple.driver.AudioDMAController-T600x 160.2
com.apple.driver.AppleT6001PMGR 1
com.apple.driver.AppleS5L8960XNCO 1
com.apple.driver.AppleS8000AES 1
com.apple.driver.AppleSamsungSerial 1.0.0d1
com.apple.driver.AppleSerialShim 1
com.apple.driver.AppleProResHW 177.0.0
com.apple.AGXG13X 190.22
com.apple.driver.AppleAVE2 560.5.0
com.apple.driver.AppleAVD 566
com.apple.driver.AppleSPIMC 1
com.apple.driver.AppleSDXC 3.2.1
com.apple.driver.usb.AppleSynopsysUSB40XHCI 1
com.apple.driver.AppleMobileDispT600X-DCP 140.0
com.apple.driver.AppleSPMIPMU 1.0.1
com.apple.driver.AppleEventLogHandler 1
com.apple.driver.AppleS8000DWI 1.0.0d1
com.apple.driver.AppleInterruptControllerV2 1.0.0d1
com.apple.driver.AppleT8110DART 1
com.apple.driver.AppleBluetoothModule 1
com.apple.driver.AppleBCMWLANBusInterfacePCIe 1
com.apple.driver.AppleS5L8940XI2C 1.0.0d2
com.apple.driver.AppleT6000DART 1
com.apple.driver.AppleT6000 1
com.apple.driver.AppleM68Buttons 1.0.0d1
com.apple.iokit.IOUserEthernet 1.0.1
com.apple.driver.usb.AppleUSBUserHCI 1
com.apple.iokit.IOKitRegistryCompatibility 1
com.apple.iokit.EndpointSecurity 1
com.apple.driver.AppleDiskImages2 126.141.2
com.apple.AppleSystemPolicy 2.0.0
com.apple.nke.applicationfirewall 402
com.apple.kec.InvalidateHmac 1
com.apple.kec.AppleEncryptedArchive 1
com.apple.driver.driverkit.serial 6.0.0
com.apple.driver.AppleActuatorDriver 5460.1
com.apple.iokit.IOAVBFamily 1040.6
com.apple.plugin.IOgPTPPlugin 1040.3
com.apple.iokit.IOEthernetAVBController 1.1.0
com.apple.driver.AppleMultitouchDriver 5460.1
com.apple.driver.AppleInputDeviceSupport 5460.1
com.apple.driver.AppleHSBluetoothDriver 5450.8
com.apple.driver.IOBluetoothHIDDriver 9.0.0
com.apple.iokit.IOAHCIBlockStorage 333.140.2
com.apple.kext.triggers 1.0
com.apple.driver.AppleUSBMergeNub 900.4.2
com.apple.driver.usb.cdc.ncm 5.0.0
com.apple.driver.usb.cdc.ecm 5.0.0
com.apple.driver.usb.cdc.acm 5.0.0
com.apple.driver.usb.serial 6.0.0
com.apple.driver.AppleMesaSEPDriver 100.99
com.apple.iokit.IOBiometricFamily 1
com.apple.iokit.IOAHCIFamily 297
com.apple.driver.AppleThunderboltPCIUpAdapter 4.1.1
com.apple.driver.AppleThunderboltDPOutAdapter 8.5.1
com.apple.driver.AppleEthernetAquantiaAqtion 1.0.64
com.apple.driver.usb.IOUSBHostHIDDevice 1.2
com.apple.driver.DiskImages.KernelBacked 493.0.0
com.apple.driver.usb.cdc 5.0.0
com.apple.driver.AppleUSBAudio 416.2
com.apple.iokit.IOAudioFamily 340.2
com.apple.vecLib.kext 1.2.0
com.apple.driver.AppleXsanScheme 3
com.apple.driver.AppleSEPHDCPManager 1.0.1
com.apple.driver.AppleTrustedAccessory 1
com.apple.iokit.AppleSEPGenericTransfer 1
com.apple.driver.usb.AppleEmbeddedUSBXHCIPCI 1
com.apple.driver.usb.AppleUSBXHCIPCI 1.2
com.apple.driver.AppleEmbeddedUSB 1
com.apple.driver.AppleConvergedIPCOLYBTControl 1
com.apple.driver.AppleConvergedPCI 1
com.apple.driver.AppleBluetoothDebug 1
com.apple.driver.usb.networking 5.0.0
com.apple.driver.AppleThunderboltUSBDownAdapter 1.0.4
com.apple.driver.AppleThunderboltDPInAdapter 8.5.1
com.apple.driver.AppleThunderboltDPAdapterFamily 8.5.1
com.apple.driver.AppleThunderboltPCIDownAdapter 4.1.1
com.apple.nke.ppp 1.9
com.apple.driver.AppleBSDKextStarter 3
com.apple.filesystems.hfs.encodings.kext 1
com.apple.driver.AppleBTM 1.0.1
com.apple.driver.AppleDiagnosticDataAccessReadOnly 1.0.0
com.apple.driver.AppleDCPDPTXProxy 1.0.0
com.apple.driver.DCPDPFamilyProxy 1
com.apple.driver.AppleHPM 3.4.4
com.apple.driver.AppleI2CEthernetAquantia 1.0.0
com.apple.driver.AppleCSEmbeddedAudio 550.3
com.apple.driver.ApplePassthroughPPM 3.0
com.apple.driver.usb.AppleUSBHostPacketFilter 1.0
com.apple.driver.AppleAOPAudio 150.2
com.apple.driver.AppleEmbeddedAudio 550.3
com.apple.iokit.AppleARMIISAudio 140.2
com.apple.AGXFirmwareKextG13XRTBuddy 190.22
com.apple.AGXFirmwareKextRTBuddy64 190.22
com.apple.driver.DCPAVFamilyProxy 1
com.apple.iokit.IONVMeFamily 2.1.0
com.apple.driver.AppleNANDConfigAccess 1.0.0
com.apple.driver.AppleMCA2-T600x 640.5
com.apple.driver.AppleEmbeddedAudioLibs 140.1
com.apple.driver.ApplePMGR 1
com.apple.driver.AppleSPU 1
com.apple.iokit.IOGPUFamily 35.29
com.apple.driver.AppleT6000TypeCPhy 1
com.apple.driver.AppleT8103TypeCPhy 1
com.apple.driver.AppleUSBXDCIARM 1.0
com.apple.driver.AppleUSBXDCI 1.0
com.apple.iokit.IOUSBDeviceFamily 2.0.0
com.apple.driver.usb.AppleSynopsysUSBXHCI 1
com.apple.driver.usb.AppleUSBXHCI 1.2
com.apple.driver.AppleEmbeddedUSBHost 1
com.apple.driver.usb.AppleUSBHub 1.2
com.apple.driver.usb.AppleUSBHostCompositeDevice 1.2
com.apple.driver.AppleStockholmControl 1.0.0
com.apple.iokit.IOMobileGraphicsFamily-DCP 343.0.0
com.apple.driver.AppleDCP 1
com.apple.driver.AppleFirmwareKit 1
com.apple.driver.AppleFirmwareUpdateKext 1
com.apple.iokit.IOMobileGraphicsFamily 343.0.0
com.apple.driver.AppleDialogPMU 1.0.1
com.apple.driver.AppleSPMI 1.0.1
com.apple.driver.AppleSART 1
com.apple.driver.AppleARMWatchdogTimer 1
com.apple.driver.AppleDisplayCrossbar 1.0.0
com.apple.iokit.IODisplayPortFamily 1.0.0
com.apple.driver.AppleTypeCPhy 1
com.apple.driver.AppleThunderboltNHI 7.2.81
com.apple.driver.AppleT6000PCIeC 1
com.apple.iokit.IOThunderboltFamily 9.3.3
com.apple.driver.ApplePIODMA 1
com.apple.driver.AppleT600xPCIe 1
com.apple.driver.AppleMultiFunctionManager 1
com.apple.driver.AppleBluetoothDebugService 1
com.apple.driver.AppleBCMWLANCore 1.0.0
com.apple.iokit.IO80211Family 1200.12.2b2
com.apple.driver.IOImageLoader 1.0.0
com.apple.driver.AppleOLYHAL 1
com.apple.driver.corecapture 1.0.4
com.apple.driver.AppleEthernetAquantiaAqtionPortMonitor 1.0.0
com.apple.driver.AppleEmbeddedPCIE 1
com.apple.driver.AppleGPIOICController 1.0.2
com.apple.driver.AppleFireStormErrorHandler 1
com.apple.driver.AppleMobileApNonce 1
com.apple.iokit.IOTimeSyncFamily 1040.3
com.apple.driver.DiskImages 493.0.0
com.apple.iokit.IOGraphicsFamily 597
com.apple.iokit.IOBluetoothSerialManager 9.0.0
com.apple.iokit.IOBluetoothHostControllerUSBTransport 9.0.0
com.apple.iokit.IOBluetoothHostControllerUARTTransport 9.0.0
com.apple.iokit.IOBluetoothHostControllerTransport 9.0.0
com.apple.driver.IOBluetoothHostControllerPCIeTransport 9.0.0
com.apple.iokit.IOBluetoothFamily 9.0.0
com.apple.driver.FairPlayIOKit 68.16.0
com.apple.iokit.CSRBluetoothHostControllerUSBTransport 9.0.0
com.apple.iokit.BroadcomBluetoothHostControllerUSBTransport 9.0.0
com.apple.driver.AppleSSE 1.0
com.apple.driver.AppleSEPKeyStore 2
com.apple.driver.AppleUSBTDM 533.120.2
com.apple.iokit.IOUSBMassStorageDriver 210.120.3
com.apple.iokit.IOPCIFamily 2.9
com.apple.iokit.IOSCSIBlockCommandsDevice 456.140.3
com.apple.iokit.IOSCSIArchitectureModelFamily 456.140.3
com.apple.driver.AppleIPAppender 1.0
com.apple.driver.AppleFDEKeyStore 28.30
com.apple.driver.AppleEffaceableStorage 1.0
com.apple.driver.AppleCredentialManager 1.0
com.apple.driver.KernelRelayHost 1
com.apple.iokit.IOUSBHostFamily 1.2
com.apple.driver.AppleUSBHostMergeProperties 1.2
com.apple.driver.usb.AppleUSBCommon 1.0
com.apple.driver.AppleSMC 3.1.9
com.apple.driver.RTBuddy 1.0.0
com.apple.driver.AppleEmbeddedTempSensor 1.0.0
com.apple.driver.AppleARMPMU 1.0
com.apple.iokit.IOAccessoryManager 1.0.0
com.apple.driver.AppleOnboardSerial 1.0
com.apple.iokit.IOSkywalkFamily 1.0
com.apple.driver.mDNSOffloadUserClient 1.0.1b8
com.apple.iokit.IONetworkingFamily 3.4
com.apple.iokit.IOSerialFamily 11
com.apple.driver.AppleSEPManager 1.0.1
com.apple.driver.AppleA7IOP 1.0.2
com.apple.driver.IOSlaveProcessor 1
com.apple.driver.AppleBiometricSensor 2
com.apple.iokit.IOHIDFamily 2.0.0
com.apple.iokit.CoreAnalyticsFamily 1
com.apple.driver.AppleANELoadBalancer 5.68.2
com.apple.driver.AppleH11ANEInterface 5.68.0
com.apple.AUC 1.0
com.apple.iokit.IOAVFamily 1.0.0
com.apple.iokit.IOHDCPFamily 1.0.0
com.apple.iokit.IOCECFamily 1
com.apple.iokit.IOAudio2Family 1.0
com.apple.driver.AppleIISController 140.1
com.apple.driver.AppleAudioClockLibs 140.1
com.apple.driver.AppleM2ScalerCSCDriver 265.0.0
com.apple.iokit.IOSurface 302.14
com.apple.driver.IODARTFamily 1
com.apple.security.quarantine 4
com.apple.security.sandbox 300.0
com.apple.kext.AppleMatch 1.0.0d1
com.apple.driver.AppleMobileFileIntegrity 1.0.5
com.apple.security.AppleImage4 4.2.0
com.apple.kext.CoreTrust 1
com.apple.iokit.IOCryptoAcceleratorFamily 1.0.1
com.apple.driver.AppleARMPlatform 1.0.2
com.apple.iokit.IOStorageFamily 2.1
com.apple.iokit.IOSlowAdaptiveClockingFamily 1.0.0
com.apple.iokit.IOReportFamily 47
com.apple.kec.pthread 1
com.apple.kec.Libm 1
com.apple.kec.corecrypto 12.0
** Stackshot Succeeded ** Bytes Traced 342095 (Uncompressed 1005120) **
Another one (different process panicking this time):
panic(cpu 1 caller 0xfffffe00299a9ab0): Kernel data abort. at pc 0xfffffe00292ad71c, lr 0x6293fe00292ad6c0 (saved state: 0xfffffe7fde02f080)
x0: 0x0000000000000000 x1: 0xfffffe166d5daae0 x2: 0xfffffe7fdbc84000 x3: 0x000000000000c000
x4: 0xfffffe14d3b92cd0 x5: 0x0000000100000000 x6: 0xfffffe7fde02f108 x7: 0xfffffe7fde02f100
x8: 0xfffffe002cec5000 x9: 0xfffffe002cc65000 x10: 0x0000000000000000 x11: 0x0000000000000000
x12: 0x0000000000000000 x13: 0xfffffe87ec0bc000 x14: 0x0000000000000001 x15: 0x00000000000000a8
x16: 0x89c9fe00292ad6a4 x17: 0xfffffe002c8cae20 x18: 0x0000000000000000 x19: 0x0000000000000000
x20: 0xfffffe166d27bc00 x21: 0xfffffe200785de60 x22: 0xfffffe166dc455c0 x23: 0xfffffe1b3cb24000
x24: 0x0000000000000000 x25: 0xfffffe002cec5000 x26: 0x0000000000000001 x27: 0x0000000000000000
x28: 0x0000000000000000 fp: 0xfffffe7fde02f3f0 lr: 0x6293fe00292ad6c0 sp: 0xfffffe7fde02f3d0
pc: 0xfffffe00292ad71c cpsr: 0x60401208 esr: 0x96000006 far: 0x000000000000000b
Debugger message: panic
Memory ID: 0x6
OS release type: User
OS version: 21G72
Kernel version: Darwin Kernel Version 21.6.0: Sat Jun 18 17:07:22 PDT 2022; root:xnu-8020.140.41~1/RELEASE_ARM64_T6000
Fileset Kernelcache UUID: 5CFF5B82BEB8FAE1094CE5733AD4234D
Kernel UUID: 43BBF43C-8008-3830-8A00-C8706889EEA1
iBoot version: iBoot-7459.141.1
secure boot?: YES
Paniclog version: 13
KernelCache slide: 0x0000000021870000
KernelCache base: 0xfffffe0028874000
Kernel slide: 0x0000000022034000
Kernel text base: 0xfffffe0029038000
Kernel text exec slide: 0x000000002211c000
Kernel text exec base: 0xfffffe0029120000
mach_absolute_time: 0x42edf334fb
Epoch Time: sec usec
Boot : 0x62e77d5a 0x0000f05d
Sleep : 0x00000000 0x00000000
Wake : 0x00000000 0x00000000
Calendar: 0x62e7ac19 0x000c5d6b
Zone info:
Zone map: 0xfffffe1006f1c000 - 0xfffffe3006f1c000
. VM : 0xfffffe1006f1c000 - 0xfffffe14d3be8000
. RO : 0xfffffe14d3be8000 - 0xfffffe166d580000
. GEN0 : 0xfffffe166d580000 - 0xfffffe1b3a24c000
. GEN1 : 0xfffffe1b3a24c000 - 0xfffffe2006f18000
. GEN2 : 0xfffffe2006f18000 - 0xfffffe24d3be4000
. GEN3 : 0xfffffe24d3be4000 - 0xfffffe29a08b0000
. DATA : 0xfffffe29a08b0000 - 0xfffffe3006f1c000
Metadata: 0xfffffe87e40bc000 - 0xfffffe87ec0bc000
Bitmaps : 0xfffffe87ec0bc000 - 0xfffffe880c0bc000
CORE 0 PVH locks held: None
CORE 1 PVH locks held: None
CORE 2 PVH locks held: None
CORE 3 PVH locks held: None
CORE 4 PVH locks held: None
CORE 5 PVH locks held: None
CORE 6 PVH locks held: None
CORE 7 PVH locks held: None
CORE 8 PVH locks held: None
CORE 9 PVH locks held: None
CORE 10 PVH locks held: None
CORE 11 PVH locks held: None
CORE 12 PVH locks held: None
CORE 13 PVH locks held: None
CORE 14 PVH locks held: None
CORE 15 PVH locks held: None
CORE 16 PVH locks held: None
CORE 17 PVH locks held: None
CORE 18 PVH locks held: None
CORE 19 PVH locks held: None
CORE 0: PC=0x00007ff899cb007c, LR=0x00007ff89bd3a0ac, FP=0x00000002048fa6e0
CORE 1 is the one that panicked. Check the full backtrace for details.
CORE 2: PC=0xfffffe00291af0c0, LR=0xfffffe00291af0bc, FP=0xfffffe7fddccff00
CORE 3: PC=0xfffffe00291af0c0, LR=0xfffffe00291af0bc, FP=0xfffffe8ff674ff00
CORE 4: PC=0xfffffe00291af0bc, LR=0xfffffe00291af0bc, FP=0xfffffe7fde08ff00
CORE 5: PC=0xfffffe00291af0c0, LR=0xfffffe00291af0bc, FP=0xfffffe8810d4ff00
CORE 6: PC=0xfffffe00291af0c0, LR=0xfffffe00291af0bc, FP=0xfffffe87c2f2ff00
CORE 7: PC=0xfffffe00291af0c0, LR=0xfffffe00291af0bc, FP=0xfffffe8ff80cff00
CORE 8: PC=0xfffffe00291af0c0, LR=0xfffffe00291af0bc, FP=0xfffffe7fddbeff00
CORE 9: PC=0xfffffe00291af0c0, LR=0xfffffe00291af0bc, FP=0xfffffe7fdda8ff00
CORE 10: PC=0xfffffe00291dfcb0, LR=0xfffffe00291dfcb0, FP=0xfffffe7fdd44f980
CORE 11: PC=0x000000019e05a0d4, LR=0x000000019e1fed8c, FP=0x000000016b0d5cc0
CORE 12: PC=0xfffffe00291af0bc, LR=0xfffffe00291af0bc, FP=0xfffffe7fdd26ff00
CORE 13: PC=0xfffffe00291af0bc, LR=0xfffffe00291af0bc, FP=0xfffffe7fdcceff00
CORE 14: PC=0xfffffe00291af0bc, LR=0xfffffe00291af0bc, FP=0xfffffe7fdd4eff00
CORE 15: PC=0xfffffe00292b3df8, LR=0xfffffe00292b3df4, FP=0xfffffe8ff6c0fe90
CORE 16: PC=0xfffffe00291af0c0, LR=0xfffffe00291af0bc, FP=0xfffffe881060ff00
CORE 17: PC=0xfffffe00291af0bc, LR=0xfffffe00291af0bc, FP=0xfffffe87c152ff00
CORE 18: PC=0xfffffe00291af0bc, LR=0xfffffe00291af0bc, FP=0xfffffe7fdc78ff00
CORE 19: PC=0xfffffe00291af0bc, LR=0xfffffe00291af0bc, FP=0xfffffe8ff948ff00
Compressor Info: 0% of compressed pages limit (OK) and 0% of segments limit (OK) with 0 swapfiles and OK swap space
Panicked task 0xfffffe1b36f56df8: 804 pages, 6 threads: pid 24350: NetAuthSysAgent
Panicked thread: 0xfffffe24d572c820, backtrace: 0xfffffe7fde02e740, tid: 204587
lr: 0xfffffe0029179268 fp: 0xfffffe7fde02e7b0
lr: 0xfffffe0029178f30 fp: 0xfffffe7fde02e820
lr: 0xfffffe00292bf180 fp: 0xfffffe7fde02e840
lr: 0xfffffe00292b0f10 fp: 0xfffffe7fde02e8b0
lr: 0xfffffe00292aeaf4 fp: 0xfffffe7fde02e970
lr: 0xfffffe00291277f8 fp: 0xfffffe7fde02e980
lr: 0xfffffe0029178bb4 fp: 0xfffffe7fde02ed20
lr: 0xfffffe0029178bb4 fp: 0xfffffe7fde02ed90
lr: 0xfffffe00299a10e0 fp: 0xfffffe7fde02edb0
lr: 0xfffffe00299a9ab0 fp: 0xfffffe7fde02ef30
lr: 0xfffffe00292b0d10 fp: 0xfffffe7fde02efa0
lr: 0xfffffe00292aec64 fp: 0xfffffe7fde02f060
lr: 0xfffffe00291277f8 fp: 0xfffffe7fde02f070
lr: 0xfffffe00292ad6c0 fp: 0xfffffe7fde02f3f0
lr: 0xfffffe00292ad6c0 fp: 0xfffffe7fde02f420
lr: 0xfffffe002be23484 fp: 0xfffffe7fde02f6e0
lr: 0xfffffe00293f1234 fp: 0xfffffe7fde02f970
lr: 0xfffffe00293f2a20 fp: 0xfffffe7fde02fd70
lr: 0xfffffe00293f26d8 fp: 0xfffffe7fde02fdb0
lr: 0xfffffe00297e2614 fp: 0xfffffe7fde02fe50
lr: 0xfffffe00292aebc8 fp: 0xfffffe7fde02ff10
lr: 0xfffffe00291277f8 fp: 0xfffffe7fde02ff20
Kernel Extensions in backtrace:
com.apple.filesystems.smbfs(4.0)[C35FF09E-CA77-3337-9F5B-39538E87FF32]@0xfffffe002be0d890->0xfffffe002be70c93
dependency: com.apple.kec.corecrypto(12.0)[CD18ACA5-5618-3CCF-9232-A8592E5CF16E]@0xfffffe002bd52000->0xfffffe002bd9e877
dependency: com.apple.kext.triggers(1.0)[7C659EAA-EEC9-3109-8DB4-E93FB895DEDD]@0xfffffe002be7acf0->0xfffffe002be7da37
last started kext at 3075925156: com.apple.filesystems.smbfs 4.0 (addr 0xfffffe002901cb50, size 65122)
loaded kexts:
org.openzfsonosx.zfs 2.1.0
com.apple.filesystems.smbfs 4.0
com.apple.driver.AppleTopCaseHIDEventDriver 5450.8
com.apple.filesystems.autofs 3.0
com.apple.AppleEthernetAquantiaAqtionFirmware 1.0.36
com.apple.driver.AppleBiometricServices 1
com.apple.driver.CoreKDL 1
com.apple.driver.AppleAHCIPort 351.100.4
com.apple.driver.usb.AppleUSBHostBillboardDevice 1.0
com.apple.driver.DiskImages.ReadWriteDiskImage 493.0.0
com.apple.driver.DiskImages.UDIFDiskImage 493.0.0
com.apple.driver.DiskImages.RAMBackingStore 493.0.0
com.apple.driver.DiskImages.FileBackingStore 493.0.0
com.apple.iokit.SCSITaskUserClient 456.140.3
com.apple.driver.BCMWLANFirmware4387.Hashstore 1
com.apple.driver.SEPHibernation 1
com.apple.driver.AppleUSBDeviceNCM 5.0.0
com.apple.driver.AppleThunderboltIP 4.0.3
com.apple.driver.AppleFileSystemDriver 3.0.1
com.apple.nke.l2tp 1.9
com.apple.filesystems.tmpfs 1
com.apple.filesystems.lifs 1
com.apple.filesystems.apfs 1934.141.2
com.apple.IOTextEncryptionFamily 1.0.0
com.apple.filesystems.hfs.kext 583.100.10
com.apple.security.BootPolicy 1
com.apple.BootCache 40
com.apple.AppleFSCompression.AppleFSCompressionTypeZlib 1.0.0
com.apple.AppleFSCompression.AppleFSCompressionTypeDataless 1.0.0d1
com.apple.AppleEmbeddedSimpleSPINORFlasher 1
com.apple.driver.ApplePMP 1
com.apple.driver.AppleSN012776Amp 550.3
com.apple.driver.AppleSmartIO2 1
com.apple.driver.AppleCS42L84Audio 550.3
com.apple.driver.AppleT6000SOCTuner 1
com.apple.driver.AppleT6002CLPC 1
com.apple.driver.AppleSmartBatteryManager 161.0.0
com.apple.driver.AppleAOPVoiceTrigger 140.1
com.apple.driver.ApplePMPFirmware 1
com.apple.driver.AppleMCDP29XXUpdateSupport 1
com.apple.driver.AppleS5L8920XPWM 1.0.0d1
com.apple.driver.AppleJPEGDriver 4.8.1
com.apple.driver.AudioDMAController-T600x 160.2
com.apple.driver.AppleT6001PMGR 1
com.apple.driver.AppleS5L8960XNCO 1
com.apple.driver.AppleS8000AES 1
com.apple.driver.AppleSamsungSerial 1.0.0d1
com.apple.driver.AppleSerialShim 1
com.apple.driver.AppleProResHW 177.0.0
com.apple.AGXG13X 190.22
com.apple.driver.AppleAVE2 560.5.0
com.apple.driver.AppleAVD 566
com.apple.driver.AppleSPIMC 1
com.apple.driver.AppleSDXC 3.2.1
com.apple.driver.usb.AppleSynopsysUSB40XHCI 1
com.apple.driver.AppleMobileDispT600X-DCP 140.0
com.apple.driver.AppleSPMIPMU 1.0.1
com.apple.driver.AppleEventLogHandler 1
com.apple.driver.AppleS8000DWI 1.0.0d1
com.apple.driver.AppleInterruptControllerV2 1.0.0d1
com.apple.driver.AppleT8110DART 1
com.apple.driver.AppleBluetoothModule 1
com.apple.driver.AppleBCMWLANBusInterfacePCIe 1
com.apple.driver.AppleS5L8940XI2C 1.0.0d2
com.apple.driver.AppleT6000DART 1
com.apple.driver.AppleT6000 1
com.apple.driver.AppleM68Buttons 1.0.0d1
com.apple.iokit.IOUserEthernet 1.0.1
com.apple.driver.usb.AppleUSBUserHCI 1
com.apple.iokit.IOKitRegistryCompatibility 1
com.apple.iokit.EndpointSecurity 1
com.apple.driver.AppleDiskImages2 126.141.2
com.apple.AppleSystemPolicy 2.0.0
com.apple.nke.applicationfirewall 402
com.apple.kec.InvalidateHmac 1
com.apple.kec.AppleEncryptedArchive 1
com.apple.driver.driverkit.serial 6.0.0
com.apple.iokit.IOAVBFamily 1040.6
com.apple.plugin.IOgPTPPlugin 1040.3
com.apple.iokit.IOEthernetAVBController 1.1.0
com.apple.iokit.IOAHCIBlockStorage 333.140.2
com.apple.driver.AppleActuatorDriver 5460.1
com.apple.driver.AppleMultitouchDriver 5460.1
com.apple.driver.AppleInputDeviceSupport 5460.1
com.apple.driver.AppleHSBluetoothDriver 5450.8
com.apple.driver.IOBluetoothHIDDriver 9.0.0
com.apple.kext.triggers 1.0
com.apple.driver.AppleUSBMergeNub 900.4.2
com.apple.driver.usb.cdc.ncm 5.0.0
com.apple.driver.usb.cdc.ecm 5.0.0
com.apple.driver.usb.cdc.acm 5.0.0
com.apple.driver.usb.serial 6.0.0
com.apple.driver.AppleMesaSEPDriver 100.99
com.apple.iokit.IOBiometricFamily 1
com.apple.iokit.IOAHCIFamily 297
com.apple.driver.AppleEthernetAquantiaAqtion 1.0.64
com.apple.driver.usb.IOUSBHostHIDDevice 1.2
com.apple.driver.AppleThunderboltPCIUpAdapter 4.1.1
com.apple.driver.AppleThunderboltDPOutAdapter 8.5.1
com.apple.driver.DiskImages.KernelBacked 493.0.0
com.apple.driver.usb.cdc 5.0.0
com.apple.driver.AppleUSBAudio 416.2
com.apple.iokit.IOAudioFamily 340.2
com.apple.vecLib.kext 1.2.0
com.apple.driver.AppleXsanScheme 3
com.apple.driver.AppleSEPHDCPManager 1.0.1
com.apple.driver.AppleTrustedAccessory 1
com.apple.iokit.AppleSEPGenericTransfer 1
com.apple.driver.AppleConvergedIPCOLYBTControl 1
com.apple.driver.AppleConvergedPCI 1
com.apple.driver.AppleBluetoothDebug 1
com.apple.driver.usb.AppleEmbeddedUSBXHCIPCI 1
com.apple.driver.usb.AppleUSBXHCIPCI 1.2
com.apple.driver.AppleEmbeddedUSB 1
com.apple.driver.usb.networking 5.0.0
com.apple.driver.AppleThunderboltUSBDownAdapter 1.0.4
com.apple.driver.AppleThunderboltPCIDownAdapter 4.1.1
com.apple.driver.AppleThunderboltDPInAdapter 8.5.1
com.apple.driver.AppleThunderboltDPAdapterFamily 8.5.1
com.apple.nke.ppp 1.9
com.apple.driver.AppleBSDKextStarter 3
com.apple.filesystems.hfs.encodings.kext 1
com.apple.driver.AppleBTM 1.0.1
com.apple.driver.AppleDiagnosticDataAccessReadOnly 1.0.0
com.apple.driver.AppleDCPDPTXProxy 1.0.0
com.apple.driver.DCPDPFamilyProxy 1
com.apple.driver.AppleI2CEthernetAquantia 1.0.0
com.apple.driver.AppleHPM 3.4.4
com.apple.driver.AppleCSEmbeddedAudio 550.3
com.apple.driver.ApplePassthroughPPM 3.0
com.apple.driver.usb.AppleUSBHostPacketFilter 1.0
com.apple.driver.AppleAOPAudio 150.2
com.apple.driver.AppleEmbeddedAudio 550.3
com.apple.iokit.AppleARMIISAudio 140.2
com.apple.AGXFirmwareKextG13XRTBuddy 190.22
com.apple.AGXFirmwareKextRTBuddy64 190.22
com.apple.driver.DCPAVFamilyProxy 1
com.apple.driver.AppleStockholmControl 1.0.0
com.apple.iokit.IONVMeFamily 2.1.0
com.apple.driver.AppleNANDConfigAccess 1.0.0
com.apple.driver.AppleMCA2-T600x 640.5
com.apple.driver.AppleEmbeddedAudioLibs 140.1
com.apple.driver.ApplePMGR 1
com.apple.driver.AppleSPU 1
com.apple.iokit.IOGPUFamily 35.29
com.apple.driver.AppleT6000TypeCPhy 1
com.apple.driver.AppleT8103TypeCPhy 1
com.apple.driver.AppleUSBXDCIARM 1.0
com.apple.driver.AppleUSBXDCI 1.0
com.apple.iokit.IOUSBDeviceFamily 2.0.0
com.apple.driver.usb.AppleSynopsysUSBXHCI 1
com.apple.driver.usb.AppleUSBXHCI 1.2
com.apple.driver.AppleEmbeddedUSBHost 1
com.apple.driver.usb.AppleUSBHub 1.2
com.apple.driver.usb.AppleUSBHostCompositeDevice 1.2
com.apple.iokit.IOMobileGraphicsFamily-DCP 343.0.0
com.apple.driver.AppleDCP 1
com.apple.driver.AppleFirmwareKit 1
com.apple.driver.AppleFirmwareUpdateKext 1
com.apple.iokit.IOMobileGraphicsFamily 343.0.0
com.apple.driver.AppleDialogPMU 1.0.1
com.apple.driver.AppleSPMI 1.0.1
com.apple.driver.AppleSART 1
com.apple.driver.AppleARMWatchdogTimer 1
com.apple.driver.AppleDisplayCrossbar 1.0.0
com.apple.iokit.IODisplayPortFamily 1.0.0
com.apple.driver.AppleTypeCPhy 1
com.apple.driver.AppleThunderboltNHI 7.2.81
com.apple.driver.AppleT6000PCIeC 1
com.apple.iokit.IOThunderboltFamily 9.3.3
com.apple.driver.ApplePIODMA 1
com.apple.driver.AppleT600xPCIe 1
com.apple.driver.AppleMultiFunctionManager 1
com.apple.driver.AppleBluetoothDebugService 1
com.apple.driver.AppleBCMWLANCore 1.0.0
com.apple.iokit.IO80211Family 1200.12.2b2
com.apple.driver.IOImageLoader 1.0.0
com.apple.driver.AppleOLYHAL 1
com.apple.driver.corecapture 1.0.4
com.apple.driver.AppleEthernetAquantiaAqtionPortMonitor 1.0.0
com.apple.driver.AppleEmbeddedPCIE 1
com.apple.driver.AppleGPIOICController 1.0.2
com.apple.driver.AppleFireStormErrorHandler 1
com.apple.driver.AppleMobileApNonce 1
com.apple.iokit.IOTimeSyncFamily 1040.3
com.apple.driver.DiskImages 493.0.0
com.apple.iokit.IOGraphicsFamily 597
com.apple.iokit.IOBluetoothSerialManager 9.0.0
com.apple.iokit.IOBluetoothHostControllerUSBTransport 9.0.0
com.apple.iokit.IOBluetoothHostControllerUARTTransport 9.0.0
com.apple.iokit.IOBluetoothHostControllerTransport 9.0.0
com.apple.driver.IOBluetoothHostControllerPCIeTransport 9.0.0
com.apple.iokit.IOBluetoothFamily 9.0.0
com.apple.driver.FairPlayIOKit 68.16.0
com.apple.iokit.CSRBluetoothHostControllerUSBTransport 9.0.0
com.apple.iokit.BroadcomBluetoothHostControllerUSBTransport 9.0.0
com.apple.driver.AppleSSE 1.0
com.apple.driver.AppleSEPKeyStore 2
com.apple.driver.AppleUSBTDM 533.120.2
com.apple.iokit.IOUSBMassStorageDriver 210.120.3
com.apple.iokit.IOPCIFamily 2.9
com.apple.iokit.IOSCSIBlockCommandsDevice 456.140.3
com.apple.iokit.IOSCSIArchitectureModelFamily 456.140.3
com.apple.driver.AppleIPAppender 1.0
com.apple.driver.AppleFDEKeyStore 28.30
com.apple.driver.AppleEffaceableStorage 1.0
com.apple.driver.AppleCredentialManager 1.0
com.apple.driver.KernelRelayHost 1
com.apple.iokit.IOUSBHostFamily 1.2
com.apple.driver.AppleUSBHostMergeProperties 1.2
com.apple.driver.usb.AppleUSBCommon 1.0
com.apple.driver.AppleSMC 3.1.9
com.apple.driver.RTBuddy 1.0.0
com.apple.driver.AppleEmbeddedTempSensor 1.0.0
com.apple.driver.AppleARMPMU 1.0
com.apple.iokit.IOAccessoryManager 1.0.0
com.apple.driver.AppleOnboardSerial 1.0
com.apple.iokit.IOSkywalkFamily 1.0
com.apple.driver.mDNSOffloadUserClient 1.0.1b8
com.apple.iokit.IONetworkingFamily 3.4
com.apple.iokit.IOSerialFamily 11
com.apple.driver.AppleSEPManager 1.0.1
com.apple.driver.AppleA7IOP 1.0.2
com.apple.driver.IOSlaveProcessor 1
com.apple.driver.AppleBiometricSensor 2
com.apple.iokit.IOHIDFamily 2.0.0
com.apple.iokit.CoreAnalyticsFamily 1
com.apple.driver.AppleANELoadBalancer 5.68.2
com.apple.driver.AppleH11ANEInterface 5.68.0
com.apple.AUC 1.0
com.apple.iokit.IOAVFamily 1.0.0
com.apple.iokit.IOHDCPFamily 1.0.0
com.apple.iokit.IOCECFamily 1
com.apple.iokit.IOAudio2Family 1.0
com.apple.driver.AppleIISController 140.1
com.apple.driver.AppleAudioClockLibs 140.1
com.apple.driver.AppleM2ScalerCSCDriver 265.0.0
com.apple.iokit.IOSurface 302.14
com.apple.driver.IODARTFamily 1
com.apple.security.quarantine 4
com.apple.security.sandbox 300.0
com.apple.kext.AppleMatch 1.0.0d1
com.apple.driver.AppleMobileFileIntegrity 1.0.5
com.apple.security.AppleImage4 4.2.0
com.apple.kext.CoreTrust 1
com.apple.iokit.IOCryptoAcceleratorFamily 1.0.1
com.apple.driver.AppleARMPlatform 1.0.2
com.apple.iokit.IOStorageFamily 2.1
com.apple.iokit.IOSlowAdaptiveClockingFamily 1.0.0
com.apple.iokit.IOReportFamily 47
com.apple.kec.pthread 1
com.apple.kec.Libm 1
com.apple.kec.corecrypto 12.0
** Stackshot Succeeded ** Bytes Traced 869802 (Uncompressed 2378112) **
lundman just released a new beta version Monterey for arm64, perhaps you can give it a shot?
#798
The new version doesn't help.
Hi, as of the start of this week I have one of these 128GiB M1 Ultras too, and now know what this problem is. Bear with us, this is the first time we've had a chance to run on a big arm Mac "in anger", and doing kernel extension development on the arm Macs is somewhat different in practice from doing it on Intel Macs.
I will commit a fix (and we will put out a package) in a couple of days, but in the meanwhile there is a workaround that should solve this for you entirely.
zsh$ sudo sysctl kstat.zfs.darwin.tunable.zfs_arc.max=$((16*1024**3))
You can also add the line
kstat.zfs.darwin.tunable.zfs_arc.max=17179869184
into /etc/zfs/zsysctl.conf. At boot time, by default, /usr/local/sbin/zsysctl will be run and will set the tunable for you.
This caps the maximum arc to 16 GiB. Almost certainly you can dial that up, but keep it well well well below 64 GIB, which is what the default settinig (0) uses. You can run the sysctl at any time the kernel extension is loaded and running, and dial it up and down; it should be safe enough to do so if you are sufficiently below half your system RAM. 16 GiB is very conservative. I have tested that value extensively and cannot get the machine to panic.
The problem in a nutshell is that we limit the entire zfs kernel extension (practically but not exactly all its allocations) to half of a system's RAM. In our case, 64 GIB. ARC is in present code capped by that 64 GiB, rather than some lower value. This leads to two potential problems.
The first problem is that because of the constraints on knowing how much memory is available from the kernel to our kernel extension, we are reactive to memory pressure. It's hard to cause a 128 GIB mac to run critically low on memory, so on my system so far I have been unable to see a pressure event at all (and I have tried!). As a result ARC may grow to its maximum extent, 64 GiB, because nothing is telling it to stop growing. Unfortunately this leaves no room for the other parts of our kernel extension which need memory.
The second problem arises when it's discovered by our code that nearly all 64GiB are in use. When that's observed some low-level code generates an artificial pressure event, and waits for that to take effect. The signal was not strong (a multiple of the desired allocation size, which translates into between kilobytes and a couple megabytes). On a machine like the M1 Studios, with their many fast I/O ports and little chance of being CPU bound when doing I/O, demands for new memory can easily outrace these attempts to free small amounts, leading to a period of allocations stalling.
The result is that a client I/O (say, HFS or APFS inside a zvol; or our command line zfs and zpool tools, or running /bin/ls on a dataset) can be delayed for potentially a long time under certain system workloads, and some of those clients are intolerant of such delays. In particular, HFS appears to decide to time out and panic in some cases.
Two probable fixes for this are in the code I am running and testing now, and will in due course commit to our trees. One change scales the artificial pressure signal up with increasing system memory. The other caps the ARC to below the 50% total extension memory threshold.
This will get rid of this exact panic, which I can reproduce easily (keep the arc_max tunable at the default, and do a time machine backup into a zvol on an otherwise quiet system). It will also avoid other panics that can arise from the same root cause.
Awesome! I will try that right now and see what happens; do let me know if I can otherwise help with testing in any way.
Now if we can just figure out what causes the boot loop when installing the same kext in Ventura... :)
I can use sysctl to set the tunable manually, but apparently /usr/local/zfs/bin/zsysctl is not running automatically on reboot, because on reboot the tunable is back to 0 (and the machine panics).
Great debugging work @rottegift, I really appreciate it.
@dmzimmerman do a "sudo launchctl list | grep -i zfs", and see if it says something like:
759 1 org.openzfsonosx.zed
143 0 org.openzfsonosx.zconfigd
145 0 org.openzfsonosx.InvariantDisks
The columns are process ID, status, and launchctl job label.
also, "sudo launchctl list org.openzfsonosx.zconfigd"
{
"StandardOutPath" = "/private/var/log/org.openzfsonosx.zconfigd.log";
"LimitLoadToSessionType" = "System";
"StandardErrorPath" = "/private/var/log/org.openzfsonosx.zconfigd.err";
"Label" = "org.openzfsonosx.zconfigd";
"OnDemand" = false;
"LastExitStatus" = 0;
"PID" = 143;
"Program" = "/usr/local/zfs/sbin/zconfigd";
"ProgramArguments" = (
"/usr/local/zfs/sbin/zconfigd";
);
};
And do look in /var/log/org.openzfsonosx.*log and /var/log/org.openzfsonosx.*err to see if there's any evidence of failures (or even zconfigd starting).
zconfigd should be running
/usr/local/zfs/sbin/zsysctl -f /etc/zfs/zsysctl.conf
You could run that by hand (use sudo) to see that it emits any output (it should say at least something like:
kstat.zfs.darwin.tunable.zfs_arc.max: 0 -> 17179869184
@jawbroken : thank you. I'm still working on a few ARM things I discovered, and will land it all at once in a few days. As far as I can tell on a 128GiB machine it is safe to use 16 GiB arc_max. With modern compressed ARC that hopefully that's not such a low constraint to suffer with for a few days.
Unfortunately, I have to report that it still panics for me, even with a 16GiB arc_max. It takes a lot longer, though, and I was pushing it pretty hard.
panic(cpu 2 caller 0xfffffe0015831f1c): Kernel data abort. at pc 0xfffffe0016bdadb4, lr 0xfffffe0016bdad9c (saved state: 0xfffffe180b38f3c0)
x0: 0x0000000000000000 x1: 0x0000000000300000 x2: 0x0000000000000000 x3: 0xfffffe1810cc6ca0
x4: 0x0000000000000000 x5: 0x000001477e03ffee x6: 0x0000000000000000 x7: 0xfffffe180b38f568
x8: 0x0000000000000000 x9: 0xb5c52280151d0028 x10: 0x0000000000000000 x11: 0x0000000000000000
x12: 0x0000000000000001 x13: 0x0000000000000014 x14: 0x0000000000153be4 x15: 0x00000000ffffffff
x16: 0xf486fe00152a1a2c x17: 0xfffffe0018294a58 x18: 0x0000000000000000 x19: 0xfffffe2cdd6f0200
x20: 0xfffffe2344933300 x21: 0x0000000000000000 x22: 0x0000000000300000 x23: 0x0000000000000000
x24: 0x0000000000000000 x25: 0xfffffe180b38f820 x26: 0xfffffe0018b35920 x27: 0x0000000000000000
x28: 0x0000000003018001 fp: 0xfffffe180b38f740 lr: 0xfffffe0016bdad9c sp: 0xfffffe180b38f710
pc: 0xfffffe0016bdadb4 cpsr: 0x60401208 esr: 0x96000046 far: 0x0000000000000000
Debugger message: panic
Memory ID: 0x6
OS release type: User
OS version: 21G115
Kernel version: Darwin Kernel Version 21.6.0: Mon Aug 22 20:19:52 PDT 2022; root:xnu-8020.140.49~2/RELEASE_ARM64_T6000
Fileset Kernelcache UUID: 39A7E336B0FAA0022B3764E49DFF29D2
Kernel UUID: 778CC57A-CF0B-3D35-8EE8-5035142D0177
iBoot version: iBoot-7459.141.1
secure boot?: YES
Paniclog version: 13
KernelCache slide: 0x000000000d6f8000
KernelCache base: 0xfffffe00146fc000
Kernel slide: 0x000000000debc000
Kernel text base: 0xfffffe0014ec0000
Kernel text exec slide: 0x000000000dfa4000
Kernel text exec base: 0xfffffe0014fa8000
mach_absolute_time: 0x1477e12d85a
Epoch Time: sec usec
Boot : 0x632ab6f2 0x000e9f90
Sleep : 0x00000000 0x00000000
Wake : 0x00000000 0x00000000
Calendar: 0x632b9bd8 0x0008165c
Zone info:
Zone map: 0xfffffe1810ca8000 - 0xfffffe3810ca8000
. VM : 0xfffffe1810ca8000 - 0xfffffe1cdd974000
. RO : 0xfffffe1cdd974000 - 0xfffffe1e7730c000
. GEN0 : 0xfffffe1e7730c000 - 0xfffffe2343fd8000
. GEN1 : 0xfffffe2343fd8000 - 0xfffffe2810ca4000
. GEN2 : 0xfffffe2810ca4000 - 0xfffffe2cdd970000
. GEN3 : 0xfffffe2cdd970000 - 0xfffffe31aa63c000
. DATA : 0xfffffe31aa63c000 - 0xfffffe3810ca8000
Metadata: 0xfffffe3810cb8000 - 0xfffffe3818cb8000
Bitmaps : 0xfffffe3818cb8000 - 0xfffffe3838cb8000
CORE 0 PVH locks held: None
CORE 1 PVH locks held: None
CORE 2 PVH locks held: None
CORE 3 PVH locks held: None
CORE 4 PVH locks held: None
CORE 5 PVH locks held: None
CORE 6 PVH locks held: None
CORE 7 PVH locks held: None
CORE 8 PVH locks held: None
CORE 9 PVH locks held: None
CORE 10 PVH locks held: None
CORE 11 PVH locks held: None
CORE 12 PVH locks held: None
CORE 13 PVH locks held: None
CORE 14 PVH locks held: None
CORE 15 PVH locks held: None
CORE 16 PVH locks held: None
CORE 17 PVH locks held: None
CORE 18 PVH locks held: None
CORE 19 PVH locks held: None
CORE 0: PC=0xfffffe001505e144, LR=0xfffffe001505e144, FP=0xfffffe180b54fd40
CORE 1: PC=0xfffffe001501c8a4, LR=0xfffffe00154f75c4, FP=0xfffffe8fedb0ff20
CORE 2 is the one that panicked. Check the full backtrace for details.
CORE 3: PC=0xfffffe00150cf504, LR=0xfffffe00150cde6c, FP=0xfffffe180ac2f950
CORE 4: PC=0xfffffe0015037258, LR=0xfffffe0015037254, FP=0xfffffe8fec44ff00
CORE 5: PC=0xfffffe001513bfb4, LR=0xfffffe001513bfb0, FP=0xfffffe8fed7efe90
CORE 6: PC=0xfffffe0015037258, LR=0xfffffe0015037254, FP=0xfffffe180f16ff00
CORE 7: PC=0xfffffe0015037254, LR=0xfffffe0015037254, FP=0xfffffe180e44ff00
CORE 8: PC=0xfffffe0015037254, LR=0xfffffe0015037254, FP=0xfffffe88069cff00
CORE 9: PC=0xfffffe0015037254, LR=0xfffffe0015037254, FP=0xfffffe8fee52ff00
CORE 10: PC=0x00000001afe9abc8, LR=0x00000001afe9ab94, FP=0x000000016f3da770
CORE 11: PC=0xfffffe001501c588, LR=0xfffffe001501c910, FP=0xfffffe880700fd10
CORE 12: PC=0xfffffe0015037254, LR=0xfffffe0015037254, FP=0xfffffe180f32ff00
CORE 13: PC=0xfffffe0015037254, LR=0xfffffe0015037254, FP=0xfffffe180f0eff00
CORE 14: PC=0xfffffe0015037254, LR=0xfffffe0015037254, FP=0xfffffe180b64ff00
CORE 15: PC=0xfffffe0015037254, LR=0xfffffe0015037254, FP=0xfffffe180e00ff00
CORE 16: PC=0xfffffe0015037254, LR=0xfffffe0015037254, FP=0xfffffe180a78ff00
CORE 17: PC=0xfffffe0015037254, LR=0xfffffe0015037254, FP=0xfffffe180b5cff00
CORE 18: PC=0xfffffe0015037254, LR=0xfffffe0015037254, FP=0xfffffe180b42ff00
CORE 19: PC=0xfffffe0015037254, LR=0xfffffe0015037254, FP=0xfffffe180e38ff00
Compressor Info: 0% of compressed pages limit (OK) and 0% of segments limit (OK) with 1 swapfiles and OK swap space
Panicked task 0xfffffe2810df66d0: 2310 pages, 4 threads: pid 1: launchd
Panicked thread: 0xfffffe31a7134820, backtrace: 0xfffffe180b38ea80, tid: 989660
lr: 0xfffffe0015001400 fp: 0xfffffe180b38eaf0
lr: 0xfffffe00150010c8 fp: 0xfffffe180b38eb60
lr: 0xfffffe001514733c fp: 0xfffffe180b38eb80
lr: 0xfffffe00151390cc fp: 0xfffffe180b38ebf0
lr: 0xfffffe0015136cb0 fp: 0xfffffe180b38ecb0
lr: 0xfffffe0014faf7f8 fp: 0xfffffe180b38ecc0
lr: 0xfffffe0015000d4c fp: 0xfffffe180b38f060
lr: 0xfffffe0015000d4c fp: 0xfffffe180b38f0d0
lr: 0xfffffe001582954c fp: 0xfffffe180b38f0f0
lr: 0xfffffe0015831f1c fp: 0xfffffe180b38f270
lr: 0xfffffe0015138ecc fp: 0xfffffe180b38f2e0
lr: 0xfffffe0015136fb4 fp: 0xfffffe180b38f3a0
lr: 0xfffffe0014faf7f8 fp: 0xfffffe180b38f3b0
lr: 0xfffffe0016bdad9c fp: 0xfffffe180b38f740
lr: 0xfffffe0016c0ac20 fp: 0xfffffe180b38f7a0
lr: 0xfffffe0016bca03c fp: 0xfffffe180b38f9a0
lr: 0xfffffe0016bcdc78 fp: 0xfffffe180b38fa10
lr: 0xfffffe0016bf8d14 fp: 0xfffffe180b38fa40
lr: 0xfffffe0015268b24 fp: 0xfffffe180b38fad0
lr: 0xfffffe0016bf5c4c fp: 0xfffffe180b38fce0
lr: 0xfffffe001527c864 fp: 0xfffffe180b38fd20
lr: 0xfffffe00152688c8 fp: 0xfffffe180b38fda0
lr: 0xfffffe001527c7ac fp: 0xfffffe180b38fdb0
lr: 0xfffffe001566aa28 fp: 0xfffffe180b38fe50
lr: 0xfffffe0015136d84 fp: 0xfffffe180b38ff10
lr: 0xfffffe0014faf7f8 fp: 0xfffffe180b38ff20
Kernel Extensions in backtrace:
com.apple.filesystems.hfs.kext(583.100.10)[45F25204-8A60-3A88-B71F-974BDDBDB3BF]@0xfffffe0016bc48a0->0xfffffe0016c134e3
dependency: com.apple.filesystems.hfs.encodings.kext(1)[4183166A-286A-3CEB-8C2C-AF85AA1F4D16]@0xfffffe0016c134f0->0xfffffe0016c1441f
last started kext at 3195486023: com.apple.filesystems.smbfs 4.0 (addr 0xfffffe0014ea4c30, size 65195)
loaded kexts:
org.openzfsonosx.zfs 2.1.99
com.apple.filesystems.smbfs 4.0
com.apple.filesystems.autofs 3.0
com.apple.driver.AppleBiometricServices 1
com.apple.driver.CoreKDL 1
com.apple.AppleEthernetAquantiaAqtionFirmware 1.0.36
com.apple.driver.AppleAHCIPort 351.100.4
com.apple.driver.DiskImages.ReadWriteDiskImage 493.0.0
com.apple.driver.DiskImages.UDIFDiskImage 493.0.0
com.apple.driver.DiskImages.RAMBackingStore 493.0.0
com.apple.driver.DiskImages.FileBackingStore 493.0.0
com.apple.driver.BCMWLANFirmware4387.Hashstore 1
com.apple.driver.SEPHibernation 1
com.apple.driver.AppleThunderboltIP 4.0.3
com.apple.driver.AppleUSBDeviceNCM 5.0.0
com.apple.driver.AppleFileSystemDriver 3.0.1
com.apple.nke.l2tp 1.9
com.apple.filesystems.tmpfs 1
com.apple.driver.ApplePMP 1
com.apple.driver.AppleSmartIO2 1
com.apple.filesystems.lifs 1
com.apple.filesystems.apfs 1934.141.2
com.apple.IOTextEncryptionFamily 1.0.0
com.apple.filesystems.hfs.kext 583.100.10
com.apple.security.BootPolicy 1
com.apple.BootCache 40
com.apple.AppleFSCompression.AppleFSCompressionTypeZlib 1.0.0
com.apple.AppleFSCompression.AppleFSCompressionTypeDataless 1.0.0d1
com.apple.AppleEmbeddedSimpleSPINORFlasher 1
com.apple.driver.AppleSN012776Amp 550.3
com.apple.driver.AppleCS42L84Audio 550.3
com.apple.driver.AppleT6000SOCTuner 1
com.apple.driver.AppleT6002CLPC 1
com.apple.driver.AppleSmartBatteryManager 161.0.0
com.apple.driver.AppleAOPVoiceTrigger 140.1
com.apple.driver.ApplePMPFirmware 1
com.apple.driver.AppleSamsungSerial 1.0.0d1
com.apple.driver.AppleSerialShim 1
com.apple.driver.AppleSDXC 3.2.1
com.apple.driver.AppleSPIMC 1
com.apple.driver.AppleMCDP29XXUpdateSupport 1
com.apple.driver.AppleS5L8960XNCO 1
com.apple.driver.AudioDMAController-T600x 160.2
com.apple.driver.AppleSPMIPMU 1.0.1
com.apple.driver.AppleAVE2 560.5.0
com.apple.driver.AppleAVD 566
com.apple.driver.AppleS8000AES 1
com.apple.driver.usb.AppleSynopsysUSB40XHCI 1
com.apple.driver.AppleEventLogHandler 1
com.apple.AGXG13X 190.22
com.apple.driver.AppleS5L8920XPWM 1.0.0d1
com.apple.driver.AppleT6001PMGR 1
com.apple.driver.AppleJPEGDriver 4.8.1
com.apple.driver.AppleProResHW 177.0.0
com.apple.driver.AppleMobileDispT600X-DCP 140.0
com.apple.driver.AppleS8000DWI 1.0.0d1
com.apple.driver.AppleInterruptControllerV2 1.0.0d1
com.apple.driver.AppleT8110DART 1
com.apple.driver.AppleBluetoothModule 1
com.apple.driver.AppleBCMWLANBusInterfacePCIe 1
com.apple.driver.AppleS5L8940XI2C 1.0.0d2
com.apple.driver.AppleT6000DART 1
com.apple.driver.AppleT6000 1
com.apple.driver.AppleM68Buttons 1.0.0d1
com.apple.iokit.IOUserEthernet 1.0.1
com.apple.driver.usb.AppleUSBUserHCI 1
com.apple.iokit.IOKitRegistryCompatibility 1
com.apple.iokit.EndpointSecurity 1
com.apple.driver.AppleDiskImages2 126.141.2
com.apple.AppleSystemPolicy 2.0.0
com.apple.nke.applicationfirewall 402
com.apple.kec.InvalidateHmac 1
com.apple.kec.AppleEncryptedArchive 1
com.apple.driver.driverkit.serial 6.0.0
com.apple.iokit.IOAVBFamily 1040.6
com.apple.plugin.IOgPTPPlugin 1040.3
com.apple.iokit.IOEthernetAVBController 1.1.0
com.apple.iokit.IOAHCIBlockStorage 333.140.2
com.apple.kext.triggers 1.0
com.apple.driver.AppleMesaSEPDriver 100.99
com.apple.iokit.IOBiometricFamily 1
com.apple.iokit.IOAHCIFamily 297
com.apple.driver.AppleEthernetAquantiaAqtion 1.0.64
com.apple.driver.AppleThunderboltPCIUpAdapter 4.1.1
com.apple.driver.AppleThunderboltDPOutAdapter 8.5.1
com.apple.driver.DiskImages.KernelBacked 493.0.0
com.apple.driver.AppleXsanScheme 3
com.apple.driver.AppleSEPHDCPManager 1.0.1
com.apple.driver.AppleTrustedAccessory 1
com.apple.iokit.AppleSEPGenericTransfer 1
com.apple.driver.usb.AppleEmbeddedUSBXHCIPCI 1
com.apple.driver.usb.AppleUSBXHCIPCI 1.2
com.apple.driver.AppleEmbeddedUSB 1
com.apple.driver.AppleConvergedIPCOLYBTControl 1
com.apple.driver.AppleConvergedPCI 1
com.apple.driver.AppleBluetoothDebug 1
com.apple.driver.usb.networking 5.0.0
com.apple.driver.AppleThunderboltUSBDownAdapter 1.0.4
com.apple.driver.AppleThunderboltPCIDownAdapter 4.1.1
com.apple.driver.AppleThunderboltDPInAdapter 8.5.1
com.apple.driver.AppleThunderboltDPAdapterFamily 8.5.1
com.apple.driver.AppleBTM 1.0.1
com.apple.nke.ppp 1.9
com.apple.driver.AppleDCPDPTXProxy 1.0.0
com.apple.driver.DCPDPFamilyProxy 1
com.apple.driver.AppleBSDKextStarter 3
com.apple.filesystems.hfs.encodings.kext 1
com.apple.driver.AppleDiagnosticDataAccessReadOnly 1.0.0
com.apple.driver.AppleCSEmbeddedAudio 550.3
com.apple.driver.AppleHPM 3.4.4
com.apple.driver.AppleI2CEthernetAquantia 1.0.0
com.apple.driver.ApplePassthroughPPM 3.0
com.apple.driver.AppleAOPAudio 150.2
com.apple.driver.AppleEmbeddedAudio 550.3
com.apple.iokit.AppleARMIISAudio 140.2
com.apple.iokit.IONVMeFamily 2.1.0
com.apple.driver.AppleNANDConfigAccess 1.0.0
com.apple.driver.AppleSART 1
com.apple.driver.AppleSPU 1
com.apple.driver.AppleDialogPMU 1.0.1
com.apple.driver.AppleStockholmControl 1.0.0
com.apple.driver.AppleSPMI 1.0.1
com.apple.driver.DCPAVFamilyProxy 1
com.apple.driver.AppleMCA2-T600x 640.5
com.apple.driver.AppleEmbeddedAudioLibs 140.1
com.apple.driver.AppleUSBXDCIARM 1.0
com.apple.driver.AppleUSBXDCI 1.0
com.apple.iokit.IOUSBDeviceFamily 2.0.0
com.apple.driver.usb.AppleSynopsysUSBXHCI 1
com.apple.driver.usb.AppleUSBXHCI 1.2
com.apple.driver.AppleEmbeddedUSBHost 1
com.apple.driver.usb.AppleUSBHub 1.2
com.apple.driver.usb.AppleUSBHostCompositeDevice 1.2
com.apple.driver.AppleT6000TypeCPhy 1
com.apple.driver.AppleT8103TypeCPhy 1
com.apple.AGXFirmwareKextG13XRTBuddy 190.22
com.apple.AGXFirmwareKextRTBuddy64 190.22
com.apple.iokit.IOGPUFamily 35.29
com.apple.driver.ApplePMGR 1
com.apple.driver.usb.AppleUSBHostPacketFilter 1.0
com.apple.iokit.IOMobileGraphicsFamily-DCP 343.0.0
com.apple.driver.AppleDCP 1
com.apple.driver.AppleFirmwareKit 1
com.apple.driver.AppleFirmwareUpdateKext 1
com.apple.iokit.IOMobileGraphicsFamily 343.0.0
com.apple.driver.AppleARMWatchdogTimer 1
com.apple.driver.AppleDisplayCrossbar 1.0.0
com.apple.iokit.IODisplayPortFamily 1.0.0
com.apple.driver.AppleTypeCPhy 1
com.apple.driver.AppleThunderboltNHI 7.2.81
com.apple.driver.AppleT6000PCIeC 1
com.apple.iokit.IOThunderboltFamily 9.3.3
com.apple.driver.ApplePIODMA 1
com.apple.driver.AppleT600xPCIe 1
com.apple.driver.AppleMultiFunctionManager 1
com.apple.driver.AppleBluetoothDebugService 1
com.apple.driver.AppleBCMWLANCore 1.0.0
com.apple.iokit.IO80211Family 1200.12.2b2
com.apple.driver.IOImageLoader 1.0.0
com.apple.driver.AppleOLYHAL 1
com.apple.driver.corecapture 1.0.4
com.apple.driver.AppleEthernetAquantiaAqtionPortMonitor 1.0.0
com.apple.driver.AppleEmbeddedPCIE 1
com.apple.driver.AppleGPIOICController 1.0.2
com.apple.driver.AppleFireStormErrorHandler 1
com.apple.driver.AppleMobileApNonce 1
com.apple.iokit.IOTimeSyncFamily 1040.3
com.apple.driver.DiskImages 493.0.0
com.apple.iokit.IOGraphicsFamily 597
com.apple.iokit.IOBluetoothSerialManager 9.0.0
com.apple.iokit.IOBluetoothHostControllerUSBTransport 9.0.0
com.apple.iokit.IOBluetoothHostControllerUARTTransport 9.0.0
com.apple.iokit.IOBluetoothHostControllerTransport 9.0.0
com.apple.driver.IOBluetoothHostControllerPCIeTransport 9.0.0
com.apple.iokit.IOBluetoothFamily 9.0.0
com.apple.driver.FairPlayIOKit 68.16.0
com.apple.iokit.CSRBluetoothHostControllerUSBTransport 9.0.0
com.apple.iokit.BroadcomBluetoothHostControllerUSBTransport 9.0.0
com.apple.driver.AppleSSE 1.0
com.apple.driver.AppleSEPKeyStore 2
com.apple.driver.AppleUSBTDM 533.120.2
com.apple.iokit.IOUSBMassStorageDriver 210.120.3
com.apple.iokit.IOPCIFamily 2.9
com.apple.iokit.IOSCSIBlockCommandsDevice 456.140.3
com.apple.iokit.IOSCSIArchitectureModelFamily 456.140.3
com.apple.driver.AppleIPAppender 1.0
com.apple.driver.AppleFDEKeyStore 28.30
com.apple.driver.AppleEffaceableStorage 1.0
com.apple.driver.AppleCredentialManager 1.0
com.apple.driver.KernelRelayHost 1
com.apple.iokit.IOUSBHostFamily 1.2
com.apple.driver.AppleUSBHostMergeProperties 1.2
com.apple.driver.usb.AppleUSBCommon 1.0
com.apple.driver.AppleSMC 3.1.9
com.apple.driver.RTBuddy 1.0.0
com.apple.driver.AppleEmbeddedTempSensor 1.0.0
com.apple.driver.AppleARMPMU 1.0
com.apple.iokit.IOAccessoryManager 1.0.0
com.apple.driver.AppleOnboardSerial 1.0
com.apple.iokit.IOSkywalkFamily 1.0
com.apple.driver.mDNSOffloadUserClient 1.0.1b8
com.apple.iokit.IONetworkingFamily 3.4
com.apple.iokit.IOSerialFamily 11
com.apple.driver.AppleSEPManager 1.0.1
com.apple.driver.AppleA7IOP 1.0.2
com.apple.driver.IOSlaveProcessor 1
com.apple.driver.AppleBiometricSensor 2
com.apple.iokit.IOHIDFamily 2.0.0
com.apple.iokit.CoreAnalyticsFamily 1
com.apple.driver.AppleANELoadBalancer 5.68.2
com.apple.driver.AppleH11ANEInterface 5.68.0
com.apple.AUC 1.0
com.apple.iokit.IOAVFamily 1.0.0
com.apple.iokit.IOHDCPFamily 1.0.0
com.apple.iokit.IOCECFamily 1
com.apple.iokit.IOAudio2Family 1.0
com.apple.driver.AppleIISController 140.1
com.apple.driver.AppleAudioClockLibs 140.1
com.apple.driver.AppleM2ScalerCSCDriver 265.0.0
com.apple.iokit.IOSurface 302.14
com.apple.driver.IODARTFamily 1
com.apple.security.quarantine 4
com.apple.security.sandbox 300.0
com.apple.kext.AppleMatch 1.0.0d1
com.apple.driver.AppleMobileFileIntegrity 1.0.5
com.apple.security.AppleImage4 4.2.0
com.apple.kext.CoreTrust 1
com.apple.iokit.IOCryptoAcceleratorFamily 1.0.1
com.apple.driver.AppleARMPlatform 1.0.2
com.apple.iokit.IOStorageFamily 2.1
com.apple.iokit.IOSlowAdaptiveClockingFamily 1.0.0
com.apple.iokit.IOReportFamily 47
com.apple.kec.pthread 1
com.apple.kec.Libm 1
com.apple.kec.corecrypto 12.0
** Stackshot Succeeded ** Bytes Traced 762271 (Uncompressed 2090224) **
zconfigd should be running
/usr/local/zfs/sbin/zsysctl -f /etc/zfs/zsysctl.conf
Well, I think I found the problem:
- first,
zsysctlis in/usr/local/zfs/binon my machine, and there is no/usr/local/zfs/sbin. - but, assuming that's a typo, when I run
sudo /usr/local/zfs/bin/zsysctl -f /etc/zfs/zsysctl.conf, it doeskstat.zfs.darwin.tunable.zfs_arc.max: 17179869184 -> 0, which is not at all what I expected to see.
The contents of my /etc/zfs/zsysctl.conf are:
kstat.zfs.darwin.tunable.zfs_arc.max=17179869184
so I really don't understand what's happening there.
Update:
tl;dr: kernel doesn't like giving our kext more than approx 32 GIB of memory on 128GiB Mac Studios.
Previously in our second lowest layer of memory allocation there was plumbing to wait in a loop for memory to appear in some circumstances. On our systems, those circumstances were always us running close to 50% of system RAM (a self-imposed limit) or learning from the kernel that system memory was recently (or even currently) in short supply.
The waiting would also happen if we received a NULL from the (essentially) malloc "system" call to the kernel.
For years we were using a since-deprecated kernel malloc call that was guaranteed never to fail: it would give you the memory requested or panic the system. More recently we switched to IOMallocAligned(), which it turns out is allowed to return a NULL whenever it likes.
As far as I can tell, we have only ever seen many NULLs being returned from IOMallocAligned() on these 128GiB Mac Studios. (I guess it could have happenend on an Intel Mac or Hack with 128GiB+ system RAM, but a brief search showed no complaints suggesting anyone had discovered this bug; it could be that on Intel platforms IOMallocAligned() returns NULL so rarely that it might as well be never -- one might never notice the effect of a NULL if retrying produced a successful allocation within a few milliseconds or a few thousand passes through the waiting loop).
I ran an experiment to simplify our second-lowest allocation layer on macOS 12+ builds. I think the 12+ condition is reasonable -- macOS Monterrey benefits from years of memory system development by Apple, and additionally all our Mac Studio 128GiB machines, when running macOS, must be running at 12+.
The results are promising. As expected, ARC grows as quickly as it can, causing (directly and indirectly) us to allocate a little more (!) than 32 GiB of memory, after which we start seeing NULL almost every time we use IOMallocAligned() for more. Nothing seems to break, and ARC correctly transitions into a much more slowly-and-tentatively-growing mode, causing us to use IOMallocAligned() much less frequently.
I can think of a few ways to try to get past the (slightly-more-than-)32GiB effective limit already, but want to UTSL and /usr/bin/zprint to see how exactly it's enforced on us.
However, give or take cosmetic changes, I have a workaround that avoids the panic without the need to set an explicit non-default arc_max at runtime via sysctl. I am likely to commit the change either tonight or Monday.
[ETA (pardon the acronym pun), more likely Monday or Tuesday, as I will want to test whether a variety of other Macs and VMs, with different OS vintages, are happy with a total removal of the relevant code, instead of an #ifdef removing it for macOS 12+ only. Over time the amount of time we have to pester xnu with requests for more memory or to return memory have reduced via changes in middle layers of our memory management. However I don't want to make a change that causes panics on 10.11 or even earlier, or on machines (including virtual machines) with only 4 GiB total RAM].
Until then, setting a sufficiently low arc_max via sysctl will virtually certainly prevent the entire kext from attempting to allocate more than 32 GiB memory, and thus will not have to deal with these strings of NULLs from IOMallocAligned().
16GiB is still my best advice for the upper limit. With this experimental code I am seeing a 26GiB ARC at the moment, and I have seen at least as much as kstat.spl.misc.spl_misc.os_mem_alloc: 35,670,573,056 memory having been obtained from osif_malloc()->IOMallocAligned(). So my kext is using somewhat, but not much, more than 33 GiB (we make some allocations via other kernel interfaces, too, and those don't pass through osif_malloc() and therefore don't increment that particular kstat).
@dmzimmerman : the only thing I can think of is that you may have installed a recent package built by lundman without first making sure that a previous install was truly gone. Personally, i prefer /usr/local/zfs/{bin,sbin,libexec,...} to /usr/local/bin et al. I did get surprised in resorting to a pacakge lundman built for me personally for bootstrapping purposes, because I had always built my own installation (rather than using a release) and had put zpool, zfs etc. in /usr/local/sbin, and so had to update the PATHs of a few scripts of mine. However I had cleaned out (post-migration-from-intel-mac) my previously installed openzfsonosx code, so zconfigd/zsysctl was doing the right thing.
I think (having done a very brief UTSL) that at least since May 2021 packages have been using /usr/local/zfs.
@rottegift: I think you may have misread my comment (or I don't understand what you're saying in response). My zsysctl is in /usr/local/zfs/bin, not /usr/local/zfs/sbin (as you expected), and definitely not in /usr/local/sbin or /usr/local/bin (which have no ZFS binaries); but the problem isn't that it doesn't exist or is in the wrong place, the problem is that zsysctl resets the tunable to 0 instead of actually doing what I have set in the configuration file, when I run it by hand. (And the other problem, of course, is that I'm still seeing the same panics even when setting the low arc_max by hand; should I go lower than 16GB?)
@dmzimmerman - The output of zsysctl should be identical to if you copied and pasted the non-commented-out lines from /etc/zfs/zsysctl.conf to just after "sudo sysctl ". What you are reporting is unexpected, and not what I have ever seen from zsysctl. It's also not a complex tool, in coding terms.
I don't know the provenance of your zsysctl. Packaging has never been my department, and it might be that the whatever installed your zsysctl correctly put it there (and likewise put all the other userland files in concordant locations). My guess was that your zsysctl is old (and maybe file and otool -L on it can prove this), but that's only a guess. Is there any chance your zsysctl is being run under rosetta? This is really a question for lundman, and frankly the subject for a different issue number / standalone problem report. (FWIW, he did scratch his head about it yesterday on IRC).
Yes, you should try going lower than 16GiB. Off the top of my head, I can think of workloads that might drive the ARC-size-to-total-memory-allocated ratio upwards. However, do verify that the sysctl has been set, in case there is any doubt about that. If you it is set before import or other activity might exercise ARC, you should see e.g.:
% sudo sysctl kstat.zfs.darwin.tunable.zfs_arc.max=429467296
kstat.zfs.darwin.tunable.zfs_arc.max: 0 -> 429467296
% sysctl -a | grep 429467296
kstat.zfs.darwin.tunable.zfs_arc.max: 429467296
kstat.zfs.misc.arcstats.c_max 429467296
(The last line reports a variable generated by ARC).
Fair enough. My zsysctl dates from September 8 (in the most recent package lundman built), and is definitely a universal binary. And I did verify that the crashes occurred with the sysctl actually set. I've lowered it to 8589934592 (8 GB) now and we'll see if that does any better. And I'll file an issue about zsysctl.