Mail.app halted on 'Unable to quarantine' when UserDirectory stored on ZFS (MacOS 14+)

Question

Mail.app halted on 'Unable to quarantine' when UserDirectory stored on ZFS (MacOS 14+)

captain-haddock17 opened this issue 4 years ago · comments

Hi all !
Since my recent (december) upgrade from MacOS 10.13.x to 10.14.6,
all my MacOS user accounts stored on ZFS can NOT have their e-mails (through Mail.app).

Short story :
at first initialization of a fresh new User Account (with same @icloud.com account though):

Mail.app is OK, and e-mails are all downloaded, visible , etc ...
if you close , and reopen Mail (juste after, or after a logout/login, or a Reboot) : Mail launches (visible in the menu bar on top), but NO (any) window is visible, or can't be brought visible.
Console message says : kernelMail[3138] Unable to quarantine: 1

Side Note :

All the iCloud apps are OK (Notes, Agenda, Contacts)
depending on some ZFS params (xattr on/sa ?) , KeyChain can be confused and has to be reinitialized ...

Long Story

MacPro 5,1 (2010) with SSD (APFS - macOS 10.14.6 (18G4032)/ Darwin 18.7.0) and 3 internal SATA drives.
sysctl {spl,zfs}.kext_version

spl.kext_version: 1.9.4-0
zfs.kext_version: 1.9.4-0

Most User accounts are stored on the internal 3xSATA Pool with ZFS FileSystem.

Since MacOS 10.14 ;
- Softlink can't be used anymore, typical. ls -s /Volumes/zDisk/Home/myser /Users/myuser
- param HomeDirectory has to be used declared (one of:
  - through SysTPref->User&Groups
  - through dscl localhost -read /Local/Default/Users/myuser NFSHomeDirectory (then use -change)

OSX xattr on Mail folders seems to be OK.

Classical User myadmin account on Apple HPFS

$ ls -lde@ Library/Mail

drwx------@ 5 myadmin staff 160 17 mar 09:12 Library/Mail
com.apple.quarantine 19

$ xattr -lv Library/Mail

Library/Mail: com.apple.quarantine: 0082;00000000;Mail;

User account on ZFS filesystem
on zPool mounted on /Volumes/zDisk/Home

$ sudo ls -lde@ myuser/Library/Mail

drwx------@ 4 myuser admin 4 6 avr 14:11 myuser/Library/Mail
com.apple.quarantine 19

$ sudo xattr -lv myuser/Library/Mail

myuser/Library/Mail: com.apple.quarantine: 0082;00000000;Mail;

Tests

Fidling with some of the ZFS parameters didn't solve the problem

com.apple.devdisk on : because with poolonly , ZFS can NOT mount the given useraccount FileSystem, MacOS is testing/preventing the mount of an external HomeDirectory ?!?! see issue #754
xattr = on (in first place) / sa (no better ?)

Common params

aclmode passthrough
aclinherit restricted
com.apple.mimic_hfs on
com.apple.browse on
com.apple.ignoreowner off
utf8only on
normalization formD
casesensitivity insensitive

_Other Fidlings _

tried to put ../Library on a jHFS formatted ZVOL with a softlink ln -s /Volumes/myuser.Library /Volumes/zDisk/Home/myuser/Library
declared Mail.app having Full Disk Access in "Security & Privacy" Prefs (no better 👎 )

Any clue ?
Thanks !

Kind regards,
William

JMoVS · Answer 1 · Mon Apr 06 2020 23:05:48 GMT+0800 (China Standard Time)

One short question: Is it HFS+ or APFS? To the best of my knowledge Apple doesn't have HPFS and the technical background and intricacies are quite a bit different between APFS and HFS+. Since this is 10.14 I would guess APFS but could you clarify?

JMoVS · Answer 2 · Mon Apr 06 2020 23:07:43 GMT+0800 (China Standard Time)

When you say it wasn't able to mount, did you make sure that:

the mountpoint was defined and empty? (No hidden files either? What is the error message of ZFS attempting to mount?)
put the zfs and the zpool binary in the Full disk access list?

William J. Franck · Answer 3 · Mon Apr 06 2020 23:52:10 GMT+0800 (China Standard Time)

Hello JMoVS,

Thanks for the follow-up :-)

Typo APFS corrected in description
MountPoint : (with com.apple.devdisk=poolonly
- yes, directory was empty. I did several tests, with 2 different Users and ZFS dataset.
- I could do a zfs mountafterwards.
- On bootime, these 2 ZFS dataset didn't mount. I figured it out with com.apple.devdisk=on and this solved the issue (maybe a side effect ?).
- I did not tried to declare an entry in fstab, hoping to force the sequence of disk mounts.

see issue #754.

Sorry, impossible to put spl.kextand zfs.kext in Full disk access. (greyed at the selection through gateKeeper)

William J. Franck · Answer 4 · Tue Apr 07 2020 00:06:12 GMT+0800 (China Standard Time)

(sorry bad manoeuvre has closed this issue !!)

JMoVS · Answer 5 · Tue Apr 07 2020 00:36:53 GMT+0800 (China Standard Time)

@captain-haddock17 Try to put the zfs and zpool binaries to full access as well as terminal.app. You'll find the binaries if you run which zfs and use that path to figure out where they are

JMoVS · Answer 6 · Tue Apr 07 2020 00:38:24 GMT+0800 (China Standard Time)

also you might run into a timing issue with your setup where mac OS runs the zpool import all launchd job after it already tried to load the user - resulting in you running into problems. We'd have to ask I think @cbreak-black or maybe @rottegift on how they're handling user home dirs on ZFS.

JMoVS · Answer 7 · Tue Apr 07 2020 00:39:15 GMT+0800 (China Standard Time)

also what is the current mountpoint? Mojave has very strict rules on where you can mount stuff and where you're not allowed to mount stuff...

William J. Franck · Answer 8 · Tue Apr 07 2020 00:51:45 GMT+0800 (China Standard Time)

Terminal.app was already in Full disk access
some more testing with cmpxat v2 from The Eclectic Light Company on some real Mail.app file 7.emlx
- xattr -lv 7.emlx

7.emlx: com.apple.quarantine: 0082;5e21acf6;Mail;

(file renamed to from-APFS for the sake)

cp -a from-APFS --> ZFS-xattr=on
cp -a from-APFS --> ZFS-xattr=sa
cmpxat from-APFS ZFS-xattr=on :: result OK
cmpxat from-APFS ZFS-xattr=sa :: result OK

JMoVS · Answer 9 · Tue Apr 07 2020 01:27:30 GMT+0800 (China Standard Time)

I have mail.app's data (~/Library/Mail) in a dataset which works fine. My settings are devdisk=on, mimic_hfs=on and xattr=on

I don't have experience with having the container stuff on ZFS though unfortunetly

JMoVS · Answer 10 · Tue Apr 07 2020 01:30:04 GMT+0800 (China Standard Time)

you could try if everything else fails to create a dataset for mail, mount that to that location with the settinhs I mentioned (and all other settings being just inherited as is) and see if that works

William J. Franck · Answer 11 · Sat Apr 11 2020 15:06:47 GMT+0800 (China Standard Time)

Some clue :

While I was trying to do some more test cases, with copy of /Users/myUser to a ZFS zvol with APFS format,
say sudo cp -a /Users/rover-yachts.5 /Volumes/zmacOS/Users

I got some strange error messages, namely:

cp: /Users/rover-yachts.5/Library/Containers/com.apple.mail/Data/DataVaults: Operation not permitted

no ls -lAnor matter -lvcould show this file /Library/Containers/com.apple.mail/Data/DataVaults

PS : needless to mention : Terminal.app has Full Disk Access on. :-)

Quote from The Eclectic Light Company page No Entry: access controls in Mojave:

Introduced in later releases of High Sierra, and used more in Mojave, are folders to which only Apple’s software has even read access, DataVaults. My account here is largely based on comments generously provided here by an anonymous source, as these don’t appear to have been mentioned anywhere by Apple (not even at WWDC 2018), nor can I find other descriptions.

DataVaults are folders to which neither the user nor third-party software has any access at all. The only software which can see and work with their contents are certain Apple-signed products which have a specific entitlement to do so. At present, all Macs running Mojave have at least one, which contains the QuickLook cache, at /var/folders/t9/[long ID]/C/com.apple.QuickLook.thumbnailcache Depending on what other features and apps you use, you may also find them at three additional locations:

~/Library/VoiceTrigger/SAT

~/Library/Containers/com.apple.mail/Data/DataVaults

/var/folders/0z/[long ID]/0/com.apple.nsurlsessiond

I found some interesting reading about Mojave's new behavior:

Access to ~/Library/Containers/com.apple.mail/Data/ denied
Apple DocRef about Entitlements

William J. Franck · Answer 12 · Sat Apr 11 2020 15:27:45 GMT+0800 (China Standard Time)

Should ZFS need a special zfs function in order to handle the very special file flag UF_DATAVAULT put on those directories ?

Ref : The Eclectic Light Company , last § in No Entry: access controls in Mojave

William J. Franck · Answer 13 · Sat Apr 11 2020 15:37:10 GMT+0800 (China Standard Time)

More info about Mail.app

$ codesign -d --entitlements - /Applications/Mail.app/

(...)
	<key>com.apple.rootless.storage.Mail</key>	<true/>
	<key>com.apple.security.app-sandbox</key>	<true/>
	<key>com.apple.security.cs.disable-library-validation</key>	<true/>
	<key>com.apple.private.security.storage.Mail</key>	<true/>

William J. Franck · Answer 14 · Sun Apr 12 2020 16:21:52 GMT+0800 (China Standard Time)

I found a workaround,
as suggested by JMoVS, and my personal test cases;

Put ~/Library/Mail and ~/Library/Containers on a ZFS zvol and making a softlink from the regular ZFS Filesystem to this jHFS (or APFS) formated zvol.

It's really a pain, because

it's not very practical to create as many zvoldisks that will be seen on your desktop as you have as accounts (5 for me).
and putting these sub-librairies on one zvoldisks will de-normalize somewhat my backups and archives per User.

Positive : At least, I can go ahead and keep ZFS as main storage solution 👍

beren12 · Answer 15 · Sun Apr 12 2020 19:56:30 GMT+0800 (China Standard Time)

if you disable SIP do you still get the error? Just wondering

JMoVS · Answer 16 · Sun Apr 12 2020 21:44:12 GMT+0800 (China Standard Time)

@captain-haddock17 You don't need to put ~/Library/Mail on APFS I think. I have SIP only partially enabled but the core problem is AFAIK in the Container. But the Emails are stored in ~/Library/Mail, so maybe try to leave ~/Library/Mail in ZFS and just put the Container on an APFS zvol ;-)

JMoVS · Answer 17 · Sun Apr 12 2020 21:44:49 GMT+0800 (China Standard Time)

also you could try to not solve this via a softlink but instead do it via specifiying the mountpoint, making the system a bit cleaner

William J. Franck · Answer 18 · Thu Apr 16 2020 14:58:17 GMT+0800 (China Standard Time)

Some more investigations:

Launching Mail.app on some empty directory (he will recreate subdirs)
Got error pannel at launch

Mail cannot open ... Mail is saving data on /Macintosh HD/Users/myuser/Librray/Mail
Note: always same directory name is labeled...

In System log

Error kernel Sandbox: Mail(41788) deny(1) file-write* /Volumes/zDisk/Users/myuser/Library/Mail

Disk info/params
Owners =Enabled or Disabled is a property shown on disk either:

diskutil info disk#.#
through "CMD+I" (with cadnass off) on the Drive shown in Finder or Desktop

Test Conditions
with SIP disabled: (huh !?! maximum chances to get some positive results...)

ZFS partition /Volumes/zDisk/Users/myuser

xattr = on / sa (sees that it doesn(t care)
Mac User HomeDirectory set to /Volumes/zDisk/Users/myuser
~/Library/Keychains softlink on main startup disk /Users/myuser
~/Library/Caches softlink on main startup disk /Users/myuser
~/Library/Containers softlink on main startup disk /Users/myuser
~/Library/Group\ Containers softlink on main startup disk /Users/myuser
~/Library/Application\ Support softlink on main startup disk /Users/myuser
varyinging Soft links on Library/Mail (with proper ownership chown -h)
as i.e ln -s /Volumes/Macintosh\ HD/Test/Mail /Volumes/zDisk/Users/myuser/Libraray/Mail

Test cases

on internal (synthesized) APFS (Owners=Enabled): OK anywhere on disk
on internal, virtual ZFS : ERROR
on external, virtual ZVOL APFS formated (Owners=__/Disabled): ERROR
on external, virtual ZVOL JHFS+ formated (Owners=__/Disabled): ERROR
on external, physical USB disk JHFS+ formated (Owners=__/Disabled): ERROR
on external, physical USB disk APFS formated (Owners=Enabled or Disabled): ERROR
on created/owned Image Disk JHFS+ formated , *.dmg stored in /Users

Did some more test on weird /Macintosh HD/ as first level of root filesystem ...
actually it should be /
I tryied a softlink ln -s /Volumes/zDisk /Macintosh\ HD
Still same issue "mail cannot open ..."

First conclusion (?)

Maybe the first clue is that we must have (at least)

Internal + Owners=Enabled

because on regular external disks, formated by regular plain MacOS APFS ou JHFS+ doesn't work.

William J. Franck · Answer 19 · Thu Apr 16 2020 16:20:08 GMT+0800 (China Standard Time)

Could be related to Sandbox (Jail) forcing special constraints on disk ?

startup disk ?
Internal
Physical ?
Owners=Enabled

Could ZFS on OSX adapt ZFS Filesystems so as to show mounted ZFS as these kind of internal disks ?

William J. Franck · Answer 20 · Thu Apr 16 2020 16:40:07 GMT+0800 (China Standard Time)

Other clue:
With Macintosh\ HD as startup-disk
Mail.app starting on empty folder ~/Library/Mail

Pointing to the regular /Users folder

ln -s /Users/mysuser/Library/Mail /Volumes/zDisk/Users/myuser/Library/Mail
is OK

Pointing to equivalent same /Users folder

ln -s /Volumes/Macintosh\ HD/Users/mysuser/Library/Mail /Volumes/zDisk/Users/myuser/Library/Mail
has same ERROR as above.

??? WTF ?

updated:

Pointing from the regular /Users/myuser/Library/Mail folder to else-where is OK as long as it is on a JHFS+ or APFS formatted filesystem (ZVOL is OK)

mv -v /Users/mysuser/Library/Mail /Users/mysuser/Library/Mail-local
ln -sv /Volumes/zLibraries/myuser/Library/Mail /Users/mysuser/Library/Mail

If .../Mail directory is located on a regular ZFS formated filesystem, Mail.app is emitting Unable to quarantine errors in the MacOS Console log.

That's where I'm at.

JMoVS · Answer 21 · Fri Apr 17 2020 02:46:59 GMT+0800 (China Standard Time)

Have you tried to run chown -R $User on the Library/Mail Dataset?

why are you using softlinks and not simply mountpoints? I have no problems with a Dataset that is mounted to ~/Library/Mail that contains my mail data. Mountpoints are more solid than links. I don't expect softlinks to work well.

chown -R $User is what I needed for my permissions of the datasets and then I just mount them to the respective place, eg ~/Library/Mail

William J. Franck · Answer 22 · Fri Apr 17 2020 03:12:06 GMT+0800 (China Standard Time)

Hi, i have checked if my ownerships where ok, but anyway I will run chown to be sure.

Thanks for mentionning that your config is ok, this gives me some more energy to go futher in my investigations :-)

JMoVS · Answer 23 · Fri Apr 17 2020 04:29:32 GMT+0800 (China Standard Time)

Just note that I don't have my home directory on ZFS. You can jump in on IRC (freenode #openzfs-osx ) or try to ask @rottegift - I think he runs his home dir on ZFS but not sure if he's on 10.14.

Good luck! ;-)

Jorgen Lundman · Answer 24 · Fri Apr 17 2020 07:52:14 GMT+0800 (China Standard Time)

Sorry I'm a bit absent, just started the big port of OpenZFS v2 to osx. More guys are generally in the irc channel as JMoVS suggested.

William J. Franck · Answer 25 · Fri Apr 17 2020 20:42:31 GMT+0800 (China Standard Time)

Something has to do with MacOS Sandboxing (Jail) of apps.

If found out this list
sudo grep -i mail /usr/share/sandbox/*.sb

/usr/share/sandbox/mds.sb: (global-name "com.apple.mdworker.mail")
/usr/share/sandbox/mdworker-mail.sb:;; Make changes just for Mail importer
/usr/share/sandbox/mdworker-mail.sb:;; Allow Mail to read mail messages anywhere
/usr/share/sandbox/mdworker-mail.sb:(allow file-read* (regex #"/Library/Mail(/|$)"))
/usr/share/sandbox/mdworker-mail.sb: (allow file-read* (home-subpath "/Library/Mail"))))
/usr/share/sandbox/mdworker-mail.sb: (allow file-read* (param-subpath "_LIBRARY" "/Mail"))))
/usr/share/sandbox/mdworker-mail.sb:(if (positive? (string-length (param "_MAIL")))
/usr/share/sandbox/mdworker-mail.sb: (allow file-read* (subpath (param "_MAIL")))))
/usr/share/sandbox/mdworker-mail.sb:(if (positive? (string-length (param "_MAIL_V2")))
/usr/share/sandbox/mdworker-mail.sb: (allow file-read* (subpath (param "_MAIL_V2")))))
/usr/share/sandbox/quicklook-satellite-general.sb:;; Mail
/usr/share/sandbox/quicklook-satellite-general.sb:(allow file-read* (home-subpath "/Library/Mail"))
/usr/share/sandbox/quicklook-satellite-general.sb:(preference-read "com.apple.mail-shared")

Could be Mail.app do comply with the new security development rules ?

I think that this issue about Mail.app has a relation-ship with how ZFS can handle user's HomeDirectory #754 and special file flags #760 .