macOS updates often break nix installation (updates replace path-hooks on multi-user install)
amckinlay opened this issue · comments
I have noticed that every once and a while /Users/andrewmckinlay/.nix-profile/bin and /nix/var/nix/profiles/default/bin disappear from my $PATH. The last time this happened, the hook nix installed was missing from /etc/zshrc.
Copying the following hook (copied from /etc/bashrc) to /etc/zshrc and restarting the shell fixed everything again:
# Nix
if [ -e '/nix/var/nix/profiles/default/etc/profile.d/nix-daemon.sh' ]; then
. '/nix/var/nix/profiles/default/etc/profile.d/nix-daemon.sh'
fi
# End Nix
My concern is that macOS updates will periodically overwrite /etc/.zshrc (and /etc/.bashrc), requiring effort on my part to reinstall nix. For example, one macOS update overwrote the entire file to fix this misspelling:
andrewmckinlay@imac ~ % diff /etc/zshrc.backup-before-nix /etc/zshrc
3c3
< # Setup user specific overrides for this in ~/.zhsrc. See zshbuiltins(1)
---
> # Setup user specific overrides for this in ~/.zshrc. See zshbuiltins(1)
Both files are read-only anyway, should they really be written to by the nix installer?
If they keep updating these files, perhaps we should file an Apple Feedback report asking them to add /etc/bashrc.d/ and /etc/zshrc.d folders whose contents get sourced by the default scripts? This way we can add configuration there that won't get overridden.
Beyond that, the only automatic solution that comes to mind would be to have nix-daemon check these files on launch and re-add the Nix block if it's missing.
I have done a few days before the upgrade to Catalina 10.15.6... and today I wanted to install something to discover that my nix installation was gone!
Well after some research, I found that the /nix was still there, but everything else was gone... no nix-env command nothing!
So after cursing a lot... I search until I found this thread...
I would like also that this problem to solved... For the moment, I have put on my 2 users the following lines in the ~/.profile
# Nix
if [ -e '/nix/var/nix/profiles/default/etc/profile.d/nix-daemon.sh' ]; then
. '/nix/var/nix/profiles/default/etc/profile.d/nix-daemon.sh'
fi
# End Nix
So I don't care if the /etc/zshrc is or not overwritten in the future... I agree this is a bad solution, but at least for the time being my workaround...
I filed feedback for Apple (FB8181728). In the meantime, nix-daemon will need to learn how to modify the files appropriately on launch.
The zsh case seems to have been fixed (but not released yet) in the course of #3456. I haven't observed this same issue with bash shell files.
Edit: this had its own problems and was reverted.
This issue has been mentioned on NixOS Discourse. There might be relevant details there:
I'm still seeing this issue as of Nix 2.3.15 and Big Sur 11.5.2. The workaround is to re-add the following to /etc/zshrc:
# Nix
if [ -e '/nix/var/nix/profiles/default/etc/profile.d/nix-daemon.sh' ]; then
source '/nix/var/nix/profiles/default/etc/profile.d/nix-daemon.sh'
fi
# End NixI'm still seeing this issue as of Nix 2.3.15 and Big Sur 11.5.2.
Edit: I see in matrix that #3608 caused its own problems documented in #4169, so I guess a release would actually be a step in the wrong direction atm... Striking through the parts that may not actually be much help...
FWIW this is roughly expected:
The fix for the ZSH case was merged in #3608, but there hasn't been a release from master since then (and it doesn't look like anyone's backported it). I'm not really in the loop, but at least per https://discourse.nixos.org/t/nix-release-schedule-and-roadmap/14204 there should be a release from master coming fairly soon?- Even after a new release is cut from master, I expect you'll continue to see this behavior on an old install until you manually migrate or manually remove Nix and reinstall it.
I would try manually migrating. I think you'd just need to create /etc/zshenv if it doesn't exist, move your source line there, and verify it works.
Also: brownie points if you (or maybe any zsh user who hasn't taken 11.5.2 yet?) remember to report back here or in #3608 if this migration helps you dodge this problem.
can't we just patch zsh to use different rcfile paths? users can then set the patched zsh as their shell with nix-darwin
@ggPeti I assume it's ideal if it still works without Nix-darwin, but tbh I have no clue how big the set of (Nix + zsh - nix-darwin) uses is.
#4169 seems to have a more robust discussion on ~zsh-specific fixes, and I'm trying to fish around for someone to add it to the 2.4 milestone; if I find someone, that might be a good place to pull on the thread...
#4169 got closed without dealing with this part of the problem.
I recently stumbled on a reference to someone saying they'd taken an update and had the update leave their modified items in place, and add the replaced items to the Relocated Items directory.
I have a spare laptop on the beta release channel and wanted to try this out, so I took an update on it this afternoon to 11.5 and noticed two things:
- It did not displace
/etc/bashrcon this update--it just placed the "official" bashrc atRelocated\ Items/Configuration/private/etc/bashrc.system_default. - It did, however, still overwrite /etc/zshrc.
I'm not sure if this means they're treating zsh init files differently since it's the default shell, or if it means they inadvertently left zshrc out of the list of files to respect instead of replace. Something to watch closely in the next few updates, I guess.
I recently stumbled on a reference to someone saying they'd taken an update and had the update leave their modified items in place, and add the replaced items to the Relocated Items directory.
I have a spare laptop on the beta release channel and wanted to try this out, so I took an update on it this afternoon to 11.5 and noticed two things:
- It did not displace
/etc/bashrcon this update--it just placed the "official" bashrc atRelocated\ Items/Configuration/private/etc/bashrc.system_default.- It did, however, still overwrite /etc/zshrc.
I'm not sure if this means they're treating zsh init files differently since it's the default shell, or if it means they inadvertently left zshrc out of the list of files to respect instead of replace. Something to watch closely in the next few updates, I guess.
I tried this with an update from 12.0 beta to 12.0.1 and, once again, zshrc was overwritten.
This issue has been mentioned on NixOS Discourse. There might be relevant details there:
This issue has been mentioned on NixOS Discourse. There might be relevant details there:
https://discourse.nixos.org/t/nix-commands-missing-after-macos-12-1-version-upgrade/16679/5
And of course this happened again on 12.2. (@frlan)
Please deprecate nix on macOS.
... please don't?!?
Every macOS update breaks nix. It's clearly not ready for macOS.
When trying to debug this problem for myself, I think I was misled by the current documentation. For example, this discussion of environment variables
https://nixos.org/manual/nix/stable/installation/env-variables.html
assumes a single-user installation. (I think?) And because I don't have prefix/etc/profile.d/nix.sh I thought my installation might be broken. Am I right in thinking there should be separate sections for single and multi user installations and the multi-user section should mention nix-daemon.sh instead of nix.sh?
assumes a single-user installation... Am I right in thinking there should be separate sections for single and multi user installations and the multi-user section should mention nix-daemon.sh instead of nix.sh?
Good catch. I think so. If none of the differences between the two scripts seem to fall within the scope of the current documentation, it might suffice to just say that each mode has a distinct script and list the path for each?
That sounds reasonable to me
Happened to me today, macOS Monterey 12.2 -> 12.3 update.
Fix that worked for me:
# 1. Remove backup files
sudo rm /etc/{bashrc,zshrc}.backup-before-nix
# 2. Run installation script again
sh <(curl -L https://nixos.org/nix/install)
# Doing 2. before 1. makes the installation script fail.
@thiagowfx You don't have to reinstall. As noted multiple times in the thread, you just need to re-add the profile/rc hook to /etc/bashrc or /etc/zshrc.
And because I don't have
prefix/etc/profile.d/nix.shI thought my installation might be broken.
Same for me. I "fixed" it by running
/nix/store/*-nix-2.*/bin/nix-env -iA nixpkgs.nix
Which creates all the same links in your per-user profile, but of course, that's only a viable workaround for a single user on a multi-user install.
This issue has been mentioned on NixOS Discourse. There might be relevant details there:
https://discourse.nixos.org/t/fixing-your-install-after-osx-upgrade/19339/6
Same issue on my side, while updating from Monterey 12.1 to Monterey 12.5 on a Apple M1 Pro...
Fixed by adding the following lines on /etc/zshrc
# Nix
if [ -e '/nix/var/nix/profiles/default/etc/profile.d/nix-daemon.sh' ]; then
. '/nix/var/nix/profiles/default/etc/profile.d/nix-daemon.sh'
fi
# End Nix
Fixed by adding the following lines on
/etc/zshrc
how can i do if i have remove the pre-installed version, Now it can't install successful ........
Fixed by adding the following lines on
/etc/zshrchow can i do if i have remove the pre-installed version, Now it can't install successful ........
@somebodyLi The instructions didn't say to remove this system file, so I don't know why you removed it.
- If you still have
/etc/zshrc.backup-before-nix, all you need to do issudo mv /etc/zshrc.backup-before-nix /etc/zshrc(as the instructions say). - If you don't have the file, you're going to have to find a replacement. Restore it from backups if you have them. Otherwise, you could re-create it using the version in Apple's org on github: https://github.com/apple-oss-distributions/zsh/blob/zsh-92/zshrc. I have no idea if this file is up-to-date.
This issue has been mentioned on NixOS Discourse. There might be relevant details there:
https://discourse.nixos.org/t/nix-install-breaks-after-every-macos-monterey-update/21353/4
I have encountered what I think is another flavour of this issue (#7106), except this time I lost /etc/fstab on update and my nix volume was modified during the upgrade to mount /Volumes/Nix\ Store instead of /nix. This breaks all the symlinks. Trying to figure out what how to correctly change the mount point back to /nix (just started looking because I just figured out roughly what was going on).
Edit: This was upgrading 12.5.1 to 12.6, btw
Edit 2: I was able to resolve the issue and think that it might have coincidentally lined up with an OS upgrade. The /etc files might have been removed by multiple runs of the install script but there was no indication of an issue until a system restart, which changed the volume mount point. That is the new theory, at least.
For future reference this is what /etc/fstab should look like:
#
# Warning - this file should only be modified with vifs(8)
#
# Failure to do so is unsupported and may be destructive.
#
UUID=XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX /nix apfs rw,noauto,nobrowse,suid,owners
With the appropriate UUID.
Edit 2: I was able to resolve the issue and think that it might have coincidentally lined up with an OS upgrade. The /etc files might have been removed by multiple runs of the install script but there was no indication of an issue until a system restart, which changed the volume mount point. That is the new theory, at least.
FWIW I'm currently on 12.6 and had no issue whatsover when upgrading to it (other than the usual /etc/zshrc).
Yeah, I used https://gist.github.com/meeech/0b97a86f235d10bc4e2a1116eec38e7e#check-you-have-nix to help me get back to a working state. It was both /etc/synthetic.confg and /etc/fstab that I needed to re-create.
I just upgraded to macOS 13.1 (22C65), it seems /nix survived.
➜ ~ diskutil list | grep Nix
7: APFS Volume Nix Store 1.4 GB disk3s7
➜ ~ diskutil info disk3s7 | grep Mount
Mounted: Yes
Mount Point: /nix
Hence the only restoration really need is ~/.zshrc or /etc/zshrc
. '/nix/var/nix/profiles/default/etc/profile.d/nix-daemon.sh'
@alphatownsman for me nix always survived but the /etc/zshrc is overwritten in every update needing manual adding the shell hook back into this file to make it work again.
As an other issue noted #3456 we should use /etc/zshenv which is global and loaded by default BUT not overwritten by an OS update since it does not exist by default.
Tested it and that way the shell hook survived the last macos update. No action was needed (adding sth back to /etc/zshrc)
IMHO we should change the installer script to use /etc/zshenv instead of /etc/zshrc for mac
@FloThinksPi we tried zshenv but reverted because it causes its own problem. See #4169
Edit: but it's a fine approach for individuals who want their system binaries to have priority. This isn't the normal case, but I know there are at least a few people out there who prefer that behavior.
@abathur Dang! Yea i also dont want that prio of binaries.
Hm there must be a way to have the shell hook somewhere that does not get overwritten, brew e.g. puts it in the users shell env by printing a snipped people have to add themselfs. Maybe we should just print the snipped in the installer too and let people figure out where to put the hook. Thats also how pyenv and other tools which mess with the shell do it.
Docker does this also to the users ~/.zshrc but instead of printing it, inserts it on its own which i feel like many people would not like since i personally dont like a tool altering my versioned dotfiles.
source /Users/<username>/.docker/init-zsh.sh || true # Added by Docker Desktop
Anyway that way people would not expect the nix installer to manage that part and we could close the issue that way.
The issue is that we can't use /etc/zshenv because the $PATH we set gets overriden by /etc/zprofile.
Instead we use /etc/zshrc, which gets overriden by macOS on upgrades.
There's a detailed explanation how the startup of zsh works: https://gist.github.com/Linerre/f11ad4a6a934dcf01ee8415c9457e7b2
The only solution I can think of is:
- prepend
/nix/var/nix/profiles/default/binto/etc/paths - when invoking
nixcheck for__ETC_PROFILE_NIX_SOURCED=1and if absent, tell the user to run a command that will populate/etc/zshrc.
Nix will still break, but the fix for the user is provided in a nice way.
I communicated this issue upstream to our devrel this past fall (it was passed along, but I haven't heard anything else).
One ~alternative solution I've seen suggested is either updating nix-daemon or installing a separate daemon that will fix-up if it goes missing.
Another is just caving and using ~/.zshrc.
I don't remember for sure, but we might also be able to try to pre-empt path_helper (which is what screws up the PATH if we use zshenv, and what processes /etc/paths and /etc/paths.d/*) by declaring a function in zshenv with its absolute path (/usr/libexec/path_helper), and then implementing it as a wrapper there that can ~fix the PATH it generates. (But I'm just assuming this is valid in zsh; function names with slashes are at least valid in bash).
~/.zshrc means that it's very hard to support multi-user installs properly, so I'd be up for shoving it into nix-daemon.
Triaged in the Nix team meeting:
- @thufschmitt: the only thing we could do is building the installer in a way that it doesn't touch files that will be messed up by macOS updates
- there is no obvious way how to do it
- to discuss, maybe we can solicit help from people close to macOS and Apple
This issue has been mentioned on NixOS Discourse. There might be relevant details there:
https://discourse.nixos.org/t/2023-03-03-nix-team-meeting-minutes-37/25998/1
Would using a LaunchDaemon be an alternative solution to modifying the fstab directly (for the fstab portion of the issue). Have looked at this as a possibility in in issue #7106 , though it is still a bit of an experiment.
For later travelers, I'm fairly certain that the macOS update did not remove your fstab. The fstab ~issue noted by a couple of late comments in this thread is a side effect of trying to fix the installation by re-running the installer (which is not currently idempotent) without completely uninstalling first.
If you run into the core problem here (macOS update overwriting the shell init hook for ZSH) and tried running the installer without uninstalling, you'll need follow the uninstall instructions before reinstalling it: https://nixos.org/manual/nix/stable/installation/installing-binary.html#macos
Edit: someone in matrix confirmed that they installed 13.3 on march 27 2023 and it did not remove fstab.
I'm responding here to part of a comment from another issue because it's more on-topic here. The relevant bit:
Edit: just looking at the zsh files and would one of the files such as /etc/zprofile by a suitable alternative, though again would need to check what files Apple indicates as being safe from overwrites?
I'm not really sure. If there's a problem with it, it'll probably be because *zshrc files should get sourced for any interactive shell, while *zprofile will only get sourced for login shells. If it is viable, I agree that we should get confirmation that updates don't clobber it before we try to change the installer.
It's a bit of an aside, but there has also been some mumbling (throughout this thread, and in the installer workgroup meetings) about Nix-side alternatives that roughly boil down to:
- making the nix CLI/daemon itself able to detect and at least warn about if not cure problems like this
- working toward reducing the scope of these init files at the Nix level (by volume they mostly deal with setting up cert envs) since this may help fix some other issues and might open up some other other paths to fixing this one (though I suspect the way Apple uses
path_helperin/etc/zprofilewill keep that from yielding any better options)
BTW I decided to ask the question about zsh related files on StackExchange, and it looks like other people deal with the same headache, as part of their sysadmin work:
I experienced this upgrading from Ventura 13.3 to the beta Ventura 13.4. I can confirm the fix as mentioned above by many worked for me - appending the following to '/etc/zshrc':
# Nix
if [ -e '/nix/var/nix/profiles/default/etc/profile.d/nix-daemon.sh' ]; then
source '/nix/var/nix/profiles/default/etc/profile.d/nix-daemon.sh'
fi
# End NixI had aliased so many commands to programs installed through Nix that zsh was unusable, I had to use bash to actually edit /etc/zshrc.
@ajmas from a comment to the answer:
launchd task / cron task that checks for the required edits, exits quietly or removes itself if there's no work to do.
Something like this seems to be the only somewhat reliable option. Maybe nix-daemon itself could do that?
I see that the nix-daemon.sh script prepends PATH items without checking to see if they're in PATH already. I know this is pretty common practice but the idea of duplicating PATH entries ad nauseam always bothered me so I wrote this Bash function to prepend but first eliminate the relevant entry from PATH if it's already there, making this operation idempotent. The function also works with other PATH-like variables such as LD_LIBRARY_PATH etc.
# function to prepend paths in an idempotent way
prepend_path() {
local dir="${1%/}" # discard trailing slash
local var="${2:-PATH}"
if [ -z "$dir" ]; then
echo "Usage: prepend_path <path_to_prepend> [name_of_path_var]" >&2
return 2 # incorrect usage return code, may be an informal standard
fi
local newvalue=${!var}
[[ $newvalue =~ ^$dir: ]] && return # quit if value already starts with $dir
newvalue=${newvalue%:$dir} # remove $dir from end of path
newvalue=${newvalue//:$dir:/:} # remove $dir from middle of path
# prepend the new entry
export ${var}="$dir:$newvalue"
}
Hm there must be a way to have the shell hook somewhere that does not get overwritten, brew e.g. puts it in the users shell env by printing a snipped people have to add themselfs.
I run fish and well… colour me surprised that the Nix installation had messed with /etc/zshrc and that a change in that file has an impact on my shell which is not zsh.
I think asking people to add it themselves to the shell they use is the way to go.
Hm there must be a way to have the shell hook somewhere that does not get overwritten, brew e.g. puts it in the users shell env by printing a snipped people have to add themselfs.
I run fish and well… colour me surprised that the Nix installation had messed with
/etc/zshrcand that a change in that file has an impact on my shell which is not zsh.I think asking people to add it themselves to the shell they use is the way to go.
I agree with this. I bet people who install nix on their mac are competent enough to edit their shell config, and plenty of other tools ask you to do so anyways, my .zshrc is full of that stuff already.
On Ventura 13.4, the file /etc/zshrc is readonly by default:
ls -la /etc/zshrc
Permissions Size User Date Modified Name
.r--r--r--@ 3.1k root 19 May 11:12 /etc/zshrc
It's one more barrier for users.
On Ventura 13.4, the file /etc/zshrc is readonly by default:
This isn't a change, FWIW. It has been true of shell init files including /etc/{bashrc,zshrc,profile,zprofile} since at least Catalina (probably earlier).
Since I am the only one using my computer, just including the necessary lines in my user's ~/.zshrc is the simplest solution for now.
for those using a home manager, you can use the following configuration:
programs.zsh.initExtra = "if [ -e '/nix/var/nix/profiles/default/etc/profile.d/nix-daemon.sh' ]; then
. '/nix/var/nix/profiles/default/etc/profile.d/nix-daemon.sh'
fi";
This issue has been mentioned on NixOS Discourse. There might be relevant details there:
https://discourse.nixos.org/t/why-can-i-not-execute-a-new-version-of-nix-with-nix-shell/31032/1
For those who manually modified their user's ~/.zshrc, you may want to change the condition to something like this:
[[ ! $(command -v nix) && -e "/nix/var/nix/profiles/default/etc/profile.d/nix-daemon.sh" ]] && source "/nix/var/nix/profiles/default/etc/profile.d/nix-daemon.sh"Otherwise, trying to test a new version of nix by using nix shell will still use the old version. See the thread linked by the discourse bot above.
The issue still persists on Sonoma 14.4 update.
persists on Sonoma 14.5 upgrade.
Has anyone considered/reviewed https://zero-to-nix.com/concepts/nix-installer ?
I will try this out with the update to 14.5 but so far it seems they dont write into the /etc/zshrc rather create a plist entry to make nix available in the shells path, interesting. May survive an update.
Edit: It survived the 14.5 update without issue it seems, when using the installer by Determinate Systems. Its not the official one though and despite the company being heavily involved in nix, its not wrong to question thrust-worthiness when you curl a script and run it as root :)
I use a Zero-to-Nix install on several Apple Silicon machines. It’s has survived several updates.
so far it seems they dont write into the /etc/zshrc rather create a plist entry to make nix available in the shells path, interesting.
This doesn't quite characterize it right. The DetSys installer does still modify shell profiles, but it also installs a launchd daemon (the plist) that runs /nix/nix-installer repair when it loads which in turn re-applies the shell profile fixes. Here's the implementation PR if you're curious:
Has anyone considered
Yep!
There's a slowly-progressing ~unofficial/experimental project in the NixOS org to work on a fork of the DetSys installer (fork mainly because the Nix team isn't yet on board with some of the ~opinionated differences between their installer and the canonical one, so we need to un-implement those or implement the official installer's approach if it differed). If that path leads to official adoption, we'll gain support for this that way.
Since the DetSys installer is leaning on its own repair command for the business end of this, implementing the same feature for the official shell installer would be a lot more complex than it looks like at a glance. One of the DetSys installer's key design decisions here is keeping a receipt of what the installer did. Without that receipt, the shell installer's version of this would probably look like one of:
- getting to (and staying at) full idempotence (and then just re-running the whole installer--with whatever user-tunable parameters were used for the first install)
- implementing full uninstall (that is able to account for user-tunable parameters), and implementing this as full uninstall and reinstall
- significantly refactor the installer to use a similar receipt approach and build something similar to the nix-installer's repair command
All of these are probably achievable, but they also smell like quite a lot of work to me. I also imagine the first two (~simpler) choices would be so slow that restoring terminal tabs with a zsh shell wouldn't have the fix when they load. My own limited energy is (at least for now) focused on getting the fork of the DetSys installer to the point where it's ~ripe for the Nix team to evaluate.
(We're close. I've delayed making some requests around this since it isn't urgent and many of the people I need to bug have been focused on urgent governance stuff.)
Around the same time as DetSys were adding this feature, I also opened a PR with a narrower (albeit weirder) workaround:
Aside: A third approach, which lilyball has advocated, is that
nix-daemonitself should be able to do this itself.
I'm a little cautious around both workarounds because they're "weird" in a (slightly different) way that might lead some people to worry their system is compromised. Graham's comment linked below and my response reflect on this property of both:
I haven't leaned on my PR much because:
- If it doesn't cause complaints or regular trouble of its own (i.e., flaky behavior due to a race condition, people disabling the service, etc.), I think the DetSys approach is better from an end-user perspective--it can repair a broader set of problems than my PR.
- Work on the fork will likely get us the same benefit, so I don't see much reason to add to the delta between the two installers until that resolves on way or another.