Support for Apple Silicon (aarch64-darwin)
arianvp opened this issue · comments
Apple will be transitioning to ARM in the near future, starting with their Big Sur release.
It would be great if we can get nixpkgs support for compiling ARM packages on Darwin
The NixOS Foundation has access to a Developer Transition Kit for testing, thanks to the friendly folks at Apple.
For now @edolstra and @rbvermaa have access to this DTK.
Note that we can get additional DTKs approved for anyone interested working on this ticket. We have a contact at Developer Relations who can help us with this. Please contact me if you would like to order a DTK to work on Nixpkgs support. (Cost is 500 US Dollars)
We also could get access to cloud machines through MacStadium. This might be interesting at a later stage for Hydra?
The original email that I got from Apple DevRel
Firstly, the most expedient way of obtaining a DTK would be for companies who have Nix maintainers and are registered as Apple developers can learn about the Universal Quick Start Program and apply for a DTK here:
https://developer.apple.com/programs/universal/
If this is the way your maintainers want to go, I would be happy to approve the applications to purchase the program. Someone noted on the thread they had applied and not been approved. I have approved then. Please let me know if there are any others.
Secondly, we have a relationship with MacStadium where qualified open source projects can simply get access to a DTK in the cloud. The maintainers of Nix should email opensource@macstadium.com.
The process for qualified OSS developers will be:
Developer signs up for a general MacStadium account so the MSA can be accepted.
MacStadium will manually add the DTK asset to their account at no cost.
Developer will have access to screen sharing on a DTK and can log in right away
My company has a DTK, so I'm planning to test Nixpkgs on it as soon as Nixpkgs is ready :)
There is already support for a aarch64-darwin platform, but it looks like that was added awhile ago for iOS support. Do we need a new platform tuple?
Yeah - we will probably have to assume aarch64-darwin means Mac instead of iOS now. With the LLVM triple we do have a way to disambiguate:
aarch64-apple-ios
vs.
aarch64-apple-darwin
Which is even more confusing, since iOS is technically still using the Darwin kernel! But nevermind that, I suspect this distinction will get even more blurred with Mac Catalyst. I'd be interested if you can just execute an iOS binary from CLI on the DTK.
Unfortunately, pure-darwin stdenv is left to wait for update of https://github.com/tpoechtrager/cctools-port. We can probably either use the native stdenv or useiOSPrebuilt stuff to get something working though.
Unfortunately, pure-darwin stdenv is left to wait for update of https://github.com/tpoechtrager/cctools-port. We can probably either use the native stdenv or
useiOSPrebuiltstuff to get something working though.
What's required from cctools-port for a pure darwin stdenv?
The code signing requirement sounds like the end of the line for macOS support in Nix - or at the very least, for having prebuilt binaries in the binary cache. Users could still build stuff locally using their local signing key (through some impure mechanism, I guess), but they wouldn't be able to share the result with other users and the binaries wouldn't be reproducible.
There's a similar thread in homebrew about this Homebrew/brew#7857 as they're gonna face similar issues with the code-signing requirement.
From what I understand; signing is done ad-hocly and needs no user interaction and thus does not pose any issues for the availability of a binary cache.
Quoting:
Yep I can confirm signing is all done automatically. It's not associated with any particular identity. The code signing seems to be purely for checksumming and not identity verification.
No further action should be needed here. Anyone here updating from beta 1-3 should note however that you may need to rebuild anything already installed. Also, as always, ensure you are running the latest version of Xcode/CLT.
So it sounds like we're good?
Or is homebrew not backed by a binary cache? i.e. always built from source?
Yep I can confirm signing is all done automatically. It's not associated with any particular identity.
Then it's not clear to me what the signing key is (or what the point of this kind of signing is...). The release notes mention "a simple ad-hoc signature issued locally" but if that's tied to the machine then you wouldn't be able to share binaries.
@arianvp There's a purity/reproducibility problem still, but I guess that's not the worst thing.
Absent proper support for detached signatures the closest thing to a "right" way to do this is to teach Nix's hashing to ignore the signature field 😭
Or is homebrew not backed by a binary cache? i.e. always built from source?
brew install downloads and installs binary bottles whenever it can unless you tell it not to.
As far as I can tell adhoc signatures are a hash of the contents of the binary and do not involve an identity, private key or any certificate chain. If so, there's no problem with reproducibility or distribution. My understanding is mostly based of a description of adhoc signing on stack overflow but I haven't found a more authoritative source.
I did a quick test of linking on a DTK. If another DTK user is able to produce the same binary from the same sources it'd be good evidence that there's no machine local key.
I've now thoroughly investigated ad-hoc signatures and implemented a minimal generator: thefloweringash/sigtool. The short version is that it is a list of sha256 hashes of the binary contents up to the start of the embedded signature. I have a work in progress branch of nixpkgs that is able to cross compile and run a basic set of utilities on a dtk. I'm currently using this tool in a setup hook (adding to fixupOutputHooks) that signs every Mach-O file in the output.
@thefloweringash Do you have a link to your branch, or instructions for other folks with a DTK to try out the currently-partially-working version of Nix?
Now that it appears Big Sur is sorted, what’s the status on this one?
Note that the foundation has one developer toolkit machine and can set it up for access if that helps anyone work on this. You can request access here :)
Once stdenv compiles, foundation will order a few machines from https://opencollective.com/nixos donations.
@thefloweringash Do you have a link to your branch, or instructions for other folks with a DTK to try out the currently-partially-working version of Nix?
I've tagged something that works as apple-silicon-1. It's incredibly messy, the commit history is equally messy, and still needs a ton of work. However, it can cross-compile bootstrap tools to aarch64-darwin, and then natively build things like hello and nix. My development branch is parrot-principles, but I make no guarantees about anything working there and will rebase and rewrite it frequently.
My cross compilation source machine is x86_64-darwin running 10.15. It should be possible to "cross compile" on aarch64 Mac via rosetta, but I haven't tried it.
To natively compile, install regular x86_64-darwin nix and update nix.conf to include
extra-platforms = aarch64-darwin
Natively building hello
$ nix-build --argstr localSystem aarch64-darwin -A hello
Cross compiling hello:
$ nix-build --argstr crossSystem aarch64-darwin -A hello
Cross compiling bootstrap tools:
$ nix-build pkgs/stdenv/darwin/make-bootstrap-tools.nix --argstr crossSystem aarch64-darwin -A build
I am not sure if this is the right issue. But would this a good moment to move away from /nix/store? We probably won't have another good opportunity like this on Darwin.
This would mean that the whole dance around creating a Nix store volume would not be necessary on Apple Silicon Macs.
As long as mac uses something like apfs, and the nix installer has a option to encrypt the partition I would like to keep a partition for the nix store. And keeping it under /nix makes the differences between oses smaller.
Another advantage of APFS is being able to opt-in to case-sensitivity, which is required for cross-compiling Linux, some nixpkgs ecosystem tools like hackage2nix, and probably other things.
Another advantage of APFS is being able to opt-in to case-sensitivity, which is required for cross-compiling Linux, some nixpkgs ecosystem tools like hackage2nix, and probably other things.
But that's regardless of whether the store lives in /nix or somewhere else. I think case-sensitive APFS works with most programs nowadays. Also, even if the Nix store is in some other location, you could still use a separate APFS volume if that's a requirement.
Sure, but I don’t think ”reinstall your entire system on a new case‐sensitive APFS volume” is a reasonable thing for us to suggest users do, and if we’re going to support creating an APFS Nix volume for that reason then we might as well also continue to use it to keep the store prefix the same across platforms. I think that if we can get an automatically‐encrypted‐on‐FileVault, case‐sensitive APFS /nix volume setup in the installer then standardizing on that for all users would be the best approach. (This is kind of off‐topic for this issue, though… AFAIK this has mostly been discussed in #nix-darwin but maybe it would be good to solicit opinions on the Discourse too?)
I am not sure if this is the right issue. But would this a good moment to move away from
/nix/store?
I think it's unrelated to apple silicon support, so this isn't the right issue.
We probably won't have another good opportunity like this on Darwin.
I'd say it's the opposite: if we were to change the path of the nix store I would prefer if we'd do it either completely before or completely after the apple silicon support: I would not want x86 macOS installations to be different from aarch64 macOS installations in ways that are not strictly necessary, and aarch64 support is likely to be buggy at the start, and I would not want to make that coincide with the change of the path of the nix store, a change that is itself very likely to introduce bugs.
I think it's unrelated to apple silicon support, so this isn't the right issue.
It is related as far as the main reason not to move to a new location has been that the transition of existing systems would be to intrusive, and that relocating only on some systems would mean they can't use the binary cache. If nix on Apple Silicon would get a new binary cache anyway, that is the window of opportunity to move the store and these changes would need to be synced.
Sure, but I don’t think ”reinstall your entire system on a new case‐sensitive APFS volume” is a reasonable thing for us to suggest users do, and if we’re going to support creating an APFS Nix volume for that reason then we might as well also continue to use it to keep the store prefix the same across platforms. I think that if we can get an automatically‐encrypted‐on‐FileVault, case‐sensitive APFS
/nixvolume setup in the installer then standardizing on that for all users would be the best approach.
From my perspective, expecting (especially new) users to partition their disks to try out nix is already unreasonable enough. It's a long way from the "just delete the /nix dir if you want to remove nix" that gave me the confidence it was worth a shot.
expecting (especially new) users to partition their disks to try out nix is already unreasonable enough.
Not that I don't roughly agree with the sentiment, but I have some nits:
- Unless they want it to be encrypted, we don't ask them to do anything. They just have to let the installer do it.
- It's just adding an APFS volume, which isn't the same as repartitioning. They're pretty simple to create and delete. We aren't allocating fixed amounts of space.
It's a long way from the "just delete the /nix dir if you want to remove nix" that gave me the confidence it was worth a shot.
The uninstall story, however, is (currently) a mess--but I'm actually feeling pretty hopeful here.
I'm working on fixing the encryption case mentioned above, and in the process it ended up making sense to me to add some code for "curing" leftover artifacts (I already feel like the installer halts and dumps directions too often--I'm trying to avoid making that problem worse). This process is actually producing some code and UI/X idioms that (I think) naturally lend themselves to the "uninstall" task. If/when this is done-and-merged, I think adding an --uninstall flag to the installer will be pretty feasible with some help.
I agree with the premise that considering we have to bootstrap a new binary cache for M1 anway; that switching the nix store path is a very good thing to consider.
However objection wasn't that a binary cache was hard to set up (it isn't) it's that we would have to run two separate hydra infrastructures as I think the path assumptions run pretty deep throughout our CI tooling.
It just means somebody needs to be willing to put in the effort to keep a second set of infra for MacOS. The current hydra maintainers didn't seem to be wanting to put that effort in.
We've already had a discussion on moving /nix, let's not repeat it here.
@thefloweringash Are you trying to solve both the code signing requirement and building natively for Apple Silicon at the same time?
Would it be reasonable to just focus on the code signing issue (and just doing it for all darwin targets) with the expectation of running Nix and Nix-installed tools under Rosetta2? That way we could get Nix functional first, and then focus on solving whatever is necessary for native compilation.
Or does Rosetta2 already allow for unsigned binaries? I wish I had an Apple Silicon machine to test with, but in my case, Nix being functional on Apple Silicon is a hard requirement for me before I'm willing to buy Apple Silicon.
As I understand it, since a recent installer change Nix and nixpkgs work fine under Rosetta 2, and binaries don’t need to be signed, but code signing of native executables is required on Apple Silicon.
Edit: (though there are still infelicities about actually building stuff on Big Sur as I understand it)
As I understand it, since a recent installer change Nix and nixpkgs work fine under Rosetta 2, and binaries don’t need to be signed, but code signing of native executables is required on Apple Silicon.
This is my understanding too. I just tested, and can build and run x86_64-darwin hello on the dtk on current nixpkgs master. I'll confirm this as soon as I get a new Mac.
Edit: (though there are still infelicities about actually building stuff on Big Sur as I understand it)
This should be resolved as soon as the channel updates.
The only thing missing is a release of the installer change in NixOS/nix#4243.
What @thefloweringash did was enough to pave the way for nix on Big Sur, and nix on Big Sur on Apple Silicon chips, when running via rosetta 2. I've posted some install instructions from test the other day here: https://gist.github.com/angerman/cbe02d814d81a8e4d4ced56b19046c19
I'm very much looking forward to having native arm64 nix, however this will likely result in a lot of breakage across the board for a while until enough software is updated and config.sub and other files are adjusted. Having Rosetta as a gateway drug is very helpful. I'm already starting to fear the time it's going to be dropped.
@rbvermaa ordered 4 mac minis with 16GB ram via foundation for the build farm, ETA Dec 16-23.
@rbvermaa ordered 4 mac minis with 16GB ram via foundation for the build farm, ETA Dec 16-23.
Ohh man, I hope no one has to work through christmas to set them up. I doubt the linux based vm solution will work.
I've backported the installer fix to 2.3-maintenance branch, so once @edolstra makes a Nix release, we have it working on Silicon chip via Rosseta.
My Apple Silicon device finally arrived and I would love to help get Nix working here (natively). @thefloweringash Given that code signing appears to be a base level requirement would it make sense to try and get that change, and I suppose any changes required for basic build tools upstreamed so that it's easier for the rest of us to try and get our favourite applications building? I'm unsure if there are any limitations with your current implementation but given we'd be going from nothing -> something it seems like a positive change.
@benpye I've confirmed a while back that @thefloweringash and I were able to produce the same (identical) ad-hoc code signature hashes on different machines. This might thus be a little less of an issue than we initially feared.
With the LLVM triple we do have a way to disambiguate:
aarch64-apple-ios
vs.
aarch64-apple-darwin
Which is even more confusing, since iOS is technically still using the Darwin kernel! But nevermind that, I suspect this distinction will get even more blurred with Mac Catalyst. I'd be interested if you can just execute an iOS binary from CLI on the DTK.
We should be able to use aarch64-apple-ios and aarch64-apple-macos. See
Lines 273 to 275 in f7876c6
I guess now is time to pull the trigger!
I've opened a pull request with what I have so far, see #105026.
The NixOS Foundation just received the first 4 M1 Mac Mini's for the buildfarm. We hope to have them online in the next 2 weeks for hydra.nixos.org usage. We will get an additional 2 M1 Mac Mini's in a couple of weeks.
Moved my comment about getting nix to work on a clean M1 to a separate ticket #106422 since most of it was non-M1 issues. But I did run into an issue with rosetta detection with the current install script on the website, that I suspect may not be an issue with install.in (either that or installing the developer tools changed something, or the install script doesn't detect the lack of developer tools and initially failed before the rosetta error).
Thanks.
We now have a total of 6 M1 Mac Minis. In the next week, I hope to have them online (our current provisioning system doesn't work with them, unfortunately). Will update this ticket once this is done.
@rbvermaa once https://github.com/AsahiLinux is done, the existing virtualized on linux/nixOS might work. But it's a long way to go. I'll just post this for reference here: https://www.youtube.com/watch?v=aMTfPSzrjXs
This issue has been mentioned on NixOS Discourse. There might be relevant details there:
Update: install took a bit longer than expected. Went through a manual install yesterday, and created a new bootstrap script for the new macminis. Will test on a newly wiped macmini tomorrow, and if all goes well, they all should be up tomorrow.
Unfortunately hit a snag restoring MacOS, the procedure of the old Intel Mac Mini doesn't work anymore. Luckily was able to restore eventually, but will only have time tomorrow to work on it further. Keep you posted.
I was finally able to bootstrap the new mac mini's properly and finished the last one yesterday evening. 🎉 They will be hooked up to hydra.nixos.org in the next hour.
Also, all mac minis from hydra.nixos.org have been moved into a new home (please don't judge me for my poor cable management ;-)
OK, the macminis are now also configured as builders in hydra.nixos.org's config. This means from now on, hydra.nixos.org can be used to build aarch64-darwin builds. To build / test on the platform, hydra jobset still needs to be created, but infrastructure wise everything should be ready .
Well, I could easily make the trunk jobset include this platform, but it probably doesn't make much sense until someone says that the builds aren't completely broken (I don't know and can't test except through Hydra.nixos.org). EDIT: I see it won't even evaluate ATM due to missing stdenv definition.
I expect we eventually aim simply for
--- a/pkgs/top-level/release.nix
+++ b/pkgs/top-level/release.nix
@@ -14 +14 @@
-, supportedSystems ? [ "x86_64-linux" "x86_64-darwin" "aarch64-linux" ]
+, supportedSystems ? [ "x86_64-linux" "x86_64-darwin" "aarch64-linux" "aarch64-darwin" ](without need to change this jobset's configuration)
Anyway, why not try the infrastructure already. It does appear to be building successfully. See #105026 (comment)
I've been using the apple-silicon branch for a while trying to build a native ghc with it. I've only got a minor patch I'm currently evaluating: https://gist.github.com/angerman/6a930357ead5b59dc1cc979539d7071c
I'm not sure I've got the wrapping for install_name_tool right yet, and we might want one for strip as well.
The native tools retain valid signatures after having strip or install_name_tool change them; the bootstrap tools we have in nix, do not retain signatures and invalidate them, which can lead to subsequent issues when those binaries are invoked again.
GHC does something nasty here, which is how I ended up running into this. GHC's whole library setup is rather non-optimal, and results in lots of runpath entries when using the default link approach. For a few releases GHC will use the linker to strip dead dylibs, but to also clean out the runpaths, ghc will do some post processing of libraries using install_name_tool. This however requires install_name_tool to re-generate the signatures for produced binaries to work; which it does with the shipped tools, but is still missing in the apple-silicon branch. I hope the above patch addresses this.
@angerman Is that patch meant to be applied to nixpkgs master or... ?
This would go on top of @thefloweringash ’s Apple-silicon Branch.
I've been wondering about the triplet, notably the aarch64 part. I'm not sure this has been discussed before (I've been crawling the threads but could don't find anything, so sorry if this has been settled already).
I did expect it to be aarch64 initially but once I got my hand on a M1 machine I realised Apple builds stuff as arm64:
$ clang --version
Apple clang version 12.0.0 (clang-1200.0.32.28)
Target: arm64-apple-darwin20.2.0
Thread model: posix
InstalledDir: /Library/Developer/CommandLineTools/usr/bin
This has consequences where e.g gems get published as such:
$ /usr/bin/ruby -e 'puts RUBY_PLATFORM' # Apple provided Ruby
universal.arm64e-darwin20
$ /usr/bin/ruby -e 'puts Gem::Platform.local'
universal-darwin-20 # rubygems will look for that binary gem platform and try to find arm64e in the fat dylib
$ ruby -e 'puts RUBY_PLATFORM' # hand-built by ruby-build
arm64-darwin20
$ ruby -e 'puts Gem::Platform.local'
arm64-darwin-20 # rubygems will look for that binary gem platform and will load an arm64 dylib
The difference between arm64 and arm64e is Pointer Authentication.
arch(1) mentions:
$ man arch
The arch_name argument must be one of the currently supported architectures:
i386 32-bit intel
x86_64 64-bit intel
x86_64h 64-bit intel (haswell)
arm64 64-bit arm
arm64e 64-bit arm (Apple Silicon)
If the binary does not contain code for arch_name, the arch command may try to select a close match. If arm64 is
specified and not found, arm64e will be tried next. If this happens, the order the architectures will be tried
is not guaranteed.
Note: I've built a binary gem for Apple's Ruby and it ended up being arm64, which (apparently, not 100% sure) then prevented the arm64 dylib to be loaded by the arm64e binary. I've yet to find a way to build as arm64e from the CLI (The only documented way I found was through Xcode).
Anyone has more insights on that triplet?
Only somewhat related, but I get an out of memory error running nix within Docker Preview on an M1 Mac Mini:
I'm using M1 mini, how is going with this issue now, any plan can build & run with arm ...
Build farm has been running OK. NixPkgs WIP: #105026 (comment)
Once #124887 is merged, we have at least two steps left:
- create an RFC and propose what Tier aarch64-darwin should belong to
- write documentation how to use Nix on Apple Silicon
#124887 was reverted.
- Evaluation fixes for aarch64-darwin: #125184
- Nix build support for aarch64-darwin: NixOS/nix#4865
@grahamc created https://hydra.nixos.org/jobset/nixpkgs/nixpkgs-unstable-aarch64-darwin
I hope to merge #125184 today.
We still need a release of Nix that supports Apple Silicon. Ideally with all the installer improvements from master.
Ideally with all the installer improvements from master.
Would definitely be nice to kick them out into the world. I reflected a little a couple weeks ago on what might need to come along with 4289:
NixOS/nix#4289 (comment)
I think there are also some small (zsh?) improvements in master.
I'd hope we can drop features from https://github.com/NixOS/nix/issues?q=is%3Aopen+is%3Aissue+milestone%3Anix-2.4 and release Nix 2.4 soon, as this extra work of porting back stuff is draining energy for all of us.
If someone does want to go ahead through all that work, that would be great. Worst case we do it with https://opencollective.com/nix-macos, since that's the whole point :)
I'd hope we can drop features from https://github.com/NixOS/nix/issues?q=is%3Aopen+is%3Aissue+milestone%3Anix-2.4 and release Nix 2.4 soon, as this extra work of porting back stuff is draining energy for all of us.
Naively, a release from master would be nice. The backport drain is a big factor in why I haven't agitated for it.
If someone does want to go ahead through all that work, that would be great. Worst case we do it with https://opencollective.com/nix-macos, since that's the whole point :)
Since there's been a lot of interest, I hoped providing a ~checklist would spur someone to try a backport PR. :/
Is anyone able to compile nix on aarch64-darwin?
I'm running into this:
error: a 'aarch64-darwin' with features {} is required to build '/nix/store/lx0k17qsdckm1y2m944nwq7y73xl7ks2-nix-2.4pre19700101_e6150de.drv', but I am a 'x86_64-darwin' with features {benchmark, big-parallel, nixos-test}
I realize that my nix is running under rosetta, but I'm unsure on how to build nix without using the emulated variant.
You have to change your nix config. (/etc/nix/nix.conf). I think the key is something like extra-platforms. There add aarch64-darwin
I use the following:
$ cat /etc/nix/nix.conf
system = aarch64-darwin
~/temp/nix $ cat /etc/nix/nix.conf
build-users-group = nixbld
system = aarch64-darwin
85.19ms ~/temp/nix $ nix-build -A defaultPackage.aarch64-darwin
these derivations will be built:
/nix/store/lx0k17qsdckm1y2m944nwq7y73xl7ks2-nix-2.4pre19700101_e6150de.drv
error: a 'aarch64-darwin' with features {} is required to build '/nix/store/lx0k17qsdckm1y2m944nwq7y73xl7ks2-nix-2.4pre19700101_e6150de.drv', but I am a 'x86_64-darwin' with features {benchmark, big-parallel, nixos-test}
Maybe a userconfig, or you have to restart your daemon.
There's an interesting thread on nix-darwin, but here's what I did with only nixpkgs:
First stage, being able to popping a nix-shell with arm64 derivations quite easily:
$ sh <(curl -L https://nixos.org/nix/install) --daemon # install as usual
$ cat /etc/nix/nix.conf # add extra-platforms
build-users-group = nixbld
extra-platforms = x86_64-darwin aarch64-darwin
$ cat shell.nix
{
pkgs ? import <nixpkgs> { localSystem = "aarch64-darwin"; },
}:
let
ruby = pkgs.ruby_3_0;
in pkgs.mkShell {
buildInputs = [
pkgs.openssl
ruby
];
}
$ nix-shell
nix$ which ruby
/nix/store/sc54smx4gqa80nxwpfvpxfw8slh7fgyd-ruby-3.0.1/bin/ruby
nix$ file $(which ruby)
/nix/store/sc54smx4gqa80nxwpfvpxfw8slh7fgyd-ruby-3.0.1/bin/ruby: Mach-O 64-bit executable arm64
But I'd rather be arm64 by default and fill in the blanks with x86_64 on a case by case basis with an additional pkgs_x86_64 ? import <nixpkgs> { localSystem = "x86_64-darwin"; }, in shell.nix, like this:
{
pkgs ? import <nixpkgs> { localSystem = "aarch64-darwin"; },
pkgs_x86_64 ? import <nixpkgs> { localSystem = "x86_64-darwin"; },
}:
let
ruby = pkgs.ruby_3_0;
in pkgs.mkShell {
buildInputs = [
pkgs.openssl
ruby
pkgs_x86_64.shellcheck
];
}
So, let's pivot into arm64 by default:
$ sudo su -l
# cat /etc/nix/nix.conf
build-users-group = nixbld
system = aarch64-darwin
extra-platforms = x86_64-darwin aarch64-darwin
# launchctl unload /Library/LaunchDaemons/org.nixos.nix-daemon.plist
# launchctl load /Library/LaunchDaemons/org.nixos.nix-daemon.plist
# nix-env -iA nixpkgs.nix
replacing old 'nix-2.3.12'
installing 'nix-2.3.12'
-----8<------
# file $(which nix-shell)
/var/root/.nix-profile/bin/nix-shell: Mach-O 64-bit executable arm64
# file /nix/var/nix/profiles/default/bin/nix-daemon
/nix/var/nix/profiles/default/bin/nix-daemon: Mach-O 64-bit executable arm64
# launchctl unload /Library/LaunchDaemons/org.nixos.nix-daemon.plist
# launchctl load /Library/LaunchDaemons/org.nixos.nix-daemon.plist
I'm a Nix beginner so I don't know if this is sensible, but it seems to work fine for me.
From there I think I can remove system = aarch64-darwin from /etc/nix/nix.conf, but everything seems alright nonetheless.
For anyone else having the issue I did, restarting the daemon after setting the new system configuration in the /etc/nix/nix.conf file worked with sudo launchctl kickstart -k system/org.nixos.nix-daemon
It would be great if someone tests NixOS/nix#4930 so that we can have the first release of Nix that supports native M1 :)
@domenkozar I only recently began experimenting with Nix. I have an M1 available which I can test on, but I'm not sure how. Instructions on how to install & test, if you're willing and have the time to write them, would be great
Nix 2.1.13 was released so installation should be simple as:
$ sh <(curl -L https://nixos.org/nix/install) --darwin-use-unencrypted-nix-store-volume --daemon
Note that while it says unencrypted, that's a misnomer as T2 chip encrypts whole storage.
@domenkozar I only recently began experimenting with Nix. I have an M1 available which I can test on, but I'm not sure how. Instructions on how to install & test, if you're willing and have the time to write them, would be great
If you follow the standard installation instructions, you can then create and/or edit /etc/nix/nix.conf to have the following:
system = x86_64-darwin
# system = aarch64-darwin
extra-platforms = x86_64-darwin aarch64-darwin
Switching between the two systems is just commenting out one and uncommenting the other, then running:
nix-env -iA nixpkgs.nix
I'm currently running x86_64-darwin since large swaths of the software I need to use doesn't build on M1, but switching is quick and easy. :)
@domenkozar I only recently began experimenting with Nix. I have an M1 available which I can test on, but I'm not sure how. Instructions on how to install & test, if you're willing and have the time to write them, would be great
If you follow the standard installation instructions, you can then create and/or edit
/etc/nix/nix.confto have the following:system = x86_64-darwin # system = aarch64-darwin extra-platforms = x86_64-darwin aarch64-darwinSwitching between the two systems is just commenting out one and uncommenting the other, then running:
nix-env -iA nixpkgs.nixI'm currently running
x86_64-darwinsince large swaths of the software I need to use doesn't build on M1, but switching is quick and easy. :)
It should be the other way around, using aarch64-darwin as system so that by default things are native. Then, fallback to rosetta for things that don't work.
It should be the other way around, using
aarch64-darwinas system so that by default things are native. Then, fallback to rosetta for things that don't work.
Understood, and I'd like that, but it doesn't work the other way for two different pieces of software I use extensively right now:
- mosh (just completely fails to compile)
- stack (fails to find a usable ghc)
So in the meantime, I just pretend I'm on an x86_64 and try again occasionally.
Mosh requires a patch while ghc support is not far.
Good stuff! I will be spending roughly a working day on testing and evaluating NixOS for work. I already did a bunch of that and can offer some time to a dev if it's interesting. This'll be between 9am-5pm CEST (Berlin/Stockholm/Amsterdam time) -- i.e. if some dev wants to try out something on Apple silicon, then I can be your test subject :) ping me if so and let me know where to appear
Nix 2.1.13 was released so installation should be simple as:
@domenkozar I am failing to install 2.1.13 for arm using your instructions. Checking the shell script I don't see arm supported yet:
# https://nixos.org/nix/install
...
Darwin.x86_64) system=x86_64-darwin; hash=81cc3f2d5b06162b43a4695b37952ef569318d44c999558b8c54a925bab24ded;;
# eventually maybe: system=arm64-darwin; hash=@binaryTarball_arm64-darwin@;;
Darwin.arm64) system=x86_64-darwin; hash=81cc3f2d5b06162b43a4695b37952ef569318d44c999558b8c54a925bab24ded;;
...
And a quick look to the releases seems to confirm arm isn't supported yet: https://releases.nixos.org/?prefix=nix/nix-2.3.13/
I'm not certain, but it looks like the install.in change in https://github.com/NixOS/nix/pull/4865/files#diff-35b540b0d1c59f2b7bcff2aa56ff2a57b3948a8e7c8c4d9521a5d604528a3d0a got left out of the port (https://github.com/NixOS/nix/pull/4867/files).
Totally missed that @dbarrosop during the review.
I've pushed the fix to 2.3-maintenance branch and opened NixOS/nix#4974 for uploading the release.
Will ping @edolstra to make another release :(
I have tested python3, node, and postgres today on a Mac with the M1 chip and my system set to aarch64-darwin. Has worked great so far :-)
🎉 Same here! finally set system = aarch64-darwin and extra-platforms = x86_64-darwin aarch64-darwin as pointed by others on my nix.conf, re-installed nixFlakes (i'm using it plus nix-darwin and home-manager on my setup) and most packages work, just did a tiny overlay for few things complaining: haskell packages (till #126195 gets merged). Thanks so much to all people involved on this :)
edit: extracted my mkDarwinSystem function as a flake from my nix configuration if hopes it can help anyone.
So I suppose we can close this then? Things that don't work yet probably warrant new issues.
So I suppose we can close this then? Things that don't work yet probably warrant new issues.
Has the Nix release mentioned here been made? Though I guess it's not a direct blocker for a Nixpkgs issue.
Has the Nix release mentioned here been made?
What about the ghc. I don't follow that. I would like to keep this open, until that is working
GHC 8.10.5 with Apple Silicon support has already been released. (Although it has other issues)
Following previous comments of attempts at a successful build, this is my attempt trying the latest release.
MacOs M1 BigSur 11.4 running latest Nix 2.3.14 nix-shell so I can setup a cabal build. The process has not introduced any of the previous mentioned errors.
I'm able to do a proper sh <(curl -L https://nixos.org/nix/install) --daemon.
Only persistent error I tend to run into even with different build attempts previously is:
error: cannot coerce null to a string, at /nix/store/sdjxjsd5phr225rs2qzklj2xci0c9gr0-source/pkgs/stdenv/generic/make-derivation.nix:192:19
Which I had attempted to resolve here with listed attempts & clean installs. I'm pretty new to Nix, its a bit difficult to understand if this would be the OS compatibility being a factor for producing the error.
So far in my setup, these don't work with aarch64-darwin:
- vscode
- comma (Shopify, update: lzma is actually what's failing)
- hub
- neovim
On the GHC side:
- 8.10.5 has been released with sufficient plumbing to have the LLVM backend support aarch64-darwin.
- 8.10.6 should be released soon with a bunch of fixes.
However, a few bugs were found (some stemming from GHC's CI using nix to build GHC, and thus having had some unwanted nix-paths leak into the build product), a crash in the aarch64-darwin linker.
In general apples aarch64 procedure calling convention is rather strict (also the reason or clang to require header files!). And there still seem to be quite a few Foreign Interface Calls, that have the wrong FFI decorations in Haskell, and thus you might see odd behaviour (or even crashes) where GHC ends up calling C functions incorrectly.
The last part is the hardest to rectify quickly, as it essentially entails auditing all Haskell code for proper FFI declarations. We are trying to come up with some automation for this.
I've opened #129427 to make sure evaluation errors don't slip into master.
The last bit missing is documentation and user testing.
I'd encourage M1 users to install Nix via:
$ sh <(curl -L https://nixos.org/nix/install) --darwin-use-unencrypted-nix-store-volume --daemon
vscode comma (Spotify) hub neovim
I would like to avoid this ticket staying open forever with reports for packages not working; and instead mark Apple silicon as "Initital basic support done" and open new tickets for any packages that don't work yet. I'm sure there are hundreds if not thousands of packages that do not compile on M1 and we can all track them in new tickets invdividually.
What about the ghc. I don't follow that. I would like to keep this open, until that is working
Similarly it seems that upstream doesn't even have the M1 story figured out completely yet. it doesn't make sense to have that as a blocker for M1 support in nixpkgs. We can open a new issue for that too.
The last bit missing is documentation and user testing.
Yes documenting M1 is supported in the manual or the website sounds like something we should at least address before closing this.
I apologize for not trying 2.3.14, but 2.4pre20210705_1f93084 from https://hydra.nixos.org/eval/1684067 installed cleanly in multi-user mode and appears to be working perfectly on my M1 MBP.
I've also had great luck with using the Flakes and the new nix commands for mixing Intel and Arm packages, e.g., nix profile install nixpkgs#ripgrep defaults to aarch64, while nix profile install nixpkgs#legacyPackages.x86_64-darwin.neovim pulls in x86_64 binaries.
I would like to avoid this ticket staying open forever with reports for packages not working
Sounds fair. :) The installation itself was flawless.
✅ TL; DR: Issue with nix-darwin which is not in the scope of this ticket.
👋 For some reason my nix is considering x86_64 whenever I apply my nix-flake and not arm64 as expected 🤔 Did anyone stumble upon this odd issue? (edit: see bottom of post, think I found the culprit...)
Checked the architecture using uname -m on my terminal (plain Terminal App):
$ uname -m
arm64
Added a custom system.activationScripts.postUserActivation.text to echo uname -m:
{
system.activationScripts.postUserActivation.text = ''
echo "What is my CPU?"
uname -m
'';
}
Build and apply my flake:
$ nix --version
nix (Nix) 2.4pre20210601_5985b8b
$ nix build .#darwinConfigurations.personal-macos.system
$ ./result/sw/bin/darwin-rebuild switch --flake .#work-macos
...
What is my CPU?
x86_64
...
The impact: I have some issues using Homebrew integration with nix-darwin and my binaries are not in the correct architecture (they are in Mach-O 64-bit executable x86_64).
As suggested I also set the system and extra-platforms:
$ cat /etc/nix/nix.conf
# WARNING: this file is generated from the nix.* options in
# your NixOS configuration, typically
# /etc/nixos/configuration.nix. Do not edit it!
max-jobs = auto
cores = 0
sandbox = false
substituters = https://cache.nixos.org/
trusted-substituters =
trusted-public-keys = cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY=
require-sigs = true
trusted-users = root
allowed-users = *
experimental-features = nix-command flakes
system = aarch64-darwin
extra-platforms = aarch64-darwin x86_64-darwin
Edit: arrrghhh:
$ file /Users/brunohenriques/.nix-profile/bin/nix
/Users/brunohenriques/.nix-profile/bin/nix: Mach-O 64-bit executable x86_64
So.. my nix executable is running on Rosetta, so this means I need to reinstall Nix to ensure that runs on Apple Native... 😢
Update2:
Reinstalled Nix and renders the same result:
$ uname -m
arm64
$ where nix
/Users/brunohenriques/.nix-profile/bin/nix
$ file /Users/brunohenriques/.nix-profile/bin/nix
/Users/brunohenriques/.nix-profile/bin/nix: Mach-O 64-bit executable arm64
Any ideas?
Last Update
Issue with nix-darwin which is out-of-scope of this ticket.
I've opened #129427 to make sure evaluation errors don't slip into master.
The last bit missing is documentation and user testing.
I'd encourage M1 users to install Nix via:
$ sh <(curl -L https://nixos.org/nix/install) --darwin-use-unencrypted-nix-store-volume --daemon
Installation works flawlessly! Thanks ! any plans on updating the documentation also ?
@fr34k8 I've updated https://nix.dev/tutorials/install-nix to have instructions per OS.
Also planning to look at fixing GHC in the following days.
This issue has been mentioned on NixOS Discourse. There might be relevant details there:
https://discourse.nixos.org/t/nixos-emulation-on-mac-mini-m1/14659/1