NixOS in Windows Store for Windows Subsystem for Linux (WSL)
ThomasMader opened this issue · comments
Nix already runs in Ubuntu via WSL afaik.
I think it would make sense to add an official NixOS image to the Windows Store.
Arch Linux is trying already. (microsoft/WSL#2253)
Are there any showstoppers?
This would be super cool. I'd like to move to using NixOS, but I can't give up my Windows machine, and I don't want to run NixOS in a virtual machine. What I'd really like to try doing is a continuous integration system using NixOS based build management to kick off compilation on the Windows side.
#28365 is an impediment, but drilling through the linked issue I saw an update yesterday from a MS staffer that they have been working on it.
Also I'm not sure if NixOS/nix#1203 is the same or different root issue.
By the way, I have a permssion denied just after the creation of /nix when I try to install nix on WSL
Any idea ?
@PierreR I also had this problem. I solved it by downloading the archive and copying the files to the directory manually.
@jtojnar Thanks ! Indeed if I run each problematic mv myself, it is going to work.
Any idea where the issue is coming from. I would like to streamline that process but it is going to be a bit cumbersome if such issue lingers on.
Just for the record I have hit microsoft/WSL#1671. Not only WSL seems to be unbearably slow, it would also fail strangely with this error:
error: setting synchronous mode: disk I/O error
My feeling is that WSL is too beta and adding an official NixOS image to the Windows Store might be premature.
@PierreR disk I/O error is NixOS/nix#1203
@jtojnar yes it is. Thanks ! Everything works fine for me except for the sluggishness.
Does NixOS the distro itself require anything special beyond what Ubuntu as a distro requires? If not, then it should be possible to get NixOS (and not just nix) onto WSL!
NixOS the distro relies pretty heavily on systemd to run services. It would still be great to use nixos-rebuild for building the user environment and making software available, but without systemd it would be difficult to use any nixos modules that expect to run daemons or cronjobs.
NixOS the distro relies pretty heavily on systemd to run services. It would still be great to use nixos-rebuild for building the user environment and making software available, but without systemd it would be difficult to use any nixos modules that expect to run daemons or cronjobs.
Other distros will be in the same boat for any required services, but I can see this being a particular problem for Nix with the Nix Daemon.
Since WSL is intended to be a single-user environment, a NixOS that runs like single-user Nix could be what is needed to make it work.
I would love to see another distro in the Windows Store. It only took Kali a couple months from request to publishing into the Windows Store (which released 3/5/2017 for record). Note that you can manually run any services as most of the syscalls are supported though I'm not too familiar with NixOS to guess what specifics might not exist yet in WSL. So while systemd won't automatically run the Nix Daemon, it's entirely possibly you "may" be able to manually run it. Do an strace and place a bug on WSL if there are any errors specific to WSL. And since emphasis is on systemd running, see microsoft/WSL#534, microsoft/WSL#1579, microsoft/WSL#2530 and vote for this UserVoice
Also try to use the latest Insider as it will greatly help in discovering what issues may still exist. For instance, you're able to perform background tasks on Insider.
Microsoft open sourced a WSL Sample for Linux Distribution maintainers. Might be very helpful.
https://blogs.msdn.microsoft.com/commandline/2018/03/26/wsl-distro-launcher/
Any news in this front? I'd manually compile myself to test it out but won't have time for it for awhile. If anybody on the NIXOS team is running build 17134 SCU (pre-release still), if they could give it a go on this build, it would be appreciated.
I'm taking a stab at this. Initial success so far @ https://twitter.com/GeoffreyHuntley/status/995977211176927232
Blocked by microsoft/WSL#3207
Thanks for the work on this. Testing out linux distros on Windows is really easy now days and you can run several side by side. Getting NixOS on to the app store could be a great gateway drug to NixOS and the excellent Nix package manager.
In private discussions with Microsoft about getting NixOS into the Microsoft App Store. Have been a little overwhelmed recently but it's high on my todo list. Microsoft are waiting on me.
Considering nixcon is on; it only makes sense to pick this back up again. I had to park it because I was overcommitted. Let's start this back up again.
Next steps are as follows:
- NixOS Foundation to sign up for a "Company" Windows Developer Account
- NixOS Foundation to review contracts and execute them
- Obtain testing plan from Microsoft
- Obtain publishing and release plan from Microsoft
- Productionise the NixOS build of
install.tar.gzand embed into an appx. - Distribute the appx to microsoft
- Party
What's the structure of install.tar.gz? I couldn't find that documented. Also, someone using WSL might want to add a package for "wslu" which has some nice things for interacting between Linux and Windows:
@ghuntley do you have a URL for the NixOS Foundation to review the documents?
Thanks for pushing NixOS towards running over WinNT! That would be handy.
The roadmap above is much about putting NixOS into the appstore. Which is quite separate a task from making NixOS runnable on WSL. I'd pretty much do with a ZIP which I'd download from nix site and install for me, not sure I even like the idea of mouseclicking it out of the fancy store instead. Especially if we have to wait for the legal stuff just for the sake of some GUI install scenario.
What's the technical status of running NixOS on WSL? Is it still blocked by the question marks and pipes in the paths? If not, are there any distributions we could try and run right away?
What's the technical status of running NixOS on WSL? Is it still blocked by the question marks and pipes in the paths? If not, are there any distributions we could try and run right away?
I don't have WSL, but in theory, it should be possible now using the "containerTarball" (nixos-system-x86_64-linux.tar.xz).
You can manipulate this into an install.tar.gz file and get it to run (and make a PR to get it in future release). There will most likely be bugs encountered doing this though. wslu looks to be a good resource to base the install.tar.gz on.
EDIT: It looks like WSL doesn't support systemd! So we will need to do a little more work to get this to work. How would we run the Nix daemon?
Another question that remains to be solved: how can we get Nix to detect that it can run Windows binaries? Nix supports "extra-platforms" so we can reuse that if there were an easy way to run a builder in "windows" mode. Can't find any sort of syscall/API to do this though.
The proprietary init does the windows interop calls to the apis. Far as daemons go, as I stated before, you dont need systemd to run services. In Debian-based distros, you just do service blah start. For example ssh. Want it to start automatically? Not supported out of box yet but you can add to your bashrc, zshrc, etc. as workaround
Now granted, WSL should support all POSIX filenames, and microsoft/WSL#3207 is definitely a bug on their side. However, would it not be relatively simple to replace all NTFS-invalid characters in Nix store path names with _?
Names clashing seems very unlikely since the input hash is prepended. There may be some derivations that have illegal characters in the package content filenames, those can be handled on a case-by-case basis.
Alternative is to wait for NixOS/rfcs#17 ;-)
Current status:
- [still waiting on nixos foundation to sign legal paperwork to allow distribution into the app store and to register a corporate account]
- systemd issues potentially resolved by today's announcement at build//
- following up with WSL product managers about getting access to WSL2 bits (under NDA if need be)
See https://devblogs.microsoft.com/commandline/announcing-wsl-2/
Developer agreement is at https://docs.microsoft.com/en-us/legal/windows/agreements/app-developer-agreement
Last correspondence with Microsoft below:
Hi Geoffrey and Graham,
Thanks for reaching out. It’s nice to virtually meet you. Here’s an overview of the process:
As outlined below, you’ve completed the first step by reaching out. I’ve pasted the other steps while adding additional details.
- Agree with the WSL team on a testing and publishing plan
- We have some basic manual tests that we conduct such as testing networking. Are there any specific compatibility concerns you have for NixOS running on WSL?
- The other part of publishing is that we ask you to send us your appx file. This will be put on a special Store CDN that will allow customers to install your app offline. You can read more on what that would look like
- Complete any required paperwork
- When you sign up as a company for a Windows Developer Account you will need to sign an App Developer Agreement. Please take a look at that agreement and let us know if it meets your legal requirements. There are a couple other documents in the aforementioned link, so please take a look at those as well.
- Sign up for a "Company" Windows Developer Account https://developer.microsoft.com/en-us/store/register. note: this can take a week or more since you'll be required to confirm your organization's identity with an independent verification service via email and/or telephone.
Once the above is complete, you can submit your app and stage it for publication. There are a couple things that happen in the background with the Store that can take a little while depending on the volume of submissions they have at the time.
Next steps in theory are:
- obtain WSL2 bits
- ???cook a nothing out of the ordinary virtual machine via CI???
- ???wrap virtual machine with WSl2 bits and package as appx [will require windows CI]???
- nixos foundation sign legal agreements and register for corporate developer account
What does "ordinary" look like? Does that include systemd?
That's the missing piece of technical detail. Discussing now with Microsoft product managers.
Not sure if we need to cook an ova or they take a tarball of a freshly installed system (WSL1) and then inject it into a VHD. (WSL2)
Not sure if they invoke grub when booting this VHD.
Not sure if initd/systemd is invoked or how to get WSL2 to invoke the equivalent.
There's no plan to implement systemd at this time. That's what was said at their MS Build booth when I asked. But init rc scripts work just fine. You can do that now without WSL2.
Yeah, NixOS has an option to disable stage 1 with boot.isContainer. We probably need to rework stage-2 a bit to avoid running systemd directly, but also keeping as much of the activation scripts as possible. The script is here: https://github.com/NixOS/nixpkgs/blob/master/nixos/modules/system/boot/stage-2-init.sh. @cleverca22 has worked on some custom init scripts through not-os, which may be useful.
At the same time, providing an installer with just multi-user Nix installed would be super useful in its own right! We don't necessarily need to go full NixOS WSL for this to be useful. It sounds like we'll need to figure out how to automatically start the daemon correctly. Single user will probably work without any modifications!
Some more technical information can be found at https://devblogs.microsoft.com/commandline/shipping-a-linux-kernel-with-windows/ that's the best info that exists at this stage.
It sounds like we'll need to figure out how to automatically start the daemon correctly.
I agree.
Single user will probably work without any modifications!
It does and it's what we should aim for.
https://twitter.com/GeoffreyHuntley/status/1007607578933714945
They're releasing a 2.0 but I presume that shouldn't affect making NixOS available there.
@ghuntley To clarify, why does the foundation need to register as an App Developer? I assume you already have the ability to upload apps to the store?
@edolstra It isn't recommended to ship an official distribution of an application under an individual's account. It brings other legal issues. Ownership of accounts and just general branding.
If he were to publish NixOS under his own account he would either have to give up ownership of his developer account to NixOS foundation or the ownership of NixOS distro on windows is handed over to Geoff
Okay, I've registered an account for the NixOS Foundation (currently pending verification). I've also reserved "NixOS" as an app name. What's the next step?
WSL2 was released today and is now available for folks who are on the Windows 10 fast ring.
https://devblogs.microsoft.com/commandline/wsl-2-is-now-available-in-windows-insiders/
Okay, I've registered an account for the NixOS Foundation (currently pending verification). I've also reserved "NixOS" as an app name.
Thank-you for registering the account and reserving the application name @edolstra.
What's the next steps?
I'd love to team up with a couple folks who are more experienced with nixos internals and ship #63049 which was raised moments ago. Stop on by the pull-request, say howdy and I'll grant you push access to the pull-request branch.
It looks like there is a way to run systemd on WSL2 https://github.com/arkane-systems/genie although I'm not sure if that is useful
@AndrewMorsillo useful. Thanks for sharing, was unaware!
@ghuntley any progress on this? Would you like to receive any help with it? Is your progress stored somewhere in a public repository?
I just came across this issue and am super excited to be able to run NixOS on WSL2!
Same, really enthousiastic about NixOS on WSL2 !! Would be the best 😄
I really look forward to seeing this! Unfortunately I don't think I have the knowledge/skills to take it on myself.
nix shell works OK on WSL 2 ubuntu FYI
@thedavidmeister The problem is not about running Nix under WSL. That is obviously possible. You can also set up X11 to work with WSL and then you have something akin to a lightweight VM on Windows using Ubuntu.
This ticket is about having NixOS as an option for WSL. Basically NixOS should be the operating system being used by WSL and not Ubuntu or whatever else.
@razvan-flavius-panda i know what this issue is about but my experience with onboarding windows people to a nixos based project is that none of this is obvious so i'm trying to make it clearer what options are for people waiting on this to land :)
Fair enough. Fair enough.
Just as a recap:
- WSL2 works more like a container than a real VM. You don't get to choose the linux kernel and there is no init system.
- NixOS is designed to run like a real linux distro, with a bootloader, selection of kernel and systemd as it's init system. Most NixOS modules depend on systemd to work properly.
I just wanted to communicate clearly that it's not just a small patch away.
From there, there are two routes possible:
- The short and hacky route. It's possible to override the nixos modules to make it mostly work. This is what arion does for example. The main downside is that things will never be 100% compatible and prone to break during channel updates.
- The proper route. This would require to go an split NixOS up properly. Define a common interface for cross-init systems. Mark modules that only works on "NixOS Linux" (vs "NixOS WSL2").
IMO option (2) is much better to avoid giving a false expectations. It's definitely a project but it's not completely insurmountable to do.
You don't get to choose the linux kernel and there is no init system
You actually can import your own kernel for the past 2 or 3 Insider builds. There's a few other things as well. See the blogs. And there is an init system. There's just not systemd without genie. Create init.rc scripts and you'll have the same as having systemd. This would be a bit more challenging for several obvious reasons. But it's doable.
Also, NX-OS would still have to work on WSL1 as that is the default even after enabling WSL2. But it's still the same as my last comment.
I don't pretend to understand enough about linux, wsl, and systemd to know if this is useful, but there's a long thread about systemd on WSL2 here that might have some useful information
I will note that I've seen plenty of init.rc scripts that including calling systemctl and other systemd functions (we have some for a few Fedora VMs/containers due to things used for managing the hosts.) So the functionality can remain exactly the same except you can add to the scripts other non-systemd stuff so if you don't have systemd, you don't run any of the systemd specific stuff. For redundancies found within a script, those should be abstracted away as much as possible to keep the file size down as well as possibly improve perf from not having to read so many lines. We do want to keep these scripts as small as possible following best coding practices.
I looked into Genie and figured out how it's running systemd. I then wrote a NixOS module to wrap systemd and systemctl instead:
It's somewhat working. Sometimes systemd starts up well and you can run systemctl to check on services, etc. It seems like nixos-rebuild switch mostly doesn't work, sadly. Still quite a bit of debugging to do.
Whilst the nixos-rebuild switch is a bit of a shame, I think it's less important on WSL2 as the reboot time is only 1-2 seconds anyway.
Looks like somebody has it working 100% here, but I haven't tried it out yet https://github.com/Trundle/NixOS-WSL
Systemd now has the --system flag which lets it run even if it doesn't have PID 1, so maybe all we need to do is set that flag?
lol? I run a non systemd distro, and I have no issues with running systemd inside containers (non pid 1) fully non degraded, so didn't even systemd needed such flag? Out of curiousity, why on EARTH would anyone want / need to run linux on windows and not vice versa? Is it basically an update to cygwin or so? 2020? ye, 20 year late MS.. 90s os :) what is WSL?
OH.. NixOS as I understood it was meant to be so 'smart about running several versions' but it can't even handle different inits? I understand one can blame systemd for that, but it's usually easier to run all good citizens together, as options, and leave the one bad one out. Still, I guess NixOS , despite having grown, isn't really all that then.
@lulcat The init system used to be upstart iirc, feel free to find those commits and propose a maintainable combined system. There's also a lot to read in previous issues on this topic.
The gist of it is that it's hard to create a wrapper around init systems with such diverse capabilities without being very limiting or re-writing systemd capabilities; as such, NixOS modules use only systemd.
Seems like it works now.
Any news on a possible release on the Microsoft Store ?
At the very least, an offline msix could be provided (that's still signed) to make usable outside of Store. The wsl --import option is great but an offline msix would be even greater.
Not Stale - still not in the Store
Are there any specific instructions for planting it into the WSL? Preferably, ones which do not involve writing an app in C++ and compiling it with Visual Studio. Everything there sounds not too hard — if you've done it before, and got all the tooling. But for the first time…
If not in the Store, a ready set of the DistroLauncher and install.tar.gz would be nice to have.
Rumors are, you can install an install.tar.gz by placing it alongside an Ubuntu.exe and running, but the part which says "It is expected that /nix/var/nix/profiles/system/activate is run during installation, for example as part of the DistributionInfo::CreateUser function" sounds as if it could be a problem.
Another issue is that you need NixOS to make a tar.gz you need to run NixOS, which in context of our task (running NixOS on Windows NT) sounds close to "unzip.zip".
@GuillaumeDesforges or @WSLUser , do you know if anyone actually have tried to get this into Windows Store? Is there a request I could give thumbs up to somewhere? (don't know anything about how windows store works) ..
@GuillaumeDesforges or @WSLUser , do you know if anyone actually have tried to get this into Windows Store? Is there a request I could give thumbs up to somewhere? (don't know anything about how windows store works) ..
From @ghuntley (#30391 (comment))
In private discussions with Microsoft about getting NixOS into the Microsoft App Store. Have been a little overwhelmed recently but it's high on my todo list. Microsoft are waiting on me.
Other than that idk.
It would help to have someone who knows the procedure to be accountable (maybe a strong word but I don't know another) on this (both follow technical issues and WS request)
Seems like it works now.
Any news on a possible release on the Microsoft Store ?
It would be great to have some manual steps to get it running as well. Is https://github.com/Trundle/NixOS-WSL the recommended way for example?
Is following the Trundle's repo still the best way to generate latest NixOS for WSL? Or are some forks or other repos that have 20.09+ images available? Possibly with WSLg, wslu support etc.
I commented in that repo that distrod claims to help with the systemd issue. Distrod relies on https://linuxcontainers.org which doesn't have anything for NixOS. Would it be difficult to create NixOS images for LXC/LXD and linuxcontainers.org so that distrod could be leveraged?
The nixos-wsl repo under nix-community is under a new maintainer and getting a lot of much-needed attention. There’s a good chance that we can get this done, soon.
Please close this and redirect discussion here.
Closing as requested. Feel free to reopen if it is not appropriate.