Support for VMs on Apple M1
pmatos opened this issue · comments
Is there any plan to release VMs in Apple M1 hardware to run GitHub Actions?
Hi @pmatos, at this moment we do not have a ETA for enabling M1 support in the GitHub Actions hosted runners, but we are actively evaluating the feasibility of this and will provide updates on our roadmap when we have something to share!
@thejoebourneidentity Any update on this yet?
AWS Now Offer It:
https://9to5mac.com/2021/02/02/m1-mac-mini-in-the-cloud/
AWS Now Offer It:
https://9to5mac.com/2021/02/02/m1-mac-mini-in-the-cloud/
It’s Scaleway who provides M1 Macs Mini, AWS doesn’t have M1 yet - only Intel. On the other hand, I believe that AWS will have M1 in a near future.
Has anyone tried the self-hosted runner on a local M1?
@ioquatix yes I have. It works but it's quite annoying in the current state of GHA:
- The runner process itself runs under Rosetta because at some point it hinges on Intel binaries.
- I have overriden Rosetta 2 by using
arch(1)on eachrunstep:arch -arch arm64 my command. It's very manual but it works perfectly to drop back into non-Rosetta land.Unfortunately without writing a full-blown custom action, GHA only allows a limited number of shells (I wish it would be possible to override the shell -command- altogether, not just the shell program from a fixed list) - The GHA UI only allows one to pick from a very limited list†, and thus I created it as
macOS x86_64, which automatically adds unremovable tagsmacOSandx86_64on the runner. It is kind of true since there's Rosetta available but it may not be what you want, as if you have other custom macOS runners on Intel it may still pick up non-arm64 jobs, but you may want to keep it free of this for resource management purposes, as well as not complicating the steps with more conditionals on thearchcommands above. - I added a manual
arm64tag to distinguish it. - It is only possible to add a runner to one GH project or one GH organisation, and with the current implementation you can really only have one runner per user on a machine††. This is fairly limiting.
- Since the runner lives as a process under a regular user (which I created one specifically for), unless you take extra measures (at the -very- least
chmod 700all other users home dirs) the runner has trivial access to everything on the machine, as well as LAN access†††. This concern is mentioned by GH when adding a new runner under the synthetic warning of "don't use custom runners on public projects". This concern would be largely solved if macOS were virtualisable on ARM as it is on Intel, but currently comes short of any VM solution to achieve that. That, or dedicate the machine to be a pure GH runner that never runs for private projects, and put it in a DMZ.
† Why they don't allow a blank "custom" platform (at the cost of being on your own) is beyond me.
†† I suppose one could hack the thing to allow more but the runner CLI does not make that easy out of the box.
††† That's ultimately why I shut it down as it's on my home LAN and contains personal data on a separate account.
@ioquatix that's basically what I used (just follow the registering steps of macOS x64 provided by GHA's UI):
name: Build
on:
- push
jobs:
try-macos-arm64:
name: Try macOS on M1
runs-on: [self-hosted, macOS, ARM64]
defaults:
run:
shell: "/usr/bin/arch -arch arm64e /bin/bash {0}"
steps:
- name: Status
run: |
clang --version
uname -a
- name: Checkout
uses: actions/checkout@v2
- name: Set Ruby metadata
id: ruby
run: |
ruby -e 'puts RUBY_PLATFORM' | tee ruby_platform
echo "::set-output name=RUBY_PLATFORM::$(cat ruby_platform)"
ruby -e 'puts Gem.platforms.last.to_s' | tee gem_platform
echo "::set-output name=GEM_PLATFORM::$(cat gem_platform)"(A note for Ruby users, please do yourself a favour and install a Ruby >= 2.7.2, as Apple's Ruby 2.6 seems to miss some important stuff)
Related issue: actions/runner#805
@QuiiBz There's MacStadium as well: https://www.macstadium.com/configure?p=minig5&c=VGS2
Anyway even if there were M1 machines on AWS the cost is absolutely ludicrous. I don't see anyone using that unless they're heavily invested (a.k.a trapped with no other option) in AWS as a management platform and need it at a huge enough scale to offset the cost.
@thejoebourneidentity do we have an ETA on this yet?!
I think the macstadium founder wrote an interesting blog about the changes in macos licensing for big sur, specifically around virtual provision of Mac hardware. Previously it was unspecified but now it's less ambiguous and I believe specifies a minimum time slice of 24hrs. If I understand that correctly I believe it means macos big sur and higher is not really usable in a cloud CI provider environment. Coincides with github actions still being on older macos with older xcode 🤔. May be an intractable problem, or an expensive one at least (clients need to fund the larger time slices / less efficient statistical multiplexing of build hardware). Would be nice to have some official explanation of the delay on all aspects of the macos infrastructure here
Coincides with github actions still being on older macos with older xcode 🤔.
Actions supports the macos-11 target already, they just haven't pointed macos-latest at the Big Sur images. https://github.com/actions/virtual-environments/blob/main/images/macos/macos-11-Readme.md
Previously it was unspecified but now it's less ambiguous and I believe specifies a minimum time slice of 24hrs.
(caveat: IANAL) There's a lot of legal verbiage (in Section 3 "Leasing for Permitted Developer Services": "leasing", "subleasing", "lessee", "end user lessee", "permitted developer services"...) to navigate through and my last read of it was that the ones renting the hardware (the lessor, e.g MacStadium) must lease it for 24hrs minimum to their customers, but that the text also allows to rent it like so to e.g. GitHub, and being the customer (or "lessee"), is then able to bill CI by the minute (that would be for "permitted developer services"). What's kind of obscure to me is that it could also count as "subleasing" from GitHub to the CI users, but then it would prevent any sensible CI billing arrangement as we know it.
I'm not sure how the Scaleway model fits in that though, as they seem to both host M1 hardware and bill by the hour (so there's no lessor/lessee arrangement).
EDIT: Ah, now I know, there's an infobox at the bottom when you try to create one saying "As required by Apple License, you must keep this instance at least 24 hours. You will be able to delete it only after 24 hours.".
I'm guessing people that are lawyers are the reason macos-latest is not macos-11 and the rollout was halted, perhaps while it was still small enough to "eat" the problem from a provisioning/cost perpective while ambiguities are ironed out. Hopefully it's obvious I'm just speculating though. I was just trying to provide some context (even if speculative) since there hasn't been any official information and simple requests were getting thumbs-ed down
More than anything, just some news, plan would be nice to have as mentioned by the comment that generated the notification which prompted my comment.
I'm extremely puzzled why Apple would be specialising and enforcing such a licensing time window of their operating system in the first place, are they trying to disincentive developers to use cloud-based methods or something? And/or only keep using local developer machines? (I wouldn't be surprised if that was the end goal, considering Apple)
They sell hardware, in general, and want to incentivize that. This is probably not the place to speculate on their motives (or at least, I hope my notifications don't fill up with that, so I can leave them enabled on this issue). Speculation on GitHub actually providing M1 services is probably far enough afield ;-)
I didn't mean to start speculation, sorry, the thought just slipped while reading this thread.
I do hope that github could be providing such a solution in the near future though, i'll leave it at that :)
@thejoebourneidentity Any update on this? maybe with macOS-12?
Kind of wondering if GitHub is waiting to see what happens with Xcode Cloud...
Why not use @sickcodes Docker-OS X under the stock GitHub Actions Ubuntu instance? Apple has never complained- XNU and LLVM are open source. What Apple doesn’t want is people loading non-headless QEMU.
@chadbrewbaker because the request is about specific CPU configuration which you cannot achieve with a virtual machine (macOS in KVM on ArchLinux in Docker).
If you need to run just macOS, you can use macos-11 or macos-10.15.
Anything new here?
This is actually pretty important!
@iMonZ we are working on it, but can not provide ETA for now
@miketimofeev thanks for the info. To manage our expectations, would you be able to provide an order of magnitude (not even a ballpark), like are we talking about a week, month, quarter, year here? I promise we won't hold you folks accountable for it ;)
@miketimofeev thanks for the info. To manage our expectations, would you be able to provide an order of magnitude (not even a ballpark), like are we talking about a week, month, quarter, year here? I promise we won't hold you folks accountable for it ;)
👋 @lloeki - We are still in evaluation phase for M1, but are taking the next step by developing plans for a preview in 2022.
Looking forward for this
Temporary workaround would be to setup a self-hosted github runner https://docs.github.com/en/actions/hosting-your-own-runners/about-self-hosted-runners
Any updates on this?
As a workaround step I'm cross-compiling builds for arm64 on x86_64 runners for some projects by adding -arch arm64 to CFLAGS, CXXFLAGS, and LDFLAGS (or its cmake equivalent -DCMAKE_OSX_ARCHITECTURES=arm64, see e.g libddwaf) but others are trickier (e.g libv8-node, because of this issue).
Of course even for those that do work there's no way to run things (e.g run unit tests or whatever). It's already quite helpful to have builds and kind of hinge on the x86_64 tests to pass.
I hope you'll be able to add support for GItHub Actions for Apple Silicon...is there any update?
+1
Hi.
we are actively evaluating the feasibility of this
I've stayed tuned and a year has passed.
Many (especially open source) projects relying on GitHub Actions to build and release are in stuck.
So end users have had to depend on Rosetta2 for quite a long time..
I hope GitHub just let us know if it's seriously planning provision of Apple Silicon in near future or not.
It'd be helpful for people to decide whether to move to another CI.
Just waiting without any information is too painful.
Thanks.
Which other CI provider supports M1?
It should surprise no one that at the same time Apple clarified their stance on what it means to do "short term rentals" of mac machines (that is, use them for things like CI/CD) such that they must be in 24-hour increments (and thus not really usable for non-enterprise CI, like open source use of Actions here) they launched 🥁 🥁 🥁 https://developer.apple.com/documentation/Xcode/About-Continuous-Integration-and-Delivery-with-Xcode-Cloud
Of note is that Xcode is a cross-compiler, I don't believe there is an issue using intel-hosted macOS to compile for arm64/apple silicon?
With all due respect to everyone explaining reasons why you might not need it are missing the point of the question. Those of us who need or want M1, A) don't want to have to explain all the details of why we do, and B) simply want clarity from GitHub.
I don't believe there is an issue using intel-hosted macOS to compile for arm64/apple silicon?
Some projects are easy to cross-compile (and I do that for those) others are not, if not impossible due to complex build systems.
In case you can then build you can still not test the resulting binaries, which matters a lot for some projects.
Which other CI provider supports M1?
CircleCI may have mentioned that support is coming (I am not affiliated with CircleCI or any other CI/CD provider and don't know anything other than what can be read at these links):
- ethereum/solidity#12291 (comment)
- https://circleci.canny.io/cloud-feature-requests/p/support-new-m1-arm-based-macs
Some other links that might suggest possible M1 CI support at some point
I'm with you, for what it's worth. Would love native M1 builds. Just trying to post possible partial solutions for the cases where those might help. Similar to the person that kicked off this last round of comments: more than anything just hearing a plan (or "there is no plan") would be cool. Cheers
Is this still happening or dead? I can't really use the Xcode stuff because it doesn't support the language I use and I also don't see myself buying into there build system which I don't want and also don't need.
Just install windows or linux in another box
Is there any update?
Is there any update? Even issues currently being grappled with?
Is there any update? Even issues currently being grappled with?
I don't know why It's being down voted
I don't know why It's being down voted
Because comments like the one you quoted are frowned upon as they generate useless notifications for people subscribed to this issue.
This is a good explanation why you should not do that
Just for the info of GitHub folks. We've just open sourced virtualization toolkit for Apple Silicon called Tart which can potentially power M1 infra for actions.
This issue has grown so long it's hard to follow. We need M1 github actions for our CI (cross-compile is not sufficient). I understand (?) that github's hosted runners are not yet supporting M1, but can we use our own (self-hosted) M1 Macs as runners yet?
Not yet: actions/runner#805
@garyo you can use self-hosted M1 runners, they work fine, I have been using them for a while. You can set them up like this: #2187 (comment)
@fkorotkov very interesting! I just read up on that site
Just for the info of GitHub folks. We've just open sourced virtualization toolkit for Apple Silicon called Tart which can potentially power M1 infra for actions.
I could not find in the documentation or issue search whether nested virt is supported. Do you know if Tart does nested virt?
Seems that Github released a support for self-hosted M1 runners:
https://github.com/actions/runner/issues/805
Is there any chance to get support for M1 on github-hosted runner? :)
Apple M2 is here, and this is still an issue, open for 18 months. If Apple care about Apple Silicon transition they should partner with GitHub and Microsoft and fix this for good. With the right focus and investment it could be fixed in less than a month.
Building arm64 binaries with qemu is incredibly slow and not always works.
With the right focus and investment it could be fixed in less than a month
Assertions without evidence are easy :-). With right focus and investment I could beat Starliner to Mars ;-)
Worth noting that CircleCI doesn't support it either https://circleci.com/docs/2.0/testing-ios/#apple-silicon-support
These are the biggest CI vendors. Perhaps it's a hard problem for many (some non-technical even) reasons.
Cirrus seems to do it https://cirrus-ci.org/guide/macOS/ as of around April 20, so maybe a month ago cirruslabs/macos-image-templates@2a3d22a
Perhaps it's a hard problem for many (some non-technical even) reasons.
Apple has a 24-hour minimum allocation period in their ToS which means that you cannot use them for a short CI run. This is also the reason you can only rent them for >=24h on AWS.
I assume that this is the biggest blocker as the hardware is already available.
Perhaps it's a hard problem for many (some non-technical even) reasons.
Apple has a 24-hour minimum allocation period in their ToS which means that you cannot use them for a short CI run. This is also the reason you can only rent them for >=24h on AWS.
I assume that this is the biggest blocker as the hardware is already available.
@BastianZim Where do you see that this applies exclusivity to Apple Silicon machines?
As far as I'm aware this language exists in the macOS TOS and doesn't specifically reference Apple Silicon machines.
AFAIK, Azure is running mac VM on Mac Pro so I would guess GitHub Actions follows the same. If the ToS applies to OS itself, I believe the reason is the available hardware choice. They probably didn't want to invest on MacMini M1 and was just waiting on Mac Pro Mx. Maybe they ordered Mac Studio for it but they probably wouldn't announce it until it's ready for the public beta even if they did.
Worth noting that CircleCI
They should partner with all major CIs
I believe the reason is the available hardware choice
That’s why I said they should talk and partner. Apple needed a strategy for Apple Silicon servers from day 1. Because a lot of the binaries are built on dedicated servers. They have Xcode Cloud now, maybe they have custom hardware powering it. I would be surprised if they used consumer hardware.
On the GitHub side there may be reasons for the delay. I’d apreciate if any GitHub employee chime in to discuss, or people (like me) keep speculating.
My guess is that the GitHub Actions product was not ready for multi-arch builds. If you look at the APIs there is no “arch” concept. Just SO (Ubuntu, MacOS, Windows).
Same in the billing system. I’m a paid customer consuming tens of thousands of build time per month. There is no breakdown per arch. Maybe price is different, and it would be ok to charge differently.
But as I said, just speculation.
Where do you see that this applies exclusivity to Apple Silicon machines?
As far as I'm aware this language exists in the macOS TOS and doesn't specifically reference Apple Silicon machines.
It applies indeed generally, that's just one argument I have heard from some providers and OSS communities but true it shouldn't be a blocker.
Apple just announced the availability of Xcode Cloud, so I doubt that they'll partner with anyone.
Can this debate of hypotheticals and complaints be moved elsewhere? Many of us are subscribed to this issue to get updates on implementation progress. Thanks! 🦄⭐️🏅👍🏻
There is now a tracking issue on the roadmap repo github/roadmap#528
I guess we should lower our expectations, the roadmap status for this issue is on future status, not even planned for this year 😢
I guess we should lower our expectations, the roadmap status for this issue is on future status, not even planned for this year 😢
Afaict, it is on the roadmap for Q2 2022:
https://github.com/orgs/github/projects/4247/views/1?filterQuery=M1
Edit: Oops. That's the self-hosted runners.
I will give up on GitHub Actions for macOS and move to the newly-launched Xcode Cloud as GitHub can't keep up with M1 / prioritize macOS
Have contacted GitHub support and they hope to have it in place by end of FY2022.
it seems that Github Actions Mac M1 support is on the roadmap: github/roadmap#528
it seems that Github Actions Mac M1 support is on the roadmap: github/roadmap#528
It's on the roadmap for almost a month, the problem is that they placed it on future implementation, therefore we have no idea when this will be done 😢
There's a typo on github/roadmap#528 however I can't comment there as that issue isn't open for public comments:
GitHub-hosted macOS runners powered by Apply silicone in an OSX arm64 runtime.
should be
GitHub-hosted macOS runners powered by Apple silicon in an OSX arm64 runtime.
OSX -> macOS
@MehdiChinoune that is a different issue. That one for self-hosted runner. This issue for GitHub hosted runner.
I think we as a community need to push for this one. We have hundreds of comments here and a lot of us are eager to get this support. It's already on the roadmap, but on future implementation 😞
Marketing:
Reality:
M1 support of hundreds of open-source projects waiting for this (look at the mentions above). I don’t know the difficulty here but GitHub should not be the one of the last ones that supports M1 hardware.
For everyone who doesn't like getting spam on this thread, you can customise watch settings to only get notified when this issue is closed.
@thejoebourneidentity it has been nearly 2 years since this issue was raised. Is there still no ETA for M1/M2 support?
<Shameless plug> Scaleway can provide on-demand M1 https://www.scaleway.com/en/hello-m1/ </Shameless Plug> You would still have to configure it with a self-hosted runner https://docs.github.com/en/actions/hosting-your-own-runners/about-self-hosted-runners but at least it could maybe help some of you.
github/roadmap#528 seems relevant?
Sorry for a shameless plug but we've tried to reach out to GitHub in this thread and in a few other places to collaborate on using our open-sourced virtualization solution to power Apple Silicon runners but haven't got any responses...
So we've built our own service with on-demand Apple Silicon runners for GitHub Actions. It's as simple as installing Cirrus Runners App and creating a subscription that includes 10 days free trial. Then you can run your actions in a macOS Ventura VM with 4 M1 cores just like that:
name: Test Suite
jobs:
test:
runs-on: ghcr.io/cirruslabs/macos-ventura-xcode:latestCirrus Runners are at least 2x more powerful than existing Intel runners provided by GitHub. Plus there is no limit on the amount of minutes a month you can use for $150 a month for each concurrently executed runner.
GitHub will never take this as a priority soon, they are earning a lot more money compared to what they will get if they support Apple Silicon 😅
Thanks for posting @fkorotkov – https://github.com/cirruslabs/tart + https://github.com/cirruslabs/macos-image-templates/pkgs/container/macos-ventura-xcode is exactly what I've been looking for.
(I'm looking to run with self-hosted runners, but using pre-built MacOS Virtual Machines)
Sorry for a shameless plug but we've tried to reach out to GitHub in this thread and in a few other places to collaborate on using our open-sourced virtualization solution to power Apple Silicon runners but haven't got any responses...
So we've built our own service with on-demand Apple Silicon runners for GitHub Actions. It's as simple as installing Cirrus Runners App and creating a subscription that includes 10 days free trial. Then you can run your actions in a macOS Ventura VM with 4 M1 cores just like that:
name: Test Suite jobs: test: runs-on: ghcr.io/cirruslabs/macos-ventura-xcode:latestCirrus Runners are at least 2x more powerful than existing Intel runners provided by GitHub. Plus there is no limit on the amount of minutes a month you can use for $150 a month for each concurrently executed runner.
@fkorotkov Is there no free tier for unfunded open source projects? https://cirrus-ci.org/pricing suggests there is…
@danielbayley we do unlimited minutes for free for open-source projects in our Cirrus CI offering. But for Cirrus Runners we only do paid plans.
Any chance this will finally happen in 2023?
@ankneis ankneis moved this from Future to Q3 2023 – Jul-Sep in GitHub public roadmap on Dec 14, 2022
It's coming. They're using Anka now for building VMs on developer MacBooks (Anka doesn't work on Mac Mini's without paying). You can see the templates in HCL in this repo.
IMO they should use Tart instead, since it's free and open source. The AGPL-3 license probably means that Microsoft can't use it, though. Microsoft, generally, can't use GPL software of any sort except in very special circumstances. I know that doesn't sound right, but I promise you, they are (for the most part, not completely) allergic to GPL licenses. Only the CirrusLabs folks can say if the AGPL-3 is right for Tart, but if they used MIT or BSD license, I'm sure MS would consider its use. Again, that's a decision for the CirrusLabs folks, and they should use the license they feel is best for them.
Tart has announced that they're changing licenses.
Tart has announced that they're changing licenses.
That's good; I don't really like the license they've chosen, given that it's not an open source license, but it's better than AGPLv3, for sure, in some ways.
At that price, though, I might as well just use VMs from AWS or Azure. My whole motivation for chasing a solution involving Tart was to save money in the long term over what AWS and Azure offer.
Hm.
In the roadmap moved to Q4/2023... So either end of year or at some undefined point in the future...
Hey @naikrovek, one of Tart’s developers here. Just want to elaborate a bit about your concerns on pricing.
Tart itself is free to use on personal workstations and until your organization reaches a certain number of server installations that utilize more than 100 CPU cores combined (running Tart on 12th M1 Mac Minis for your CI is free, for example). And even after reaching the limit, our pricing is very competitive and in our assessment it’s still several times cheaper than AWS or Azure like you mentioned.
We also have a managed service where for $150 a month you can get an M1 powered GitHub Action Runner with no limit on minutes used each month. This pricing is also the best out there. For comparison, $150 is equal to 32 hours of CI workflows on Intel runners.
We give up, just bought a Mac mini M2 and going self hosted.
Any update on this? All new macs are Apple Silicon, x86 is pretty much dead.
anyone knows how it goes now?
Can the maintainers lock or limit interaction on this issue? It’s just people asking the same question over and over (these comments are being downvoted and deleted). Maybe create and link to a discussion where people can do that, if necessary. I’m subscribed to know when this issue is updated - not to hear folks whine.
Adrressed as #8439 and M1 is now available!