GitHub Releases: Status 503 Egress is over the account limit.

Question

GitHub Releases: Status 503 Egress is over the account limit.

AustinTSchaffer opened this issue 3 years ago · comments

Hello! I tried installing sharp for a new project this morning and ran into an issue with this repository.

What I did:

npm install sharp

What I got:

> sharp@0.29.3 install {my new project directory}
> (node install/libvips && node install/dll-copy && prebuild-install) || (node install/can-compile && node-gyp rebuild && node install/dll-copy)

sharp: Downloading https://github.com/lovell/sharp-libvips/releases/download/v8.11.3/libvips-8.11.3-darwin-x64.tar.br
sharp: Installation error: Status 503 Egress is over the account limit.
sharp: Please see https://sharp.pixelplumbing.com/install for required dependencies

I reran the install command with the AWS Lambda instructions, which then worked, though it looks like that command also downloaded resources from this repository. For future reference, should I just keep retrying the install command until GitHub sends the binaries?

Lovell Fuller · Answer 1 · Wed Dec 08 2021 00:06:46 GMT+0800 (China Standard Time)

Egress is over the account limit

This message is from (the Microsoft-owned) Azure and says (the Microsoft-owned) GitHub has reached the egress limit on their blob storage account.

The Azure docs mention an egress limit of 120Gbps, which I guess might apply to the whole of GitHub Releases.

https://docs.microsoft.com/en-us/azure/storage/common/scalability-targets-standard-account

@MylesBorins Is this a known limitation of GitHub Releases?

Lovell Fuller · Answer 2 · Wed Dec 08 2021 00:21:35 GMT+0800 (China Standard Time)

The only other example I can find of this specific error message in relation to GitHub Releases is from 3 days ago, which suggests this might be a new problem.

codecov/uploader#532 (comment)

Myles Borins · Answer 3 · Wed Dec 08 2021 00:23:36 GMT+0800 (China Standard Time)

@lovell would you be able to open a ticket with GitHub support so we can track this internally. I'll escalate to the appropriate folks

Lovell Fuller · Answer 4 · Wed Dec 08 2021 01:07:54 GMT+0800 (China Standard Time)

@MylesBorins Thank you for the quick response, support ticket 1419586 created for this.

Lovell Fuller · Answer 5 · Wed Dec 08 2021 17:43:03 GMT+0800 (China Standard Time)

I got a reply from GitHub support, which quotes the same URL I provided:

Thanks for reaching out to GitHub Support!

The 503: Egress is over the account limit. error is pretty uncommon and usually transient.

You can find more information related to this error here:

https://docs.microsoft.com/en-us/azure/storage/common/scalability-targets-standard-account

If the error persists though, please let us know and we can ask our engineers to investigate further.

This appears to confirm that GitHub Releases:

serves assets directly from Azure blob storage
has a peak traffic level greater than the 120Gbps egress limit

Many others are reporting the same problem at community/community#8535

Anyone else experiencing this should contact GitHub support to provide more evidence via https://support.github.com

Matt Kane · Answer 6 · Wed Dec 08 2021 21:01:38 GMT+0800 (China Standard Time)

I wonder if it would make sense to load the binary via e.g. Cloudflare instead of direct from GitHub

Lovell Fuller · Answer 7 · Wed Dec 08 2021 21:03:31 GMT+0800 (China Standard Time)

Follow up response from GitHub support:

We have indeed had multiple reports of this bug over the last couple of hours and our engineers are working on a fix. It's very useful for us to have more data points, so thank you for submitting this report! I'll pass it on to the team.

Alexandros Gkesos · Answer 8 · Wed Dec 08 2021 21:05:34 GMT+0800 (China Standard Time)

Got the same while downloading kubectl

Error: Failed to install provider
Error while installing gavinbunney/kubectl v1.13.1: unsuccessful request to
https://github.com/gavinbunney/terraform-provider-kubectl/releases/download/v1.13.1/terraform-provider-kubectl_1.13.1_linux_amd64.zip:
503 Egress is over the account limit.

Lovell Fuller · Answer 9 · Wed Dec 08 2021 22:49:37 GMT+0800 (China Standard Time)

Further response from GitHub support:

Just wanted to update you on this: the team has identified and resolved the underlying issue, so you should no longer see any of these 503 errors.

Lovell Fuller · Answer 10 · Wed Dec 08 2021 23:02:42 GMT+0800 (China Standard Time)

It looks like release assets are being served via S3 again, specifically the us-east-1 region, so my best guess would be that this was a cascading failure situation related to yesterday's AWS outage.

Austin Schaffer · Answer 11 · Thu Dec 09 2021 06:36:06 GMT+0800 (China Standard Time)

@lovell

Thank you for keeping this thread updated on what was going on with GitHub, Azure, and AWS, despite it being well beyond the scope of this project. Best wishes and happy holidays.

Julien Goux · Answer 12 · Wed Dec 15 2021 18:03:39 GMT+0800 (China Standard Time)

It doesn't seem fully resolved, we just hit the error in our CI right now. 😅

sharp: Downloading https://github.com/lovell/sharp-libvips/releases/download/v8.11.3/libvips-8.11.3-linux-x64.tar.br
sharp: Installation error: Status 503 Egress is over the account limit.

Lovell Fuller · Answer 13 · Wed Dec 15 2021 18:13:08 GMT+0800 (China Standard Time)

If you are experiencing this problem, please contact GitHub support via https://support.github.com

Aleksandar · Answer 14 · Fri May 13 2022 18:54:22 GMT+0800 (China Standard Time)

Experiencing this issue for a few days now - never ocurred before. Builds on Azure DevOps have been failing due to:

error node-pre-gyp ERR! install response status 503 Egress is over the account limit. on 
https://github.com/Automattic/node-canvas/releases/download/v2.8.0/canvas-v2.8.0-node-v93-linux-glibc-x64.tar.gz