GitHub Rate Limit Errors
atiq-cs opened this issue · comments
Statiq.Web uses octokit.net which seems to easily hit 'GitHub Rate Limit'.
This often happens with few hundreds of megabytes of output,
[DBUG] Deployment/Output » ExecuteIf » ExecuteIf » DeployGitHubPages » Uploading git blob for D:/statiq-blog/output\vendor\fontawesome-free\webfonts\fa-solid-900.ttf
[DBUG] Deployment/Output » ExecuteIf » ExecuteIf » DeployGitHubPages » Uploading git blob for D:/statiq-blog/output\vendor\fontawesome-free\webfonts\fa-solid-900.woff
[DBUG] Deployment/Output » ExecuteIf » ExecuteIf » DeployGitHubPages » Uploading git blob for D:/statiq-blog/output\vendor\fontawesome-free\webfonts\fa-solid-900.woff2
[DBUG] Deployment/Output » ExecuteIf » ExecuteIf » DeployGitHubPages » Creating a new git tree
[ERRO] Deployment/Output » ExecuteIf » ExecuteIf » DeployGitHubPages » Server Error
[DBUG] Exception while executing pipeline Deployment/Output: Octokit.ApiException: Server Error
at Octokit.Connection.HandleErrors(IResponse response) in /Users/shiftkey/src/octokit.net/Octokit/Http/Connection.cs:line 665
at Octokit.Connection.RunRequest(IRequest request, CancellationToken cancellationToken) in /Users/shiftkey/src/octokit.net/Octokit/Http/Connection.cs:line 643
at Octokit.Connection.Run[T](IRequest request, CancellationToken cancellationToken) in /Users/shiftkey/src/octokit.net/Octokit/Http/Connection.cs:line 628
at Octokit.ApiConnection.Post[T](Uri uri, Object data, String accepts, String contentType) in /Users/shiftkey/src/octokit.net/Octokit/Http/ApiConnection.cs:line 275
at Statiq.Web.GitHub.GitHubClientExtensions.ThrottleAsync[T](GitHubClient client, Func`2 operation, CancellationToken cancellationToken, Boolean throwIfRateLimitExceeded)
at Statiq.Web.GitHub.DeployGitHubPages.ExecuteConfigAsync(IDocument input, IExecutionContext context, IMetadata values)
at Statiq.Common.MultiConfigModule.ExecuteContextAsync(IExecutionContext context)
at Statiq.Common.Module.ExecuteAsync(IExecutionContext context)
at Statiq.Common.Module.ExecuteAsync(IExecutionContext context)
at Statiq.Core.Engine.ExecuteModulesAsync(ExecutionContextData contextData, IExecutionContext parent, IEnumerable`1 modules, ImmutableArray`1 inputs, ILogger logger)
{
"message": "Server Error"
}
When we do git push manually those output files that works fine. However, octokit used by Statiq.Web always seems to hit Rate Limit issues for my cases.
related to octokit rate limit exceeded,
http://coderdiaries.com/github-error-rate-limit-exceeded
This is a known problem and despite many, many attempts at getting around it I'm basically out of ideas. You can see in the call stack where a method ThrottleAsync is called - that's supposed to throttle the requests and add time/silently retry when the rate limit is hit.
It seems to trigger more frequently for larger sites or under accounts that are already using the GitHub API for other purposes. Is it at all possible some other error could be at fault? I.e. does this always seem to happen right at the "Creating a new git tree" step (which might suggest something else - I'd expect rate limit errors to be somewhat random on where they happen)?
Replacing Octokit with libgit2sharp should fix this.
ref, Statiq.Web.GitHub/DeployGitHubPages.cs
You mean because then we'd be interacting directly via git instead of through the GitHub API using git-like operations? Early on my first attempt at a built-in GitHub Pages upload capability was via libgit2sharp and it didn't go well. The problem is that we can't actually be guaranteed of the output being a repository on disk that's cloned from the GitHub Pages branch. Unlike the git-like APIs in the GitHub REST API, you can't use a git library like libgit2sharp to create a changeset without cloning the repository. I tried a few variations of cloning the GitHub Pages branch in order to generate a changeset and push the new output content but couldn't ever get anything to work very well. Git permissions, conflicting git repositories (because the output folder is inside the main repository), etc. were all issues.
Not to mention git and GitHub REST via Octokit are two very different APIs. I can't just drop in libgit2sharp as a one-for-one and call it replaced. It would take a fair amount of work to switch over even if it was worth trying again.
Unless I've totally misunderstood the suggestion here?
This is a known problem and despite many, many attempts at getting around it I'm basically out of ideas. You can see in the call stack where a method
ThrottleAsyncis called - that's supposed to throttle the requests and add time/silently retry when the rate limit is hit.It seems to trigger more frequently for larger sites or under accounts that are already using the GitHub API for other purposes. Is it at all possible some other error could be at fault? I.e. does this always seem to happen right at the "Creating a new git tree" step (which might suggest something else - I'd expect rate limit errors to be somewhat random on where they happen)?
I ran into the same issue. My site is not big. Last couple of updates would work 50% of the time. Today I tried to update some layout. I finally gave up and pushed everything manually to my GitHub Page. Hope a fix can be found.
@Feiko Do you by change happen to have other things calling the GitHub APIs (bots, etc.) or is it just the publishing step at any given time? Maybe I'll try doubling the waiting time when the GitHub API reports back the limiting threshold has been reached. Keep an eye out for the next release and let's see if that helps.
@daveaglick I have it hooked up to Netlify, Because I'm using Netlify.CMS to update my content. Something I posted about on my webpage https://www.feiko.io/posts/2021-05-07-adding-netlify-cms-to-your-net-powered-static-website. However, I did not use Netlify for my latest update, because I needed to change layout, not content. So I don't think this is the issue in my case.
I'll try again in the next release.
I'll go ahead and close this issue for now since some work was done around the rate limiting that (hopefully) mitigates the problem. Please comment so I can open it back up if it turns out that didn't resolve the problem in the latest releases.
@daveaglick, I recently updated my GitHub solution to the latest version of Statiq.Web. I was able to publish new content without any problem. Thank you for your help and great support on this. Keep up the good work!
@Feiko Awesome! Glad to hear it. This has been a particularly long-running and tricky problem so hopefully it's solved once and for all 🤞