Convert Markdown parsing to use GitHub's markdown parsing
mattstauffer opened this issue · comments
Sample from Giscus:
tighten/giscus@bb8f124
Problem: 5000/hour rate limit.
@mattstauffer This is an interesting comment on a way to get around the 5000 hour rate limit
you could have multiple tokens and switch them out depending on how many requests you have left for the hour.
Also reference this if making multiple tokens possibly
https://developer.github.com/changes/2012-12-06-create-authorization-for-app/
Hey @mattstauffer, what's the reason to use github's api to convert to markdown? Is it primarily the issues with the current parser?
How soon would you expect the 5k/hr limit to be a factor? Or are you hitting it already?
@mattstauffer I may have gone way to simple as i did not address the api rate limit, but super simple change to use github markdown:
<?php namespace Gistlog\ContentParser;
use Github\Client;
use Github\HttpClient\Message\ResponseMediator;
class MarkdownTransformer implements Transformer
{
private $client;
public function __construct()
{
$this->client = new Client;
}
public function transform($content)
{
return $this->client->api('markdown')->render($content);
}
}Fork here: https://github.com/quickliketurtle/gistlog/tree/github-markdown
Let me know if this is worth doing without addressing the rate limit and i'll submit a PR.
@quickliketurtle The current parser has a ton of little implementation details that differ from GitHub's and fixing every little one will take a lot of work. Using GitHub's API will be much faster.
However, that means we have to hope that we don't have any more than 5000 new conversion requests per hour; otherwise we'll hit the limit and all of a sudden some page won't convert.
I doubt we'll run into that at our current traffic load, though, so that should be fine.
Your branch looks like a good start (although it looks like you removed the .env.example file, which was probably an accident?). My biggest concern would be to make sure we're not going to run into the rate limit; can you test that it's A) caching correctly and B) authenticating with our GitHub credentials correctly. If you find it working correctly and can help verify those two things are happening correctly, I'd love for you to PR this!
I started on this here as well a while ago 😳https://github.com/tightenco/gistlog/commits/aw-switch-to-github-markdown
Thanks. I changed it around to be more inline with what Adam was working on.
I left the original transformer and added new one for github (rather than changing the existing one). added back the .env.example file ( I renamed it to .env originally rather than copying it ).
I've got caching working ( verified by looking in the github-api-cache directory ). Created a Service Provider similar to GistClientServiceProvider.
Regarding the rate limit. It's 60 requests / hour if you don't authenticate so i'm hitting that now. ( i clicked refresh a lot ) :-)
Looking at the GistClientServiceProvider, there is a check for client_id and client_secret, however those are commented out in config/services.php and do not exist in .env.example.
Doesn't that mean the current api requests are not authenticated and should be hitting the 60 req/hr limit?
How should i test the authentication? I can use a personal access token, but for the client_id and client_secret, since secret is in the name I'm not sure you should share it. Could i register a new app on my account? What do you recommend?
@mattstauffer I made a change so auth is working using just a github token. I also added a test to get the rate limit response confirming it is 5000, not 60.
I extended the github api package with a rate_limit endpoint which is what my test is using. I opened on issue on that repo asking it they would be interested in a PR for that. But until that happens maybe this test should be different...
So a recap:
- Using github api for markdown transformation.
- Response is cached.
- API Auth with token with 5000 rate limit.
Let me know your thoughts (any needed changes etc). Appreciate being able to work on this, it's fun. :-)
@quickliketurtle Not sure which version has the id and secret commented out; they're working on master for sure. But token may be fine, too? Unsure of the diff, honestly.
MY DUDE writing a test for the 5000. Thank you! Been wanting a test like that for a while.
I'm curious to see what you're doing. Why don't you PR it in with your notes and I can review it and we can go from there?
@mattstauffer this can be closed now