pushedAt is not reliable
kelset opened this issue · comments
Description
In the LibraryListItemColumnThree component, to calculate the "Updated at" field, to the getTimeSinceToday method the value props.library.github.stats.pushedAt gets passed.
But this field (provided by the GitHub API) is actually not "reliable": what I mean by that is that the field pushedAt is not the datetime of the last commit pushed to master - which to me should define when the repo has been updated.
To prove it I'll use the react-native-push-notification repo - if you check via native.directory (here) you can see that it says that it has been updated for the last time "2 weeks ago" because it uses
"pushed_at": "2017-06-30T17:58:21Z",
as datetime to use for the calculation.
But if you check instead the commit history it will say that the last commit to master was done on May 15th (2017-05-15T18:43:43Z) - which is way more than 2 weeks ago.
I can't actually figure out where the GitHub API gets that value for pushed_at, but yeah it's not the last commit to master.
Proposed solution
While searching around, I managed to use the GraphQL explorer from GitHub itself to search (in APIv4) for the last commit date using this query:
query {
repository(owner:"zo0r", name:"react-native-push-notification") {
ref(qualifiedName: "master") {
target {
... on Commit {
id
committedDate
}
}
}
}
}So my proposal would be to use, instead of the APIv3 (REST) for GitHub, actually move to its v4 and, by using GraphQL, fetch all the necessary data plus the datetime of the last commit to master, and then use that one to calculate the "Updated XXXX ago" field.
But it may be "a big deal" so I wanted to open an issue to discuss with you guys if it would be a good idea or if it would be better to stick with v3 - in that case, I guess it would be necessary a secondary GET to the API for the last commit for each repo.
If you greenlight it I'd love to try to do it and PR (but I can't give any ETD atm).
I'm a fan of GraphQL and so are the members on our team. So if you want to commit the time to creating a PR to migrating our code over to GraphQL I'm in.
Thanks for thinking about this! Really appreciate you helping us move forward.
I think you should thank @brentvatne for pointing me to the repo ;)
Anyway glad to hear, I will give it a shot - consider that my experience with actual GraphQL (and React web tbh) is limited to a 48hrs Hackathon so it will probably take me a bit of time to send the actual PR 😇
Any preferences over what to use to parse the data via GraphQL? I think that the standard GraphQL js lib should be enough for querying Github APIv4.
In that case, be sure to take your time, really think about the problem, and start with instances where we use isomorphic-fetch. You don't have to touch any of the React web components to prove that GraphQL is a better abstraction over using fetch calls.
I also think the standard GraphQL JS lib is fine :)
Hey @kelset!
I have put together PR #38 which implements a new scoring system. One of the rules checks the pushedAt date and awards a bonus for recently updated libraries. This issue currently causes an issue with that rule being incorrectly applied.
Are you currently working on a PR to fix this issue, or is it something that someone else is able to pick up?
Hi @matt-oakes,
yeah I'm still trying to make it work locally, didn't have enough spare time to complete it yet ><'
Hopefully this weekend I'd be able to submit at least a WIP PR with the "correct" pushedAtinformation your scoring system relies now.
Since we are speaking about it, I think it's worth saying that technically pushedAt is correct: it's GitHub that probably consider that field in a way we can't understand - so instead my idea was having a new field called something like lastCommitDate or lastCommitOnMasterDateand use that instead of pushedAt.
@kelset Sounds great! I think changing the name of the property to something like lastMasterCommitData is a good idea.
Just so you know, my PR also fixes a few small issues with the Github data:
watchershas been changed tosubscribers. This is because "watchers" in the Github API is always the same number as "stargazers". The number of "watchers" at the top of each Github repo is actually called "subscribers" in the API (how straight forward...).- I have added
fullNameproperty, which is the repo name in the formatreact-community/native-directory. This is just to save parsing thegithhubUrlagain. - You should also be able to get the
licenceinformation in a single GraphQL command and avoid the need for the additional API calls that my PR creates.
Quick update: didn't submit the PR 'cause couldn't find a way to do a "single" GraphQl call - most tutorials I've found are all just using Apollo to wrap everything, so yeah I haven't found a way yet 😶. Any link appreciated ;)
It looks like from this page of the official docs that you can just make a standard HTTP request, rather than using a library.
This page from the Github v4 GraphQL API docs should have all the details you need, along with the query you posted above. You just need to convert the requests to Javascript rather than the curl examples they give. It will return JSON which you should just be able to use. Should be no need for Apollo or anything more complex.
@kelset @matt-oakes I've landed: #38, so:
license has been added
fullName has been added
watchers is now subscribers
Ok I'm back on track ;)
Finally the awesome guys at Apollo published an article that makes it quite easy for me to understand how to do it :)
I think that adding their apollo-fetch is worth for readability - gonna try to submit a PR tonight :3
Nice @kelset -- glad you're still working on this!
If the work is clean, simple and easy to understand, it will definitely land in native.directory! We use GraphQL on expo.io and I wouldn't mind using it here too.
PR submitted (#50) but it is not completed yet - I preferred to start creating it to be able to discuss it along the way.
Fixed by #333
👏👏👏👏👏