Investigate and implement new Twitter API
IBBoard opened this issue · comments
After a delay, Twitter has announced new APIs that apparently include things like seeing polls and fetching conversation threads. It might even have streaming again! (although that might be limited to searches and not home timelines).
It also appears to use a new "bearer token" for authentication, which we'd have to look in to (but Baedert appears to be implementing for a Mastadon fork of Corebird)
Once it's out of Early Access and reliably available then we should look at the new API and implement it.
Looks like the structure of tweet objects is going to change significantly, but also be far more flexible (so we can pull just what we need). That probably means we can't easily do this piecemeal.
It also appears to use a new "bearer token" for authentication, which we'd have to look in to (but Baedert appears to be implementing for a Mastadon fork of Corebird).
From what i have seen so far, this "bearer token" will not replace the "user token" in our case. Using the "bearer token" for example would limit us to an app-wide (rather than an user-specific) rate limit.
Now the conversation at the bottom of tweet detail, can't be shown sometimes.
At the Mentions view, even worse, never succeed.
Cawbird verson: cawbird-git 1.2.1.r5.gefd5e6c7-1 built with -D video=false -D gspell=false
Did you tried to access the conversation for an tweet older than seven days? If so, this is a known limitation.
Maybe the historical search will be allowed for standard access with version 2 of the Twitter API, but until then we are stuck with what we have.
Did you tried to access the conversation for an tweet older than seven days? If so, this is a known limitation.
Yes, you're right. Thx
@IBBoard, a little bit off-topic, but while looking in the code to understand how the backend is currently implemented, I noticed it is written in C?
You know the reason for using C instead of Vala?
@20NE - As @CodedOre said, there's a known limitation with replies that's imposed by Twitter's API. We'll see what the new API gets us. But regardless, "I'm missing tweets" is a completely separate bug that should be in its own issue rather than being added to a feature request for using the new API.
@CodedOre Not a clue. I always assumed that Baedert just liked pain and suffering 😉 Or he thought there was some micro-optimisation in performance that Vala transpiling missed. The commit that changed CbUserStream to C doesn't give any clues and I think most of the others don't either, but there's occasionally a reason.
I've often thought that it would be nice to ditch all of the C code and move to pure Vala. It would probably solve so many problems. But it's a big job and I've never had time to do it. And it's not going to be a particularly obvious improvement for users either, even if it does make life easier for developers.
I've now made Cawbird a "Project" rather than a "Stand-alone App", so we should be able to start using v2 API calls.
I think the fun will be that the message structure has changed a little bit, so even the slightest update to API usage might have a lot of knock-ons throughout the code.
One thing that I've noticed that concerns me is that the filtered stream (which we don't use) and the search API have app-wide monthly caps on the number of tweets we can fetch (docs). 500K tweets is a lot of tweets, especially when we fetch in batches of 35. BUT we also use the search API for threading.
And then the "user mentions" timeline also says that it has a monthly limit of 500K tweets, despite it only saying "filtered stream and search".
This could get tricky, depending on how many users we actually have and how much they search/view mentions/get mentions
I'm almost ready to write off the v2 API. More and more endpoints are coming out and more and more of them say that they're subject to the tweet cap.
500,000 tweets per month is under 17,000 per day, and just under 1400 per hour (assuming people have Cawbird running for 12 hours per day - which is about what I used to do between a work and home machine). That means that 100 users would only be able to fetch 14 tweets per hour before we ran out of tweet cap. And that includes searches, mention timelines, and user timelines shown in profiles (and may include your main subscribed timeline, when they release that). That's barely any tweets, or we've got to have only a handful of users.
The only alternative is that I stop shipping default keys (or keep shipping them and have a huge warning saying "half of the functionality may not work because Twitter hates third-party clients"). But anything that says "here's the app, sorry but you've got to get your own key" runs the risk of falling foul of Twitter's "no white label apps that people put their own keys in". Out current approach is obviously 100% above board because we've just chosen to separate out dev (my personal keys), pre-prod (unstable) and production (official builds and Flatpak), which we're allowed to do, and the fact that this also lets other distros choose to package their own key is entirely their own choice 😁
Well, let's hope that there will be a solution available soon...
Would be a bummer if I finish Cawbird 2.0 and then nobody can use it because of tweet caps...
I think it might depend how long the v1.1 API is retained for. And making it more generic to support Mastadon as well will give it more life (but more complexity!)
Worst case is that it becomes like my post-Corebird-pre-Cawbird patch set and it's just a niche app for the developers 😁
Tweeting has finally arrived! Includes the ability to post polls (that's going to be good, but awkward for UI!) and reply settings (probably a priority feature - people need the control these days, and we should enable that).
There's also some rubbish about "Super Follows" tweets, but no-one wants that and we won't implement it on principle!
https://twittercommunity.com/t/announcing-manage-tweets-endpoints-for-the-twitter-api-v2/161501
Interestingly, Twitter have just announced "Elevated" access, which gets you 2m tweets per month and… my account is clearly old enough that Cawbird is counted as Elevated 🙂 (Possibly because v2 is the new default and v1.1 APIs aren't enabled by default, but as we're on v1.1 then we get grandfathered into the Elevated tier)
Also, I'm wondering whether we can limit the number of posts we pull to make it work if we do things like not having an auto-updating "Likes" tab, and caching @-mentions etc (so we only load new ones each start-up). The big killer is probably still thread replies, if there's not an API for that yet. If they still come out of search then we'll have to do a "latest 20" or something and really limit what people see.