PrivacyDevel / nitter

One of my instances uses this fork, and they have changed likes endpoint (as far as I can tell) upstream sometime around 1/19.

I'm not finding the issue looking at the code, but I've created a python script to get likes for a user. I'm not seeing this here only 'favoriters'?

Also the user I used to test to see if hidden likes are truly hidden, are now hidden.

For likes I've been using

https://twitter.com/i/api/graphql/G_zHbTiwSqLm0TAK_3sNWQ/Likes

My script utilizes the same method of copying over x-csrf-token and 2 cookie elements for now.

Again I'm probably way off base using 'likes' but things are broken in this area now.

ETA https://gist.github.com/cmj/adfd541dde30585d861d28fd58bec9f0

Well, this change happened on the heals of them shutting various elements down in the past week. Guest accounts have been disabled. Closing.

zedeus#983 (comment)

Thank you for bringing this to my attention. I have looked into it a while ago but was not able to fix it without a full account. It is however still an issue and I would therefore like to reopen it so that it doesn't get forgotten, if I or somebody else wants to work on it in the future.

I've been testing around with postman and it seems you need to add the bearer token from the same account you use the authentication cookies and xcsrfToken

I've tried adding it to my personal nitter instance but still got empty results, but at least it's an initial attempt at getting it to work.

bearerToken variable was added to the following files for the attempt

Now I'm unsure on what is going wrong, since I have no idea how to properly debug nitter

Changed "proc genHeaders*" in src/apiutils.nim from

 proc genHeaders* (url, oauthToken, oauthTokenSecret: string): HttpHeaders =
   let header = getOauthHeader(url, oauthToken, oauthTokenSecret)
 
   result = newHttpHeaders({
     "connection": "keep-alive",
     "authorization" : header,
     "content-type": "application/json",
     "x-twitter-active-user": "yes",
     "authority": "api.twitter.com",
     "accept-encoding": "gzip",
     "accept-language": "en-US,en;q=0.9",
     "accept": "*/*",
     "DNT": "1"
   })

to

 proc genHeaders* (url, oauthToken, oauthTokenSecret: string): HttpHeaders =
   let header = getOauthHeader(url, oauthToken, oauthTokenSecret)
 
   result = newHttpHeaders({
     "connection": "keep-alive",
     "authorization" : cfg.bearerToken,
     "content-type": "application/json",
     "x-twitter-active-user": "yes",
     "authority": "api.twitter.com",
     "accept-encoding": "gzip",
     "accept-language": "en-US,en;q=0.9",
     "accept": "*/*",
     "DNT": "1"
   })

And removed from "fetch*" in same file what I added previously

 if len(cfg.bearerToken) != 0:
        additional_headers.add("Authorizateion", cfg.bearerToken)

This makes the likes tab work, but the media tab stop working and nitter fails find accounts you haven't visited previously.
Likes tab working:

Media tab not working:

Never previously visited account before not found:

Managed to get it working

Turned "genHeaders*" in src/apiutils.nim into this

proc genHeaders* (url, oauthToken, oauthTokenSecret: string): HttpHeaders =
  if "api.twitter.com/2/timeline/favorites" in url:
    let header = cfg.bearerToken
    result = newHttpHeaders({
      "connection": "keep-alive",
      "authorization" : header,
      "content-type": "application/json",
      "x-twitter-active-user": "yes",
      "authority": "api.twitter.com",
      "accept-encoding": "gzip",
      "accept-language": "en-US,en;q=0.9",
      "accept": "*/*",
      "DNT": "1"
    })
  else:
    let header = getOauthHeader(url, oauthToken, oauthTokenSecret)
    result = newHttpHeaders({
      "connection": "keep-alive",
      "authorization" : header,
      "content-type": "application/json",
      "x-twitter-active-user": "yes",
      "authority": "api.twitter.com",
      "accept-encoding": "gzip",
      "accept-language": "en-US,en;q=0.9",
      "accept": "*/*",
      "DNT": "1"
    })

Crude, but gets the job done. Maybe someone with more skill in nim could do something better.

2024-02-24.17-18-50-new.mp4

Note:
RSS feeds don't seem to be working with the likes tab, they are generating completely empty.

I have no idea how to get the RSS feed to generate for the likes tab. Was it even working before?

I have no idea how to get the RSS feed to generate for the likes tab. Was it even working before?

~~Looking back, it seems I was just scraping the /favorites page, then manipulating the data for RSS output.. So apparently not.~~
Yes, it was working. I personally only used it to track one account so it was easier to scrape the likes page and manipulate the data how I needed it for a bot.

Also I seem to be having some issues with the addition of bearer tokens. I had it print out the headers on fetching the favorites URL and everything seems to line up OK. When you have time, could you create a patch? I must be missing something.

This is what I'm working with going off your findings: https://gist.github.com/cmj/3f11ef5a202696c5ae2a5e50f17b9543

Thanks for your effort!

Sure thing, here's a patch with all changes I made.
https://gist.github.com/rotorot0/f84db5d6bbc1982b598c4157ffa15de5

Yeah it's still not working here. Even using bearer token from the account listed in the jsonl. I'll try and take a deeper dive later this evening... Something is afoot.

Yes, I can see it just fine.

Here's their page with my twitter account

Yeah it's still not working here. Even using bearer token from the account listed in the jsonl. I'll try and take a deeper dive later this evening... Something is afoot.

Okay, I noticed my mistake here. The way I grabbed the bearer token was with the OldTweetDeck browser extension.

Gotta add a likes tab and then grab the bearer token from that

Thank you for your work.

Here's their page with my twitter account

You can't see their likes on twitter, but you can with nitter still, correct?

You can't see their likes on twitter, but you can with nitter still, correct?

Exactly, I don't even have a likes tab when I access their profile on twitter.

I jumped back on this issue today. Since I was continuously getting {"errors":[{"code":220,"message":"Your credentials do not allow access to this resource."}]}. I've tried every option you point out @rotorot0 using the standard favorites endpoint. It must be the bearer token not authorized in conjunction with the account to access that endpoint, even though I have triple checked and with various accounts. Tweetdeck uses the old url and only changes I see on my end are api versions 1.1 and 2 between tweetdeck and nitter, respectively, however I'm not sure that matters.

Anyways, I thought I'd try switching over to the graphql query option. RSS works (sorry, it was working before too). One downside is you can't see "hidden likes".

A few changes need to be made to the parser so it can read the slightly different structure. I'll try and clean it up later.

Strange, I'm having none of those issues. How old are the accounts you're using? The one I use is from December 2022

OK, I had to make sure. Did a fresh checkout with your patch and double checked with tweetdeck. I might've missed the likes tab of another user. It then added one character to the end of the bearer token.

Everything works.

The unfortunate part of all this is having to install Tweetdeck. Since the regular browser interface uses graphql, this might be the only way to get a token for the old API endpoint for now. The benefit is actually viewing hidden likes and less changes to be made.

Thanks again.

Did a fresh checkout with your patch and double checked with tweetdeck. I might've missed the likes tab of another user. It then added one character to the end of the bearer token.

I see, so that's what happened. Thought it was maybe a restriction on newer accounts. I even made a new one to test and it worked fine.

It very well could've been a bad paste of the token on my part; it was early. Either way I think this issue is resolved. I'll leave it open until @PrivacyDevel approves.

ETA: I think the only additional part I can add is the migration to graphql for likes, which doesn't require extra headers, but does limit views. I'd rather keep full potential of likes open.

Just wanted to add, OldTweetDeck has a hardcoded public Bearer token:
Bearer AAAAAAAAAAAAAAAAAAAAAFQODgEAAAAAVHTp76lzh3rFzcHbmHVvQxYYpTw%3DckAlMINMjmCwxUcaXbAN4XqJVdgMJaHqNOFgPMK0zN1qLqLQCF

This works for every account I've tried, so there's no need to go hunt it down, we can add this directly.

https://github.com/dimdenGD/OldTweetDeck/blob/c9a3f6c4c76ca7f0f0ccc75afee7fdc641bb913a/src/interception.js#L3

I see. Then I'll just do minimal changes and hard code the bearer token too.

I tried to do some digging and found out that the X-Csrf-Token keeps rotating every now and then. Is that normal? And at the end of the day, got stuck at Postman working and in nitter instance doesn't work.

Nevermind, worked fine after fixing a slight mistake on my part.

What happens in case I have more than 1 account in guest_accounts.json file? The cookie and x-csrf-token should be different for each account no? If it's different then how can I make it to use different cookie and csrf cause I can only provide those for a single account.

Using the csrf and auth token, you bypass the oauth authentication and use the auth cookie, csrf (and bearer token) instead.

They can be from an entirely different account listed in your accounts file.

This is just for the likes tab, so it will only be using csrf and auth tokens for that request only. It will cycle through all other accounts listed in the json file as usual for everything else. I use this for a few Likes RSS feeds and haven't hit limits.

There is no method now to cycle through csrf and auth yet...

Here is a basic example of how the likes page is populated:
https://gist.github.com/cmj/62ab48eebb4a5c599fa322d5f6850689

I think on my side the pressure on the Likes endpoint will be higher. Probably 1 account won't be able to handle that much pressure evetually. That's why I was thinking how can I input multiple accounts to be used for the Likes tab.

This v1.1 endpoint was the easiest method to use, right now there isn't a simple way to manage those tokens. That could be done, but if you expect to see more that the current rate-limits on GET favorites/list which stands at 75 requests in a 15 minute window(?), you can use the v2/graphql endpoint.

That was my first attempt in getting Likes working again, the only "issue" is you won't see Likes from users who hide them. However, you will be able to use multiple accounts and have Nitter manage rate-limits properly, and you don't even need csrf or auth tokens, if i remember correctly.

I have that working somewhere, I'll try and clean it up soon.

but if you expect to see more that the current rate-limits on GET favorites/list which stands at 75 requests in a 15 minute window(?), you can use the v2/graphql endpoint.

I tried to make the graphql work work but was not able to achieve that. Couldn't even find the correct graphql Likes endpoint url. Can you maybe share?

I set that endpoint as favorites* = graphql / "eSSNbhECHHWWALkkQq-YTA/Likes" in consts.nim

There's a lot more that has to be done as well. I think there was a few parsing changes, etc.

I set that endpoint as favorites* = graphql / "eSSNbhECHHWWALkkQq-YTA/Likes" in consts.nim

There's a lot more that has to be done as well. I think there was a few parsing changes, etc.

I won't be needing the parsing changes, to be honest I am using nitter as an API provider. Have completely removed frontend, only have API endpoints. So as long as I can pull the raw data with fetchRaw that's more than enough for me.

Wanted to ask, other than the api url change to graphql, all other things like variables and features will be the same?

Looks like just 2 more features are required:
https://gist.github.com/cmj/98bd8ed1a1645c5af321c5f195ce11d0#file-consts-patch-L23

Looks like just 2 more features are required: https://gist.github.com/cmj/98bd8ed1a1645c5af321c5f195ce11d0#file-consts-patch-L23

Should I replace this line in apiutils.nim -

let header = if "favorites" in url: cfg.bearerToken
               else: getOauthHeader(url, oauthToken, oauthTokenSecret)

back to this? -

let header = getOauthHeader(url, oauthToken, oauthTokenSecret)

Yes, you can leave it as original. You could even remove these lines when using the graphql endpoint:

nitter/src/apiutils.nim

Lines 152 to 156 in cb0d360

    
           if len(cfg.cookieHeader) != 0: 
        
               additional_headers.add("Cookie", cfg.cookieHeader) 
        
           if len(cfg.xCsrfToken) != 0: 
        
               additional_headers.add("x-csrf-token", cfg.xCsrfToken)

Thank you very much @cmj ! That worked perfectly! But wanted to ask, if there's any downside to using the graphql endpoint for Likes? Curious cause why you guys didn't implement this by default?

@muhitrhn The only downside to using the graphql endpoint is it has been updated with a new feature that allows users to hide their likes, see Twitter user @ehikian for an example. If we stick with the old 1.1 endpoint they are visible.

The benefit of the graphql endpoint would be you can use multiple accounts if you are exceeding rate-limits. We settled on 1.1 because it was an easier route and most Nitter instances are private; not much need for the account management. And the benefit of bypassing the new Twitter Likes-hiding features.

That being said, for others interested, I'll fork upstream later and add the graphql option so no extra tokens are needed.

master...cmj:nitter:master

So I don't have anything to worry. Cause I need users likes to be visible. Otherwise they can't do what they need to do with my bot.

@cmj thank you very much for your contribution! I was now finally able to view the likes of other users as well with that graphql change! And the addition of the token generation script and the cleanup of some legacy code from me is also much appreciated! I merged your changes and consider this issue solved for now. I hope you don't mind :)

	if len(cfg.cookieHeader) != 0:
	additional_headers.add("Cookie", cfg.cookieHeader)
	if len(cfg.xCsrfToken) != 0:
	additional_headers.add("x-csrf-token", cfg.xCsrfToken)

Likes/Favorites has changed.