How do you learn about the twitter "API"?
kentcdodds opened this issue · comments
The twitter API used here seems unofficial. Where do you learn about the possible options?
these apis can break at anytime
You can discover the unofficial apis by looking at blogs/repos from people that have done it before eg: the fa0311/TwitterInternalAPIDocument repo or checking the devtools network tab, specifically the Fetch/XHR tab.
some examples
cdn.syndication.twimg.com/tweet-result?
Used at publish.twitter.com for the official embed but not documented.
Usage:
Make GET request to cdn.syndication.twimg.com/tweet-result?id=20&lang=en
- No special headers requred, just a valid tweet id. (You can visit the url from a browser)
- Doesn't include all the tweet data eg: spaces, longform tweets, rt count
Example using javascript
async function getTweet(id) {
const url = `https://cdn.syndication.twimg.com/tweet-result?id=${id}`
const response = await fetch(url)
const json = await response.json()
return json
}api.twitter.com/*
Used by the official twitter web client so it's very rich. It's always changing and fields can be deleted without warning.
eg: the co-tweet feature was removed from all api responses.
Usage:
Get a (public) bearer token from devtools in an incognito twitter.com tab. The current one is
AAAAAAAAAAAAAAAAAAAAANRILgAAAAAAnNwIzUejRCOuH5E6I8xnZz4puTs=1Zv7ttfk8LF81IUq16cHjhLTvJu4FA33AGWWjCpTnA
but it might be changed.
Make POST Request to api.twitter.com/1.1/guest/activate.json with the bearer token in the Authorization header.
The response will be something like this.
{"guest_token": "123"}The guest_token value is used to set a valid X-Guest-Token header. It can be reused but it expires after some time (not sure how long it lasts), so you need to renew it.
An example of getting a tweet from the statuses/lookup.json endpoint using curl
#!/bin/bash
bearer="AAAAAAAAAAAAAAAAAAAAANRILgAAAAAAnNwIzUejRCOuH5E6I8xnZz4puTs=1Zv7ttfk8LF81IUq16cHjhLTvJu4FA33AGWWjCpTnA"
# get guest token & parse json with jq
GT=$(curl -X 'POST' "https://api.twitter.com/1.1/guest/activate.json" -H "Authorization: Bearer $bearer" | jq -r .guest_token )
# Get tweet with hardcoded id of 20, & pretty print output with jq .
curl "https://api.twitter.com/1.1/statuses/lookup.json?include_entities=1&id=20" -H "X-Guest-Token: $GT" -H "Authorization: Bearer $bearer" | jq .With these headers, you can do most read-only operations. ie: the same experience as a logged out twitter user.
It's also very verbose and has lots of flags.
Here's an example using the TweetResultByRestId graphql endpoint.
// main.ts
export async function TweetResultByRestID(tweetId: string, headers: HeadersInit): Promise<any> {
let url = "https://api.twitter.com/graphql/ncDeACNGIApPMaqGVuF_rw/TweetResultByRestId"
const variables = {
tweetId: tweetId,
includePromotedContent: true,
withBirdwatchNotes: true,
withCommunity: true,
withDownvotePerspective: true,
withReactionsMetadata: true,
withReactionsPerspective: true,
withSuperFollowsTweetFields: true,
withSuperFollowsUserFields: true,
withVoice: true,
}
const features = {
freedom_of_speech_not_reach_fetch_enabled: true,
graphql_is_translatable_rweb_tweet_is_translatable_enabled: true,
interactive_text_enabled: true,
longform_notetweets_consumption_enabled: true,
longform_notetweets_richtext_consumption_enabled: true,
responsive_web_edit_tweet_api_enabled: true,
responsive_web_enhance_cards_enabled: true,
responsive_web_graphql_exclude_directive_enabled: true,
responsive_web_graphql_skip_user_profile_image_extensions_enabled: true,
responsive_web_graphql_timeline_navigation_enabled: true,
responsive_web_text_conversations_enabled: true,
responsive_web_twitter_blue_verified_badge_is_enabled: true,
standardized_nudges_misinfo: true,
tweet_awards_web_tipping_enabled: true,
tweet_with_visibility_results_prefer_gql_limited_actions_policy_enabled: true,
tweetypie_unmention_optimization_enabled: true,
verified_phone_label_enabled: true,
vibe_api_enabled: true,
view_counts_everywhere_api_enabled: true,
}
url = `${url}?variables=${encodeURI(JSON.stringify(variables))}&features=${encodeURI(JSON.stringify(features))}`
const response = await fetch(url, { method: 'GET', headers: headers })
const json = await response.json();
return json
}
async function genHeaders() {
const url = "https://api.twitter.com/1.1/guest/activate.json";
const bearer =
`AAAAAAAAAAAAAAAAAAAAANRILgAAAAAAnNwIzUejRCOuH5E6I8xnZz4puTs=1Zv7ttfk8LF81IUq16cHjhLTvJu4FA33AGWWjCpTnA`;
const response = await fetch(url, {
method: "POST",
headers: {
"Authorization": `Bearer ${bearer}`
},
});
const json = await response.json();
return {
"Authorization": `Bearer ${bearer}`,
"X-Guest-Token": json.guest_token,
};
}
console.log(JSON.stringify(await TweetResultByRestID("20", await genHeaders()), null, " "))You can run it by copying the code to a .ts file then running it with deno (it will ask for permissons)
deno run main.tsI found a useful repo automatically documenting all the graphql endpoints at https://github.com/fa0311/TwitterInternalAPIDocument.
You can also use the genereated headers in some offical & documented REST endpoints that don't requre authentication eg: https://api.twitter.com/1.1/statuses/lookup.json?include_entities=1&id=20 which is
documented at https://developer.twitter.com/en/docs/twitter-api/v1/tweets/post-and-engage/api-reference/get-statuses-lookup
Links
- p1atdev/whisper - Deno http client
- zedeus/nitter - Twitter web client using the unofficial apis
- twintproject/twint - [archived] Python library to scrape twitter
- p1atdev/twitterql - [archived] deno twitter unofficial api client
- Tweet: Making a collage of 16k profile pictures using the the graphql twitter api
- Video: so how does twitter work? - Stream archive of George Hotz reversing the api
Thanks a lot!
Are you sure that token you shared isn't your personal token?
Yeah, I'm sure it's not anyone's personal token.
This blog post from 2020 includes it, the only difference is that the = is encoded to %3D (not sure why)
https://www.mufaddal.dev/scraping/2020/11/23/Scraping-Twitter.html
How?
The authorization : Bearer
seems to be invariant, incase if it ever changes, just inspect one of the API requests headers in your (incognito) browser to get it, make sure to include it in every request. The next important bit is the x-guest-token header, that’s what permits you to make any tweet fetching API requests. Acquire it by making a POST the /1.1/guest/activate.json endpoint, don’t forget to include the authorization header.
You can also find the exact string in some code search tools
Sourcegraph
- 251 results, including projects like
yt-dlp&nitter
Without encoding the = to %3D it's just 16 results
Github Search
- 500+ files
- 10 issues (oldest one from jan 2022)
Without encoding the = to %3D , it's 72 results
https://github.com/search?q=AAAAAAAAAAAAAAAAAAAAANRILgAAAAAAnNwIzUejRCOuH5E6I8xnZz4puTs%3D1Zv7ttfk8LF81IUq16cHjhLTvJu4FA33AGWWjCpTnA&type=code
The token with & without encoding compared
-AAAAAAAAAAAAAAAAAAAAANRILgAAAAAAnNwIzUejRCOuH5E6I8xnZz4puTs=1Zv7ttfk8LF81IUq16cHjhLTvJu4FA33AGWWjCpTnA
+AAAAAAAAAAAAAAAAAAAAANRILgAAAAAAnNwIzUejRCOuH5E6I8xnZz4puTs%3D1Zv7ttfk8LF81IUq16cHjhLTvJu4FA33AGWWjCpTnA
@kentcdodds Twitter's embed makes a request for the data it uses. You can see that data for any tweet under the network tab in https://publish.twitter.com/. What I basically did was get some tweets with different types of content and build all the types for it manually (https://github.com/vercel-labs/react-tweet/tree/main/packages/react-tweet/src/api/types).
As it's an unofficial API it can change at any moment and break something.