Cannot get tweets
gridikono opened this issue · comments
Gridi Kono commented
Describe the bug
I am trying to extract tweets for a given profile, but I got the error below:
here is my code:
def scrape_tweets(self, username):
for tweet in sntwitter.TwitterSearchScraper(f'from:{username}').get_items():
dal = DAL()
dal.insert_tweet(tweet=tweet)
self.scrape_replies(tweet=tweet)
It worked until at the mid of April 2023. Below is my error that I got:
Error retrieving https://api.twitter.com/1.1/guest/activate.json: non-200 response (401)
return self._request('POST', *args, **kwargs)
File "C:\Users\Gridi\Desktop\nlp\osfa2022\Source code\ASSA.Scraper.Twitter\snscrape\base.py", line 247, in _request
raise ScraperException(msg)
snscrape.base.ScraperException: 4 requests to https://api.twitter.com/1.1/guest/activate.json failed, giving up.
I know that twitter has applied restriction in its public APIs, but there is any fix for it ?
How to reproduce
def scrape_tweets(self, username):
for tweet in sntwitter.TwitterSearchScraper(f'from:{username}').get_items():
dal = DAL()
dal.insert_tweet(tweet=tweet)
self.scrape_replies(tweet=tweet)
Expected behaviour
To get the tweets
Screenshots and recordings
No response
Operating system
Windows 10
Python version: output of python3 --version
3.9
snscrape version: output of snscrape --version
latest version installed from pip
Scraper
TwitterSearchScraper
How are you using snscrape?
Module (import snscrape.modules.something
in Python code)
Backtrace
No response
Log output
No response
Dump of locals
No response
Additional context
No response
Mazen Tayseer commented