Bug in TwitterSearch when looking for more tweets

[Rob Poulter]

Line 115 of TwitterSearch:

self.__nextMaxID = min(self.__response['content']['statuses'], key=lambda i: i['id'])['id'] - 1

Since I've only just started looking at this I'm essentially following your getting started guide in the README for a keyword search I'm looking at (single search term: 'ECAWA'), and this line is throwing a value error since the argument to min() is an empty sequence.

[Christian Koepp]

Thanks for opening this issue. I will have a look at it within the next days...

[Christian Koepp]

I not able to reproduce your report. Can you please post your code? Please do not forget to mask your credentials - happens to me frequently, that's why I mention it :)

If you're still experiencing this issue please also add the following code to yours and run it again:

try:
    ts = TwitterSearch( ... )
    (...)

except Exception as e:
    import pprint
    pprint.pprint(ts.getTweets())

...this will output quite a few lines containing all the response you've got from the Twitter Search API so I can figure out why min() with it's lambda function isn't working as it's supposed to do.

If you'll need a fix quickly you could switch to an older version of TwitterSearch (for example v0.78.1) as I changed the for-loop to the min-lambda-stuff just recently.

[Rob Poulter]

Here's the code I'm running (basically what you have in the sample with different language and search term).

from TwitterSearch import *

def report_error(error: list) -> None:
  source, e = error
  with open("error.txt","a") as err:
    err.write("Error: {} Exception: {}\n".format(source,e))

def get_tweets(keywords: list) -> list:
  tweets = []
  try:
    tso = TwitterSearchOrder()
    tso.setKeywords(keywords)
    tso.setLanguage('en')
    tso.setCount(1)
    tso.setIncludeEntities(False)

    ts = TwitterSearch(
        consumer_key = '-',
        consumer_secret = '-',
        access_token = '-',
        access_token_secret = '-'
      )
    ts.authenticate()
    for tweet in ts.searchTweetsIterable(tso):
      tweets.append(tweet)
  except TwitterSearchException as e:
    report_error(["TwitterSearchException",e])

  return tweets

keywords = ["ECAWA"]
t = get_tweets(keywords)
print(len(t))

With the exception handling wrapped around the search, this is the only bit that got printed out:

{'search_metadata': {'completed_in': 0.004,
                     'count': 1,
                     'max_id': 367582437343113215,
                     'max_id_str': '367582437343113215',
                     'query': 'ECAWA',
                     'refresh_url': '?since_id=367582437343113215&q=ECAWA&lang=en',
                     'since_id': 0,
                     'since_id_str': '0'},
 'statuses': []}

I hope that's helpful.

I'm avoiding the error for the time being by just checking for a value error around that min() call while I get a handle on what I want to do with my script (thanks for TwitterSearch btw, from that playing around I've done with the data I have it's pretty neat :)

[Christian Koepp]

Yep, this is what I needed - seems like there are no results and therefore min() isn't able to gather a minimal ID because of the empty list.

But it's quite odd that there is no result available as the Twitter Search API should return some - even the search_metadata list returned from Twitter thinks that there is a count of 1. I've never seen such a strange response so far ^^

Do you get some results, don't you? I assume this exception happens only at the very last tweet available, right? In this case a small check for len(self.__response['content']['statuses']) should be sufficient to get rid of this annoying bug.

I'll have a closer look at this when I'm at home again.

[Rob Poulter]

Yep, handling the exception gives me 9 valid tweets for that keyword search. It is only happening on the last one.