Flexget / Flexget

Expected behaviour:

Pass the popcorn discover search searches correctly.

Actual behaviour:

AttributeError: 'NoneType' object has no attribute 'json' and a crash. Log below.

Steps to reproduce:

Have a bad passthepopcorn login cookie, but what kind of bad exactly I do not know.
Try to do a discover with the passthepopcorn search plugin.

In my case:

Have a task which does a discover from the passthepopcorn search plugin, configured correctly, working fine.
PTP disappeared for months, and then came back.
I changed no configuration but I was now getting crashes, I think due to a bad login.

Config:

discover:
  from:
    - passthepopcorn:
        grouping: true
        order_by: Seeders
        order_desc: true
        passkey: ****
        password: '****'
        username: ****
  interval: 12 hour
  release_estimations:
    optimistic: 30 days
  what:
    - movie_list: ****

Log:

(click to expand)

2023-10-01 21:33:25 VERBOSE  discover      get_ptp_search  Searching for `****` with plugin `passthepopcorn` (1 of 12)
2023-10-01 21:33:26 DEBUG    passthepopcorn get_ptp_search  Using search params: {'order_by': 'seeders', 'order_way': 'desc', 'action': 'advanced', 'json': 'noredirect', 'grouping': 1, 'searchstr': '****'}
2023-10-01 21:33:26 DEBUG    passthepopcorn get_ptp_search  Found valid login cookie
2023-10-01 21:33:26 DEBUG    utils.requests get_ptp_search  GETing URL https://passthepopcorn.me/torrents.php with args () and kwargs {'params': {'order_by': 'seeders', 'order_way': 'desc', 'action': 'advanced', 'json': 'noredirect', 'grouping': 1, 'searchstr': '****'}, 'cookies': {'PHPSESSID': '****', 'session': '****'}, 'allow_redirects': True, 'timeout': 30}
2023-10-01 21:33:27 ERROR    passthepopcorn get_ptp_search  PassThePopcorn request failed: 400 Client Error: Bad Request for url: https://passthepopcorn.me/login.php?return_to=%2Ftorrents.php%3Forder_by%3Dseeders%26order_way%3Ddesc%26action%3Dadvanced%26json%3Dnoredirect%26grouping%3D1%26searchstr%3D****
2023-10-01 21:33:27 CRITICAL task          get_ptp_search  BUG: Unhandled error in plugin discover: 'NoneType' object has no attribute 'json'
Traceback (most recent call last):

  File "/usr/lib/python3.10/threading.py", line 973, in _bootstrap
    self._bootstrap_inner()
    │    └ <function Thread._bootstrap_inner at 0x7f0461149cf0>
    └ <Thread(task_queue, started daemon 139656673461824)>
  File "/usr/lib/python3.10/threading.py", line 1016, in _bootstrap_inner
    self.run()
    │    └ <function Thread.run at 0x7f0461149a20>
    └ <Thread(task_queue, started daemon 139656673461824)>
  File "/usr/lib/python3.10/threading.py", line 953, in run
    self._target(*self._args, **self._kwargs)
    │    │        │    │        │    └ {}
    │    │        │    │        └ <Thread(task_queue, started daemon 139656673461824)>
    │    │        │    └ ()
    │    │        └ <Thread(task_queue, started daemon 139656673461824)>
    │    └ <bound method TaskQueue.run of <flexget.task_queue.TaskQueue object at 0x7f045b3d94b0>>
    └ <Thread(task_queue, started daemon 139656673461824)>
  File "/usr/local/lib/python3.10/dist-packages/flexget/task_queue.py", line 46, in run
    self.current_task.execute()
    │    │            └ <function Task.execute at 0x7f045e8e6f80>
    │    └ <flexget.task.Task object at 0x7f045aef6740>
    └ <flexget.task_queue.TaskQueue object at 0x7f045b3d94b0>
  File "/usr/local/lib/python3.10/dist-packages/flexget/task.py", line 87, in wrapper
    return func(self, *args, **kw)
           │    │      │       └ {}
           │    │      └ ()
           │    └ <flexget.task.Task object at 0x7f045aef6740>
           └ <function Task.execute at 0x7f045e8e6ef0>
  File "/usr/local/lib/python3.10/dist-packages/flexget/task.py", line 722, in execute
    self._execute()
    │    └ <function Task._execute at 0x7f045e8e6e60>
    └ <flexget.task.Task object at 0x7f045aef6740>
  File "/usr/local/lib/python3.10/dist-packages/flexget/task.py", line 691, in _execute
    self.__run_task_phase(phase)
    │                     └ 'input'
    └ <flexget.task.Task object at 0x7f045aef6740>
  File "/usr/local/lib/python3.10/dist-packages/flexget/task.py", line 514, in __run_task_phase
    response = self.__run_plugin(plugin, phase, args)
               │                 │       │      └ (<flexget.task.Task object at 0x7f045aef6740>, {'what': [{'movie_list': 'trakt-film-priv'}], 'from': [{'passthepopcorn': {'us...
               │                 │       └ 'input'
               │                 └ <PluginInfo(name=discover)>
               └ <flexget.task.Task object at 0x7f045aef6740>
> File "/usr/local/lib/python3.10/dist-packages/flexget/task.py", line 547, in __run_plugin
    result = method(*args, **kwargs)
             │       │       └ {}
             │       └ (<flexget.task.Task object at 0x7f045aef6740>, {'what': [{'movie_list': 'trakt-film-priv'}], 'from': [{'passthepopcorn': {'us...
             └ <Event(name=plugin.discover.input,func=on_task_input,priority=128)>
  File "/usr/local/lib/python3.10/dist-packages/flexget/event.py", line 20, in __call__
    return self.func(*args, **kwargs)
           │    │     │       └ {}
           │    │     └ (<flexget.task.Task object at 0x7f045aef6740>, {'what': [{'movie_list': 'trakt-film-priv'}], 'from': [{'passthepopcorn': {'us...
           │    └ <bound method Discover.on_task_input of <flexget.plugins.input.discover.Discover object at 0x7f045cb65b10>>
           └ <Event(name=plugin.discover.input,func=on_task_input,priority=128)>
  File "/usr/local/lib/python3.10/dist-packages/flexget/plugins/input/discover.py", line 311, in on_task_input
    return self.execute_searches(config, entries, task)
           │    │                │       │        └ <flexget.task.Task object at 0x7f045aef6740>
           │    │                │       └ [<Entry(title=****,state=undecided)>, <Entry(title=****,state=undecid...
           │    │                └ {'what': [{'movie_list': 'trakt-film-priv'}], 'from': [{'passthepopcorn': {'username': '****', 'password': '****...
           │    └ <function Discover.execute_searches at 0x7f045e32b2e0>
           └ <flexget.plugins.input.discover.Discover object at 0x7f045cb65b10>
  File "/usr/local/lib/python3.10/dist-packages/flexget/plugins/input/discover.py", line 134, in execute_searches
    search_results = search.search(task=task, entry=entry, config=plugin_config)
                     │      │           │           │             └ {'username': '****', 'password': '****', 'passkey': '****', 'orde...
                     │      │           │           └ <Entry(title=****,state=undecided)>
                     │      │           └ <flexget.task.Task object at 0x7f045aef6740>
                     │      └ <function internet.__call__.<locals>.wrapped_func at 0x7f045b42e3b0>
                     └ <flexget.components.sites.sites.passthepopcorn.SearchPassThePopcorn object at 0x7f045b3bc4c0>
  File "/usr/local/lib/python3.10/dist-packages/flexget/plugin.py", line 131, in wrapped_func
    return func(*args, **kwargs)
           │     │       └ {'task': <flexget.task.Task object at 0x7f045aef6740>, 'entry': <Entry(title=****,state=un...
           │     └ (<flexget.components.sites.sites.passthepopcorn.SearchPassThePopcorn object at 0x7f045b3bc4c0>,)
           └ <function SearchPassThePopcorn.search at 0x7f045b42e320>
  File "/usr/local/lib/python3.10/dist-packages/flexget/components/sites/sites/passthepopcorn.py", line 279, in search
    ).json()

AttributeError: 'NoneType' object has no attribute 'json'
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/flexget/task.py", line 547, in __run_plugin
    result = method(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/flexget/event.py", line 20, in __call__
    return self.func(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/flexget/plugins/input/discover.py", line 311, in on_task_input
    return self.execute_searches(config, entries, task)
  File "/usr/local/lib/python3.10/dist-packages/flexget/plugins/input/discover.py", line 134, in execute_searches
    search_results = search.search(task=task, entry=entry, config=plugin_config)
  File "/usr/local/lib/python3.10/dist-packages/flexget/plugin.py", line 131, in wrapped_func
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/flexget/components/sites/sites/passthepopcorn.py", line 279, in search
    ).json()
AttributeError: 'NoneType' object has no attribute 'json'

Additional information:

FlexGet version: 3.9.10
Python version: 3.10.12
Installation method: pip
Using daemon (yes/no): yes
OS and version: Ubuntu 22.04.2
Link to crash log: it's above. I worked around the issue before getting a debug-level log.

From the log, this part seemed most relevant:

PassThePopcorn request failed: 400 Client Error: Bad Request for url: https://passthepopcorn.me/login.php?return_to=%2Ftorrents.php%3Forde...

That seems like a logging in issue to me. I double checked my config against my profile on PTP and everything was correct.
I read the source code to see what was happening, and I'm pretty sure what was happening is that PTP was returning a redirection to the login page, and the plugin is not handling this correctly. It's trying to decode an HTML page as JSON.
This probably meant my cookie was bad.
I looked at the source code more.

Flexget/flexget/components/sites/sites/passthepopcorn.py

Lines 152 to 162 in b73abba

    
           except RequestException as e: 
        
               if e.response is not None and e.response.status_code == 429: 
        
                   logger.error('Saved cookie is invalid and will be deleted. Error: {}', str(e)) 
        
                   # cookie is invalid and must be deleted 
        
                   with Session() as session: 
        
                       session.query(PassThePopcornCookie).filter( 
        
                           PassThePopcornCookie.username == username 
        
                       ).delete() 
        
                   invalid_cookie = True 
        
               else: 
        
                   logger.error('PassThePopcorn request failed: {}', str(e))

The error message looks like it came from line 162, which means a RequestException was caught on line 152 but the status code was not the 429 which it expects on line 153 which would trigger the cookie to be deleted.
So we just get an error message and the cookie is not deleted.

I couldn't figure out how to force the cookie to delete, I couldn't find any command line options to tell it to log in freshly to PTP. I didn't want to dig in the database. So I edited the source code temporarily to default force to True and restarted and ran it again. It logged in and searched correctly.
I changed it back and restarted again, and it's happy.

So I think the root of the issue here is that bad cookies are not always deleted when they should be.

If the status code can't be relied on to detect a bad cookie, maybe it would be better to detect a redirection to the login page?

This is possibly resolved by #3882

Indeed. Login is now a new method, I think we are safe to close this.

	except RequestException as e:
	if e.response is not None and e.response.status_code == 429:
	logger.error('Saved cookie is invalid and will be deleted. Error: {}', str(e))
	# cookie is invalid and must be deleted
	with Session() as session:
	session.query(PassThePopcornCookie).filter(
	PassThePopcornCookie.username == username
	).delete()
	invalid_cookie = True
	else:
	logger.error('PassThePopcorn request failed: {}', str(e))