Add check whether Terracotta is running in multithreaded context
kiksekage opened this issue · comments
I am seeing some weird behaviour when trying to use the RelationalMetaStore
class, specifically the db connection handling. Since the connection is created once and yield
ed down the call stack it seems we might run into some issues when trying to do the commit
or the rollback if any of the downstream calls errored out. We essentially end up with a lot of AttributeError: 'NoneType' object has no attribute 'rollback'/'commit'
, which seems weird to me since these are only set to None
in the finally
block in the top-most connect
call.
I tried adding a little if self.connected:
check before commiting or rollbacking which seems to reduce errors and smooth loading of the PNGs, but that seems a little icky since we are in a if not self.connected
loop.
Am I looking at this wrong? Has anyone tried using the RelationalMetaStore
implementation and seen/not seen this before? I tried thinking about a reproducible example but I am struggling to figure out how that could be done.
Do you have a full traceback?
I forgot to mention that I am experimenting with deploying TC on Azure Functions, and im seeing these errors when trying to deploy and debug locally. I didnt think to try and serve from TC itself before now, but I can confirm that terracotta serve -d <SAME DB>
works without hitches. Here is the traceback, even though it isnt so juicy:
Traceback (most recent call last):
File "/home/thns/bathy/bss-tc/.venv/lib/python3.8/site-packages/flask/app.py", line 2525, in wsgi_app
response = self.full_dispatch_request()
File "/home/thns/bathy/bss-tc/.venv/lib/python3.8/site-packages/flask/app.py", line 1822, in full_dispatch_request
rv = self.handle_user_exception(e)
File "/home/thns/bathy/bss-tc/.venv/lib/python3.8/site-packages/flask/app.py", line 1820, in full_dispatch_request
rv = self.dispatch_request()
File "/home/thns/bathy/bss-tc/.venv/lib/python3.8/site-packages/flask/app.py", line 1796, in dispatch_request
return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)
File "/home/thns/bathy/bss-tc/misc/terracotta/terracotta/server/metadata.py", line 54, in get_metadata
payload = metadata(parsed_keys)
File "/usr/lib/python3.8/contextlib.py", line 75, in inner
return func(*args, **kwds)
File "/home/thns/bathy/bss-tc/misc/terracotta/terracotta/handlers/metadata.py", line 18, in metadata
metadata = driver.get_metadata(keys)
File "/home/thns/bathy/bss-tc/misc/terracotta/terracotta/drivers/base_classes.py", line 43, in inner
return fun(self, *args, **kwargs)
File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__
next(self.gen)
File "/home/thns/bathy/bss-tc/misc/terracotta/terracotta/drivers/relational_meta_store.py", line 150, in connect
self.connection.commit()
AttributeError: 'NoneType' object has no attribute 'commit'
Is the Azure environment using multithreading under the hood? I have a hard time seeing how this could occur without race conditions.
Lo and behold, that was the exact issue. Here is the setting for managing the thread count for any future reference, thanks @dionhaefner 🙏
Glad it's solved. Let's be more defensive and add a check for this.