Reload doesn't work
MarSoft opened this issue · comments
I have enabled code reloading (with either DEBUG=True
or --reload
option). Now whenever I change any code, worker is getting reloaded, but it is serving old code.
I encountered such behaviour before with Flask when Gunicorn was start with --preload
option which is incompatible with --reload
.
Investigated somewhat further. Seems that preloading is not enabled, but Muffin application is imported during startup (by Manager.__init__
). According to Gunicorn docs, it seems that this may interfere app reloading:
The reloader is incompatible with application preloading. When using a paste configuration be sure that the server block does not import any application code or the reload will not work as designed. - https://gunicorn-docs.readthedocs.org/en/19.1/settings.html#debugging
For now I have no idea on how to eliminate this, or how to actually confirm this.
Today I've started a little project with Muffin and thought it will be ok to have the feature works. So I made some changes to make it alive. Thank you for the feedback.
Seems that this fix doesn't fix reload (at least for me), and even worse - it breaks setups where application is not within its own package.
Let's consider this setup:
project/
project/app.py
project/views.py
app.py:
from muffin import Application
app = Application('test')
import views
views.py:
from app import app
app.register('/')
def index(req):
return 'Hello'
Now when I go to directory project
and do muffin app run
, app is started but without its views, returning 404
for any url. This happens because your code unloads app
module and reloads it again, but views
module is not reloaded because it is already in sys.modules
!
This can be "fixed" by adding a dummy module test.py
with the only line from app import app
, but such "fix" just breaks reloading completely.
AFAIR, the setup you were testing on was like this:
project/
project/__init__.py
project/views.py
init.py:
from muffin import Application
app = Application('test')
from . import views
and you call it from the parent directory as muffin project run
, or probably install your application as a module.
With such setup your approach works almost fine, but there is still a problem. Reload event happens if any file used by application is changed, including e.g. gunicorn
's modules or muffin
's ones; sometimes I want to temporarily change some file not belonging to an app but lying in my virtualenv, like add a print statement somewhere. After such change, worker is being reloaded which I can see in console, but modules stay obsolete which is very annoying.
So, to summarize, there are 2 problems with current approach:
- It breaks setups where the whole app code is not contained within a single module (probably with submodules);
- It reloads worker for each
.py
file change (it takes list of all.py
files fromsys.modules
, seegunicorn.reloader.Reloader.get_files()
method), but only application's submodules are actually updated.
You can take a look at my attempt of fixing reload issue: https://github.com/MarSoft/muffin/compare/develop...fix-reload?expand=1
I had to rework manage.py
module to handle run
action specially: for that action it doesn't load application, and as a result doesn't use application's config which is not good, but app's module is not loaded and reload can work fine.
Alternative method (which only fixes problem 1 but not 2) is to unload not only app's module with submodules, but all modules with __file__
being in the same directory as app's module, along with their submodules. Here is how it can look:
def load(self):
"""Load a Muffin application."""
# Fix paths
os.chdir(self.cfg.chdir)
sys.path.insert(0, self.cfg.chdir)
app = self.app_uri
if not isinstance(app, Application):
module, *_ = self.app_uri.split(':', 1)
if module in sys.modules:
moddir = os.path.dirname(sys.modules[module].__file__)
appmods = [
mod for mod in sys.modules
if os.path.dirname(getattr(sys.modules[mod], '__file__', '')) == moddir
]
for module in appmods:
print('pop:', module)
sys.modules.pop(module)
paths = [p for p in sys.modules if p.startswith('%s.' % module)]
for path in paths:
print('subpop:', path)
sys.modules.pop(path)
app = import_app(app)
return app
Doesn't actual for version > 0.45