Reload doesn't work

[Семён Марьясин]

I have enabled code reloading (with either DEBUG=True or --reload option). Now whenever I change any code, worker is getting reloaded, but it is serving old code.
I encountered such behaviour before with Flask when Gunicorn was start with --preload option which is incompatible with --reload.

[Семён Марьясин]

Investigated somewhat further. Seems that preloading is not enabled, but Muffin application is imported during startup (by Manager.__init__). According to Gunicorn docs, it seems that this may interfere app reloading:

The reloader is incompatible with application preloading. When using a paste configuration be sure that the server block does not import any application code or the reload will not work as designed. - https://gunicorn-docs.readthedocs.org/en/19.1/settings.html#debugging

For now I have no idea on how to eliminate this, or how to actually confirm this.

[Kirill Klenov]

Today I've started a little project with Muffin and thought it will be ok to have the feature works. So I made some changes to make it alive. Thank you for the feedback.

[Семён Марьясин]

Seems that this fix doesn't fix reload (at least for me), and even worse - it breaks setups where application is not within its own package.
Let's consider this setup:

project/
project/app.py
project/views.py

app.py:

from muffin import Application
app = Application('test')
import views

views.py:

from app import app
app.register('/')
def index(req):
    return 'Hello'

Now when I go to directory project and do muffin app run, app is started but without its views, returning 404 for any url. This happens because your code unloads app module and reloads it again, but views module is not reloaded because it is already in sys.modules!
This can be "fixed" by adding a dummy module test.py with the only line from app import app, but such "fix" just breaks reloading completely.
AFAIR, the setup you were testing on was like this:

project/
project/__init__.py
project/views.py

init.py:

from muffin import Application
app = Application('test')
from . import views

and you call it from the parent directory as muffin project run, or probably install your application as a module.

With such setup your approach works almost fine, but there is still a problem. Reload event happens if any file used by application is changed, including e.g. gunicorn's modules or muffin's ones; sometimes I want to temporarily change some file not belonging to an app but lying in my virtualenv, like add a print statement somewhere. After such change, worker is being reloaded which I can see in console, but modules stay obsolete which is very annoying.

[Семён Марьясин]

So, to summarize, there are 2 problems with current approach:

1. It breaks setups where the whole app code is not contained within a single module (probably with submodules);
2. It reloads worker for each .py file change (it takes list of all .py files from sys.modules, see gunicorn.reloader.Reloader.get_files() method), but only application's submodules are actually updated.

You can take a look at my attempt of fixing reload issue: https://github.com/MarSoft/muffin/compare/develop...fix-reload?expand=1
I had to rework manage.py module to handle run action specially: for that action it doesn't load application, and as a result doesn't use application's config which is not good, but app's module is not loaded and reload can work fine.

[Семён Марьясин]

Alternative method (which only fixes problem 1 but not 2) is to unload not only app's module with submodules, but all modules with __file__ being in the same directory as app's module, along with their submodules. Here is how it can look:

    def load(self):
        """Load a Muffin application."""
        # Fix paths
        os.chdir(self.cfg.chdir)
        sys.path.insert(0, self.cfg.chdir)

        app = self.app_uri
        if not isinstance(app, Application):
            module, *_ = self.app_uri.split(':', 1)
            if module in sys.modules:
                moddir = os.path.dirname(sys.modules[module].__file__)
                appmods = [
                    mod for mod in sys.modules
                    if os.path.dirname(getattr(sys.modules[mod], '__file__', '')) == moddir
                ]
                for module in appmods:
                    print('pop:', module)
                    sys.modules.pop(module)
                    paths = [p for p in sys.modules if p.startswith('%s.' % module)]
                    for path in paths:
                        print('subpop:', path)
                        sys.modules.pop(path)
            app = import_app(app)

        return app

[Семён Марьясин]

Please see https://github.com/MarSoft/muffin/tree/fix-reload-2

[Kirill Klenov]

Doesn't actual for version > 0.45