Giters

Miserlou / flask-zappa

Serverless Flask on AWS Lambda + API Gateway

Repository from Github https://github.comMiserlou/flask-zappaRepository from Github https://github.comMiserlou/flask-zappa

proposed usage

cariaso opened this issue · comments

commented

@Miserlou says

please try creating a README file of what the project's usage would look like, then we can start to build the product based on that.

I feel I'm not in a strong position to begin that. I do my current flask work by doing a git clone into a particular directory on a server, and then restarting the gunicorn server. I'm sure there are better ways, but I'm unfamiliar and that has served my needs well enough.

I'm aware of heroku as being slightly more like the 'run this command to deploy' interface you seem to anticipate, but from what I see heroku seems to work equally well for flask and django, so I'm not certain it's any better of a model.

Looking at django-zappa I see

$ python manage.py deploy production
Deploying..
Your application is now live at: https://7k6anj0k99.execute-api.us-east-1.amazonaws.com/production

$ python manage.py update production
Updating..
Your application is now live at: https://7k6anj0k99.execute-api.us-east-1.amazonaws.com/production

$ python manage.py invoke production check
System check identified no issues (0 silenced).

and I don't see anything wrong with that.

I know that the soon-to-be released flask 1.0 uses click to provide command line interfaces.
http://flask.pocoo.org/docs/dev/cli/
and that probably is the way to achieve the 'production check' behavior you suggest above. I am using click for some of my flask work and it's simple and effective.

So unless someone can enlighten me, sticking with the same django-flask interface as above seems like a worthwhile goal.

commented

I know that the soon-to-be released flask 1.0 uses click to provide command line interfaces.

This is really the main issue. Certainly, this would be much nicer to use for end users, but, there are two issues here:

  • Does it make sense to target against a version that doesn't exist yet?
  • When will 1.0 officially be released?
  • Does this mean that Zappa will never be usable for old Flask installs (which means, essentially every Flask project ever written?)
commented
commented

I just mentioned this project on irc efnet #pocoo and was pointed to
https://github.com/apex/apex
https://medium.com/@tjholowaychuk/introducing-apex-800824ffaa70#.z8r4d4l2n
which to my eye is far enough away to not be an issue, but I'll let other decide for themselves

commented

Apex really has nothing to do with Zappa.. To me, it seems like Apex is just another tool like Kappa - just a CLI utility for managing Lambda functions. Zappa replaces the need for such a tool entirely, but that is not its primary selling point

Something I have mentioned before that people don't really seem to understand is that the bulk of Zappa is the work dealing with API Gateway, not AWS Lambda. Lambda is fairly straightforward relative to API GW, and Lambda alone is not enough to power a web application.

commented

From the mailing list:

This has already been discussed extensively on GitHub. In summary:

The last update from mitsuhiko on that issue is from the middle of January stating that he is working on creating an org to better manage development and releases. I can confirm this is in progress.

Target 0.10.1 if you need to target a released version, it is stable. Most significant changes are in Werkzeug and Jinja, which saw releases relatively recently.

Write tests, and run them against 0.10.1 and 1.0-dev. I don't recall there being breaking changes, although I'm not sure. If you have failures, please report them so we can either fix or document them.

Click integration can be added to 0.10 with Flask-Cli. Flask-Script is also fine for most use cases.

Finally, if Flask is a major part of your product and you can't consider 0.10 stable, you can help make Flask 1.0 a reality. Contributions, whether bug reports, pull requests, reviews, or discussion, are always welcome and helpful.

commented

So, I guess the right thing to do is to target 0.10.1 with Flask-Cli as a dependency.

commented

We currently use flask-cli to get shell support in our 0.10 projects so that sounds great to me 👍

commented

I just pushed an attempt at it on the branch basic_example. It is, as the name suggest, pretty basic and somewhat hackish, but the flask-app can be reached.

Usage is:

python manage.py deploy production test_settings.json

and

python manage.py update production test_settings.json

where test_settings.json
is a config file with e.g.:

{
    "production": {
       "s3_bucket": "flask-test",
       "settings_file": "production_settings.py",
       "project_name": "MyTestProject2"
    },
    "staging": {
       "s3_bucket": "staging-bucket",
       "settings_file": "staging_settings.py",
       "project_name": "MyTestProject3"
    }
}

and production_settings.py:

APP_MODULE="test_app"
APP_OBJECT="app"

I used a small flask-app in test_app.py:

# -*- coding: utf-8 -*-
import flask


app = flask.Flask(__name__)


@app.route('/')
def index():
    return "Hello, world!", 200

@app.route('/myresource/<resource_id>')
def myresource(resource_id):
    return "This is myresource: %s. Trailing slashes are disabled on all resources." % resource_id, 200

@app.route('/myresource/<resource_id>/')
def myresource2(resource_id):
    return "This is myresource2: %s. Trailing slashes are enabled on all resources." % resource_id, 200

Let me know what you think.

commented

Ah, excellent!

Is there a URL that I can see it working at?

For test applications, I found that having a POST echo, a cookie-setting page, a 301 redirect, a 404, and an application/json content-type page were all useful, as they all introduce API-Gateway related quirks.

commented

Right now it's up here and here.

Good check-list!

I wanted to throw something out there as soon as possible so others could chime in!

commented

Awesome! Feels snappy too. This seems like a good starting point.

commented

I've tried to reproduce this. With only the trivial change to the s3_bucket names, it runs with this as output

python manage.py deploy production test_settings.json
deploying to production
{u'production': {u's3_bucket': u'mc-flask-test', u'project_name': u'MyTestProject2', u'settings_file': u'production_settings.py'}, u'staging': {u's3_bucket': u'mc-staging-bucket', u'project_name': u'MyTestProject3', u'settings_file': u'staging_settings.py'}}
current_file /Users/cariaso/dev/zappastuff/flask-zappa
Packaging project as zip..

WARNING!

You do not have ZappaMiddleware in your remote settings's MIDDLEWARE_CLASSES.

This means that some aspects of your application may not work!

MyTestProject2-production-1455799532.zip
Uploading zip (3.5MiB)..
Creating API Gateway routes..
Deploying API Gateway..
Your Zappa deployment is live!: https://xac8qo35w0.execute-api.us-east-1.amazonaws.com/production

That endpoint is listening, but generates no output. Presumably this relates to the warning message
"You do not have ZappaMiddleware in your remote settings's MIDDLEWARE_CLASSES."
can you suggest what I might be missing?

commented

@cariaso - this is what's going on with the middleware - Miserlou/django-zappa#9 - we should make the Django Middleware into a plain WSGI Middleware that we use with ALL of the projects. Then we can remove the warning entirely, projects won't have to add anything as the *-zappa handler will use the correct middleware automatically.

I don't know if @codingjoe is going to work on this or not. Either way, I don't think it'll be too hard to implement..

commented

@cariaso It works without middleware for me. (Simple stuff anyway).

Could I convince you to check your cloudwatch logs for me? Did you install all needed packages into your virtual env?

commented

After seeing that you'd already provided test_settings.json and production_settings.py I assumed that test_app.py was in place without checking. The cloudwatch logs make it clear that wasn't the case.

No module named test_app: ImportError Traceback (most recent call last): File "/var/task/handler.py", line 22, in lambda_handler app_module = importlib.import_module(settings.APP_MODULE) File "/usr/lib64/python2.7/importlib/init.py", line 37, in import_module import(name) ImportError: No module named test_app

after dropping in the test_app.py I have similar symptoms, but the log message is

create_wsgi_request() got an unexpected keyword argument 'trailing_slash': TypeError Traceback (most recent call last): File "/var/task/handler.py", line 55, in lambda_handler trailing_slash=False) TypeError: create_wsgi_request() got an unexpected keyword argument 'trailing_slash'

which seems more likely to be an issue with some unpushed code?

commented

Yes, you need zappa==0.10.4, which can be installed through the git repo.

Try:

pip install git+https://github.com/Miserlou/Zappa.git
commented

I was in a fresh virtualenv, and the requirements.txt indicates 0.10.4 . When I try to re-run the install I got this message, confirming 0.10.4 was in place:

Requirement already satisfied (use --upgrade to upgrade): zappa==0.10.4 in /Users/cariaso/dev/zappastuff/anenv27/lib/python2.7/site-packages (from -r requirements.txt (line 4))

after running
pip install git+https://github.com/Miserlou/Zappa.git
I get the same result.

commented

Perhaps it's because the version number haven't been bumped. Try to force a reinstall with pip install --upgrade --force-reinstall git+https://github.com/Miserlou/Zappa.git

commented

@Miserlou what's the version-bumping policy? Do you want to do stuff like that, or would it be ok for the rest of us to do it?

commented

I'd like to stay on top of it until we have significant (>90%) test coverage, which is going to require tying Travis to a locked-down AWS IAM user on my account. Right now we're only testing the non-AWS stuff, which is pretty insignificant. This is a big TODO, and will hopefully come in the next week. But once we have good test coverage we can figure out a process for doing group semver publishing.

In the meantime, just let me know when you want to version bump and I'll look more into automated testing of AWS stuff.

commented

It's probably possible to mock out boto calls for unit testing and not need to have integration tests on live AWS creds.

https://github.com/spulec/moto looks promising

commented

@Moto looks awesome, but unfortunately doesn't provide APIGW support and probably never will, and that's the thing that needs the most testing.

I think I can make a locked-down IAM user and then use encrypted credentials on Travis to make sure that everything works end to end.

commented

pip install --upgrade --force-reinstall git+https://github.com/Miserlou/Zappa.git
solved it. See for yourself at
https://dh9b8jz2qd.execute-api.us-east-1.amazonaws.com/production

commented

Awesome! Hello, world!

commented

Awesome! I'm pushing a small update later which will give some of the checklist scenarios.

On 19/02/2016, at 11.11, Rich Jones notifications@github.com wrote:

Awesome! Hello, world!

—
Reply to this email directly or view it on GitHub.