azioca / django-celery-sqs

An example Django App with Celery using Amazon SQS as the broker

Celery-SQS

A sample Django App that shows how to use Celery with Amazon SQS as the Broker.

Quick how to

• clone this repo
• create a virtualenv & install requirements
• NOTE python 3 is required

$> mkvirtualenv -p `which python3` celery-sqs
$> pip install -r requirements.txt

• Add celerysqs/secret.py with the following template
  • NOTE: the access/secret keys must have both IAM Policies listed below attached

KEY = 'some secret key'  # for Django

SQS = {
    'access_key': 'your aws_access_key',
    'secret_key': 'your aws_secret_key',
}

• run the server

  $> DJANGO_SETTINGS_MODULE=celerysqs.conf.aws python manage.py runserver

• run the celery worker

  $> DJANGO_SETTINGS_MODULE=celerysqs.conf.aws celery worker -A celerysqs --concurrency=1 -l info

• queue some tasks

  $> ./scripts/curls.sh

IAM Policy Requirements

NOTE: Both are required

ListQueues (*)

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "Stmt1234567890000",
            "Effect": "Allow",
            "Action": [
                "sqs:ListQueues"
            ],
            "Resource": [
                "*"
            ]
        }
    ]
}

CRUD (prefix-)

NOTE: replace {region} and {prefix} with your amazon region and desired prefix

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "Stmt1234567890000",
            "Effect": "Allow",
            "Action": [
                "sqs:ChangeMessageVisibility",
                "sqs:CreateQueue",
                "sqs:DeleteMessage",
                "sqs:DeleteQueue",
                "sqs:GetQueueAttributes",
                "sqs:GetQueueUrl",
                "sqs:ReceiveMessage",
                "sqs:SendMessage",
                "sqs:SetQueueAttributes"
            ],
            "Resource": [
                "arn:aws:sqs:{region}:*:{prefix}-*"
            ]
        }
    ]
}