Django-async-image-upload

Async image upload using presigned url for Django.

This module provide async processing capability using cloud architecture for Django server. Using this module, you can decouple code from Django api server and easily extend image processing capability.

There are already plenty of good thumbnail packages such as sorl-thumbnail, django-imagekit. However most of the packages process image on django server. Some are partially asynchrounous with celery and other technique, but server still needs to recieve files from client to proecess them.

Using S3(and others storages in the futures) presigned-url feature, client can directly uploads to S3 without involving backend server. Once image is uploaded, api server gets callback once image processing in finished elsewhere(serverless function in this case.

TODO

Current version only provide asynchronous upload with lambda function. There are a lot more to do with this module.

Unittest and github action
Multiple presigned url creation
Support for Django 2.X (Jsonfield)
Support for azure, gcp

In order to use this module you need to deploy cloud counter part. ** Current version only support AWS.

Architecture (AWS)

Client request for presigned url for image upload.
Server generate Presigned URL for upload.
Server returns presigned url credential.
Client uploads files in multipart/form with credential.
As soon as file is created on s3 bucket, lambda is triggered for image processing.
Lambda calls api server for callback for success or failure.

Refer to sequence diagram for more detail.

Requirements

Because this module supports native django json fields. (will support 2.X in the future)

Django 3.1 and up
Python 3.6 and up
DB: MariaDB 10.2.7+, MySQL 5.7.8+, Oracle, PostgreSQL, and SQLite

python packages

djangorestframework
drf-yasg
boto3

Instruction

Cloud module

You need to deploy cloud module(storage and serverless function) first. I provided SAM template to automate deployment. Refer to following page for detailed instruction.

Django API Module

Install Django-async-image-upload::

pip install django-async-image-upload

Add it to your INSTALLED_APPS::

INSTALLED_APPS = (

    'django_async_image_upload.apps.AsyncImageUploadAppConfig',

)

Add Django-async-image-upload's URL patterns::

from django_async_image_upload import urls as django_async_image_upload_urls

urlpatterns = [
    url(r'^', include(django_async_image_upload_urls)),
]

Settings

Available settings are:

ASYNC_IMAGE_UPLOAD_BUCKET

name of bucket where presigned url is generated
default: None

ASYNC_IMAGE_UPLOAD_TEMPORARY_IMAGE_PATH

folder path where image is uploaded using presigned url
default: "temp/"

ASYNC_IMAGE_UPLOAD_PRESIGNED_URL_EXPIRATION

Presigned url expiration in second
default: 1000

ASYNC_IMAGE_UPLOAD_PRESIGNED_URL_REUSE_WINDOW

Time in second where you can reuse already generated presigned url
default: 30

ASYNC_IMAGE_UPLOAD_PRESIGNED_URL_CONTENT_LENGTH

Max size of image in byte
default: 10485760 # 10MiB

PROCESSED_IMAGE_MODEL

Image model that will be created when success callback is issued.
default: "ProcessedImage"

AWS_ACCESS_KEY_ID and AWS_ACCESS_KEY_SECRET

Assuming you are already using Boto3, you need to set following AWS credentials

Create presigned url

POST to /async_image/presignedurl/ to generate presignedurl credential.

Swagger

For more information regarding api, you can access swagger on http:///async_image/docs/swagger?format=openapi

Upload image file to bucket

Client can use multipart/form upload to given url and fields in credential in with image file to upload to s3 bucket.

Order of the key matters, I recommend you to send in order of "key, acl, x-amz-meta-id... file".

How to check successful process from client

Because of the nature of the async operation, you won't be able to find out the result of the upload right after you upload to S3. You have following options.

Long polling - GET request to /async_image/presignedurl/<uuid> for completion.
Open connection with sever using technics such as websocket, graphql.
Assume operation is successful workaround it.

It's your choice. (There might be better solution out there. Please let me know if you have simpler solution.)

Extra note

If you are deploying on EC2, your EC2 should have appropriate IAM credential or EC2 role with s3:putOject permission on particular bucket.

Running Tests

TBD

Credits

Tools used in rendering this package:

kokospapa8 / django-async-image-upload