torchbox / wagtail-storages

Use AWS S3 with private documents in Wagtail

wagtail-storages

This package fills the missing gap in using AWS S3 together with Wagtail. This package will be useful if you want to:

• Use AWS S3 bucket for hosting Wagtail documents.
• Put the bucket behind the CDN so that the bucket is not called directly each time.
• Allow editors to use privacy controls on documents, whilst using CDN.

• Avoid time-outs because of downloads being proxied through Wagtail views.

  Note: you cannot use the document redirect view if you want your documents to be truly private.

What does it do?

The package is a collection of signal handlers and Wagtail hooks.

• Sets per-object ACLs on S3 whenever privacy settings change on a Wagtail document.
• Replaces the current document view with a redirect. Either to a signed S3 bucket URL for private documents or public custom domain URL for public ones.
• Purges CDN for documents that have changed.

Requirements

• django-storages with the S3Boto3Storage storage backend configured in a Wagtail project.
• CDN supported by Wagtail front-end cache invalidator.

Management commands

django-admin fix_document_acls

The package provider a management command that sets all the documents' ACLs according to the their collection permissions. This must be called if there had been documents in a bucket before the package was used to make sure the ACLs in the bucket are correct.

Settings

WAGTAIL_STORAGES_DOCUMENTS_FRONTENDCACHE

Using the same format as Wagtail's WAGTAILFRONTENDCACHE setting, but to be only used by the wagtail-storages to purge the documents. If not set, the purge won't happen. Read more on how to format it in the Wagtail docs, e.g.

WAGTAIL_STORAGES_DOCUMENTS_FRONTENDCACHE = {
    'cloudfront': {
        'BACKEND': 'wagtail.contrib.frontend_cache.backends.CloudfrontBackend',
        'DISTRIBUTION_ID': 'your-distribution-id',
     },
}

WAGTAIL_STORAGES_DOCUMENT_HOOK_ORDER

Set a custom order for the document hook order. It's set to 100 by default. It's important that it runs after any of your hooks since it returns a response, e.g.

WAGTAIL_STORAGES_DOCUMENT_HOOK_ORDER = 900

Recommended S3 setup with Wagtail

The following guide explains the recommended setup for using S3 with Wagtail. This guide assumes that:

• You serve your main website at llamasavers.com (replace llamasavers.com with your actual domain name).
• Your S3 bucket is called media.llamasavers.com and you host it from that domain name.
• You are using CDN on that domain, this guide will assume Cloudflare.

Set up S3 bucket

First, set up your S3 bucket. It must be configured to:

• Have a name that matches the domain name, e.g. media.llamasavers.com.

• Allow the user to perform the following actions on the bucket:

  • s3:ListBucket
  • s3:GetBucketLocation
  • s3:ListBucketMultipartUploads
  • s3:ListBucketVersions

• Allow the user to perform all the actions (s3:*) on the objects within the bucket.
• Allow the internet traffic to access Wagtail image renditions (images/*).

• Allow use of public ACLs, by disabling:

  • "Block public access to buckets and objects granted through new access control lists (ACLs)"
  • "Block public access to buckets and objects granted through any access control lists (ACLs)"

The user permissions can be set in the IAM or via a bucket policy. See example of all of those points being achieved in the bucket policy below.

{
     "Version": "2012-10-17",
     "Statement": [
         {
             "Sid": "PublicGetObject",
             "Effect": "Allow",
             "Principal": "*",
             "Action": "s3:GetObject",
             "Resource": "arn:aws:s3:::[BUCKET NAME]/images/*"
         },
         {
             "Sid": "AllowUserManageBucket",
             "Effect": "Allow",
             "Principal": {
                 "AWS": "arn:aws:iam::[USER ARN]"
             },
             "Action": [
                 "s3:ListBucket",
                 "s3:GetBucketLocation",
                 "s3:ListBucketMultipartUploads",
                 "s3:ListBucketVersions"
             ],
             "Resource": "arn:aws:s3:::[BUCKET NAME]"
         },
         {
             "Sid": "AllowUserManageBucketObjects",
             "Effect": "Allow",
             "Principal": {
                 "AWS": "arn:aws:iam::[USER ARN]"
             },
             "Action": "s3:*",
             "Resource": "arn:aws:s3:::[BUCKET NAME]/*"
         }
     ]
 }

After the S3 bucket is set up on AWS, you can configure the Wagtail project to use it.

Set up django-storages

Install django-storages and boto3.

pip install django-storages[boto3]

Set up your S3 bucket with django-storages. The following code allows configuration via environment variables.

# settings.py
import os


if "AWS_STORAGE_BUCKET_NAME" in os.environ:
    # Add django-storages to the installed apps
    INSTALLED_APPS = INSTALLED_APPS + ["storages"]

    # https://docs.djangoproject.com/en/stable/ref/settings/#default-file-storage
    DEFAULT_FILE_STORAGE = "storages.backends.s3boto3.S3Boto3Storage"

    AWS_STORAGE_BUCKET_NAME = os.environ["AWS_STORAGE_BUCKET_NAME"]

    # Disables signing of the S3 objects' URLs. When set to True it
    # will append authorization querystring to each URL.
    AWS_QUERYSTRING_AUTH = False

    # Do not allow overriding files on S3 as per Wagtail docs recommendation:
    # https://docs.wagtail.io/en/stable/advanced_topics/deploying.html#cloud-storage
    # Not having this setting may have consequences such as losing files.
    AWS_S3_FILE_OVERWRITE = False

    # Default ACL for new files should be "private" - not accessible to the
    # public. Images should be made available to public via the bucket policy,
    # where the documents should use wagtail-storages.
    AWS_DEFAULT_ACL = "private"

    # We generally use this setting in production to put the S3 bucket
    # behind a CDN using a custom domain, e.g. media.llamasavers.com.
    # https://django-storages.readthedocs.io/en/latest/backends/amazon-S3.html#cloudfront
    if "AWS_S3_CUSTOM_DOMAIN" in os.environ:
        AWS_S3_CUSTOM_DOMAIN = os.environ["AWS_S3_CUSTOM_DOMAIN"]

    # When signing URLs is enabled, the region must be set.
    # The global S3 endpoint does not seem to support signed URLS.
    # Set this only if you will be using signed URLs.
    if "AWS_S3_REGION_NAME" in os.environ:
        AWS_S3_REGION_NAME = os.environ["AWS_S3_REGION_NAME"]

    # This settings lets you force using http or https protocol when generating
    # the URLs to the files. Set https as default.
    # https://github.com/jschneier/django-storages/blob/10d1929de5e0318dbd63d715db4bebc9a42257b5/storages/backends/s3boto3.py#L217
    AWS_S3_URL_PROTOCOL = os.environ.get("AWS_S3_URL_PROTOCOL", "https:")

If you use the above snippet, you can set the following environment variables:

• AWS_STORAGE_BUCKET_NAME - set to media.llamasavers.com.
• AWS_S3_CUSTOM_DOMAIN - set to media.llamasavers.com.
• AWS_S3_REGION_NAME - set to your AWS region name, e.g. eu-west-2.

You can use one of the methods to provide boto3 with credentials. We suggest you stick with the environment variables. To do that, you need to set the following variables:

• AWS_ACCESS_KEY_ID
• AWS_SECRET_ACCESS_KEY

Now the storage should be configured and working. Editors should be able to upload images and documents in Wagtail admin.

Set up wagtail-storages

Install wagtail-storages itself.

pip install wagtail-storages

Add wagtail_storages to your INSTALLED_APPS in your settings file.

# settings.py

INSTALLED_APPS = [
    # ... Other apps
    "wagtail_storages.apps.WagtailStoragesConfig",
    # ... Other apps
]

With that, ACLs should be updated if documents are moved to private collections.

If you already have files in your S3 bucket, run django-admin fix_document_acls to make sure all documents have the right ACLs set up.

Set up front-end cache invalidation

If edge cache is set up on the custom domain (media.llamasavers.com) you should set up the CDN purging to avoid having outdated or private documents available to users via the CDN endpoint. For example, for Cloudflare you want to use a configuration similar to the one below:

# settings.py
import os


if "S3_CACHE_CLOUDFLARE_TOKEN" in os.environ:
     WAGTAIL_STORAGES_DOCUMENTS_FRONTENDCACHE = {
         "default": {
             "BACKEND": "wagtail.contrib.frontend_cache.backends.CloudflareBackend",
             "EMAIL": os.environ["S3_CACHE_CLOUDFLARE_EMAIL"],
             "TOKEN": os.environ["S3_CACHE_CLOUDFLARE_TOKEN"],
             "ZONEID": os.environ["S3_CACHE_CLOUDFLARE_ZONEID"],
         },
     }

Then set the following environment variables:

• S3_CACHE_CLOUDFLARE_EMAIL
• S3_CACHE_CLOUDFLARE_TOKEN
• S3_CACHE_CLOUDFLARE_ZONEID

Once set up, the documents will be purged from cache when they are modified or their privacy settings have changed.

The setting follows configuration format of the front-end cache invalidator configuration in Wagtail. See the details here. The only difference is the setting name, which for wagtail-storages is WAGTAIL_STORAGES_DOCUMENTS_FRONTENDCACHE.

All done!