django-syzygy

Django application providing database migration tooling to automate their deployment.

Inspired by a 2015 post from Ludwig Hähne and experience dealing with migration at Zapier.

Installation

pip install django-syzygy

Usage

Add 'syzygy' to your INSTALLED_APPS

# settings.py
INSTALLED_APPS = [
    ...
    'syzygy',
    ...
]

Setup you deployment pipeline to run migrate --pre-deploy before rolling out your code changes and migrate afterwards to apply the postponed migrations.

Concept

Syzygy introduces a notion of prerequisite and postponed migrations with regards to deployment.

A migration is assumed to be a prerequisite to deployment unless it contains a destructive operation or the migration has its stage class attribute set to Stage.POST_DEPLOY. When this attribute is defined it will bypass operations based heuristics.

e.g. this migration would be considered a prerequisite

class Migration(migrations.Migration):
    operations = [
        AddField('model', 'field', models.IntegerField(null=True))
    ]

while the following migrations would be postponed

class Migration(migrations.Migration):
    operations = [
        RemoveField('model', 'field'),
    ]

from syzygy import Stage

class Migration(migrations.Migration):
    stage = Stage.POST_DEPLOY

    operations = [
        RunSQL(...),
    ]

In order to prevent the creation of migrations mixing operations of different stages this package registers system checks. These checks will generate an error for every migration with an ambiguous stage.

e.g. this migration would result in a check error

class Migration(migrations.Migration):
    operations = [
        AddField('model', 'other_field', models.IntegerField(null=True)),
        RemoveField('model', 'field'),
    ]

Migration revert are also supported and result in inverting the nature of migrations. A migration that is normally considered a prerequisite would then be postponed when reverted.

With this new notion of migration stage it's possible for the migrate command to target only migrations meant to be run before a deployment using the --pre-deploy flag or error out in the case on an ambiguous plan.

Third-party migrations

As long as the adoption of migration stages concept not generalized your project might depend on third-party apps containing migrations with an ambiguous sequence of operations.

Since an explicit stage cannot be explicitly assigned by editing these migrations a fallback or an override stage can be specified through the respective MIGRATION_STAGES_FALLBACK and MIGRATION_STAGES_OVERRIDE settings.

By default third-party app migrations with an ambiguous sequence of operations will fallback to Stage.PRE_DEPLOY but this behavior can be changed by setting MIGRATION_THIRD_PARTY_STAGES_FALLBACK to Stage.POST_DEPLOY or disabled by setting it to None.

Note

The third-party app detection logic relies on the site Python module and is known to not properly detect all kind of third-party Django applications. You should rely on MIGRATION_STAGES_FALLBACK and MIGRATION_STAGES_OVERRIDE to configure stages if it doesn't work for your setup.

Migration quorum

When deploying migrations to multiple clusters sharing the same database it's important that:

Migrations are applied only once
Pre-deployment migrations are applied before deployment in any clusters is takes place
Post-deployment migrations are only applied once all clusters are done deploying

The built-in migrate command doesn't offer any guarantees with regards to serializability of invocations, in other words naively calling migrate from multiple clusters before or after a deployment could cause some migrations to be attempted to be applied twice.

To circumvent this limitation Syzygy introduces a --quorum <N:int> flag to the migrate command that allow clusters coordination to take place.

When specified the migrate --quorum <N:int> command will wait for at least N number invocations of migrate for the planned migrations before proceeding with applying them once and blocking on all callers until the operation completes.

In order to use the --quorum feature you must configure the MIGRATION_QUORUM_BACKEND setting to point to a quorum backend such as cache based one provided by Sygyzy

MIGRATION_QUORUM_BACKEND = 'syzygy.quorum.backends.cache.CacheQuorum'

CACHES = {
    ...,
    'quorum': {
        ...
    },
}
MIGRATION_QUORUM_BACKEND = {
    'backend': 'syzygy.quorum.backends.cache.CacheQuorum',
    'alias': 'quorum',
}

Note

In order for CacheQuorum to work properly in a distributed environment it must be pointed at a backend that supports atomic incr operations such as Memcached or Redis.

Development

Make your changes, and then run tests via tox:

tox

ogdhruv / django-syzygy