jamescooke / factory_audit

Testing different factory libraries with Django

Factory Audit

Checking how various object factories perform in Django with respect to creating valid instances out of the box.

See the full write-up (with pretty diagram).

Definitions

• Factory libraries

  The following factory libraries have been explored:

  • Django Fakery
  • Factory Boy
  • Factory Djoy
  • Hypothesis[django]
  • Mixer
  • Model Mommy

  Disclosure: Factory Djoy is my factory library. It's a thin wrapper around Factory Boy which does the hard work.

• Models

  Two factories have been created with each factory library:

  • ItemFactory: to create and save instances of plant.models.Item, a test model defined in the 'plant' app.
  • UserFactory: to create and save instances of the default django.contrib.auth User Model.

• Grading

  Each factory is graded based on how its default configuration behaves.

  The gradings are based on the definition of "valid". Valid instances are ones which will pass Django's full_clean and not raise a ValidationError. For example, using the ItemFactory a generated item passes validation with:

  item = ItemFactory()
  item.full_clean()

  The gradings are:

  • 🔴 RED - Factory creates invalid instances of the model and saves them to the database.
  • 💛 YELLOW - Factory raises an exception and does not save any instances. Preferably this would be a ValidationError, but I've also allowed IntegrityError here.
  • 💚 GREEN - Factory creates multiple valid instances with no invalid instances created or skipped. Running factory n times generates n valid instances.

Results

Library	ItemFactory	UserFactory
Django Fakery	🔴 RED	💛 YELLOW
Factory Boy	🔴 RED	🔴 RED
Factory Djoy	💛 YELLOW	💚 GREEN
Hypothesis[django]	🔴 RED	🔴 RED
Mixer	💚 GREEN	💚 GREEN
Model Mommy	💛 YELLOW	💚 GREEN

For more detailed notes about each factory including some of the grey areas around grading please see Notes about each library in the full write-up.

Development

To install and run:

$ git clone [e-mail]:jamescooke/factory_audit.git
$ cd factory_audit
$ make venv
$ . venv/bin/activate
$ make install
$ cd factory_audit/
$ make test

Contributions

Please add your own factory library and run it through the tests - pull requests very welcome.

Please help me write a better wrapper for Hypothesis! I hypothesise that it is possible!