This project aims to be an exhaustive collection of examples of theses mistakes, as many as possible, and also of the solution(s) to these performance loses caused by N+1 issues.
N+1 is the common way to reference a database performance issue that Django is susceptible to, when using certain database calls, in certain ways.
For convenience of poking and proding, a basic Django project named nplus1_nomore is available.
Each app focuses on one given type of problem, group of problems or alternative solutions.
- Python 3.6
- Django 2.2
Optional:
- Docker (for use with Postgres metrics module)
To install the required Python modules:
pip install -r requirements.txtI have used a number of online resources when researching this topic. Here are some of them:
- https://blog.sentry.io/2020/09/14/finding-and-fixing-django-n-1-problems
- https://scoutapm.com/blog/django-and-the-n1-queries-problem
- https://www.valentinog.com/blog/n-plus-one/
- https://apirobot.me/posts/django-graphql-solving-n-1-problem-using-dataloaders
From the folder containing this README file, and from an appropiate virtualenv, execute the following:
# install requirements
pip install -r requirements.txt
# go to project folder
cd n_plus_one
# migrations
python manage.py makemigrations autopre example_0 example_1 example_2 example_3 rest graph
python manage.py migrate
Model instances can be created using the custom management commands provided, or by loading the included fixtures.
Collection of examples of different ways in which the N+1 problem can arise in Django.
N+1 by direct ForeingKey relationship.
To create extra instances for this app, execute:
python manage.py create_example_0_data $INTTo load the data fixture:
python manage.py loaddata --app example_0 example_0_dump.jsonN+1 by reverse ForeignKey relationship.
This is an implementation of the example used on this very good explanation of the n+1 issue in Django.
To create extra instances for this app, execute:
python manage.py create_example_1_data $INTTo load the data fixture:
python manage.py loaddata --app example_1 example_1_dump.jsonIn this example we explore:
- N+1 issues of accesing data from a
ForeignKeyField - Optimizations using database functions instead of Python code
2N+1 issues with double-nested model access through ForeighKeyField
Inspired by the 2N+1 explanation on https://scoutapm.com/blog/django-and-the-n1-queries-problem
To create extra instances for this app, execute:
python manage.py create_example_2_data $INTTo load the data fixture:
python manage.py loaddata --app example_2 example_2_dump.jsonExploration of N+1 issues when using ManyToManyField in your models.
To create extra instances for this app, execute:
python manage.py create_example_3_data $INTTo load the data fixture:
python manage.py loaddata --app example_3 example_3_dump.jsonIn this case we explore the N+1 issues when developing REST endpoints with DjangoRESTFramework
To create extra instances for this app, execute:
python manage.py create_rest_data $INTTo load the data fixture:
python manage.py loaddata --app rest rest_dump.jsonThis example is inspired by the tutorial at https://apirobot.me/posts/django-graphql-solving-n-1-problem-using-dataloaders
There are a number of modules to help us with this as well as keep an eye on the overall performance of our project.
To enable any module, start the server with the appropiate settings.
Links to disabled modules will return an error.
nplusone is a library for detecting the n+1 queries problem in Python ORMs, including SQLAlchemy, Peewee, and the Django ORM.
To enable nplusone:
python manage.py migrate --settings=nplus1_nomore.settings.nplusone
python manage.py runserver --settings=nplus1_nomore.settings.nplusoneFor more details: https://github.com/jmcarp/nplusone
Silk is a live profiling and inspection tool for the Django framework. Silk intercepts and stores HTTP requests and database queries before presenting them in a user interface for further inspection.
To enabledjango-silk:
python manage.py migrate --settings=nplus1_nomore.settings.silk
python manage.py runserver --settings=nplus1_nomore.settings.silk
Note: you'll need to uncomment the decorators and the import statement on views to enable profiling
For more details: https://github.com/jazzband/django-silk
A Django application that exposes a bunch of PostgreSQL database metrics.
To enable django-postgres-metrics:
# start local docker PG server
./start_pg.sh
python manage.py migrate --settings=nplus1_nomore.settings.pg_metrics
# create superuser (requires admin access)
python manage.py createsuperuser --settings=nplus1_nomore.settings.pg_metrics
# create some instances
python manage.py loaddata --app example_1 example_1_dump.json --settings=nplus1_nomore.settings.pg_metrics
# start server
python manage.py runserver --settings=nplus1_nomore.settings.pg_metrics
Remember to create the .env file from .env_example, and fill the required data.
For more details: https://github.com/django-postgres-metrics/django-postgres-metrics
This is a simple profiling middleware for Django applications.
To use add ?prof or &prof at the end of your query to see the results of the query analysis.
To enable cprofile:
# start server
python manage.py runserver --settings=nplus1_nomore.settings.cprofile
The Django Debug Toolbar is a configurable set of panels that display various debug information about the current request/response and when clicked, display more details about the panel’s content.
To enable debug_toolbar:
# start server
python manage.py runserver --settings=nplus1_nomore.settings.debug
For more details: https://django-debug-toolbar.readthedocs.io/en/latest/
Checks for various conditions and provides reports when anomalous behavior is detected.
To enable django-health-check:
python manage.py migrate --settings=nplus1_nomore.settings.health
# start server
python manage.py runserver --settings=nplus1_nomore.settings.health
For more details: https://github.com/KristianOellegaard/django-health-check
To load the data for django-auto-prefetch: python manage.py loaddata --app autopre autopre_dump.json
An example page can be reached at /autopre/
For more details: https://github.com/tolomea/django-auto-prefetch
- django-live-profiler