ElasticSearch Integration with Django (1.9.3) Tutorial

Objectives

Learn how to integrate ElasticSearch into a Django (v1.9.3) codebase.

Part 1: Basic Setup

The tutorial uses a custom project template that is no loger available, so recreating manually to match tutorial structure.

Target Structure

This is the iproject structure the tutorial uses (not a typical setup for Django).

.
├── README.md
├── project/
│   ├── apps/
│   │   └── core/
│   │       ├── admin.py
│   │       ├── apps.py
│   │       ├── management
│   │       │   └── commands
│   │       │       ├── dummy-data.py
│   │       │       └── push-to-index.py
│   │       ├── migrations
│   │       │   └── 0001_initial.py
│   │       ├── models.py
│   │       ├── urls.py
│   │       └── views.py
│   ├── conf/
│   │   ├── __init__.py
│   │   └── base.py
│   ├── manage.py
│   ├── static/
│   │   ├── bootstrap-theme.min.css
│   │   ├── bootstrap.min.css
│   │   ├── bootstrap.min.js
│   │   ├── dashboard.css
│   │   ├── jquery-ui.css
│   │   ├── jquery-ui.min.css
│   │   ├── jquery-ui.min.js
│   │   └── jquery.min.js
│   ├── templates/
│   │   ├── base.html
│   │   ├── index.html
│   │   └── student-details.html
│   ├── urls.py
│   └── wsgi.py
└── requirements.txt

Starting Structure

create a VirtualEnvironment to isolate the project and activate it.

Pip install these dependencies for the project:


$ pip install Django==1.9.3 elasticsearch requests django_extensions debug_toolbar django-debug-toolbar

Generate a new Django project: $ django-admin startproject project

This is the default project structure generated by manage.py when starting a new project.


.
|_____project/
|_____|_______init__.py
|_____|_____settings.py
|_____|_____urls.py
|_____|_____wsgi.py
|_____manage.py

Now that we have a project, the base files will need LOTS of tweaking.

Refactoring Project Structure

Rename the nested project/project folder to project/project.bak. We will not be using this directory (thus we break with django best practices).

Create the following directory structures:


$ mkdir project/apps/core
$ mkdir conf
$ mkdir db
$ mkdir log
$ mkdir media
$ mkdir static
$ mkdir templates

Copy the default settings.py to the new project root: $ cp project/project.bak/settings.py project/conf/base.py
Copy the default urls.py to the project root: $ cp project/project.bak/urls.py project/urls.py
Copy the default wsgi.py to the project root: $ cp project/project.bak/wsgi.py project/wsgi.py

Edit project/manage.py to match this:


#!/usr/bin/env python
import os
import sys

if __name__ == "__main__":
    # os.environ.setdefault("DJANGO_SETTINGS_MODULE", "project.settings")
    os.environ.setdefault("DJANGO_SETTINGS_MODULE", "conf")

    from django.core.management import execute_from_command_line

    execute_from_command_line(sys.argv)

Edit project/wsgi.py to match this:


"""
WSGI config for project project.

It exposes the WSGI callable as a module-level variable named ``application``.

For more information on this file, see
https://docs.djangoproject.com/en/1.9/howto/deployment/wsgi/
"""

import os
from os.path import abspath, dirname
from sys import path

SITE_ROOT = dirname(dirname(abspath(__file__)))
path.append(SITE_ROOT)

os.environ.setdefault("DJANGO_SETTINGS_MODULE", "src.conf")

from django.core.wsgi import get_wsgi_application
application = get_wsgi_application()

Edit project/urls.py to match this:


"""project URL Configuration

The `urlpatterns` list routes URLs to views. For more information please see:
    https://docs.djangoproject.com/en/1.9/topics/http/urls/
Examples:
Function views
    1. Add an import:  from my_app import views
    2. Add a URL to urlpatterns:  url(r'^$', views.home, name='home')
Class-based views
    1. Add an import:  from other_app.views import Home
    2. Add a URL to urlpatterns:  url(r'^$', Home.as_view(), name='home')
Including another URLconf
    1. Import the include() function: from django.conf.urls import url, include
    2. Add a URL to urlpatterns:  url(r'^blog/', include('blog.urls'))
"""

from django.conf import settings
from django.conf.urls import include, url
from django.contrib import admin
# from core.views import autocomplete_view, student_detail, HomePageView

urlpatterns = [
    url(r'^admin/', admin.site.urls),
    # url(r'^autocomplete/', autocomplete_view, name='autocomplete-view'),
    # url(r'^student', student_detail, name='student-detail'),
    # url(r'^$', HomePageView.as_view(), name='index-view'),
]

if settings.DEBUG:
    from django.conf.urls.static import static
    urlpatterns += static(settings.MEDIA_URL, document_root=settings.MEDIA_ROOT)
    urlpatterns += static(settings.STATIC_URL, document_root=settings.STATIC_ROOT)

    import debug_toolbar
    urlpatterns = [
        url(r'^__debug__/', include(debug_toolbar.urls)),
    ] + urlpatterns

Edit project/conf/base.py to math this:


"""
Django settings for project project.

Generated by 'django-admin startproject' using Django 1.9.3.

For more information on this file, see
https://docs.djangoproject.com/en/1.9/topics/settings/

For the full list of settings and their values, see
https://docs.djangoproject.com/en/1.9/ref/settings/
"""

import os
import sys

from elasticsearch import Elasticsearch, RequestsHttpConnection

# Build paths inside the project like this: os.path.join(BASE_DIR, ...)
BASE_DIR = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
ROOT_DIR = os.path.dirname(BASE_DIR)
sys.path.append(
    os.path.join(BASE_DIR, 'apps')
)

# Quick-start development settings - unsuitable for production
# See https://docs.djangoproject.com/en/1.9/howto/deployment/checklist/

# SECURITY WARNING: keep the secret key used in production secret!
SECRET_KEY = 'alun^uucv3t=(9%z=mt5q085x^xn$n5+h=tbjta-lfm5qmikra'

# SECURITY WARNING: don't run with debug turned on in production!
DEBUG = True

# ALLOWED_HOSTS = []
ALLOWED_HOSTS = ['*']


# Application definition

INSTALLED_APPS = [
    'django.contrib.admin',
    'django.contrib.auth',
    'django.contrib.contenttypes',
    'django.contrib.sessions',
    'django.contrib.messages',
    'django.contrib.staticfiles',
    # ADDED:
    'core',
    'django_extensions',
    'debug_toolbar',
]

MIDDLEWARE_CLASSES = [
    'django.middleware.security.SecurityMiddleware',
    'django.contrib.sessions.middleware.SessionMiddleware',
    'django.middleware.common.CommonMiddleware',
    'django.middleware.csrf.CsrfViewMiddleware',
    'django.contrib.auth.middleware.AuthenticationMiddleware',
    'django.contrib.auth.middleware.SessionAuthenticationMiddleware',
    'django.contrib.messages.middleware.MessageMiddleware',
    'django.middleware.clickjacking.XFrameOptionsMiddleware',
    # ADDED:
    'debug_toolbar.middleware.DebugToolbarMiddleware',
]

# ROOT_URLCONF = 'project.urls'
ROOT_URLCONF = 'urls'

TEMPLATES = [
    {
        'BACKEND': 'django.template.backends.django.DjangoTemplates',
        # 'DIRS': [],
        'DIRS': [
            os.path.normpath(os.path.join(BASE_DIR, 'templates')),
        ],
        'APP_DIRS': True,
        'OPTIONS': {
            'context_processors': [
                'django.template.context_processors.debug',
                'django.template.context_processors.request',
                'django.template.context_processors.media', # Added
                'django.template.context_processors.static', # Added
                'django.contrib.auth.context_processors.auth',
                'django.contrib.messages.context_processors.messages',
            ],
        },
    },
]

# WSGI_APPLICATION = 'project.wsgi.application'
WSGI_APPLICATION = 'wsgi.application'


# Database
# https://docs.djangoproject.com/en/1.9/ref/settings/#databases

DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.sqlite3',
        # 'NAME': os.path.join(BASE_DIR, 'db.sqlite3'),
        'NAME': os.path.join(BASE_DIR, 'db/development.db'),  # Added.
    }
}


# Password validation
# https://docs.djangoproject.com/en/1.9/ref/settings/#auth-password-validators

AUTH_PASSWORD_VALIDATORS = [
    {
        'NAME': 'django.contrib.auth.password_validation.UserAttributeSimilarityValidator',
    },
    {
        'NAME': 'django.contrib.auth.password_validation.MinimumLengthValidator',
    },
    {
        'NAME': 'django.contrib.auth.password_validation.CommonPasswordValidator',
    },
    {
        'NAME': 'django.contrib.auth.password_validation.NumericPasswordValidator',
    },
]


# Internationalization
# https://docs.djangoproject.com/en/1.9/topics/i18n/

LANGUAGE_CODE = 'en-us'

TIME_ZONE = 'UTC'

USE_I18N = True

USE_L10N = True

USE_TZ = True


# Static files (CSS, JavaScript, Images)
# https://docs.djangoproject.com/en/1.9/howto/static-files/

STATIC_URL = '/static/'

# EVERYTHING BELOW HERE ADDED.

# STATIC FILE CONFIGURATION
# See: https://docs.djangoproject.com/en/dev/ref/settings/#static-root
STATIC_ROOT = os.path.join(ROOT_DIR, 'assets')

# See:
# https://docs.djangoproject.com/en/dev/ref/contrib/staticfiles/#std:setting-STATICFILES_DIRS
STATICFILES_DIRS = (
    os.path.join(BASE_DIR, 'static'),
)

STATICFILES_FINDERS = (
    'django.contrib.staticfiles.finders.FileSystemFinder',
    'django.contrib.staticfiles.finders.AppDirectoriesFinder',
)
# END STATIC FILE CONFIGURATION

# MEDIA CONFIGURATION
# See: https://docs.djangoproject.com/en/dev/ref/settings/#media-root
MEDIA_ROOT = os.path.normpath(os.path.join(ROOT_DIR, 'media'))

# See: https://docs.djangoproject.com/en/dev/ref/settings/#media-url
MEDIA_URL = '/media/'
# END MEDIA CONFIGURATION

LOGGING = {
    'version': 1,
    'disable_existing_loggers': False,
    'filters': {
        'require_debug_true': {
            '()': 'django.utils.log.RequireDebugTrue',
        },
        'require_debug_false': {
            '()': 'django.utils.log.RequireDebugFalse',
        },
    },
    'formatters': {
        'verbose': {
            'format': "[%(asctime)s] %(levelname)s [%(name)s:%(lineno)s] %(message)s",
            'datefmt': "%d/%b/%Y %H:%M:%S"
        },
        'simple': {
            'format': '%(levelname)s %(message)s'
        },
    },
    'handlers': {
        'console': {
            'level': 'DEBUG',
            'filters': ['require_debug_true'],
            'class': 'logging.StreamHandler',
        },
        'file': {
            'level': 'ERROR',
            'class': 'logging.FileHandler',
            'filters': ['require_debug_false'],
            'filename': 'log/error.log',
            'formatter': 'verbose'
        },
    },
    'loggers': {
        'django.db.backends': {
            'level': 'DEBUG',
            'handlers': ['console'],
        },
        'django.request': {
            'handlers': ['file'],
            'level': 'ERROR',
            'propagate': True,
        },
    }
}

# from elasticsearch import Elasticsearch, RequestsHttpConnection

ES_CLIENT = Elasticsearch(
    ['http://127.0.0.1:9200/'],
    connection_class=RequestsHttpConnection
)
ES_AUTOREFRESH = True

Create an init file for the conf app: $ touch project/apps/conf/__init__.py

Edit the conf/__init__.py file to match this:


from .base import *

try:
    from .local import *
except ImportError:
    pass

Create the project/apps/core/models.py file: $ touch project/apps/core/models.py

Populate the new models.py file as follows:


from django.db import models
from django.core.validators import MinValueValidator, MaxValueValidator

class University(models.Model):
    name = models.CharField(max_length=255, unique=True)

class Course(models.Model):
    name = models.CharField(max_length=255, unique=True)

class Student(models.Model):
    YEAR_IN_SCHOOL_CHOICES = (
        ('FR', 'Freshman'),
        ('SO', 'Sophomore'),
        ('JR', 'Junior'),
        ('SR', 'Senior'),
    )
    # note: incorrect choice in MyModel.create leads to creation of incorrect record
    year_in_school = models.CharField(
        max_length=2, choices=YEAR_IN_SCHOOL_CHOICES)
    age = models.SmallIntegerField(
        validators=[MinValueValidator(1), MaxValueValidator(100)]
    )
    first_name = models.CharField(max_length=50)
    last_name = models.CharField(max_length=50)
    # various relationships models
    university = models.ForeignKey(University, null=True, blank=True)
    courses = models.ManyToManyField(Course, null=True, blank=True)

Make migrations for the new models: $ python project/manage.py makemigrations core
This results in a new file: project/apps/core/migrations/0001_initial.py
Make a directory for the db: $ mkdir project/db
Build the db tables from the migration: $ python project/manage.py migrate
Create a new file for the admin interface: $ tpuch project/apps/core/admin.py

Edit the new admin.py file to match this:


from django.contrib import admin
from .models import University, Course, Student

admin.site.register(University)
admin.site.register(Course)
admin.site.register(Student)

Create a superuser for the admin interface: $ python project/manage.py createsuperuser
- Provide a username and password as prompted.
Test the project: $ python project/manage.py runserver
- You can see the running project in a browser at: http://localhost:8000/admin/
- If you access http://localhost:8000 you will get a 404 error because we have not define any urls to handle that route.
Login to the admin panel to verify everything is working.

Working Django Application Structure

The state of the project structure after adapting the default django project to match tutorial, installing dependencies, configuring middleware, etc.

Note: I am intentionally ommitting the __Pycache__, .git and project.bak dirs from this tree output.

$ tree -I ".git|__pycache__|project.bak"
.
├── README.md
├── project
│   ├── apps
│   │   └── core
│   │       ├── admin.py
│   │       ├── migrations
│   │       │   ├── 0001_initial.py
│   │       │   └── __init__.py
│   │       └── models.py
│   ├── conf
│   │   ├── __init__.py
│   │   └── base.py
│   ├── db
│   │   └── development.db
│   ├── log
│   │   └── error.log
│   ├── manage.py
│   ├── media
│   ├── static
│   ├── templates
│   ├── urls.py
│   └── wsgi.py
└── requirements.txt

Create A Reusable Command To Populate DB Tables

...

References

Tutorial

Additional Refs

Troubleshooting Refs

taoteg / es-django-tut

ElasticSearch Integration with Django (1.9.3) Tutorial

Objectives

Part 1: Basic Setup

References

About

Languages