django-elasticsearch-metrics

Django app for storing time-series metrics in Elasticsearch.

[Work in progress] Below is a sci-fi API. This isn't working yet!

Pre-requisites

Python 2.7 or >=3.6
Django 1.11 or 2.0
Elasticsearch 6

Quickstart

Add "elasticseach_metrics" to INSTALLED_APPS.

INSTALLED_APPS += [
    "elasticsearch_metrics",
]

Define the ELASTICSEARCH_DSL setting.

ELASTICSEARCH_DSL = {
    "default": {"hosts": "localhost:9200"}
}

This setting is passed to elasticsearch_dsl.connections.configure so it takes the same parameters.

In one of your apps, define a new metric in metrics.py.

A Metric is a subclass of elasticsearch_dsl.Document.

# myapp/metrics.py

from elasticsearch_metrics import Metric
from elasticsearch_dsl import Integer

class PageView(Metric):
    user_id = Integer()

Use the sync_metrics management command to ensure that the index template for your metric is created in Elasticsearch.

# This will create an index template called myapp_pageview
python manage.py sync_metrics

Now add some data:

from myapp.metrics import PageView

user = User.objects.latest()

view = PageView(user_id=user.id)
# By default we create an index for each day.
# Therefore, this will persist the document
# to an index called, e.g. "myapp_pageview-2020.02.04"
view.save()

Go forth and search!

# perform a search across all page views
PageView.search()

Index settings

You can configure the index template settings by setting Metric.Index.settings.

class PageView(Metric):
    user_id = Integer()

    class Index:
        settings = {
            "number_of_shards": 2,
            "refresh_interval": "5s",
        }

Overriding the default template name and pattern

class PageView(Metric):
    user_id = Integer()

    class Index:
        settings = {
            "number_of_shards": 2,
            "refresh_interval": "5s",
        }

    class Meta:
        template_name = "myapp_pviews"
        template = "myapp_pviews-*"

Configuration

ELASTICSEARCH_DSL: Required. Connection settings passed to elasticsearch_dsl.connections.configure.
ELASTICSEARCH_METRICS_DATEFORMAT: Dateformat to use when creating indexes. Default: %Y.%m.%d (same dateformat Elasticsearch uses for date math)

Management commands

sync_metrics: Ensure that all index templates have been created for your metrics.

python manage.py sync_metrics

clean_metrics : Clean old data using curator.

python manage.py clean_metrics myapp.PageView --older-than 45 --time-unit days

Signals

Signals are located in the elasticsearch_metrics.signals module.

pre_index_template_create(Metric, index_template): Sent before PUTting a new index template into Elasticsearch.

Caveats

_source and _all are disabled by default on metric indices in order to save disk space. For most metrics use cases, Users will not need to retrieve the source JSON documents. Be sure to understand the consequences of this: https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping-source-field.html#_disabling_source . To enable _source, you can override it in class Meta.

class MyMetric(Metric):
    class Meta:
        source = MetaField(enabled=True)

License

MIT Licensed.

erinspace / django-elasticsearch-metrics