Elastix

A simple Elastic REST client written in Elixir.

Preface

This library talks to the Elastic(search) server through the HTTP/REST/JSON API. Its methods almost always return a HTTPoison request object.

When needed, the payload can be provided as an Elixir Map, which is internally converted to JSON. The library does not assume anything else regarding the payload and also does not (and will never) provide a magic DSL to generate the payload. That way users can directly manipulate the API data, that is sent to the Elastic server.

Overview

Elastix has 5 main modules and one utility module, that can be used, if the call/feature you want is not implemented (yet). However – please open issues or provide pull requests so I can improve the software for everybody. The modules are:

Elastix.Index corresponding to: this official API Documentation
Elastix.Document corresponding to: this official API Documentation
Elastix.Search corresponding to: this official API Documentation
Elastix.Bulk corresponding to: this official API Documentation
Elastix.Mapping corresponding to: this official API Documentation
and Elastix.HTTP – a thin HTTPoison wrapper

I will try and provide documentation and examples for all of them with time, for now just consult the source code.

Simple Example

start elastix application dependencies (or define it as an application dependency in mix.exs):

Elastix.start()

create the Elastic index

Elastix.Index.create("http://127.0.0.1:9200", "sample_index_name", %{})

assuming you have a model product create a document, search, or delete

# Elastic Server URL
elastic_url = "http://127.0.0.1:9200"

# Elastic Index Name
index_name = "shop_api_production"

# Elastic Document Type
doc_type = "product"

index_data = %{
  name: product.name,
  item_number: product.item_number,
  inserted_at: product.inserted_at,
  updated_at: product.updated_at
}

# Add mapping
mapping = %{
  properties: %{
    name: %{type: "text"},
    item_number: %{type: "integer"},
    inserted_at: %{type: "date"},
    updated_at: %{type: "date"}
  }
}

# add some search params according to Elastic JSON API
search_payload = %{}

# which document types should be included in the search?
search_in = [doc_type]

Elastix.Mapping.put(elastic_url, index_name, doc_type, mapping)
Elastix.Document.index(elastic_url, index_name, doc_type, product.id, index_data)
Elastix.Search.search(elastic_url, index_name, search_in, search_payload)
Elastix.Document.delete(elastic_url, index_name, doc_type, product.id)

Bulk request

It is possible to execute bulk requests with elastix.

Bulk requests take as parameters the list of lines to send to Elasticsearch. You can also optionally give them options. Available options are:

index the index of the request
type the document type of the request. (you can't specify type without specifying index)

Examples

lines = [
  %{index: %{_id: "1"}},
  %{field: "value1"},
  %{index: %{_id: "2"}},
  %{field: "value2"}
]

# Send bulk data
Elastix.Bulk.post elastic_url, lines, index: "my_index", type: "my_type"
# Send your lines by transforming them to iolist
Elastix.Bulk.post_to_iolist elastic_url, lines, index: "my_index", type: "my_type"

# Send raw data directly to the API
data = Enum.map(lines, fn line -> Poison.encode!(line) <> "\n" end)

Elastix.Bulk.post_raw elastic_url, data, index: "my_index", type: "my_type"

# Finally, you can specify the index or the type directly in you lines
lines = [
  %{index: %{_id: "1", _index: "my_index", _type: "my_type"}},
  %{field: "value1"},
  %{index: %{_id: "2", _index: "my_other_index", _type: "my_other_type"}},
  %{field: "value2"}
]

Elastix.Bulk.post elastic_url, lines

Configuration

Currently we can

pass options to the JSON decoder used by Elastix (poison)
optionally use shield for authentication (shield)
optionally pass along custom headers for every request made to the elasticsearch server(s)s
optionally pass along options to HTTPoison
optionally use a different JSON library

by setting the respective keys in your config/config.exs

config :elastix,
  json_options: [keys: :atoms],
  shield: true,
  username: "username",
  password: "password",
  httpoison_options: [hackney: [pool: :elastix_pool]]

The above for example will

lead to the HTTPoison responses being parsed into maps with atom keys instead of string keys (be careful as most of the time this is not a good idea as stated here: https://github.com/devinus/poison#parser).
use shield for authentication

Custom headers

To add custom headers to a request you must pass in the custom_headers option.

For example:

config :elastix,
  custom_headers: {MyModule, :add_aws_signature, ["us-east"]}

This must be a {Module, :function, [args]} tuple. The request will be added to the head of the args list. The args list may be empty. The request is a map with the method, headers, url, and body keys.

The function you define should return the full set of headers you want to send, including any headers passed in. For example:

defmodule MyModule do
  def add_aws_signature(request, region) do
    [{"Authorization", generate_aws_signature(request, region)} | request.headers]
  end

  defp generate_aws_signature(request, region) do
    # See: https://github.com/bryanjos/aws_auth or similar
  end
end

Override the default JSON library

To use a different JSON library you must pass in the json_codec option.

For example:

config :elastix,
  json_codec: JiffyCodec,
  json_options: [:return_maps]

This must be a module that implements the Elastix.JSON.Codec behavior. For example:

defmodule JiffyCodec do
  @behaviour Elastix.JSON.Codec

  def encode!(data), do: :jiffy.encode(data)
  def decode(json, opts \\ []), do: {:ok, :jiffy.decode(json, opts)}
end

Running tests

You need elastic search running locally on port 9200. A quick way of running any version of elastic, is via docker:

$ docker run -p 9200:9200 -it --rm elasticsearch:5.1.2

Then clone the repo and fetch its dependencies:

$ git clone git@github.com:werbitzky/elastix.git
$ cd elastix
$ mix deps.get
$ mix test

License

Copyright © 2017 El Werbitzky werbitzky@gmail.com This work is free. You can redistribute it and/or modify it under the terms of the Do What The Fuck You Want To Public License, Version 2, as published by Sam Hocevar. See http://www.wtfpl.net/ for more details.

juanperi / elastix