Giters

IBMStreams / streamsx.elasticsearch

Toolkit to enable application developers to store streaming data into Elastic Search database

Home Page:https://ibmstreams.github.io/streamsx.elasticsearch/

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Incompatible changes for version 2 of the toolkit

brandtol opened this issue · comments

commented

We plan to remove some functionality from the next version of the toolkit, for two reasons:

  • simplify code and dependencies
  • Some features used so far have been deprecated by Elasticsearch, or will be deprecated and removed in ES6/7

The details are :

Operators

  1. The ElasticsearchIndex operator will be removed, because it uses the TransportClient which will be deprecated/removed in ES7/8
  2. The ElasticsearchRestIndex operator (using the JEST client) will be renamed to ElasticsearchIndex

Parameters

  1. Parameter sizeMetricsEnabled will be removed as these metrics will not be provided anymore (see below)
  2. Parameters typeName and typeNameAttribute will be deprecated, because document types are deprecated in ES6

Metrics

  1. The following metrics (enabled via the sizeMetricsEnabled parameter) will be removed: avgInsertSizeBytes, maxInsertSizeBytes, minInsertSizeBytes and sumInsertSizeBytes. The reasons for removal are :
    • the metrics are polled from the ES server (potentially after each insert) which may affect throughput
    • to support these metrics a plugin needs to be installed on the ES server
    • the metrics do not work well with the existing JEST client
    • If there is a need for these metrics they can be calculated/estimated on the client side

If you have objections with any of the changes, please comment on this issue.

commented

New proposal : remove typeName and typeNameAttribute parameters, instead of deprecating.

Rationale: we are targeting ES6/7 with the v2 release anyway, users of ES5 can still use v1.X versions of the toolkit to get support for the type parameters.

commented

Regarding removal of typeName and typeNameAttribute parameters. We did not do it for this release. We only mark them as deprecated. The reason is, in ES6 each index still needs to have a type, but there is only ONE type allowed per index. So the typeNameAttribute parameter does not make much sense anymore, but the typeName is still sort of useful.

We changed the behavoiur, so if the user does not specify a type, a default value of "_doc" is assigned. That is what the ES documentation recommends.

commented

all changes done