Giters

mravi / kafka-connect-hbase

Kafka Connect to Hbase

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Using for complex hbase format

pranny opened this issue · comments

commented

I have a hbase table with multiple column families. From the given example, I am not sure how do I create my configs.

My table looks like

column family: main, column name: value
column family: main, column name: timestamp
column family: meta, column name: datatype

The Kafka Topic has Value schema that goes like

fields:
  - key: Corresponds to the hbase key
  - value: Corresponds to the "main:value"
  - timestamp: Corresponds to "main:timestamp"
  - datatype: Corresponds to "meta:datatype"

Is something like below a valid config?

tasks.max=1
topics=telemetry
zookeeper.quorum=localhost:2181
event.parser.class=io.svectors.hbase.parser.AvroEventParser
hbase.telemetry.rowkey.columns=value, timestamp, datatype
hbase.telemetry.rowkey.delimiter=:
hbase.telemetry.family=main, meta
commented

It seems, it supports one column family as of now. I have enhanced the code with multiple column family support and custom column mapping. You can refer this for your use case:
https://github.com/nishutayal/kafka-connect-hbase