auto.create.tables.replicated in v2.1.0-lt

[tellienne]

Hi!
We are testing connector v2.1.0-lt to transfer data from Postgres to clickhouse.
Using "auto.create.tables.replicated: "true" tables are not created in clickhouse.
In the logs we see only messages like:
"java.sql.sqlexception: code: 390. db::exception: table TABLE_NAME doesn't exist. (cannot_get_create_table_query)".

If we remove the line and leave only "auto.create.tables: "true", then the tables begin to be created in the clickhouse

Does this option work in this version of the connector?
Perhaps we should use some other parameters in the config?

[Kenny]

could u try to set both:

"auto.create.tables: "true"
"auto.create.tables.replicated: "true"

worked in my side.

[tellienne]

yes, we use both options, but the tables are not created in clickhouse.
Here is part of the config:

name: "debezium-embedded-postgres"
database.server.name: "ER54"
schema.include.list: "public"
plugin.name: "pgoutput"
table.include.list: "public.messages, public.users, public.debezium_heartbeat"
heartbeat.interval.ms: 15000
heartbeat.action.query: "update debezium_heartbeat SET date_column = now();"
snapshot.mode: "initial"
offset.flush.interval.ms: 5000
enable.snapshot.ddl: "true"
auto.create.tables: "true"
auto.create.tables.replicated: "true"
publication.autocreate.mode: "filtered"

when we comment

auto.create.tables.replicated: "true"

it start work

[Kenny]

with configure:

auto.create.tables.replicated: "true"

have u set up clickhouse with macro cluster in xml?
it will create table with DDL: create table on cluster {cluster}...

[tellienne]

You are absolutely right, the macro was missing.
We have added macros and a table with the new engine is created, but only on one host.
Our clickhouse cluster has two hosts in one shard.
We tried to create a test table manually
using the request:

CREATE TABLE chat.test ON CLUSTER '{cluster}' (
    log_date date,
    user_name String) ENGINE = ReplicatedMergeTree ('/test', '{replica}'
)
PARTITION BY log_date
ORDER BY
    (log_date, user_name);

The table was successfully created on all hosts.

But the tables that the connector creates are not available on the second host :(

Any ideas about this?

[Kenny]

u can check log to see the query for creation table.
try to run manually CREATE TABLE chat.test ON CLUSTER '{cluster}' on both 2 nodes with different table name.
-> make sure tables created on all node in cluster.

[tellienne]

u can check log to see the query for creation table.

in the logs of the second node there is no request to create a table from the connector

try to run manually CREATE TABLE chat.test ON CLUSTER '{cluster}' on both 2 nodes with different table name.

made a request CREATE TABLE chat.test ON CLUSTER '{cluster} on the first node, the request worked well, the table was created on all hosts in the cluster.

made a request CREATE TABLE chat.test3 ON CLUSTER '{cluster} on the second node - everything is fine too, the table was created on all hosts

Is it possible to see somewhere the complete query with which the connector creates the table?
I only see entries in the logs like

{136a3f68-93d4-4ec5-9b5f-0d671eb80ad4} <Debug> executeQuery: (from 10.131.0.25:42946, user: USER) SHOW CREATE TABLE chat.table_name (stage: Complete)

but I can't find the full ddl

[Kenny]

in the logs of the second node there is no request to create a table from the connector

So, does the table already exist in both 2 nodes?.
Have u try to trigger a CDC event from the source table?

how do u set up CH clickhouse?. is there a load-balance to receive DB request? (like CHproxy)

[tellienne]

how do u set up CH clickhouse?. is there a load-balance to receive DB request? (like CHproxy)

the cluster consists of:
one Zukeeper node
two clickhouse nodes in one shard (node A, node B)

The connector config specifies the IP address of node A and port 8123.

So, does the table already exist in both 2 nodes?.

No, only in first node. maybe I've already confused you

here are our steps:

1. write IP clickhouse node A in the connector config. the cluster has macros cluster, shard and replica
2. add a list of tables to the table.include.list parameter and

auto.create.tables: "true"
auto.create.tables.replicated: "true"

3. start connector with

 snapshot.mode: "initial"

4. check the result

expect - tables from the list were created on node A and node B in the clickhouse because it is used ON CLUSTER '{cluster}

reality - the table was created only on node A.

[tellienne]

UPD: I updated the connector version to 2.2.0 and now the connector creates tables on all hosts in the cluster