snapshot of new tables in Postgres
tellienne opened this issue · comments
Hello!
Previously we described a problem when a new table is created in Postgres - #438
There is the same problem in the version 2.1.0-lt.
The old issue is closed, but it seems that the situation has not changed.
Is there any news about this?
Hi @tellienne , The snapshot.mode configuration parameter controls the functionality of replication of existing data
https://debezium.io/documentation/reference/stable/connectors/postgresql.html#postgresql-property-snapshot-mode
Did you have the snapshot.mode set to initial?
Did you have the snapshot.mode set to initial?
yes, we use this mode.
Our config:
table.include.list: "public.companies, public.messages"
snapshot.mode: "initial"
offset.flush.interval.ms: 5000
enable.snapshot.ddl: "true"
auto.create.tables: "true"
publication.autocreate.mode: "filtered"
this works well the first time you run the connector
We would like to add several more tables to the table.include.list and so that when the connector is restarted, a snapshot is taken for these tables first, and then replicated.
Is it possible?
Looks like the problem described here https://debezium.io/blog/2021/10/07/incremental-snapshots/
Yes correct, Im adding a README page on how to add new tables based on the above blog post. We are also working on a feature to simplify this process , #606
this will be great, thanks!
Hello again.
We are trying to use incremental snapshots from the Debezium documentation. It works, but there are strange things - the connector starts to consume a lot of CPU.
we deploy the connector in k8s and after the start of the incremental snapshot, the container consumes 6-8 vCPU. there are only 100 thousand rows in the table, but the connector is loaded and cannot copy them.
Perhaps you have some ideas about this?
Hi @tellienne
Update production_setup.md for incremental snapshot size
You can try reducing the incremental.snapshot.chunk.size property and increasing the snapshot.max.threads property to higher value.
Added documentation here - Update production_setup.md for incremental snapshot size