Migrate unmanaged tablet to Vitess keyspace

Question

Migrate unmanaged tablet to Vitess keyspace

zehweh opened this issue 3 years ago · comments

Hi,

for the last couple of days I've been trying to migrate data from an unmanaged tablet to a vitess cluster in kubernetes using the vitess orchestrator.

Setting up a replicated, unsharded test keyspace together with an unmanaged tablet works like a charm but when it comes to migrating the data, it fails and I see a lot of connection errors:

# date
# Tue 15 Mar 2022 08:14:56 AM UTC
# vtctlclient MoveTables -source testdb -all Create vitesstest.testremote2vitess

MoveTables Error: rpc error: code = Unknown desc = TabletManager.ApplySchema on zone1-1728353218 error: EOF (errno 2013) (sqlstate HY000) during query: SELECT
			table_name AS table_name,
			ordinal_position AS ordinal_position,
			column_name AS column_name
		FROM information_schema.key_column_usage
		WHERE table_schema = 'vt_vitesstest'
			AND table_name IN ('wp_commentmeta', 'wp_comments', 'wp_links', 'wp_options', 'wp_postmeta', 'wp_posts', 'wp_term_relationships', 'wp_term_taxonomy', 'wp_termmeta', 'wp_terms', 'wp_usermeta', 'wp_users')
			AND constraint_name='PRIMARY'
		ORDER BY table_name, ordinal_position
EOF (errno 2013) (sqlstate HY000) during query: SELECT * FROM `vt_vitesstest`.`wp_links` WHERE 1=0
EOF (errno 2013) (sqlstate HY000) during query: SELECT * FROM `vt_vitesstest`.`wp_posts` WHERE 1=0
EOF (errno 2013) (sqlstate HY000) during query: SELECT * FROM `vt_vitesstest`.`wp_term_relationships` WHERE 1=0
EOF (errno 2013) (sqlstate HY000) during query: SHOW CREATE TABLE `vt_vitesstest`.`wp_comments`
EOF (errno 2013) (sqlstate HY000) during query: SHOW CREATE TABLE `vt_vitesstest`.`wp_postmeta`
EOF (errno 2013) (sqlstate HY000) during query: SHOW CREATE TABLE `vt_vitesstest`.`wp_term_taxonomy`
EOF (errno 2013) (sqlstate HY000) during query: SHOW CREATE TABLE `vt_vitesstest`.`wp_termmeta`
context canceled: EOF (errno 2013) (sqlstate HY000) during query: SELECT
			table_name AS table_name,
			ordinal_position AS ordinal_position,
			column_name AS column_name
		FROM information_schema.key_column_usage
		WHERE table_schema = 'vt_vitesstest'
			AND table_name IN ('wp_commentmeta', 'wp_comments', 'wp_links', 'wp_options', 'wp_postmeta', 'wp_posts', 'wp_term_relationships', 'wp_term_taxonomy', 'wp_termmeta', 'wp_terms', 'wp_usermeta', 'wp_users')
			AND constraint_name='PRIMARY'
		ORDER BY table_name, ordinal_position
EOF (errno 2013) (sqlstate HY000) during query: SELECT * FROM `vt_vitesstest`.`wp_links` WHERE 1=0
EOF (errno 2013) (sqlstate HY000) during query: SELECT * FROM `vt_vitesstest`.`wp_posts` WHERE 1=0
EOF (errno 2013) (sqlstate HY000) during query: SELECT * FROM `vt_vitesstest`.`wp_term_relationships` WHERE 1=0
EOF (errno 2013) (sqlstate HY000) during query: SHOW CREATE TABLE `vt_vitesstest`.`wp_comments`
EOF (errno 2013) (sqlstate HY000) during query: SHOW CREATE TABLE `vt_vitesstest`.`wp_postmeta`
EOF (errno 2013) (sqlstate HY000) during query: SHOW CREATE TABLE `vt_vitesstest`.`wp_term_taxonomy`
EOF (errno 2013) (sqlstate HY000) during query: SHOW CREATE TABLE `vt_vitesstest`.`wp_termmeta`
context canceled
E0315 08:14:57.172988 4085950 main.go:76] remote error: rpc error: code = Unknown desc = TabletManager.ApplySchema on zone1-1728353218 error: EOF (errno 2013) (sqlstate HY000) during query: SELECT
			table_name AS table_name,
			ordinal_position AS ordinal_position,
			column_name AS column_name
		FROM information_schema.key_column_usage
		WHERE table_schema = 'vt_vitesstest'
			AND table_name IN ('wp_commentmeta', 'wp_comments', 'wp_links', 'wp_options', 'wp_postmeta', 'wp_posts', 'wp_term_relationships', 'wp_term_taxonomy', 'wp_termmeta', 'wp_terms', 'wp_usermeta', 'wp_users')
			AND constraint_name='PRIMARY'
		ORDER BY table_name, ordinal_position
EOF (errno 2013) (sqlstate HY000) during query: SELECT * FROM `vt_vitesstest`.`wp_links` WHERE 1=0
EOF (errno 2013) (sqlstate HY000) during query: SELECT * FROM `vt_vitesstest`.`wp_posts` WHERE 1=0
EOF (errno 2013) (sqlstate HY000) during query: SELECT * FROM `vt_vitesstest`.`wp_term_relationships` WHERE 1=0
EOF (errno 2013) (sqlstate HY000) during query: SHOW CREATE TABLE `vt_vitesstest`.`wp_comments`
EOF (errno 2013) (sqlstate HY000) during query: SHOW CREATE TABLE `vt_vitesstest`.`wp_postmeta`
EOF (errno 2013) (sqlstate HY000) during query: SHOW CREATE TABLE `vt_vitesstest`.`wp_term_taxonomy`
EOF (errno 2013) (sqlstate HY000) during query: SHOW CREATE TABLE `vt_vitesstest`.`wp_termmeta`
context canceled: EOF (errno 2013) (sqlstate HY000) during query: SELECT
			table_name AS table_name,
			ordinal_position AS ordinal_position,
			column_name AS column_name
		FROM information_schema.key_column_usage
		WHERE table_schema = 'vt_vitesstest'
			AND table_name IN ('wp_commentmeta', 'wp_comments', 'wp_links', 'wp_options', 'wp_postmeta', 'wp_posts', 'wp_term_relationships', 'wp_term_taxonomy', 'wp_termmeta', 'wp_terms', 'wp_usermeta', 'wp_users')
			AND constraint_name='PRIMARY'
		ORDER BY table_name, ordinal_position
EOF (errno 2013) (sqlstate HY000) during query: SELECT * FROM `vt_vitesstest`.`wp_links` WHERE 1=0
EOF (errno 2013) (sqlstate HY000) during query: SELECT * FROM `vt_vitesstest`.`wp_posts` WHERE 1=0
EOF (errno 2013) (sqlstate HY000) during query: SELECT * FROM `vt_vitesstest`.`wp_term_relationships` WHERE 1=0
EOF (errno 2013) (sqlstate HY000) during query: SHOW CREATE TABLE `vt_vitesstest`.`wp_comments`
EOF (errno 2013) (sqlstate HY000) during query: SHOW CREATE TABLE `vt_vitesstest`.`wp_postmeta`
EOF (errno 2013) (sqlstate HY000) during query: SHOW CREATE TABLE `vt_vitesstest`.`wp_term_taxonomy`
EOF (errno 2013) (sqlstate HY000) during query: SHOW CREATE TABLE `vt_vitesstest`.`wp_termmeta`
context canceled

After checking the logs, I noticed that the mysql master on the test keyspace gets killed (by vttablet?) immediately after starting the migration.
I tested the migration with vitess versions 12, 13 and latest with percona as well as plain mysql (5.7) using a wordpress db dump.

Here are my configs and logs:

Hope that helps!

Thanks,
chris

jawabuu · Answer 1 · Thu Mar 17 2022 02:17:10 GMT+0800 (China Standard Time)

Hey @zehweh
Could you try this
https://gist.github.com/jawabuu/da27bb6bbb562d9caa5d1692955c892a

zehweh · Answer 2 · Thu Mar 17 2022 23:53:49 GMT+0800 (China Standard Time)

Thanks @jawabuu for your quick response.

Unfortunately, I get connection errors again.
You can see the logs here: https://gist.github.com/zehweh/c55ebe1aedcee5acd498e954baa1cd27

I noticed, that vttablet complains about mysql running in read-only mode... This looks suspicious.

Some additional information I forgot to add:

kubernetes version 1.22.4
flannel for networking

jawabuu · Answer 3 · Fri Mar 18 2022 00:05:55 GMT+0800 (China Standard Time)

@zehweh Are you on #vitess slack channel?

jawabuu · Answer 4 · Fri Mar 18 2022 00:06:49 GMT+0800 (China Standard Time)

I'd like to see your
kubectl get po after running my gist

zehweh · Answer 5 · Tue Jun 07 2022 13:34:48 GMT+0800 (China Standard Time)

I found out what was causing the issues:

1: The resource limits were set too low in my VitessCluster manifest which resulted in occasional OOM kills of the mysql container and the logs you see above.

2: I'm working with a master-master replication on the external database (unmanaged tablet). The master I'm connecting to has multiple sets of GTIDs:

mysql> SHOW SLAVE STATUS\G

*************************** 1. row ***************************
               Slave_IO_State: Waiting for master to send event
[...]
            Executed_Gtid_Set: 01af65be-d03d-11ec-a14f-0cc47a6af5da:1-28538331:544939690-544939696,
d048674a-e588-11ec-80cd-0cc47a6c65c8:1-65287695
[...]

Because of this, the MoveTables workflow wouldn't start:

❯ vtctlclient MoveTables  --source testdb -all Create vitesstest.testremote2vitess                                          
Waiting for workflow to start:
0% ... 0% ... 0% ... 0% ... 0% ... 0% ... 0% ... 0% ... 0% ... 0% ... 0% ... 0% ... 0% ... 0% ... 0% ... 0% ... 0% ... 0% ... 0% ... 0% ... 0% ... 0% ... 0% ... 0% ... 0% ... 0% ... 0% ... 0% ... 0% ... 0% ... E0606 09:56:06.404528    5131 main.go:67] E0606 09:56:06.404251 vtctl.go:2582] workflow did not start within 30s
MoveTables Error: rpc error: code = Unknown desc = workflow did not start within 30s
E0606 09:56:06.406961    5131 main.go:76] remote error: rpc error: code = Unknown desc = workflow did not start within 30s

After switching to a master-slave setup and resetting the GTIDs, I was able to successfully start the VReplication.

Thanks @jawabuu for your help!