Replication is not working from appication/client side
Dwijad opened this issue · comments
Hi
I have three node dynomite cluster setup in VMWARE along with Redis server running in each of those three nodes. The replication is working fine if i use redis-cli. However one of the application that i am trying to integrate with dynomite is unable to read/write to the cluster.
Dynomite Version: dynomite-v0.6.21-rc2-19-gb56cf6a
Redis version: 6.2.6
redis-cli 6.2.6
CentOS 7
Node 1: 172.16.10.242
dyn_o_mite:
datacenter: dc-a
rack: rack1
dyn_listen: 0.0.0.0:7379
dyn_seed_provider: simple_provider
dyn_seeds:
- 172.16.10.48:7379:rack1:dc-b:0
- 172.16.10.49:7379:rack1:dc-c:0
listen: 172.16.10.242:8379
servers:
- 0.0.0.0:6379:1
tokens: '0'
secure_server_option: datacenter
pem_key_file: /root/dynomite/conf/dynomite.pem
data_store: 0
stats_listen: 172.16.10.242:22222
read_consistency : DC_QUORUM
write_consistency : DC_QUORUM
Node 2: 172.16.10.48
dyn_o_mite:
datacenter: dc-b
rack: rack1
dyn_seed_provider: simple_provider
dyn_listen: 0.0.0.0:7379
dyn_seeds:
- 172.16.10.49:7379:rack1:dc-c:0
- 172.16.10.242:7379:rack1:dc-a:0
listen: 172.16.10.48:8379
servers:
- 0.0.0.0:6379:1
tokens: '0'
secure_server_option: datacenter
pem_key_file: /root/dynomite/conf/dynomite.pem
data_store: 0
stats_listen: 172.16.10.48:22222
read_consistency : DC_QUORUM
write_consistency : DC_QUORUM
Node 3: 172.16.10.48
dyn_o_mite:
datacenter: dc-c
rack: rack1
dyn_listen: 0.0.0.0:7379
dyn_seed_provider: simple_provider
dyn_seeds:
- 172.16.10.242:7379:rack1:dc-a:0
- 172.16.10.48:7379:rack1:dc-b:0
listen: 172.16.10.49:8379
servers:
- 0.0.0.0:6379:1
tokens: '0'
secure_server_option: datacenter
pem_key_file: /root/dynomite/conf/dynomite.pem
data_store: 0
stats_listen: 172.16.10.49:22222
read_consistency : DC_QUORUM
write_consistency : DC_QUORUM
The replication is working fine if i use the redis-cli
[root@redis ~]# redis-cli -h 172.16.10.242 -p 8379
172.16.10.242:8379> SET C 1
OK
[root@redis-2 ~]# redis-cli -h 172.16.10.48 -p 8379
172.16.10.48:8379>
172.16.10.48:8379> KEYS *
1) "V"
2) "X"
3) "C"
However if i use my application to write to one of the node(172.16.10.242:8379) of dynomite cluster, i am getting following error:
[2022-04-23 10:04:37.154] redis_parse_req:2387 parsed bad req 2980 res 1 type 0 state 5
00000000 2a 31 0d 0a 24 35 0d 0a 4d 55 4c 54 49 0d 0a 2a |*1..$5..MULTI..*|
00000010 32 0d 0a 24 34 0d 0a 4c 4c 45 4e 0d 0a 24 35 36 |2..$4..LLEN..$56|
00000020 0d 0a 33 34 33 64 35 37 30 33 2d 66 30 65 32 2d |..343d5703-f0e2-|
00000030 33 66 33 64 2d 39 64 39 36 2d 30 31 65 35 38 37 |3f3d-9d96-01e587|
00000040 61 32 37 30 30 33 2e 72 65 70 6c 79 2e 63 65 6c |a27003.reply.cel|
00000050 65 72 79 2e 70 69 64 62 6f 78 0d 0a 2a 32 0d 0a |ery.pidbox..*2..|
00000060 24 34 0d 0a 4c 4c 45 4e 0d 0a 24 35 39 0d 0a 33 |$4..LLEN..$59..3|
00000070 34 33 64 35 37 30 33 2d 66 30 65 32 2d 33 66 33 |43d5703-f0e2-3f3|
00000080 64 2d 39 64 39 36 2d 30 31 65 35 38 37 61 32 37 |d-9d96-01e587a27|
00000090 30 30 33 2e 72 65 70 6c 79 2e 63 65 6c 65 72 79 |003.reply.celery|
000000a0 2e 70 69 64 62 6f 78 06 16 33 0d 0a 2a 32 0d 0a |.pidbox..3..*2..|
000000b0 24 34 0d 0a 4c 4c 45 4e 0d 0a 24 35 39 0d 0a 33 |$4..LLEN..$59..3|
000000c0 34 33 64 35 37 30 33 2d 66 30 65 32 2d 33 66 33 |43d5703-f0e2-3f3|
000000d0 64 2d 39 64 39 36 2d 30 31 65 35 38 37 61 32 37 |d-9d96-01e587a27|
000000e0 30 30 33 2e 72 65 70 6c 79 2e 63 65 6c 65 72 79 |003.reply.celery|
000000f0 2e 70 69 64 62 6f 78 06 16 36 0d 0a 2a 32 0d 0a |.pidbox..6..*2..|
00000100 24 34 0d 0a 4c 4c 45 4e 0d 0a 24 35 39 0d 0a 33 |$4..LLEN..$59..3|
00000110 34 33 64 35 37 30 33 2d 66 30 65 32 2d 33 66 33 |43d5703-f0e2-3f3|
00000120 64 2d 39 64 39 36 2d 30 31 65 35 38 37 61 32 37 |d-9d96-01e587a27|
00000130 30 30 33 2e 72 65 70 6c 79 2e 63 65 6c 65 72 79 |003.reply.celery|
00000140 2e 70 69 64 62 6f 78 06 16 39 0d 0a 2a 31 0d 0a |.pidbox..9..*1..|
00000150 24 34 0d 0a 45 58 45 43 0d 0a |$4..EXEC..|
[2022-04-23 10:04:37.154] core_close:418 close <CONN_CLIENT 0xa7f920 16 from '172.16.10.215:42820'> on event 00FF eof 0 done 0 rb 685 sb 20: Invalid argument
[2022-04-23 10:04:37.154] client_unref_internal_try_put:97 <CONN_CLIENT 0xa7f920 -1 from '172.16.10.215:42820'> unref owner <POOL 0xa31bb0 'dyn_o_mite'>
[2022-04-23 10:04:37.155] conn_recv_data:366 <CONN_CLIENT 0xa80360 15 from '172.16.10.215:42818'> recv eof rb 28 sb 14
[2022-04-23 10:04:37.155] core_close:418 close <CONN_CLIENT 0xa80360 15 from '172.16.10.215:42818'> on event 00FF eof 1 done 1 rb 28 sb 14
[2022-04-23 10:04:37.156] client_unref_internal_try_put:97 <CONN_CLIENT 0xa80360 -1 from '172.16.10.215:42818'> unref owner <POOL 0xa31bb0 'dyn_o_mite'>
[2022-04-23 10:04:37.169] proxy_accept:203 <CONN_PROXY 0xa333b0 8 listening on '172.16.10.242:8379'> accepted <CONN_CLIENT 0xa80360 15 from '172.16.10.215:42824'>
[2022-04-23 10:04:37.171] proxy_accept:203 <CONN_PROXY 0xa333b0 8 listening on '172.16.10.242:8379'> accepted <CONN_CLIENT 0xa7f920 16 from '172.16.10.215:42826'>
[2022-04-23 10:04:37.173] conn_recv_data:366 <CONN_CLIENT 0xa54a20 13 from '172.16.10.215:42822'> recv eof rb 28 sb 14
[2022-04-23 10:04:37.173] core_close:418 close <CONN_CLIENT 0xa54a20 13 from '172.16.10.215:42822'> on event 00FF eof 1 done 1 rb 28 sb 14
[2022-04-23 10:04:37.173] client_unref_internal_try_put:97 <CONN_CLIENT 0xa54a20 -1 from '172.16.10.215:42822'> unref owner <POOL 0xa31bb0 'dyn_o_mite'>
[2022-04-23 10:04:37.185] proxy_accept:203 <CONN_PROXY 0xa333b0 8 listening on '172.16.10.242:8379'> accepted <CONN_CLIENT 0xa54a20 13 from '172.16.10.215:42828'>
[2022-04-23 10:04:37.187] redis_parse_req:1583 parsed unsupported command 'MULTI'
[2022-04-23 10:04:37.187] redis_parse_req:2387 parsed bad req 3007 res 1 type 0 state 5
The error in the application side is:
redis.exceptions.ConnectionError: Connection closed by server.
I can reach the dynomite port from remote:
# nc -vz 172.16.10.242 8379
Ncat: Version 7.50 ( https://nmap.org/ncat )
Ncat: Connected to 172.16.10.242:8379.
Ncat: 0 bytes sent, 0 bytes received in 0.02 seconds.
It looks dynomite cluster is working. However i am not sure why the application is not able to write to the dynomite cluster.
Thanks
It was an issue within the application which was unable to write to the dynomite. Dynomite is working fine.
Also, It looks dynomite is not updating its codebase from last few years. After exploring alternatives to master-master redis cluster, i have found keydb a better option for my requirements.