Are Move Commands supported?
eugenemiretsky opened this issue · comments
Getting the error following error
Caused by: scredis.exceptions.RedisClusterErrorResponseException: MOVED 14769 172.21.150.26:6379
at scredis.protocol.Request.complete(Request.scala:49)
at scredis.io.DecoderActor$$anonfun$receive$1.applyOrElse(DecoderActor.scala:34)
at akka.actor.Actor.aroundReceive(Actor.scala:517)
at akka.actor.Actor.aroundReceive$(Actor.scala:515)
at scredis.io.DecoderActor.aroundReceive(DecoderActor.scala:12)
at akka.actor.ActorCell.receiveMessage(ActorCell.scala:588)
at akka.actor.ActorCell.invoke(ActorCell.scala:557)
at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:258)
at akka.dispatch.Mailbox.run(Mailbox.scala:225)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:748)
State of cluster is
redis-cli cluster nodes
3be984a5f88b479ac8817aeec6093e2caccb43fd 172.21.149.85:6379@16379 slave 6d37ca21137d672cd88725b453c447c72634371a 0 1572564674580 12 connected
01902483cac3551c338c5e45efc1aa0c4639aad3 172.21.149.69:6379@16379 slave a17660047bd1f4fa75b98ab3e6ebc9ad7813ba30 0 1572564674079 17 connected
4b97a2f4b8a785a9fbfd96e5729d490a119ed692 172.21.150.21:6379@16379 slave 9468adead5cae7b3dde35ecb5224a5871bd324d2 0 1572564674679 16 connected
ecde9ec841ea3e51e4f0d9450dea4effadfe5629 172.21.149.87:6379@16379 slave 2b1e370c436643d71c57a31045bf9766fb664558 0 1572564673000 18 connected
2b1e370c436643d71c57a31045bf9766fb664558 172.21.150.29:6379@16379 master - 0 1572564673576 7 connected 8937-10425
9468adead5cae7b3dde35ecb5224a5871bd324d2 172.21.150.17:6379@16379 master - 0 1572564673676 5 connected 5958-7446
5c1ba84f4c0aa23cf038077ac30f5dfcd44a6b54 172.21.149.105:6379@16379 slave b6d4081ec29735702ffcec80c88813c9b46aa6fd 0 1572564673000 20 connected
d74469591cebae23d6fd3f50148a37df0b7ab2ac 172.21.150.26:6379@16379 master - 0 1572564673000 10 connected 13405-14894
fc26735a287f07a701b0397618b660e90a734aec 172.21.150.193:6379@16379 master - 0 1572564674579 2 connected 1489-2978
6d37ca21137d672cd88725b453c447c72634371a 172.21.150.168:6379@16379 master - 0 1572564674579 1 connected 0-1488
b6d4081ec29735702ffcec80c88813c9b46aa6fd 172.21.150.191:6379@16379 myself,master - 0 1572564672000 9 connected 11916-13404
a7ad6a789227e9456a850378a0b65bdbdb757bbc 172.21.150.110:6379@16379 master - 0 1572564673576 3 connected 2979-4467
23afc8b5e32f3952ac13df54ea3869648436a419 172.21.150.131:6379@16379 master - 0 1572564674000 8 connected 10426-11915
7bbb324c0afe056ca00b0e488563ca18b51eae5b 172.21.149.98:6379@16379 slave fe7848852c7b942a1cce6456af8440d57d32eaf5 0 1572564674580 15 connected
30a77a4713e99b34f298dbddf32183859f803f17 172.21.149.227:6379@16379 slave fc26735a287f07a701b0397618b660e90a734aec 0 1572564673577 13 connected
fe7848852c7b942a1cce6456af8440d57d32eaf5 172.21.150.133:6379@16379 master - 0 1572564673576 4 connected 4468-5957
cc17cafbd5358ebb991e0b2882c29c80a0e409b7 172.21.149.60:6379@16379 slave d74469591cebae23d6fd3f50148a37df0b7ab2ac 0 1572564674000 21 connected
a17660047bd1f4fa75b98ab3e6ebc9ad7813ba30 172.21.150.146:6379@16379 master - 0 1572564673000 6 connected 7447-8936
e16127a1cf8fdd7d360bf41a57015159a27aa84a 172.21.149.245:6379@16379 slave 23afc8b5e32f3952ac13df54ea3869648436a419 0 1572564674079 19 connected
6c3e19d7469c5dc9bf3fd7a3b03d90e476ccfc3e 172.21.149.83:6379@16379 slave a7ad6a789227e9456a850378a0b65bdbdb757bbc 0 1572564674000 14 connected
9f90ba6cf83eb413c0c0ddf4610c343d39afc09c 172.21.149.106:6379@16379 slave d5e761de03636bd5c60b93c85cd3c9f9406bae49 0 1572564673000 22 connected
d5e761de03636bd5c60b93c85cd3c9f9406bae49 172.21.149.181:6379@16379 master - 0 1572564674000 11 connected 14895-16383
What could be the issue?
Yes MOVED commands should be supported by scredis. They indicate that data should be found on another node and request should be retired to that node.
There might be a bug because those kind of interactions are not easy to test but overall it is supported.
What version of scredis / Redis do you use?
What actions do you invoke to trigger this error?
Can you paste also logs? There should be information what actually raised this exception.
Version 2.1.7
Application logs? That's really the only thing we got.
We started getting these errors after setting receive-timeout=150ms.
Two things started happening
- The number of Redis connections grow linearly, until Redis runs out of connections and CPU spikes up (the limit can be increased by increasing ulimit, but the growth in connections is still a problem). Only way to resolve it is to restart the applications nodes
- After recovery, the application nodes that have no been restarted show the MOVE error from above, all new applications nodes are working fine.
@kpbochenek I think ClusterConnection.updateServerErrors is leaking connections. It removes an entry from connections, but never deletes it - hence all the actors created by that connections Actor continue running
Thanks for investigating it!
Could you try to make a PR to fix this?
It should be enough to changeconnections to be a Map[..., (AkkaNonBlockingConnection, Int)]
define a function to properly terminate actor and call it in a place you found on AkkaNonBlockingConnection?
@kpbochenek can you give me permissions to create a branch and push?
Make a fork (top right "Fork" button), clone your fork, make a branch and make changes, commit and make cross repo PR.
That's a standard process when you contribute to open source projects.