[BUG] redis eval error: "this Redis command is not allowed from scripts"
momentumtools opened this issue · comments
Hi,
locally the asynqmon works perfectly, however now that I just deployed it to my heroku setup (using a redis enterprise cloud instance the chrome console log returns these errors:
UNKNOWN: UNKNOWN: redis eval error: ERR Error running script
(call to f_7574d838a4ddc18d626139a5521903c904f1404a):
@user_script:19: @user_script: 19: This Redis command is not allowed from scripts
I googled a bit and this is most likely due to the REDIS server being relatively strict.
Any idea how to circumvent this through an asynqmon configuration (because on the redis enterprise cloud I don't see any EVAL related configurations)
Greetings,
PJ
PS: love your queuing tool and have been in the game for quite some time now :)!
@momentumtools Thank you for opening this issue!
Would you mind providing additional info?
- Redis version number
- Which page (e.g. URL path) in the Web UI triggers this error
Hi @hibiken ,
thanks for the quick reply, here are the parameters
- Redis 6.2.3
- paths with errors: /jobs/api/queues
- paths without errors: /jobs/api/servers, /jobs/api/redis_info, /jobs/api/scheduler_entries, /jobs/api/queue_stats
PS: the support folks at redis enterprise ask me the specific LUA script that we are trying to execute to help me figure out which command it is that is blocked on my DB
Thanks!
I haven't confirmed it on my side, but my first guess is this MEMORY USAGE command we are calling in this Lua script . Specifically these two lines
https://github.com/hibiken/asynq/blob/dff2e3a33639338c648599bf6ea83f835feec4bf/internal/rdb/inspect.go#L236
https://github.com/hibiken/asynq/blob/dff2e3a33639338c648599bf6ea83f835feec4bf/internal/rdb/inspect.go#L253
I need to do a bit more research but my initial thought is that the issue is due to the restriction around non-deterministic commands in script; non-deterministic commands are not allowed in Lua script when replication is enabled. As documented in https://redis.io/commands/eval, we may want to disable replication when calling MEMORY USAGE from that script (https://redis.io/commands/eval#selective-replication-of-commands) Or use read-only EVAL command instead (this EVAL_RO is only supported from redis v7+ though, so maybe not realistic to use this). I will look into this issue in the next few days but if you have any updates, please let me know in this thread :)
Actually I take that back, since the redis version is above v4 in this case, script effect replication is used and since the Lua script is read-only, it should be allowed. Not sure where this error is coming from. Let me know if you hear back from the support team.
Hi @hibiken , thanks again for such fast support. I'm amazed :)!
I've forwarded https://github.com/hibiken/asynq/blob/dff2e3a33639338c648599bf6ea83f835feec4bf/internal/rdb/inspect.go#L225-L266 to the redis enterprise support, maybe it will ring a bell for them.
Is there a way for us to be surer which LUA script is triggering the error? a kind of debug mode to enable?
This is a good opportunity to find a bug so thank you for providing the info!
There's no debug mode to quickly figure out the problematic lua script unfortunately, but based on the fact that the error is returned when we make API calls to /api/queues endpoint, it tells me that this handler is causing the error. And if you look at the handler definition, it's most likely this call to inspector.GetQueueInfo is a culprit, which in turn calls this CurrentStat and this function executes two lua scripts this one and this one.
Hi @hibiken , I'll pass the first script to the redis enterprise support team as well, as I only pasted them the 2nd one you thought the error was in when you wrote #210 (comment)
@momentumtools Any update on this?
Nope, redis enterprise support is dead silent...
most likely I'll try with another redis provider and see if the same error occurs, if not it's redis enterprise their loss sadly...
Long story short, this EVAL error did not pop up yet with any other users, right?
@hibiken ok I got back from their support
The issue is in the script you mentioned, there are some MEMORY commands. Most MEMORY commands are blocked on Redis Enterprise Cloud due to the architecture differences between Enterprise and Open Source Redis.
I'll thus have to pick another redis provider, but so now when other people bump into this they'll know where to look ;)
@momentumtools Thank you for the update! Switching to another redis provider sounds like a non-trivial change, especially for some tooling issues. Maybe we should support turning off memory usage command in the script via a flag; For example, we could add a flag --disable_memory_usage_command to this Web UI binary. Let me know what you think!
In the api/queues interface, eval will be called twice and an error will be reported the second time
I rewrote the eval of redisClient, so I can print the logs
hey @hibiken how easy can you get that flag in to disable the check? Im running asynqmon on AWS with elastic cache and getting
UNKNOWN: UNKNOWN: redis eval error: ERR This Redis command is not allowed from script script: 0c80fc1868e3aae205155ae2017ce995a3a7cec0, on @user_script:19.