Infrequent and weird #<RuntimeError: NOAUTH Authentication required> in heroku/rediscloud environment

Question

Infrequent and weird #<RuntimeError: NOAUTH Authentication required> in heroku/rediscloud environment

chriso0710 opened this issue 5 years ago · comments

Hi guys,

ohm has been working great (still using version 2.3) in my Sinatra app for years.

But I'm seeing some weird periodic error in a heroku environment. I already opened a ticket with heroku, but they have no idea at the moment, other than switching to a different redis client.

My app uses the Redis Cloud heroku addon. The app does a redis fetch for every request it gets. Redis Cloud has also been working great for months.
I have the REDIS config var set for heroku. The app reads it via ENV to get the redis config url. There is only a single place in the code where the redis config gets set:

Ohm.redis = Redic.new(ENV['REDIS'])

But I'm seeing daily errors from my app. The error occurs only after, before or during the heroku daily dyno restart. So only once per day!

NoMethodError: undefined method `each' for #<RuntimeError: NOAUTH Authentication required>
File "/app/vendor/bundle/ruby/2.3.0/gems/ohm-2.3.0/lib/ohm.rb" line 144 in block in fetch
File "/app/vendor/bundle/ruby/2.3.0/gems/ohm-2.3.0/lib/ohm.rb" line 699 in synchronize
File "/app/vendor/bundle/ruby/2.3.0/gems/ohm-2.3.0/lib/ohm.rb" line 699 in synchronize
File "/app/vendor/bundle/ruby/2.3.0/gems/ohm-2.3.0/lib/ohm.rb" line 143 in fetch
File "/app/vendor/bundle/ruby/2.3.0/gems/ohm-2.3.0/lib/ohm.rb" line 457 in sort
File "/app/vendor/bundle/ruby/2.3.0/gems/ohm-2.3.0/lib/ohm.rb" line 498 in first

I would appreciate any ideas or hints for debugging or resolving this issue. Thanks.

Christian

Michel Martens · Answer 1 · Tue Jan 29 2019 04:48:21 GMT+0800 (China Standard Time)

I know what's happening from the point of view of the server, and I think the client should retry. I'll check this and get back to you.

Michel Martens · Answer 2 · Tue Jan 29 2019 06:14:30 GMT+0800 (China Standard Time)

Is there a chance for you to try with the latest version?

Christian Ott · Answer 3 · Tue Jan 29 2019 13:42:10 GMT+0800 (China Standard Time)

Yeah, I know. I still have the update on my todo list, see #223. But ohm has been running smoothly most of the time, so I was delaying the update.
Looks like I need to tackle this quite soon. Though it still might take some days until I find the time.

Do you have any ideas to mitigate the error for 2.3 in the meantime?

Michel Martens · Answer 4 · Tue Jan 29 2019 17:31:27 GMT+0800 (China Standard Time)

Can you tell me which version of Redic you have installed?

Christian Ott · Answer 5 · Tue Jan 29 2019 17:41:01 GMT+0800 (China Standard Time)

redic 1.5.0

Michel Martens · Answer 6 · Wed Jan 30 2019 15:28:16 GMT+0800 (China Standard Time)

What is stopping you from upgrading?

Christian Ott · Answer 7 · Wed Jan 30 2019 17:24:36 GMT+0800 (China Standard Time)

:-) No time, too much stuff going on.
But will do it for our stage environment.

Michel Martens · Answer 8 · Wed Jan 30 2019 17:31:44 GMT+0800 (China Standard Time)

I had troubles yesterday installing msgpack, needed for Ohm 2.3.0. And I was thinking about just that: upgrading Ohm in staging to see how it goes. I wasn't able to reproduce it with the latest version, so that would be one idea. I can as try again with msgpack later if your experiment fails.

Christian Ott · Answer 9 · Wed Jan 30 2019 17:34:05 GMT+0800 (China Standard Time)

Thank you @soveran, I will keep you posted.

Christian Ott · Answer 10 · Thu Jan 31 2019 01:19:00 GMT+0800 (China Standard Time)

I did the gem update (ohm 3.1.1, redic 1.5.1) and a lot of my tests started to fail immediately.

The reason seems to be the following. I have this model:

class User < Ohm::Model

    attribute :uid
    attribute :name
    attribute :brand
    ...

    index :uid
    index :provider
    index :brand
...
end

In ohm 2.3 this worked

User.find(uid: "409590").first

and returned a valid user object.

In ohm 3.1 the same command fails

User.find(uid: "409590").first

and returns nil, which breaks my tests.

I have to do a

User.find(uid: "409590").to_a.first
#<User:0x007fb6ab933c30 @attributes={:uid=>"409590", :name=>"Christian Ott", ...>

to get a user object. Is this expected behaviour or am I missing something here?

Christian Ott · Answer 11 · Thu Jan 31 2019 03:52:14 GMT+0800 (China Standard Time)

Similar problem with ohm3:

My app uses

User.all.size
0

But correct size gets returned only with

User.all.to_a.size
3

Michel Martens · Answer 12 · Thu Jan 31 2019 16:47:48 GMT+0800 (China Standard Time)

Regarding User.find(uid: "409590").first, it may have to do with how nil values are handled when indexing and the changes between the two versions you are testing. You may need to iterate over all objects and save them so that the indices get updated.

Michel Martens · Answer 13 · Thu Jan 31 2019 16:48:38 GMT+0800 (China Standard Time)

(If that's not the case, let me know. I'm offline a lot these days and until February 6th, but I'm following up on this issue every morning.)

Christian Ott · Answer 14 · Thu Jan 31 2019 17:01:56 GMT+0800 (China Standard Time)

I do not have nil values for the field uid in my data. Nevertheless I saved all objects to rebuild the index.
But the problem remains: The first method on the find always returns nil.

Michel Martens · Answer 15 · Thu Jan 31 2019 17:09:38 GMT+0800 (China Standard Time)

OK, I may need to see the definition of your model. If you want to keep it private, contact me by email.

Christian Ott · Answer 16 · Thu Jan 31 2019 17:17:37 GMT+0800 (China Standard Time)

Full ohm definition, without any methods:

class User < Ohm::Model

    attribute :uid
    attribute :name
    attribute :provider
    attribute :email
    attribute :image_url
    attribute :url
    attribute :lastlogin
    attribute :registered
    attribute :admin
    attribute :brand
    attribute :address
    attribute :locked
    attribute :locale
    attribute :stripeid
    attribute :stripecountrycode

    index :uid
    index :provider
    index :brand

    collection :projects, :Project
    reference :plan, :Plan

end

Michel Martens · Answer 17 · Thu Jan 31 2019 17:32:25 GMT+0800 (China Standard Time)

Cool. I created some instances and the result is what I expected. Can you create some instances that will allow me to reproduce the error?

Christian Ott · Answer 18 · Thu Jan 31 2019 17:56:40 GMT+0800 (China Standard Time)

This is what I get in irb:

> u = User.new(uid: "4711", provider: "github", name: "Chris", brand: "sosdev")
#<User:0x007fa1acd21310 @attributes={:uid=>"4711", :provider=>"github", :name=>"Chris", :brand=>"sosdev", :locale=>"en"}, @_memo={}>
> u.save
#<User:0x007fa1acd21310 @attributes={:uid=>"4711", :provider=>"github", :name=>"Chris", :brand=>"sosdev", :locale=>"en"}, @_memo={}, @id="43">
> u.inspect
"#<User:0x007fa1acd21310 @attributes={:uid=>\"4711\", :provider=>\"github\", :name=>\"Chris\", :brand=>\"sosdev\", :locale=>\"en\"}, @_memo={}, @id=\"43\">"
> User.all.size
0
> User.find(uid: "4711").first
nil
> User.find(uid: "4711").to_a.first
#<User:0x007fa1acf42180 @attributes={:uid=>"4711", :name=>"Chris", :provider=>"github", :brand=>"sosdev", :locale=>"en"}, @_memo={}, @id="43">

Michel Martens · Answer 19 · Thu Jan 31 2019 18:11:13 GMT+0800 (China Standard Time)

If I do exactly that, I get the right results. If you try that after running FLUSHDB, does it work?

Christian Ott · Answer 20 · Thu Jan 31 2019 18:23:18 GMT+0800 (China Standard Time)

No, sorry, now it gets weird. I have the same problems with a totally empty DB.
I'm using redis 2.8 locally on a Mac. Have not changed anything on the server.

127.0.0.1:6379> flushdb
OK
127.0.0.1:6379> keys *
(empty list or set)

 > u = User.new(uid: "4711", provider: "github", name: "Chris", brand: "sosdev")
#<User:0x007ff284283bf0 @attributes={:uid=>"4711", :provider=>"github", :name=>"Chris", :brand=>"sosdev", :locale=>"en"}, @_memo={}>
> u.save
#<User:0x007ff284283bf0 @attributes={:uid=>"4711", :provider=>"github", :name=>"Chris", :brand=>"sosdev", :locale=>"en"}, @_memo={}, @id="1">
> User.all.size
0
> User.find(uid: "4711").first
nil
> User.all.to_a.size
1

# Server
redis_version:2.8.24
redis_git_sha1:00000000
redis_git_dirty:0
redis_build_id:2828360fe221885d
redis_mode:standalone
os:Darwin 17.3.0 x86_64
arch_bits:64
multiplexing_api:kqueue
gcc_version:4.2.1
process_id:57152
run_id:f6fed8973a89d3f1d24ac2564f24e60a60af6659
tcp_port:6379
uptime_in_seconds:4843
uptime_in_days:0
hz:10
lru_clock:5426057
config_file:/Users/co/.redis/redis.conf

Thanks for your help BTW!

Michel Martens · Answer 21 · Thu Jan 31 2019 19:07:16 GMT+0800 (China Standard Time)

I've just tried with that same version (2.8.24) and got the right results. Try to run this file:

require "ohm"

Ohm.redis.call("FLUSHDB")
Ohm.redis.call("SCRIPT", "FLUSH")

class Plan < Ohm::Model
  attribute :name
end

class User < Ohm::Model
  attribute :uid
  attribute :name
  attribute :provider
  attribute :email
  attribute :image_url
  attribute :url
  attribute :lastlogin
  attribute :registered
  attribute :admin
  attribute :brand
  attribute :address
  attribute :locked
  attribute :locale
  attribute :stripeid
  attribute :stripecountrycode

  index :uid
  index :provider
  index :brand

  collection :projects, :Project
  reference :plan, :Plan
end

p User.create(uid: "4711", provider: "github", name: "Chris", brand: "sosdev")
p User.all.size
p User.find(uid: "4711").first
p User.find(uid: "4711").to_a.first

Christian Ott · Answer 22 · Thu Jan 31 2019 19:11:22 GMT+0800 (China Standard Time)

Yep, this works!

#<User:0x007fb077b3f040 @attributes={:uid=>"4711", :provider=>"github", :name=>"Chris", :brand=>"sosdev"}, @_memo={}, @id="1">
1
#<User:0x007fb077b3cac0 @attributes={:uid=>"4711", :name=>"Chris", :provider=>"github", :brand=>"sosdev"}, @_memo={}, @id="1">
#<User:0x007fb077007c68 @attributes={:uid=>"4711", :name=>"Chris", :provider=>"github", :brand=>"sosdev"}, @_memo={}, @id="1">

Only difference seems the

Ohm.redis.call("SCRIPT", "FLUSH")

Christian Ott · Answer 23 · Thu Jan 31 2019 19:16:55 GMT+0800 (China Standard Time)

But back in my irb console the problem returns. Maybe it has something to do with json gems or model methods?

> User.all.to_a
[
    [0] #<User:0x007ff93b2f0ff8 @attributes={:uid=>"4711", :name=>"Chris", :provider=>"github", :brand=>"sosdev", :locale=>"en"}, @_memo={}, @id="1">
]
> User.find(uid: "4711").first
nil

This is my initialize method in the User class. Looks harmless to me:

    def initialize(data)
        super(data)
        @attributes[:locale] = "en" if !attributes[:locale]
    end

Michel Martens · Answer 24 · Thu Jan 31 2019 20:03:18 GMT+0800 (China Standard Time)

Can you try flushing the DB and the scripts when you enter IRB?

>> Ohm.redis.call("FLUSHDB")
>> Ohm.redis.call("SCRIPT", "FLUSH")

Christian Ott · Answer 25 · Thu Jan 31 2019 20:03:43 GMT+0800 (China Standard Time)

We are getting closer. I think I found the reason for the error. I am requiring

require 'active_support/all'

because I am using methods from active_support, although I am not using rails for this project.
I think this must have something to do with JSON processing in active_support.

Michel Martens · Answer 26 · Thu Jan 31 2019 20:24:10 GMT+0800 (China Standard Time)

Ah, that could be the culprit. Let me know if it works fine without requiring active_support/all.

Christian Ott · Answer 27 · Thu Jan 31 2019 20:32:56 GMT+0800 (China Standard Time)

Yes, without active_support/all it all works! Thank you very much for your help so far.
Unfortunately I need active_support. Do you have an idea how to prevent active_support interfering with JSON operations?

Michel Martens · Answer 28 · Thu Jan 31 2019 20:34:13 GMT+0800 (China Standard Time)

I guess it is overriding to_json, but we can track down the problem. Which version of activesupport are you using?

Michel Martens · Answer 29 · Thu Jan 31 2019 21:03:26 GMT+0800 (China Standard Time)

I remembered something similar that was fixed a while ago: soveran/nest#7

Christian Ott · Answer 30 · Thu Jan 31 2019 21:05:55 GMT+0800 (China Standard Time)

Still using an old version: activesupport (3.2.22.5)
20+ comments later the original problem seems to turn into some kind of dependency hell... That's exactly the reason I was hesitant with updating ;-)

Christian Ott · Answer 31 · Thu Jan 31 2019 21:16:59 GMT+0800 (China Standard Time)

When doing this after requiring active_support it works again:
https://stackoverflow.com/a/12324793/2520015

This might be a somewhat reasonable solution for me...

Michel Martens · Answer 32 · Thu Jan 31 2019 21:23:52 GMT+0800 (China Standard Time)

Try this, first locate Nest:

$ gem which nest

Then modify it and add this method:

def to_json(*args)
  @ns.to_json(*args)
end

Retry after that and let me know if it works.

Christian Ott · Answer 33 · Thu Jan 31 2019 21:28:24 GMT+0800 (China Standard Time)

Yes, patching nest works together with requiring active_supprt/all !

Michel Martens · Answer 34 · Thu Jan 31 2019 21:35:10 GMT+0800 (China Standard Time)

Great, I will check if that change doesn't interfere with anything else.

Christian Ott · Answer 35 · Thu Jan 31 2019 21:50:07 GMT+0800 (China Standard Time)

Thanks again @soveran for your help! That was fantastic.

I'm going to run my tests again and then push to the staging environment at heroku to see if this solves the original NOAUTH error.

Michel Martens · Answer 36 · Thu Jan 31 2019 21:53:48 GMT+0800 (China Standard Time)

I just pushed a new version of Nest with that change. I'm pesimistic about this change solving the NOAUTH error, but in any case it feels like we are making progress :-)

Christian Ott · Answer 37 · Fri Feb 01 2019 19:44:13 GMT+0800 (China Standard Time)

All tests are successful.
I pushed the update to our stage env yesterday evening. NOAUTH error has not been seen yet.

Michel Martens · Answer 38 · Sat Feb 02 2019 16:31:53 GMT+0800 (China Standard Time)

Excellent! Keep me posted :-)

Michel Martens · Answer 39 · Wed Feb 13 2019 00:08:08 GMT+0800 (China Standard Time)

@chriso0710 Have you run into any issues yet? I suddendly remembered and was wondering about what happened.

Christian Ott · Answer 40 · Wed Feb 13 2019 01:26:35 GMT+0800 (China Standard Time)

Thank you for asking.
No issues so far. Currently the ohm update is installed only in the stage environment. Not in production, which gets a lot more requests. Will probably take some weeks of testing before we move the update into production, though...

Do you have any idea what exactly could have caused the NOAUTH error in the 2.3 release?

Michel Martens · Answer 41 · Mon Mar 11 2019 21:58:07 GMT+0800 (China Standard Time)

I think we can close this for now. My guess is that the NOAUTH error was related to what we fixed. Feel free to reopen this issue if the problem returns. Thanks a lot for helping me out with this!

Christian Ott · Answer 42 · Mon Mar 11 2019 22:03:44 GMT+0800 (China Standard Time)

Yeah, ok. Thanks for your help!