conflict with opentelemetry-instrumentation-redis 0.15

Question

conflict with opentelemetry-instrumentation-redis 0.15

mjobin-mdsol opened this issue 3 years ago · comments

I am encountering a weird race condition involving adding the scout_apm gem to an app with uses delayed_jobs and opentelemetry-instrumentation-redis

seems like it creates an infinite loop between the two gems


51 | 03/23/202113:30:17.793 +0900 | /home/app/web/vendor/bundle/ruby/2.6.0/gems/scout_apm-4.0.4/lib/scout_apm/tracer.rb:34:in `instrument'
-- | -- | --
52 | 03/23/202113:30:17.793 +0900 | /home/app/web/vendor/bundle/ruby/2.6.0/gems/scout_apm-4.0.4/lib/scout_apm/instruments/redis.rb:32:in `block in call_with_scout_instruments'
53 | 03/23/202113:30:17.793 +0900 | /home/app/web/vendor/bundle/ruby/2.6.0/gems/opentelemetry-instrumentation-redis-0.15.0/lib/opentelemetry/instrumentation/redis/patches/client.rb:18:in `call'
54 | 03/23/202113:30:17.793 +0900 | /home/app/web/vendor/bundle/ruby/2.6.0/gems/opentelemetry-api-0.15.0/lib/opentelemetry/trace/tracer.rb:29:in `in_span'
55 | 03/23/202113:30:17.793 +0900 | /home/app/web/vendor/bundle/ruby/2.6.0/gems/opentelemetry-api-0.15.0/lib/opentelemetry/trace.rb:82:in `with_span'
56 | 03/23/202113:30:17.793 +0900 | /home/app/web/vendor/bundle/ruby/2.6.0/gems/opentelemetry-api-0.15.0/lib/opentelemetry/context.rb:62:in `with_value'
57 | 03/23/202113:30:17.793 +0900 | /home/app/web/vendor/bundle/ruby/2.6.0/gems/opentelemetry-api-0.15.0/lib/opentelemetry/trace.rb:82:in `block in with_span'
58 | 03/23/202113:30:17.793 +0900 | /home/app/web/vendor/bundle/ruby/2.6.0/gems/opentelemetry-api-0.15.0/lib/opentelemetry/trace/tracer.rb:29:in `block in in_span'
59 | 03/23/202113:30:17.793 +0900 | /home/app/web/vendor/bundle/ruby/2.6.0/gems/opentelemetry-instrumentation-redis-0.15.0/lib/opentelemetry/instrumentation/redis/patches/client.rb:23:in `block in call'
60 | 03/23/202113:30:17.793 +0900 | /home/app/web/vendor/bundle/ruby/2.6.0/gems/scout_apm-4.0.4/lib/scout_apm/instruments/redis.rb:31:in `call_with_scout_instruments'
61 | 03/23/202113:30:17.793 +0900 | /home/app/web/vendor/bundle/ruby/2.6.0/gems/scout_apm-4.0.4/lib/scout_apm/tracer.rb:44:in `instrument'
62 | 03/23/202113:30:17.793 +0900 | /home/app/web/vendor/bundle/ruby/2.6.0/gems/scout_apm-4.0.4/lib/scout_apm/tracer.rb:34:in `instrument'
63 | 03/23/202113:30:17.793 +0900 | /home/app/web/vendor/bundle/ruby/2.6.0/gems/scout_apm-4.0.4/lib/scout_apm/instruments/redis.rb:32:in `block in call_with_scout_instruments'
64 | 03/23/202113:30:17.793 +0900 | /home/app/web/vendor/bundle/ruby/2.6.0/gems/opentelemetry-instrumentation-redis-0.15.0/lib/opentelemetry/instrumentation/redis/patches/client.rb:18:in `call'
65 | 03/23/202113:30:17.793 +0900 | /home/app/web/vendor/bundle/ruby/2.6.0/gems/opentelemetry-api-0.15.0/lib/opentelemetry/trace/tracer.rb:29:in `in_span'
66 | 03/23/202113:30:17.793 +0900 | /home/app/web/vendor/bundle/ruby/2.6.0/gems/opentelemetry-api-0.15.0/lib/opentelemetry/trace.rb:82:in `with_span'
67 | 03/23/202113:30:17.793 +0900 | /home/app/web/vendor/bundle/ruby/2.6.0/gems/opentelemetry-api-0.15.0/lib/opentelemetry/context.rb:62:in `with_value'
68 | 03/23/202113:30:17.793 +0900 | /home/app/web/vendor/bundle/ruby/2.6.0/gems/opentelemetry-api-0.15.0/lib/opentelemetry/trace.rb:82:in `block in with_span'
69 | 03/23/202113:30:17.793 +0900 | /home/app/web/vendor/bundle/ruby/2.6.0/gems/opentelemetry-api-0.15.0/lib/opentelemetry/trace/tracer.rb:29:in `block in in_span'
70 | 03/23/202113:30:17.793 +0900 | /home/app/web/vendor/bundle/ruby/2.6.0/gems/opentelemetry-instrumentation-redis-0.15.0/lib/opentelemetry/instrumentation/redis/patches/client.rb:23:in `block in call'
71 | 03/23/202113:30:17.793 +0900 | /home/app/web/vendor/bundle/ruby/2.6.0/gems/scout_apm-4.0.4/lib/scout_apm/instruments/redis.rb:31:in `call_with_scout_instruments'
72 | 03/23/202113:30:17.793 +0900 | /home/app/web/vendor/bundle/ruby/2.6.0/gems/scout_apm-4.0.4/lib/scout_apm/tracer.rb:44:in `instrument'
73 | 03/23/202113:30:17.793 +0900 | /home/app/web/vendor/bundle/ruby/2.6.0/gems/scout_apm-4.0.4/lib/scout_apm/tracer.rb:34:in `instrument'
74 | 03/23/202113:30:17.793 +0900 | /home/app/web/vendor/bundle/ruby/2.6.0/gems/scout_apm-4.0.4/lib/scout_apm/instruments/redis.rb:32:in `block in call_with_scout_instruments'
75 | 03/23/202113:30:17.793 +0900 | /home/app/web/vendor/bundle/ruby/2.6.0/gems/opentelemetry-instrumentation-redis-0.15.0/lib/opentelemetry/instrumentation/redis/patches/client.rb:18:in `call'

Mathieu Jobin · Answer 1 · Thu Mar 25 2021 16:26:10 GMT+0800 (China Standard Time)

This is obviously a mix-&-match for Module#prepend and legacy method aliasing.

opentelemetry/instrumentation/redis/patches/client.rb:23 calls super, which is call_with_scout_instruments
and then call_with_scout_instruments is now an alias of OpenTelemetry method prepend patching, which calls Scout again .. and again, creating the infinite loop

Please update to use Module#prepend which requires ruby 2.0 as a private method, 2.1 as a public method ;)

Dave Anderson · Answer 2 · Thu Mar 25 2021 22:53:41 GMT+0800 (China Standard Time)

@mjobin-mdsol Thanks for the report. You're correct that it's a conflict in which method we use to instrument. We'll update the instrumentation to use prepend.

Mathieu Jobin · Answer 3 · Wed Mar 31 2021 09:44:36 GMT+0800 (China Standard Time)

@dlanderson if that can help, I came up with this master...mjobin-mdsol:testing-with-opentelemetry

Mathieu Jobin · Answer 4 · Tue Apr 06 2021 16:10:47 GMT+0800 (China Standard Time)

any updates here?

Mathieu Jobin · Answer 5 · Thu Apr 22 2021 14:24:03 GMT+0800 (China Standard Time)

@dlanderson your dev branch works for this
problem solved, but expecting an official release

Mathieu Jobin · Answer 6 · Mon May 24 2021 09:24:21 GMT+0800 (China Standard Time)

any plans on merging https://github.com/scoutapp/scout_apm_ruby/tree/prepend_redis for this ?

Mathieu Jobin · Answer 7 · Thu Jul 15 2021 12:59:28 GMT+0800 (China Standard Time)

works fine with v4.1.1

Mathieu Jobin · Answer 8 · Thu Aug 26 2021 21:08:11 GMT+0800 (China Standard Time)

heads up, you'll hit this problem again with newrelic_rpm (7.2.0)