Usage with smallrye-messaging results in Timeout exception for the first message
priyanknarvekar opened this issue · comments
for kubernetes service-discovery, If we use stork to resolve and select instances in the consumer of the smallrye-messaging,
during the consumption of the first message stork fails to select an instance throwing Timeout Exception.
during all the subsequent messages, selectInstance works fine. Timeout Exception is reproducible and doesn't matter the amount of duration.
Sample code for the mq consumer which uses stork in the body :
https://github.com/priyanknarvekar/rmq-consumer-stork/blob/main/src/main/java/com/example/RMQConsumer.java
Corresponding mq publisher
https://github.com/priyanknarvekar/rmq-publisher/blob/main/src/main/java/com/example/RMQPublisher.java
Log from the consumer :
$ quarkus dev
[INFO] Scanning for projects...
[INFO]
[INFO] -------------------< com.example:rmq-consumer-stork >-------------------
[INFO] Building rmq-consumer-stork 1.0
[INFO] --------------------------------[ jar ]---------------------------------
[INFO]
[INFO] --- quarkus-maven-plugin:2.12.2.Final:dev (default-cli) @ rmq-consumer-stork ---
[INFO] Invoking org.apache.maven.plugins:maven-resources-plugin:2.6:resources) @ rmq-consumer-stork
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 1 resource
[INFO] Invoking io.quarkus.platform:quarkus-maven-plugin:2.12.2.Final:generate-code) @ rmq-consumer-stork
[INFO] Invoking org.apache.maven.plugins:maven-compiler-plugin:3.8.1:compile) @ rmq-consumer-stork
[INFO] Nothing to compile - all classes are up to date
[INFO] Invoking org.apache.maven.plugins:maven-resources-plugin:2.6:testResources) @ rmq-consumer-stork
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory /Users/someuser/workspaces/someworkspace/rmq-consumer-stork/src/test/resources
[INFO] Invoking io.quarkus.platform:quarkus-maven-plugin:2.12.2.Final:generate-code-tests) @ rmq-consumer-stork
[INFO] Invoking org.apache.maven.plugins:maven-compiler-plugin:3.8.1:testCompile) @ rmq-consumer-stork
[INFO] Changes detected - recompiling the module!
[ERROR] Port 5005 in use, not starting in debug mode
[
[34;1mINFO [m] Checking for existing resources in: /Users/someuser/workspaces/someworkspace/rmq-consumer-stork/src/main/kubernetes.
__ ____ __ _____ ___ __ ____ ______
--/ __ \/ / / / _ | / _ \/ //_/ / / / __/
-/ /_/ / /_/ / __ |/ , _/ ,< / /_/ /\ \
--\___\_\____/_/ |_/_/|_/_/|_|\____/___/
2022-10-17 05:26:33,427 INFO [io.qua.grp.run.GrpcServerRecorder] (Quarkus Main Thread) Registering gRPC reflection service
2022-10-17 05:26:33,582 INFO [io.qua.grp.run.GrpcServerRecorder] (vert.x-eventloop-thread-0) gRPC Server started on 0.0.0.0:9000 [SSL enabled: false]
2022-10-17 05:26:34,238 INFO [io.sma.rea.mes.rabbitmq] (Quarkus Main Thread) SRMSG17036: RabbitMQ broker configured to localhost:57057 for channel from-rabbitmq
2022-10-17 05:26:34,254 INFO [io.sma.rea.mes.rabbitmq] (Quarkus Main Thread) SRMSG17006: Establishing connection with RabbitMQ broker for channel `from-rabbitmq`
2022-10-17 05:26:34,259 INFO [io.sma.rea.mes.rabbitmq] (Quarkus Main Thread) SRMSG17007: Connection with RabbitMQ broker established for channel `from-rabbitmq`
2022-10-17 05:26:34,260 INFO [io.ver.rab.imp.RabbitMQClientImpl] (Quarkus Main Thread) Starting rabbitmq client
2022-10-17 05:26:34,269 INFO [io.quarkus] (Quarkus Main Thread) rmq-consumer-stork 1.0 on JVM (powered by Quarkus 2.12.2.Final) started in 5.068s.
2022-10-17 05:26:34,270 INFO [io.quarkus] (Quarkus Main Thread) Profile dev activated. Live Coding activated.
2022-10-17 05:26:34,270 INFO [io.quarkus] (Quarkus Main Thread) Installed features: [cdi, grpc-server, kubernetes, kubernetes-client, smallrye-context-propagation, smallrye-reactive-messaging, smallrye-reactive-messaging-rabbitmq, vertx]
2022-10-17 05:26:34,388 INFO [io.sma.rea.mes.rabbitmq] (vert.x-eventloop-thread-0) SRMSG17000: RabbitMQ Receiver listening address rmq-stork-test
2022-10-17 05:26:40,139 ERROR [io.sma.rea.mes.provider] (vert.x-worker-thread-0) SRMSG00200: The method com.example.RMQConsumer#consume has thrown an exception: io.smallrye.mutiny.TimeoutException
at io.smallrye.mutiny.operators.uni.UniBlockingAwait.await(UniBlockingAwait.java:64)
at io.smallrye.mutiny.groups.UniAwait.atMost(UniAwait.java:65)
at com.example.RMQConsumer.consume(RMQConsumer.java:28)
at com.example.RMQConsumer_Subclass.consume$$superforward1(Unknown Source)
at com.example.RMQConsumer_Subclass$$function$$1.apply(Unknown Source)
at io.quarkus.arc.impl.AroundInvokeInvocationContext.proceed(AroundInvokeInvocationContext.java:53)
at io.quarkus.arc.runtime.devconsole.InvocationInterceptor.proceed(InvocationInterceptor.java:62)
at io.quarkus.arc.runtime.devconsole.InvocationInterceptor.monitor(InvocationInterceptor.java:49)
at io.quarkus.arc.runtime.devconsole.InvocationInterceptor_Bean.intercept(Unknown Source)
at io.quarkus.arc.impl.InterceptorInvocation.invoke(InterceptorInvocation.java:41)
at io.quarkus.arc.impl.AroundInvokeInvocationContext.perform(AroundInvokeInvocationContext.java:40)
at io.quarkus.arc.impl.InvocationContexts.performAroundInvoke(InvocationContexts.java:32)
at com.example.RMQConsumer_Subclass.consume(Unknown Source)
at com.example.RMQConsumer_ClientProxy.consume(Unknown Source)
at com.example.RMQConsumer_SmallRyeMessagingInvoker_consume_e1bfbd0242536245fce5ab1c04b4b7e251cc66cd.invoke(Unknown Source)
at io.smallrye.reactive.messaging.providers.AbstractMediator.lambda$invokeBlocking$4(AbstractMediator.java:116)
at io.smallrye.context.impl.wrappers.SlowContextualConsumer.accept(SlowContextualConsumer.java:21)
at io.smallrye.mutiny.operators.uni.builders.UniCreateWithEmitter.subscribe(UniCreateWithEmitter.java:22)
at io.smallrye.mutiny.operators.AbstractUni.subscribe(AbstractUni.java:36)
at io.smallrye.mutiny.operators.uni.builders.UniCreateFromDeferredSupplier.subscribe(UniCreateFromDeferredSupplier.java:36)
at io.smallrye.mutiny.operators.AbstractUni.subscribe(AbstractUni.java:36)
at io.smallrye.mutiny.groups.UniSubscribe.withSubscriber(UniSubscribe.java:52)
at io.smallrye.mutiny.groups.UniSubscribe.with(UniSubscribe.java:112)
at io.smallrye.mutiny.groups.UniSubscribe.with(UniSubscribe.java:89)
at io.vertx.mutiny.core.Context$1.handle(Context.java:170)
at io.vertx.mutiny.core.Context$1.handle(Context.java:168)
at io.vertx.core.impl.ContextBase.lambda$null$0(ContextBase.java:137)
at io.vertx.core.impl.ContextInternal.dispatch(ContextInternal.java:264)
at io.vertx.core.impl.ContextBase.lambda$executeBlocking$1(ContextBase.java:135)
at io.vertx.core.impl.TaskQueue.run(TaskQueue.java:76)
at org.jboss.threads.ContextHandler$1.runWith(ContextHandler.java:18)
at org.jboss.threads.EnhancedQueueExecutor$Task.run(EnhancedQueueExecutor.java:2449)
at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1478)
at org.jboss.threads.DelegatingRunnable.run(DelegatingRunnable.java:29)
at org.jboss.threads.ThreadLocalResettingRunnable.run(ThreadLocalResettingRunnable.java:29)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.base/java.lang.Thread.run(Thread.java:833)
2022-10-17 05:26:40,144 WARN [io.sma.rea.mes.rabbitmq] (vert.x-eventloop-thread-0) SRMSG17013: A message sent to channel `from-rabbitmq` has been nacked, ignoring the failure and marking the RabbitMQ message as rejected
2022-10-17 05:26:40,628 INFO [com.exa.RMQConsumer] (vert.x-worker-thread-0) service instance : 10.42.76.12
2022-10-17 05:26:40,638 INFO [com.exa.RMQConsumer] (vert.x-worker-thread-0) price = 22
2022-10-17 05:26:40,639 INFO [io.sma.rea.mes.rabbitmq] (vert.x-eventloop-thread-0) SRMSG17033: A message sent to channel `from-rabbitmq` has been ack'd
2022-10-17 05:26:40,643 INFO [com.exa.RMQConsumer] (vert.x-worker-thread-0) service instance : 10.42.76.12
2022-10-17 05:26:40,644 INFO [com.exa.RMQConsumer] (vert.x-worker-thread-0) price = 23
2022-10-17 05:26:40,644 INFO [io.sma.rea.mes.rabbitmq] (vert.x-eventloop-thread-0) SRMSG17033: A message sent to channel `from-rabbitmq` has been ack'd
2022-10-17 05:26:40,646 INFO [com.exa.RMQConsumer] (vert.x-worker-thread-0) service instance : 10.42.76.12
2022-10-17 05:26:40,647 INFO [com.exa.RMQConsumer] (vert.x-worker-thread-0) price = 24
2022-10-17 05:26:40,648 INFO [io.sma.rea.mes.rabbitmq] (vert.x-eventloop-thread-0) SRMSG17033: A message sent to channel `from-rabbitmq` has been ack'd
2022-10-17 05:26:40,650 INFO [com.exa.RMQConsumer] (vert.x-worker-thread-0) service instance : 10.42.76.12
2022-10-17 05:26:40,650 INFO [com.exa.RMQConsumer] (vert.x-worker-thread-0) price = 25
2022-10-17 05:26:40,651 INFO [io.sma.rea.mes.rabbitmq] (vert.x-eventloop-thread-0) SRMSG17033: A message sent to channel `from-rabbitmq` has been ack'd
2022-10-17 05:26:40,652 INFO [com.exa.RMQConsumer] (vert.x-worker-thread-0) service instance : 10.42.76.12
2022-10-17 05:26:40,653 INFO [com.exa.RMQConsumer] (vert.x-worker-thread-0) price = 26
2022-10-17 05:26:40,654 INFO [io.sma.rea.mes.rabbitmq] (vert.x-eventloop-thread-0) SRMSG17033: A message sent to channel `from-rabbitmq` has been ack'd
2022-10-17 05:26:41,059 INFO [com.exa.RMQConsumer] (vert.x-worker-thread-0) service instance : 10.42.76.12
2022-10-17 05:26:41,060 INFO [com.exa.RMQConsumer] (vert.x-worker-thread-0) price = 27
2022-10-17 05:26:41,061 INFO [io.sma.rea.mes.rabbitmq] (vert.x-eventloop-thread-0) SRMSG17033: A message sent to channel `from-rabbitmq` has been ack'd
2022-10-17 05:26:42,059 INFO [com.exa.RMQConsumer] (vert.x-worker-thread-0) service instance : 10.42.76.12
2022-10-17 05:26:42,060 INFO [com.exa.RMQConsumer] (vert.x-worker-thread-0) price = 28
2022-10-17 05:26:42,061 INFO [io.sma.rea.mes.rabbitmq] (vert.x-eventloop-thread-0) SRMSG17033: A message sent to channel `from-rabbitmq` has been ack'd
2022-10-17 05:26:43,060 INFO [com.exa.RMQConsumer] (vert.x-worker-thread-0) service instance : 10.42.76.12
2022-10-17 05:26:43,060 INFO [com.exa.RMQConsumer] (vert.x-worker-thread-0) price = 29
2022-10-17 05:26:43,061 INFO [io.sma.rea.mes.rabbitmq] (vert.x-eventloop-thread-0) SRMSG17033: A message sent to channel `from-rabbitmq` has been ack'd
2022-10-17 05:26:44,058 INFO [com.exa.RMQConsumer] (vert.x-worker-thread-0) service instance : 10.42.76.12
2022-10-17 05:26:44,058 INFO [com.exa.RMQConsumer] (vert.x-worker-thread-0) price = 30
2022-10-17 05:26:44,059 INFO [io.sma.rea.mes.rabbitmq] (vert.x-eventloop-thread-0) SRMSG17033: A message sent to channel `from-rabbitmq` has been ack'd
2022-10-17 05:26:45,062 INFO [com.exa.RMQConsumer] (vert.x-worker-thread-0) service instance : 10.42.76.12
2022-10-17 05:26:45,063 INFO [com.exa.RMQConsumer] (vert.x-worker-thread-0) price = 31
2022-10-17 05:26:45,064 INFO [io.sma.rea.mes.rabbitmq] (vert.x-eventloop-thread-0) SRMSG17033: A message sent to channel `from-rabbitmq` has been ack'd
2022-10-17 05:26:46,058 INFO [com.exa.RMQConsumer] (vert.x-worker-thread-0) service instance : 10.42.76.12
2022-10-17 05:26:46,059 INFO [com.exa.RMQConsumer] (vert.x-worker-thread-0) price = 32
2022-10-17 05:26:46,059 INFO [io.sma.rea.mes.rabbitmq] (vert.x-eventloop-thread-0) SRMSG17033: A message sent to channel `from-rabbitmq` has been ack'd
2022-10-17 05:26:47,055 INFO [com.exa.RMQConsumer] (vert.x-worker-thread-0) service instance : 10.42.76.12
2022-10-17 05:26:47,056 INFO [com.exa.RMQConsumer] (vert.x-worker-thread-0) price = 33
2022-10-17 05:26:47,056 INFO [io.sma.rea.mes.rabbitmq] (vert.x-eventloop-thread-0) SRMSG17033: A message sent to channel `from-rabbitmq` has been ack'd
2022-10-17 05:26:48,062 INFO [com.exa.RMQConsumer] (vert.x-worker-thread-0) service instance : 10.42.76.12
2022-10-17 05:26:48,063 INFO [com.exa.RMQConsumer] (vert.x-worker-thread-0) price = 34
2022-10-17 05:26:48,064 INFO [io.sma.rea.mes.rabbitmq] (vert.x-eventloop-thread-0) SRMSG17033: A message sent to channel `from-rabbitmq` has been ack'd
2022-10-17 05:26:49,056 INFO [com.exa.RMQConsumer] (vert.x-worker-thread-0) service instance : 10.42.76.12
2022-10-17 05:26:49,057 INFO [com.exa.RMQConsumer] (vert.x-worker-thread-0) price = 35
2022-10-17 05:26:49,057 INFO [io.sma.rea.mes.rabbitmq] (vert.x-eventloop-thread-0) SRMSG17033: A message sent to channel `from-rabbitmq` has been ack'd
2022-10-17 05:26:49,772 INFO [io.ver.rab.imp.RabbitMQClientImpl] (Shutdown thread) Stopping rabbitmq client
--
Tests paused
Press [r] to resume testing, [o] Toggle test output, [:] for the terminal, [h] for more options>
2022-10-17 05:26:49,785 INFO [io.ver.rab.imp.RabbitMQClientImpl] (pool-11-thread-3) Stopping rabbitmq client
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 28.006 s
[INFO] Finished at: 2022-10-17T05:26:50-04:00
[INFO] ------------------------------------------------------------------------
I was unable to reproduce it outside of Kubernetes. My guess is the ordered aspect of @Blocking
Try replacing @Blocking with: @io.smallrye.reactive.messaging.annotations.Blocking(ordered = false)
ping @priyanknarvekar
Closing - no feedback.