Giters

kafka-ops / julie

A solution to help you build automation and gitops in your Apache Kafka deployments. The Kafka gitops!

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Julie MDSClient Timeout

yremmet opened this issue · comments

commented

Describe the bug
I use Julie in several environments. All were deployed in the same way with similar configuration. For some reason, on the newer environments, the MDSClient no longer works and times out during authentication.
However, sending the request manually via curl works. I have already tried different versions of java and julie. The broker and Julie run on the same VM

Runtime (please complete the following information):

  • OS: Red Hat Enterprise Linux release 8.4 (Ootpa)
  • JVM version: openjdk version "11.0.13" 2021-10-19 LT
  • Version 4.2.3

Additional context

[DEBUG] 2022-05-04 09:54:48.134 [main] AccessControlProviderFactory - Function get
[DEBUG] 2022-05-04 09:54:48.134 [main] AccessControlProviderFactory - Function get
[DEBUG] 2022-05-04 09:54:48.134 [main] AccessControlProviderFactory - Function apiClientLogIn create mdsclient
[DEBUG] 2022-05-04 09:54:48.134 [main] AccessControlProviderFactory - Function apiClientLogIn create mdsclient
[DEBUG] 2022-05-04 09:54:48.157 [main] MDSApiClientBuilder - Connecting to an MDS server at https://xyz:8090
[DEBUG] 2022-05-04 09:54:48.157 [main] MDSApiClientBuilder - Connecting to an MDS server at https://xyz:8090
[DEBUG] 2022-05-04 09:54:48.157 [main] MDSApiClientBuilder - Connecting to an MDS server at https://xyz:8090
[DEBUG] 2022-05-04 09:54:48.158 [main] MDSApiClient - Authenticate client
[DEBUG] 2022-05-04 09:54:48.158 [main] MDSApiClient - Authenticate client
[DEBUG] 2022-05-04 09:54:48.158 [main] MDSApiClient - Authenticate client
[DEBUG] 2022-05-04 09:54:48.165 [main] JulieHttpClient - method: GET request.uri: https://xyz:8090/security/1.0/authenticate
[DEBUG] 2022-05-04 09:54:48.165 [main] JulieHttpClient - method: GET request.uri: https://xyz:8090/security/1.0/authenticate
[ERROR] 2022-05-04 09:55:48.176 [main] MDSApiClient - java.io.IOException: java.net.http.HttpTimeoutException: request timed out
[ERROR] 2022-05-04 09:55:48.176 [main] MDSApiClient - java.io.IOException: java.net.http.HttpTimeoutException: request timed out
[ERROR] 2022-05-04 09:55:48.176 [main] MDSApiClient - java.io.IOException: java.net.http.HttpTimeoutException: request timed out
Exception in thread "main" java.io.IOException: java.io.IOException: java.io.IOException: java.net.http.HttpTimeoutException: request timed out
        at com.purbon.kafka.topology.AccessControlProviderFactory.get(AccessControlProviderFactory.java:64)
        at com.purbon.kafka.topology.JulieOps.build(JulieOps.java:90)
        at com.purbon.kafka.topology.CommandLineInterface.processTopology(CommandLineInterface.java:206)
        at com.purbon.kafka.topology.CommandLineInterface.run(CommandLineInterface.java:156)
        at com.purbon.kafka.topology.CommandLineInterface.main(CommandLineInterface.java:146)
Caused by: java.io.IOException: java.io.IOException: java.net.http.HttpTimeoutException: request timed out
        at com.purbon.kafka.topology.api.mds.MDSApiClient.authenticate(MDSApiClient.java:51)
        at com.purbon.kafka.topology.AccessControlProviderFactory.get(AccessControlProviderFactory.java:57)
        ... 4 more
Caused by: java.io.IOException: java.net.http.HttpTimeoutException: request timed out
        at com.purbon.kafka.topology.clients.JulieHttpClient.doGet(JulieHttpClient.java:132)
        at com.purbon.kafka.topology.clients.JulieHttpClient.doGet(JulieHttpClient.java:117)
        at com.purbon.kafka.topology.api.mds.MDSApiClient.authenticate(MDSApiClient.java:40)
        ... 5 more
Caused by: java.net.http.HttpTimeoutException: request timed out
        at java.net.http/jdk.internal.net.http.HttpClientImpl.send(HttpClientImpl.java:559)
        at java.net.http/jdk.internal.net.http.HttpClientFacade.send(HttpClientFacade.java:119)
        at com.purbon.kafka.topology.clients.JulieHttpClient.doGet(JulieHttpClient.java:128)
        ... 7 more

[root@xyz julie]# time curl  https://xyz:8090/security/1.0/authenticate -k -u 'user:password' --fail
{"auth_token":"aGVsbG8gZnJpZW5k","token_type":"Bearer","expires_in":3600}
real    0m0.213s
user    0m0.009s
sys     0m0.004s
commented

Some questions:

  • which version of the confluent platform you using?
  • can you share more information about your logs and reponse codes within your MDS?
  • a timeout would be that the request never come back, so there should be more data missing.

can you add julie.debug.mode=true in your config using the last 4.2.5 and share the results?

commented

Hi sorry it took me so long. The systems are airgapped and it takes some time to get data in and out.

which version of the confluent platform you using?
6.2.0
can you share more information about your logs and reponse codes within your MDS?
As I said I can recreate the auth request with curl and it works just fine and the control center works as well.
I added the logs of two run one with the correct credentials and one time it tried to run it with wrong credentials.
It seems to me the answer of the MDS doesn't make it to julie.

[2022-05-06 07:02:21,646] DEBUG Login failed for julie_wrong (io.confluent.rbacapi.login.MdsLoginService)
[2022-05-06 07:02:21,651] INFO 127.0.0.1 - - [06/May/2022:07:02:21 +0000] "GET /security/1.0/authenticate HTTP/1.1" 401 101  25 (io.confluent.rest-utils.requests)
[2022-05-06 07:02:21,651] INFO 127.0.0.1 - - [06/May/2022:07:02:21 +0000] "GET /security/1.0/authenticate HTTP/1.1" 401 101  25 (io.confluent.rest-utils.requests)
[2022-05-06 07:02:21,651] INFO 127.0.0.1 - - [06/May/2022:07:02:21 +0000] "GET /security/1.0/authenticate HTTP/1.1" 401 101  25 (io.confluent.rest-utils.requests)
[2022-05-06 07:02:24,435] INFO 10.175.17.150 - - [06/May/2022:07:02:24 +0000] "GET /v1/metadata/id HTTP/1.1" 200 105  1 (io.confluent.rest-utils.requests)
[2022-05-06 07:02:24,435] INFO 10.175.17.150 - - [06/May/2022:07:02:24 +0000] "GET /v1/metadata/id HTTP/1.1" 200 105  1 (io.confluent.rest-utils.requests)
[2022-05-06 07:02:24,436] INFO 10.175.17.150 - - [06/May/2022:07:02:24 +0000] "GET /v1/metadata/id HTTP/1.1" 200 105  2 (io.confluent.rest-utils.requests)

[2022-05-06 07:08:32,485] DEBUG Login succeeded for julie (io.confluent.rbacapi.login.MdsLoginService)
[2022-05-06 07:08:32,486] INFO 10.175.17.150 - srv_de-odskaf-l-t-01 [06/May/2022:07:08:32 +0000] "GET /security/1.0/metadataClusterId HTTP/1.1" 200 22  194 (io.confluent.rest-utils.requests)
[2022-05-06 07:08:32,486] INFO 10.175.17.150 - srv_de-odskaf-l-t-01 [06/May/2022:07:08:32 +0000] "GET /security/1.0/metadataClusterId HTTP/1.1" 200 22  194 (io.confluent.rest-utils.requests)
[2022-05-06 07:08:32,486] INFO 10.175.17.150 - srv_de-odskaf-l-t-01 [06/May/2022:07:08:32 +0000] "GET /security/1.0/metadataClusterId HTTP/1.1" 200 22  194 (io.confluent.rest-utils.requests)
[2022-05-06 07:08:36,052] INFO 10.175.17.150 - - [06/May/2022:07:08:36 +0000] "GET /v1/metadata/id HTTP/1.1" 200 105  2 (io.confluent.rest-utils.requests)
[2022-05-06 07:08:36,052] INFO 10.175.17.150 - - [06/May/2022:07:08:36 +0000] "GET /v1/metadata/id HTTP/1.1" 200 105  2 (io.confluent.rest-utils.requests)
[2022-05-06 07:08:36,052] INFO 10.175.17.150 - - [06/May/2022:07:08:36 +0000] "GET /v1/metadata/id HTTP/1.1" 200 105  2 (io.confluent.rest-utils.requests)

I did run the latest release with the debug.mode but the logs mean me nothing to me.

log4j:WARN No appenders could be found for logger (org.apache.kafka.clients.admin.AdminClientConfig).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
shouldRetry: count=1
shouldRetry: count=2
shouldRetry: count=3
shouldRetry: count=4
shouldRetry: count=5
shouldRetry: count=6
shouldRetry: count=7
shouldRetry: count=8
shouldRetry: count=9
shouldRetry: count=10
shouldRetry: count=11
shouldRetry: count=12

I appreciate your help
thank you