tls: bad record MAC
dicolasi opened this issue · comments
I having the above error with the following configuration:
kowl:
# Config.yaml is required for Kowl and Kowl Business.
# See reference config: https://github.com/cloudhut/kowl/blob/master/docs/config/kowl.yaml)
config:
kafka:
brokers:
- kafka-prod-kafka-bootstrap.kafka.svc:9093
tls:
enabled: true
caFilepath: /etc/kowl/secrets/kafka-tls-ca
certFilepath: /etc/kowl/secrets/kafka-tls-cert
keyFilepath: /etc/kowl/secrets/kafka-tls-key
insecureSkipTlsVerify: true
schemaRegistry:
enabled: true
urls: [ "http://cp-schema-registry:8081" ] # Url with scheme is required, e.g. ["http://localhost:8081"]
logger:
level: debug
Secret is correctly created and contains the right certificate:
apiVersion: v1
data:
kafka-sasl-password: [...]
kafka-tls-ca: IiI=
kafka-tls-cert: [...]
kafka-tls-key: [...]
kafka-tls-passphrase: IiI=
kind: Secret
metadata:
creationTimestamp: "2020-12-17T09:12:13Z"
labels:
app.kubernetes.io/instance: kowl
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/name: kowl
app.kubernetes.io/version: v1.2.2
helm.sh/chart: kowl-1.2.0
name: kowl
namespace: kafka
ownerReferences:
- apiVersion: kubernetes-client.io/v1
controller: true
kind: ExternalSecret
name: kowl
uid: 39e02a07-e535-4aa9-9346-a828e87013b6
- apiVersion: apps/v1
blockOwnerDeletion: true
controller: false
kind: Deployment
name: kowl
uid: bb895019-d2ab-43ea-a20e-9baa1ec527f9
resourceVersion: "13463764"
selfLink: /api/v1/namespaces/kafka/secrets/kowl
uid: 36c34e14-d9ff-4d91-868a-8c2e3ef95caf
type: Opaque
here the full log:
oauth2-proxy [2020/12/17 09:10:40] [logger.go:490] mapping path "/" => upstream "http://127.0.0.1:8080"
oauth2-proxy [2020/12/17 09:10:40] [logger.go:490] OAuthProxy configured for Keycloak Client ID: kafka-client
oauth2-proxy [2020/12/17 09:10:40] [logger.go:490] Cookie settings: name:_oauth2_proxy secure(https):true httponly:true expiry:168h0m0s domains: path:/ samesite: refresh:disabled
oauth2-proxy [2020/12/17 09:10:40] [logger.go:490] HTTP: listening on :4180
kowl {"level":"info","msg":"started Kowl","version":"v1.2.2","built":"2020-11-23T15:49:59Z","git_sha":"284eb140e520ee647f8801992c54e7ad05b3c0c3"}
kowl {"level":"info","ts":"2020-12-17T09:16:43.972Z","msg":"connecting to Kafka cluster"}
kowl {"level":"debug","ts":"2020-12-17T09:16:43.972Z","msg":"Initializing new client","source":"sarama"}
kowl {"level":"debug","ts":"2020-12-17T09:16:43.972Z","msg":"client/metadata fetching metadata for all topics from broker kafka-prod-kafka-bootstrap.kafka.svc:9093","source":"sarama"}
kowl {"level":"debug","ts":"2020-12-17T09:16:43.974Z","msg":"Connected to broker at kafka-prod-kafka-bootstrap.kafka.svc:9093 (unregistered)","source":"sarama"}
kowl {"level":"debug","ts":"2020-12-17T09:16:44.004Z","msg":"client/metadata got error from broker -1 while fetching metadata: local error: tls: bad record MAC","source":"sarama"}
kowl {"level":"debug","ts":"2020-12-17T09:16:44.004Z","msg":"Closed connection to broker kafka-prod-kafka-bootstrap.kafka.svc:9093","source":"sarama"}
kowl {"level":"debug","ts":"2020-12-17T09:16:44.004Z","msg":"client/metadata no available broker to send metadata request to","source":"sarama"}
kowl {"level":"debug","ts":"2020-12-17T09:16:44.004Z","msg":"client/brokers resurrecting 1 dead seed brokers","source":"sarama"}
kowl {"level":"debug","ts":"2020-12-17T09:16:44.004Z","msg":"client/metadata retrying after 250ms... (3 attempts remaining)","source":"sarama"}
kowl {"level":"debug","ts":"2020-12-17T09:16:44.255Z","msg":"client/metadata fetching metadata for all topics from broker kafka-prod-kafka-bootstrap.kafka.svc:9093","source":"sarama"}
kowl {"level":"debug","ts":"2020-12-17T09:16:44.256Z","msg":"Connected to broker at kafka-prod-kafka-bootstrap.kafka.svc:9093 (unregistered)","source":"sarama"}
kowl {"level":"debug","ts":"2020-12-17T09:16:44.267Z","msg":"client/metadata got error from broker -1 while fetching metadata: local error: tls: bad record MAC","source":"sarama"}
kowl {"level":"debug","ts":"2020-12-17T09:16:44.267Z","msg":"Closed connection to broker kafka-prod-kafka-bootstrap.kafka.svc:9093","source":"sarama"}
kowl {"level":"debug","ts":"2020-12-17T09:16:44.268Z","msg":"client/metadata no available broker to send metadata request to","source":"sarama"}
kowl {"level":"debug","ts":"2020-12-17T09:16:44.268Z","msg":"client/brokers resurrecting 1 dead seed brokers","source":"sarama"}
kowl {"level":"debug","ts":"2020-12-17T09:16:44.268Z","msg":"client/metadata retrying after 250ms... (2 attempts remaining)","source":"sarama"}
kowl {"level":"debug","ts":"2020-12-17T09:16:44.518Z","msg":"client/metadata fetching metadata for all topics from broker kafka-prod-kafka-bootstrap.kafka.svc:9093","source":"sarama"}
kowl {"level":"debug","ts":"2020-12-17T09:16:44.519Z","msg":"Connected to broker at kafka-prod-kafka-bootstrap.kafka.svc:9093 (unregistered)","source":"sarama"}
kowl {"level":"debug","ts":"2020-12-17T09:16:44.530Z","msg":"client/metadata got error from broker -1 while fetching metadata: local error: tls: bad record MAC","source":"sarama"}
kowl {"level":"debug","ts":"2020-12-17T09:16:44.530Z","msg":"Closed connection to broker kafka-prod-kafka-bootstrap.kafka.svc:9093","source":"sarama"}
kowl {"level":"debug","ts":"2020-12-17T09:16:44.530Z","msg":"client/metadata no available broker to send metadata request to","source":"sarama"}
kowl {"level":"debug","ts":"2020-12-17T09:16:44.530Z","msg":"client/brokers resurrecting 1 dead seed brokers","source":"sarama"}
kowl {"level":"debug","ts":"2020-12-17T09:16:44.530Z","msg":"client/metadata retrying after 250ms... (1 attempts remaining)","source":"sarama"}
kowl {"level":"debug","ts":"2020-12-17T09:16:44.781Z","msg":"client/metadata fetching metadata for all topics from broker kafka-prod-kafka-bootstrap.kafka.svc:9093","source":"sarama"}
kowl {"level":"debug","ts":"2020-12-17T09:16:44.782Z","msg":"Connected to broker at kafka-prod-kafka-bootstrap.kafka.svc:9093 (unregistered)","source":"sarama"}
kowl {"level":"debug","ts":"2020-12-17T09:16:44.811Z","msg":"client/metadata got error from broker -1 while fetching metadata: local error: tls: bad record MAC","source":"sarama"}
kowl {"level":"debug","ts":"2020-12-17T09:16:44.811Z","msg":"Closed connection to broker kafka-prod-kafka-bootstrap.kafka.svc:9093","source":"sarama"}
kowl {"level":"debug","ts":"2020-12-17T09:16:44.811Z","msg":"client/metadata no available broker to send metadata request to","source":"sarama"}
kowl {"level":"debug","ts":"2020-12-17T09:16:44.811Z","msg":"client/brokers resurrecting 1 dead seed brokers","source":"sarama"}
kowl {"level":"debug","ts":"2020-12-17T09:16:44.811Z","msg":"Closing Client","source":"sarama"}
kowl {"level":"fatal","ts":"2020-12-17T09:16:44.811Z","msg":"failed to create kafka service","error":"failed to create kafka client: kafka: client has run out of available brokers to talk to (Is your cluster reachable?)"}
kowl stream closed
The certificate is a self-signed one created by strimzi. Without insecureSkipTlsVerify: true kwol won't start since the certificate it is not signed by a trusted CA.
Here the error:
│ kowl {"level":"debug","ts":"2020-12-17T09:25:59.813Z","msg":"client/metadata got error from broker -1 while fetching metadata: x509: certificate signed by unknown authority","source":"sarama"} │
I haven't seen that error yet. What TLS protocols are allowed on the server side? Maybe any helpful log messages on the server side?
Regarding insecureSkipTlsVerify this only means that the certificate will be checked for validity. If issued correctly and used with the right DNS you don't need to enable this for self signed certificates. You can configure which CAs you trust by passing a CA file into Kowl.
@weeco this is the log I got server side:
│ 2020-12-17 09:27:36,654 INFO [SocketServer brokerId=0] Failed authentication with /10.0.1.43 (SSL handshake failed) (org.apache.kafka.common.network.Selector) [data-plane-kafka-network-thread-0-ListenerName(INTERNAL-9093)-SSL-3]
I haven't seen that error yet. What TLS protocols are allowed on the server side? Maybe any helpful log messages on the server side?
Regarding
insecureSkipTlsVerifythis only means that the certificate will be checked for validity. If issued correctly and used with the right DNS you don't need to enable this for self signed certificates. You can configure which CAs you trust by passing a CA file into Kowl.
I am using an internal listener, not an exposed one.
listeners: │
│ - authentication: │
│ type: tls │
│ name: internal │
│ port: 9093 │
│ tls: true │
│ type: internal
@weeco this is what is allowed server side:
https://strimzi.io/docs/0.8.1/#type-KafkaListenerAuthenticationScramSha512-reference
@dicolasi That shouldn't matter. I'd recommend you take a look at your certificates again and see what SANs are configured. There are multiple ways to connect internally to your cluster. (e. g. kafka:9093 or kafka.namespace.svc.cluster.local:9093, ...)
Above Kafka server error message seem to indicate that endpoint identification is required which does not work. I hope that gives you something to research for. I'm not very familar with Strimzi and I won't be able to assist with that issue, sorry.
@weeco I am already connected properly:
kafka:
brokers:
- kafka-prod-kafka-bootstrap.kafka.svc:9093
Thanks anyway :)
@dicolasi The DNS is probably used to verify the SSL certificate though. Hence I mentioned that your specific DNS might not be one of the SANs in your TLS certificates