Cross-account AWS Glue access for deserialization

[Robbert Hofman]

Issue submitter TODO list

• I've looked up my issue in FAQ
• I've searched for an already existing issues here
• I've tried running master-labeled docker image and the issue still persists there
• I'm running a supported version of the application which is listed here

Describe the bug (actual behavior)

I get Fallback serde was used errors when I use the GlueSchemaRegistry serde method from a different AWS account.

We have 2 AWS accounts:

• Account A where MSK and AWS Glue are running
• Account B where we have a machine that we log in to, and we docker-compose up kafka-ui

We did the necessary steps to allow account B to connect to MSK in account A.
Connecting from account B to the MSK in account A works, I can list all the topics just fine and see messages.
However, deserialization with GlueSchemaRegistry does not work in kafka-ui in account B.
If we run the same docker-compose in account A, everything works fine.

Here is our docker-compose file when run in account A:

version: '2'
services:
  kafka-ui:
    container_name: kafka-ui
    image: provectuslabs/kafka-ui:latest
    network_mode: "host"
    environment:
      kafka.clusters.0.name: xxx
      kafka.clusters.0.bootstrapServers: b-3.xxxxxxx.xxxxxxx.xx.kafka.eu-west-3.amazonaws.com:9098,b-1.xxxxxx.xxxxxx.xx.kafka.eu-west-3.amazonaws.com:9098,b-2.xxxxxxxxxx.xxxxxx.xx.kafka.eu-west-3.amazonaws.com:9098
      kafka.clusters.0.properties.security.protocol: SASL_SSL
      kafka.clusters.0.properties.sasl.mechanism: AWS_MSK_IAM
      kafka.clusters.0.properties.sasl.client.callback.handler.class: software.amazon.msk.auth.iam.IAMClientCallbackHandler
      kafka.clusters.0.properties.sasl.jaas.config: software.amazon.msk.auth.iam.IAMLoginModule required;
      kafka.clusters.0.serde.0.name: GlueSchemaRegistry
      kafka.clusters.0.serde.0.filePath: /glue-serde/kafkaui-glue-serde-v1.2.1-jar-with-dependencies.jar
      kafka.clusters.0.serde.0.className: com.provectus.kafka.ui.serdes.glue.GlueSerde
      kafka.clusters.0.serde.0.properties.region: eu-west-3
      kafka.clusters.0.serde.0.properties.registry: i-can-put-random-stuff-here-and-it-still-works
      kafka.clusters.0.serde.0.properties.keySchemaNameTemplate: "%s-key" # template that will be used to find schema name for topic key. Optional, default is null (not set).
      kafka.clusters.0.serde.0.properties.valueSchemaNameTemplate: "%s-value" # template that will be used to find schema name for topic value. Optional, default is  '%s'

    volumes:
      - ./libs:/glue-serde

Another weird thing here is that the value of kafka.clusters.0.serde.0.properties.registry does not seem to matter, however it needs to be set. On account A, I can put whatever in there and it will work. However, commenting out the line gives errors and refuses to start the container.

A very similar docker-compose file in account B:

---
version: '2'
services:
  kafka-ui:
    container_name: kafka-ui
    image: provectuslabs/kafka-ui:latest
    environment:
      kafka.clusters.0.name: xxx
      kafka.clusters.0.bootstrapServers: b-3.xxxxxx.xxxxxx.xx.kafka.eu-west-3.amazonaws.com:9098,b-1.xxxxxx.xxxxxx.xx.kafka.eu-west-3.amazonaws.com:9098,b-2.xxxxxx.xxxxxx.xx.kafka.eu-west-3.amazonaws.com:9098
      kafka.clusters.0.properties.security.protocol: SASL_SSL
      kafka.clusters.0.properties.sasl.mechanism: AWS_MSK_IAM
      kafka.clusters.0.properties.sasl.client.callback.handler.class: software.amazon.msk.auth.iam.IAMClientCallbackHandler
      kafka.clusters.0.properties.sasl.jaas.config: software.amazon.msk.auth.iam.IAMLoginModule required;
      kafka.clusters.0.serde.0.name: GlueSchemaRegistry
      kafka.clusters.0.serde.0.filePath: /glue-serde/kafkaui-glue-serde-v1.2.1-jar-with-dependencies.jar
      kafka.clusters.0.serde.0.className: com.provectus.kafka.ui.serdes.glue.GlueSerde
      kafka.clusters.0.serde.0.properties.region: eu-west-3
      kafka.clusters.0.serde.0.properties.registry: msk-dev
      kafka.clusters.0.serde.0.properties.keySchemaNameTemplate: "%s-key" # template that will be used to find schema name for topic key. Optional, default is null (not set).
      kafka.clusters.0.serde.0.properties.valueSchemaNameTemplate: "%s-value" # template that will be used to find schema name for topic value. Optional, default is  '%s'
      AWS_SHARED_CREDENTIALS_FILE: "/kafkaui/.aws/credentials"
      LOGGING_LEVEL_ROOT: debug
    volumes:
      - ./libs:/glue-serde
      - ~/.aws/:/kafkaui/.aws:ro
    user: 0:0
    ports:
      - 8080:8080

Where we set logging level to debug, mount AWS credentials, run as root, and expose 8080 instead of network_mode host. Here, we do specify the correct name of the AWS Glue registry, msk-dev.

Expected behavior

No response

Your installation details

56fa824, v0.7.1

Steps to reproduce

• Spin up a machine in another AWS account than the MSK cluster and Glue registry
• Try to deserialize messages using GlueSchemaRegistry

Screenshots

No response

Logs

No response

Additional context

No response

[github-actions]

Hello there RobbertDM! 👋

Thank you and congratulations 🎉 for opening your very first issue in this project! 💖

In case you want to claim this issue, please comment down below! We will try to get back to you as soon as we can. 👀

[Robbert Hofman]

Ah, this had nothing to do with cross-account. It was a permissions error. The AWS role that we assumed from account B (a role defined in account A) was not exactly the same as the AWS role that we used in account A. The difference between the two roles were some glue permissions. Mistery solved, however, clearer debugging messages could have saved me some time. Something like No permission to read glue schema in addition to Fallback serde was used.