provectus / kafka-ui

Issue submitter TODO list

I've looked up my issue in FAQ
I've searched for an already existing issues here
I've tried running master-labeled docker image and the issue still persists there
I'm running a supported version of the application which is listed here

Describe the bug (actual behavior)

Hello,

I've encountered an issue with the Kafka-UI configuration when utilizing Protobuf files for serialization/deserialization in a Kubernetes environment.
The Protobuf files are mounted using a Kubernetes ConfigMap. However, Kubernetes creates hidden files and links (e.g., ..data, ..timestamp.file) in the mounted directory. These hidden files and links are inadvertently being processed by Kafka-UI, leading to errors related to multiple enums sharing the same constant due to the presence of duplicate Protobuf files.

Expected behavior

Kafka-UI should ignore hidden files and links created by Kubernetes when mounting ConfigMaps, or provide a mechanism to exclude specific files or patterns from being processed.

Your installation details

The configuration is specified as follows:

yamlApplicationConfig:
  kafka:
    clusters:
      - name: kafka
        bootstrapServers: kafka-cluster-kafka-bootstrap:9092
        serde:
          - name: ProtobufFile
            properties:
              protobufFilesDir: /protofiles/
              protobufMessageName: my.Value

And I was mounting protobuf files using configmap (by ksutomization patch)

helmCharts:
- name: kafka-ui
  releaseName: kafka-ui
  version: 0.7.5
  repo: https://provectus.github.io/kafka-ui-charts
  valuesInline:
    yamlApplicationConfig:
      kafka:
        clusters:
          - name: kafka
            bootstrapServers: kafka-cluster-kafka-bootstrap:9092
            serde:
              - name: ProtobufFile
                properties:
                  protobufFilesDir: /protofiles/mypackage
                  protobufMessageName: mypackage.v1.MyType
      auth:
        type: disabled

patches:
- patch: |-
    apiVersion: apps/v1
    kind: Deployment
    metadata:
      name: kafka-ui
    spec:
      revisionHistoryLimit: 1
      template:
        spec:
          containers:
          - name: kafka-ui
            volumeMounts:
            - name: protofiles
              mountPath: /protofiles/
          volumes:
          - name: protofiles
            configMap:
              name: proto-mypackage-v1

---
configMapGenerator:
- name: proto-package-v1
  options:
    disableNameSuffixHash: true
  files:
  - my_message.proto
  - my_common.proto

Steps to reproduce

Configure a Kafka cluster in Kafka-UI with Protobuf serialization/deserialization, specifying a directory mounted via Kubernetes ConfigMap for Protobuf files.
Observe the logs/errors in Kafka-UI related to duplicate enums or other Protobuf schema-related errors.

Screenshots

Logs

org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'messagesController' defined in URL [jar:file:/kafka-ui-api.jar!/BOOT-INF/classes!/com/provectus/kafka/ui/controller/MessagesController.class]: Unsatisfied dependency expressed through constructor parameter 0: Error creating bean with name 'messagesService' defined in URL [jar:file:/kafka-ui-api.jar!/BOOT-INF/classes!/com/provectus/kafka/ui/service/MessagesService.class]: Unsatisfied dependency expressed through constructor parameter 1: Error creating bean with name 'deserializationService' defined in URL [jar:file:/kafka-ui-api.jar!/BOOT-INF/classes!/com/provectus/kafka/ui/service/DeserializationService.class]: Failed to instantiate [com.provectus.kafka.ui.service.DeserializationService]: Constructor threw exception
	at org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:800)
	at org.springframework.beans.factory.support.ConstructorResolver.autowireConstructor(ConstructorResolver.java:245)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.autowireConstructor(AbstractAutowireCapableBeanFactory.java:1352)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1189)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:560)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:520)
	at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:326)
	at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:234)
	at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:324)
	at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:200)
	at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:973)
	at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:917)
	at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:584)
	at org.springframework.boot.web.reactive.context.ReactiveWebServerApplicationContext.refresh(ReactiveWebServerApplicationContext.java:66)
	at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:732)
	at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:434)
	at org.springframework.boot.SpringApplication.run(SpringApplication.java:310)
	at com.provectus.kafka.ui.KafkaUiApplication.startApplication(KafkaUiApplication.java:24)
	at com.provectus.kafka.ui.KafkaUiApplication.main(KafkaUiApplication.java:17)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:568)
	at org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:49)
	at org.springframework.boot.loader.Launcher.launch(Launcher.java:95)
	at org.springframework.boot.loader.Launcher.launch(Launcher.java:58)
	at org.springframework.boot.loader.JarLauncher.main(JarLauncher.java:65)
Caused by: org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'messagesService' defined in URL [jar:file:/kafka-ui-api.jar!/BOOT-INF/classes!/com/provectus/kafka/ui/service/MessagesService.class]: Unsatisfied dependency expressed through constructor parameter 1: Error creating bean with name 'deserializationService' defined in URL [jar:file:/kafka-ui-api.jar!/BOOT-INF/classes!/com/provectus/kafka/ui/service/DeserializationService.class]: Failed to instantiate [com.provectus.kafka.ui.service.DeserializationService]: Constructor threw exception
	at org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:800)
	at org.springframework.beans.factory.support.ConstructorResolver.autowireConstructor(ConstructorResolver.java:245)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.autowireConstructor(AbstractAutowireCapableBeanFactory.java:1352)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1189)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:560)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:520)
	at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:326)
	at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:234)
	at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:324)
	at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:200)
	at org.springframework.beans.factory.config.DependencyDescriptor.resolveCandidate(DependencyDescriptor.java:254)
	at org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1417)
	at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:1337)
	at org.springframework.beans.factory.support.ConstructorResolver.resolveAutowiredArgument(ConstructorResolver.java:887)
	at org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:791)
	... 26 common frames omitted
Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'deserializationService' defined in URL [jar:file:/kafka-ui-api.jar!/BOOT-INF/classes!/com/provectus/kafka/ui/service/DeserializationService.class]: Failed to instantiate [com.provectus.kafka.ui.service.DeserializationService]: Constructor threw exception
	at org.springframework.beans.factory.support.ConstructorResolver.instantiate(ConstructorResolver.java:326)
	at org.springframework.beans.factory.support.ConstructorResolver.autowireConstructor(ConstructorResolver.java:314)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.autowireConstructor(AbstractAutowireCapableBeanFactory.java:1352)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1189)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:560)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:520)
	at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:326)
	at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:234)
	at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:324)
	at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:200)
	at org.springframework.beans.factory.config.DependencyDescriptor.resolveCandidate(DependencyDescriptor.java:254)
	at org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1417)
	at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:1337)
	at org.springframework.beans.factory.support.ConstructorResolver.resolveAutowiredArgument(ConstructorResolver.java:887)
	at org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:791)
	... 40 common frames omitted
Caused by: org.springframework.beans.BeanInstantiationException: Failed to instantiate [com.provectus.kafka.ui.service.DeserializationService]: Constructor threw exception
	at org.springframework.beans.BeanUtils.instantiateClass(BeanUtils.java:224)
	at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:110)
	at org.springframework.beans.factory.support.ConstructorResolver.instantiate(ConstructorResolver.java:323)
	... 54 common frames omitted
Caused by: com.squareup.wire.schema.SchemaException: multiple enums share constant KIND_UNSPECIFIED:
  1. mypackage.v1.Kind.KIND_UNSPECIFIED (/protofiles/core_sdk_message.proto:89:3)
  2. mypackage.v1.Kind.KIND_UNSPECIFIED (/protofiles/..2024_03_18_18_43_09.906008176/core_sdk_message.proto:89:3)
  for file /protofiles/migration_message.proto
multiple enums share constant KIND_SESSION:
  1. mypackage.v1.Kind.KIND_SESSION (/protofiles/core_sdk_message.proto:90:3)
  2. mypackage.v1.Kind.KIND_SESSION (/protofiles/..2024_03_18_18_43_09.906008176/core_sdk_message.proto:90:3)
  for file /protofiles/migration_message.proto
multiple enums share constant KIND_EVENT:
  1. mypackage.v1.Kind.KIND_EVENT (/protofiles/core_sdk_message.proto:91:3)
  2. mypackage.v1.Kind.KIND_EVENT (/protofiles/..2024_03_18_18_43_09.906008176/core_sdk_message.proto:91:3)
  for file /protofiles/migration_message.proto
multiple enums share constant KIND_PAGEVIEW:
  1. mypackage.v1.Kind.KIND_PAGEVIEW (/protofiles/core_sdk_message.proto:92:3)
  2. mypackage.v1.Kind.KIND_PAGEVIEW (/protofiles/..2024_03_18_18_43_09.906008176/core_sdk_message.proto:92:3)
  for file /protofiles/migration_message.proto
multiple enums share constant KIND_USER:
  1. mypackage.v1.Kind.KIND_USER (/protofiles/core_sdk_message.proto:93:3)
  2. mypackage.v1.Kind.KIND_USER (/protofiles/..2024_03_18_18_43_09.906008176/core_sdk_message.proto:93:3)
  for file /protofiles/migration_message.proto
multiple enums share constant ENDPOINT_UNSPECIFIED:
  1. mypackage.v1.Endpoint.ENDPOINT_UNSPECIFIED (/protofiles/..2024_03_18_18_43_09.906008176/pre_enrich_enums.proto:8:3)
  2. mypackage.v1.Endpoint.ENDPOINT_UNSPECIFIED (/protofiles/pre_enrich_enums.proto:8:3)
  for file /protofiles/migration_message.proto
multiple enums share constant ENDPOINT_PIXEL:
  1. mypackage.v1.Endpoint.ENDPOINT_PIXEL (/protofiles/..2024_03_18_18_43_09.906008176/pre_enrich_enums.proto:9:3)
  2. mypackage.v1.Endpoint.ENDPOINT_PIXEL (/protofiles/pre_enrich_enums.proto:9:3)
  for file /protofiles/migration_message.proto
multiple enums share constant ENDPOINT_ANDROID:
  1. mypackage.v1.Endpoint.ENDPOINT_ANDROID (/protofiles/..2024_03_18_18_43_09.906008176/pre_enrich_enums.proto:11:3)
  2. mypackage.v1.Endpoint.ENDPOINT_ANDROID (/protofiles/pre_enrich_enums.proto:11:3)
  for file /protofiles/migration_message.proto
multiple enums share constant ENDPOINT_IOS:
  1. mypackage.v1.Endpoint.ENDPOINT_IOS (/protofiles/..2024_03_18_18_43_09.906008176/pre_enrich_enums.proto:13:3)
  2. mypackage.v1.Endpoint.ENDPOINT_IOS (/protofiles/pre_enrich_enums.proto:13:3)
  for file /protofiles/migration_message.proto
multiple enums share constant ENDPOINT_IDENTIFY_V2:
...

Additional context

Currently I'm using an init containers to copy the content over instead of mounting the configmap directly, which works for me.

patches:
- patch: |-
    apiVersion: apps/v1
    kind: Deployment
    metadata:
      name: kafka-ui
    spec:
      revisionHistoryLimit: 1
      template:
        spec:
          initContainers:
          - name: copy-protofiles
            image: busybox
            command: ['sh', '-c', 'cp -L -r /proto-src/* /protofiles/ && find /protofiles/ -type d -name ".*" -exec rm -rf {} +']
            volumeMounts:
            - name: proto-mypackage-v1-src
              mountPath: /proto-src/mypackage/v1
            - name: protofiles
              mountPath: /protofiles
          containers:
          - name: kafka-ui
            volumeMounts:
            - name: protofiles
              mountPath: /protofiles/
          volumes:
          - name: proto-mypackage-v1-src
            configMap:
              name: proto-mypackage-v1
          - name: protofiles
            emptyDir: {}

Hello there kehao95! 👋

Thank you and congratulations 🎉 for opening your very first issue in this project! 💖

In case you want to claim this issue, please comment down below! We will try to get back to you as soon as we can. 👀

similar issue spring-projects/spring-boot#23232

Not sure, but maybe Files.walk with option FOLLOW_LINKS can resolve this issue.
Otherwise, we may need an extra filter in this function

kafka-ui/kafka-ui-api/src/main/java/com/provectus/kafka/ui/serdes/builtin/ProtobufFileSerde.java

Lines 399 to 415 in 53a6553

    
             private Map<String, ProtoFile> loadFilesWithLocations() { 
        
               Map<String, ProtoFile> filesByLocations = new HashMap<>(); 
        
               try (var files = Files.walk(baseLocation)) { 
        
                 files.filter(p -> !Files.isDirectory(p) && p.toString().endsWith(".proto")) 
        
                     .forEach(path -> { 
        
                       // relative path will be used as "import" statement 
        
                       String relativePath = baseLocation.relativize(path).toString(); 
        
                       var protoFileElement = ProtoParser.Companion.parse( 
        
                           Location.get(baseLocation.toString(), relativePath), 
        
                           readFileAsString(path) 
        
                       ); 
        
                       filesByLocations.put(relativePath, ProtoFile.Companion.get(protoFileElement)); 
        
                     }); 
        
               } 
        
               return filesByLocations; 
        
             } 
        
           }

@kehao95 hey, as this repo is no longer maintained (see #4255) I'd be happy to help you out here: kafbat/kafka-ui#262

@Haarolean Thank you for your contribution and fix I'll try it out! I hope there will be a deprecation notice on this project though.

	private Map<String, ProtoFile> loadFilesWithLocations() {
	Map<String, ProtoFile> filesByLocations = new HashMap<>();
	try (var files = Files.walk(baseLocation)) {
	files.filter(p -> !Files.isDirectory(p) && p.toString().endsWith(".proto"))
	.forEach(path -> {
	// relative path will be used as "import" statement
	String relativePath = baseLocation.relativize(path).toString();
	var protoFileElement = ProtoParser.Companion.parse(
	Location.get(baseLocation.toString(), relativePath),
	readFileAsString(path)
	);
	filesByLocations.put(relativePath, ProtoFile.Companion.get(protoFileElement));
	});
	}
	return filesByLocations;
	}
	}

Issue with Protobuf Files Configuration when using Kubernetes ConfigMap