kafka-ops / julie

A solution to help you build automation and gitops in your Apache Kafka deployments. The Kafka gitops!

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Quotas not working out the box

sionsmith opened this issue · comments

Describe the bug
Reading the docs I should be able set quotas using the YAML examples provided in the documentation:

platform:
   kafka:
     quotas:
       - principal: "User:App0"
         producer_byte_rate: 1024
         consumer_byte_rate: 1024
         request_percentage: 50.0
       - principal: "User:App1"
         producer_byte_rate: 2048
         consumer_byte_rate: 2048
       - principal: "User:App2"
         request_percentage: 80.0

However quotas is not supported which throws the following error:

➜  julie git:(jenkins) ✗ java -jar julie-ops.jar --brokers localhost:29092 --clientConfig sasl_plain.properties --topology sasl-descriptor-topics-quotas.yaml --dryRun
log4j:WARN No appenders could be found for logger (org.apache.kafka.clients.admin.AdminClientConfig).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
com.purbon.kafka.topology.exceptions.TopologyParsingException: Failed to deserialize topology from sasl-descriptor-topics-quotas.yaml
	at com.purbon.kafka.topology.serdes.TopologySerdes.deserialise(TopologySerdes.java:42)
	at com.purbon.kafka.topology.TopologyObjectBuilder.parseListOfTopologies(TopologyObjectBuilder.java:98)
	at com.purbon.kafka.topology.TopologyObjectBuilder.build(TopologyObjectBuilder.java:36)
	at com.purbon.kafka.topology.TopologyObjectBuilder.build(TopologyObjectBuilder.java:30)
	at com.purbon.kafka.topology.JulieOps.build(JulieOps.java:127)
	at com.purbon.kafka.topology.JulieOps.build(JulieOps.java:87)
	at com.purbon.kafka.topology.CommandLineInterface.processTopology(CommandLineInterface.java:211)
	at com.purbon.kafka.topology.CommandLineInterface.run(CommandLineInterface.java:161)
	at com.purbon.kafka.topology.CommandLineInterface.main(CommandLineInterface.java:147)
Caused by: com.fasterxml.jackson.databind.exc.UnrecognizedPropertyException: Unrecognized field "quotas" (class com.purbon.kafka.topology.model.users.platform.Kafka), not marked as ignorable (2 known properties: "rbac", "instances"])
 at [Source: UNKNOWN; line: -1, column: -1] (through reference chain: com.purbon.kafka.topology.model.users.platform.Kafka["quotas"])
	at com.fasterxml.jackson.databind.exc.UnrecognizedPropertyException.from(UnrecognizedPropertyException.java:61)
	at com.fasterxml.jackson.databind.DeserializationContext.handleUnknownProperty(DeserializationContext.java:987)
	at com.fasterxml.jackson.databind.deser.std.StdDeserializer.handleUnknownProperty(StdDeserializer.java:1974)
	at com.fasterxml.jackson.databind.deser.BeanDeserializerBase.handleUnknownProperty(BeanDeserializerBase.java:1701)
	at com.fasterxml.jackson.databind.deser.BeanDeserializerBase.handleUnknownVanilla(BeanDeserializerBase.java:1679)
	at com.fasterxml.jackson.databind.deser.BeanDeserializer.vanillaDeserialize(BeanDeserializer.java:330)
	at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:187)
	at com.fasterxml.jackson.databind.deser.DefaultDeserializationContext.readRootValue(DefaultDeserializationContext.java:322)
	at com.fasterxml.jackson.databind.ObjectMapper._readValue(ObjectMapper.java:4569)
	at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:2798)
	at com.fasterxml.jackson.databind.ObjectMapper.treeToValue(ObjectMapper.java:3261)
	at com.purbon.kafka.topology.serdes.TopologyCustomDeserializer.parse(TopologyCustomDeserializer.java:177)
	at com.purbon.kafka.topology.serdes.TopologyCustomDeserializer.deserialize(TopologyCustomDeserializer.java:131)
	at com.purbon.kafka.topology.serdes.TopologyCustomDeserializer.deserialize(TopologyCustomDeserializer.java:47)
	at com.fasterxml.jackson.databind.deser.DefaultDeserializationContext.readRootValue(DefaultDeserializationContext.java:322)
	at com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:4593)
	at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:3413)
	at com.purbon.kafka.topology.serdes.TopologySerdes.deserialise(TopologySerdes.java:39)
	... 8 more

To Reproduce
java -jar julie-ops.jar --brokers localhost:29092 --clientConfig sasl_plain.properties --topology /julie/src/test/resource/descriptor-with-quotas.yaml

Expected behavior
A quota would be added to the user

Runtime (please complete the following information):

  • OS: mac
  • JVM version: Java 17
  • JulieOps Version 4.3.0

Additional context
Add any other context about the problem here.

I have just noticed that Quotas were not added until 4.4.0 my bad!