existed ACLs are not deleted

Question

existed ACLs are not deleted

tolikkk opened this issue 2 years ago · comments

Hello.

I have problems with ACL in the topology configuration - it is only created (added), but not deleted. I also tried using options "allow.delete.bindings=true" in config.properties or "ALLOW_DELETE_BINDINGS=true" in env with no success.

config.properties:

julie.debug.mode=false
topology.topic.prefix.separator=_
topology.topic.prefix.format={{topic}}
topology.project.prefix.format=

Run command:
julie-ops-cli.sh --topology topology.yml --brokers broker.example.com:9093 --clientConfig config.properties

How to reproduce

add ACL for principal User:Bob in topic bar1

topics.yml:

context: "example"
projects:
  - name: "kafka"
    topics:
      - name: "bar1"
        config:
          retention.ms: "604800000"
        consumers:
          - principal: "User:Bob"

ACL successfully created.

execution log:

...
List of ACLs:
bar1
'TOPIC', 'bar1', '*', 'READ', 'User:Bob', 'LITERAL'
'TOPIC', 'bar1', '*', 'DESCRIBE', 'User:Bob', 'LITERAL'
*
'GROUP', '*', '*', 'READ', 'User:Bob', 'LITERAL'

kafka-acl list (kafka-acls.sh --bootstrap-server=IP_ADDRESS:9092 --list):

Current ACLs for resource `ResourcePattern(resourceType=TOPIC, name=bar1, patternType=LITERAL)`:
 	(principal=User:Bob, host=*, operation=DESCRIBE, permissionType=ALLOW)
	(principal=User:Bob, host=*, operation=READ, permissionType=ALLOW)

Current ACLs for resource `ResourcePattern(resourceType=GROUP, name=*, patternType=LITERAL)`:
 	(principal=User:Bob, host=*, operation=READ, permissionType=ALLOW)

update ACL - remove User:Bob, add User:Alice in the same topic bar1

topics.yml:

context: "example"
projects:
  - name: "kafka"
    topics:
      - name: "bar1"
        config:
          retention.ms: "604800000"
        consumers:
          - principal: "User:Alice"

New ACL for User:Alice added, but old ACL for User:Bob is still present. It is not deleted as expected.

execution log:

...
List of ACLs:
bar1
'TOPIC', 'bar1', '*', 'DESCRIBE', 'User:Bob', 'LITERAL'
'TOPIC', 'bar1', '*', 'DESCRIBE', 'User:Alice', 'LITERAL'
'TOPIC', 'bar1', '*', 'READ', 'User:Bob', 'LITERAL'
'TOPIC', 'bar1', '*', 'READ', 'User:Alice', 'LITERAL'
*
'GROUP', '*', '*', 'READ', 'User:Bob', 'LITERAL'
'GROUP', '*', '*', 'READ', 'User:Alice', 'LITERAL'

kafka-acl list (kafka-acls.sh --bootstrap-server=IP_ADDRESS:9092 --list):

Current ACLs for resource `ResourcePattern(resourceType=TOPIC, name=bar1, patternType=LITERAL)`:
 	(principal=User:Bob, host=*, operation=DESCRIBE, permissionType=ALLOW)
	(principal=User:Alice, host=*, operation=DESCRIBE, permissionType=ALLOW)
	(principal=User:Bob, host=*, operation=READ, permissionType=ALLOW)
	(principal=User:Alice, host=*, operation=READ, permissionType=ALLOW)

Current ACLs for resource `ResourcePattern(resourceType=GROUP, name=*, patternType=LITERAL)`:
 	(principal=User:Bob, host=*, operation=READ, permissionType=ALLOW)
	(principal=User:Alice, host=*, operation=READ, permissionType=ALLOW)

Pere Urbón · Answer 1 · Wed Aug 03 2022 15:41:13 GMT+0800 (China Standard Time)

security.protocol=SASL_PLAINTEXT
sasl.mechanism=PLAIN
sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required \
  username="kafka" \
  password="kafka";

allow.delete.topics=true
allow.delete.bindings=true
julie.debug.mode=false

topology.topic.prefix.separator=_
topology.topic.prefix.format={{topic}}
topology.project.prefix.format=

context: "c"
projects:
  - name: "k"
    topics:
      - name: "bar1"
        config:
          retention.ms: "604800000"
        consumers:
          - principal: "User:Bob"

$ java -jar target/julie-ops.jar --brokers localhost:9092 --clientConfig example/topology-builder-sasl-plain.properties --topology example/t.yaml
log4j:WARN No appenders could be found for logger (org.apache.kafka.clients.admin.AdminClientConfig).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
[WARN ] 2022-08-03 09:37:47.475 [main] TopicManager - Remote state verification disabled, this is not a good practice, be awarein future versions, this check is going to become mandatory.
[WARN ] 2022-08-03 09:37:47.475 [main] TopicManager - Remote state verification disabled, this is not a good practice, be awarein future versions, this check is going to become mandatory.
[WARN ] 2022-08-03 09:37:47.491 [main] AccessControlManager - Remote state verification disabled, this is not a good practice, be awarein future versions, this check is going to become mandatory.
[WARN ] 2022-08-03 09:37:47.491 [main] AccessControlManager - Remote state verification disabled, this is not a good practice, be awarein future versions, this check is going to become mandatory.
[WARN ] 2022-08-03 09:37:47.494 [main] ArtefactManager - Remote state verification disabled, this is not a good practice, be awarein future versions, this check is going to become mandatory.
[WARN ] 2022-08-03 09:37:47.494 [main] ArtefactManager - Remote state verification disabled, this is not a good practice, be awarein future versions, this check is going to become mandatory.
[WARN ] 2022-08-03 09:37:47.495 [main] ArtefactManager - Remote state verification disabled, this is not a good practice, be awarein future versions, this check is going to become mandatory.
[WARN ] 2022-08-03 09:37:47.495 [main] ArtefactManager - Remote state verification disabled, this is not a good practice, be awarein future versions, this check is going to become mandatory.
{
  "Operation" : "com.purbon.kafka.topology.actions.topics.CreateTopicAction",
  "Topic" : "bar1",
  "Action" : "create"
}
{
  "Operation" : "com.purbon.kafka.topology.actions.topics.DeleteTopics",
  "topics" : [ "julieops" ]
}
{
  "Operation" : "com.purbon.kafka.topology.actions.access.CreateBindings",
  "Bindings" : [ {
    "resourceType" : "TOPIC",
    "resourceName" : "bar1",
    "host" : "*",
    "operation" : "READ",
    "principal" : "User:Bob",
    "pattern" : "LITERAL",
    "scope" : null
  }, {
    "resourceType" : "GROUP",
    "resourceName" : "*",
    "host" : "*",
    "operation" : "READ",
    "principal" : "User:Bob",
    "pattern" : "LITERAL",
    "scope" : null
  }, {
    "resourceType" : "TOPIC",
    "resourceName" : "bar1",
    "host" : "*",
    "operation" : "DESCRIBE",
    "principal" : "User:Bob",
    "pattern" : "LITERAL",
    "scope" : null
  } ]
}
List of Topics:
_confluent-controlcenter-7-0-1-1-MonitoringMessageAggregatorWindows-ONE_MINUTE-repartition
_confluent-controlcenter-7-0-1-1-monitoring-message-rekey-store
.... redacted....
_confluent-controlcenter-7-0-1-1-cluster-rekey
_confluent-controlcenter-7-0-1-1-MonitoringStream-THREE_HOURS-changelog
List of ACLs:
bar1
'TOPIC', 'bar1', '*', 'DESCRIBE', 'User:Bob', 'LITERAL'
'TOPIC', 'bar1', '*', 'READ', 'User:Bob', 'LITERAL'
kafka-cluster
'CLUSTER', 'kafka-cluster', '*', 'ALL', 'User:kafka', 'LITERAL'
*
'GROUP', '*', '*', 'READ', 'User:Bob', 'LITERAL'
List of Principles:
List of Connectors:
List of KSQL Artifacts:
Kafka Topology updated

$ docker exec kafka kafka-acls --bootstrap-server kafka:29092 --list                                                     2.7.0
Current ACLs for resource `ResourcePattern(resourceType=TOPIC, name=bar1, patternType=LITERAL)`:
 	(principal=User:Bob, host=*, operation=DESCRIBE, permissionType=ALLOW)
	(principal=User:Bob, host=*, operation=READ, permissionType=ALLOW)

Current ACLs for resource `ResourcePattern(resourceType=CLUSTER, name=kafka-cluster, patternType=LITERAL)`:
 	(principal=User:kafka, host=*, operation=ALL, permissionType=ALLOW)

Current ACLs for resource `ResourcePattern(resourceType=GROUP, name=*, patternType=LITERAL)`:
 	(principal=User:Bob, host=*, operation=READ, permissionType=ALLOW)

context: "c"
projects:
  - name: "k"
    topics:
      - name: "bar1"
        config:
          retention.ms: "604800000"
        consumers:
          - principal: "User:Alice"

... redacted....
_confluent-controlcenter-7-0-1-1-cluster-rekey
_confluent-controlcenter-7-0-1-1-MonitoringStream-THREE_HOURS-changelog
List of ACLs:
bar1
'TOPIC', 'bar1', '*', 'READ', 'User:Alice', 'LITERAL'
'TOPIC', 'bar1', '*', 'DESCRIBE', 'User:Alice', 'LITERAL'
kafka-cluster
'CLUSTER', 'kafka-cluster', '*', 'ALL', 'User:kafka', 'LITERAL'
*
'GROUP', '*', '*', 'READ', 'User:Alice', 'LITERAL'
List of Principles:
List of Connectors:
List of KSQL Artifacts:
Kafka Topology updated

$ docker exec kafka kafka-acls --bootstrap-server kafka:29092 --list                                                     2.7.0
Current ACLs for resource `ResourcePattern(resourceType=TOPIC, name=bar1, patternType=LITERAL)`:
 	(principal=User:Alice, host=*, operation=READ, permissionType=ALLOW)
	(principal=User:Alice, host=*, operation=DESCRIBE, permissionType=ALLOW)

Current ACLs for resource `ResourcePattern(resourceType=CLUSTER, name=kafka-cluster, patternType=LITERAL)`:
 	(principal=User:kafka, host=*, operation=ALL, permissionType=ALLOW)

Current ACLs for resource `ResourcePattern(resourceType=GROUP, name=*, patternType=LITERAL)`:
 	(principal=User:Alice, host=*, operation=READ, permissionType=ALLOW)

feel free to reopen with more information if you have it, but as of now I'm unable to reproduce successfully your situation.

Looking forward to hear from you.

Pere Urbón · Answer 2 · Wed Aug 03 2022 15:45:03 GMT+0800 (China Standard Time)

I think I know what might have happen, this is due to the way AclsManagerFetch ACLs from the current state. You need to use topology.state.cluster.enabled=true if you aim to fetch ACLs from the cluster otherwise you have to keep your state local (file, s3, kafka, etc) with you.

If you do this, you will solve your problem.

Let me know how it goes.

tolikkk · Answer 3 · Wed Aug 03 2022 16:01:02 GMT+0800 (China Standard Time)

Yes, I miss option "topology.state.cluster.enabled".

This 2 options in config.properties did the trick. Thanks a lot!

allow.delete.bindings=true
topology.state.cluster.enabled=true