rdomenzain / kafka-ease

A library for the easy and automated creation of topics and acls in Kafka.

Kafka Ease

Automating the creation of topics and ACLs is a common task for Kafka administrators. This repository contains a set of utilities to automate these tasks.

• Create a topic with a given number of partitions and replication factor
• Create ACLs for a given topic

See the documentation for more information.

Contents

• Kafka Ease
  • Contents
  • Getting started
    • Installation
      • requirements.txt
      • Pip install
  • How to use
    • File format
    • Validate configuration
    • Apply configuration
    • Help
  • Changelog

Getting started

Installation

requirements.txt

To install the library from the GitLab package registry, write the following lines in the file requirements.txt:

kafka-ease==[VERSION]

Pip install

To install the library, run the following command:

pip install kafka-ease==[VERSION]

Check PEP 440 for version schemes and version specifiers.

How to use

File format

The file format is YAML or JSON. The file must contain a list of topics and a list of ACLs.

# acl.yaml
topics:
  - name: topic.name
    num_partitions: 1
    replication_factor: 1
    cleanup_policy: delete
    retention_days: 7

acls:
  - resource_type: TOPIC
    resource_name: topic.name
    principal: User:kafka-user
    host: "*"
    operation: READ
    permission_type: ALLOW
    pattern_type: LITERAL

{
  "topics": [
    {
      "name": "topic.name",
      "num_partitions": 1,
      "replication_factor": 1,
      "cleanup_policy": "delete",
      "retention_days": 7
    }
  ],
  "acls": [
    {
      "resource_type": "TOPIC",
      "resource_name": "topic.name",
      "principal": "User:kafka-user",
      "host": "*",
      "operation": "READ",
      "permission_type": "ALLOW",
      "pattern_type": "LITERAL"
    }
  ]
}

• Topics

  • name: Topic name
  • num_partitions: Number of partitions
  • replication_factor: Replication factor
  • cleanup_policy: Cleanup policy (delete, compact)
  • retention_days: Delete retention days

• ACLs

  • resource_type: Resource type (UNKNOWN, ANY, CLUSTER, TOPIC, DELEGATION_TOKEN, GROUP, TRANSACTIONAL_ID)
  • resource_name: Resource name
  • principal: Principal (User:, Group:)
  • host: Host (*)
  • operation: Operation (ANY, ALL, READ, WRITE, CREATE, DELETE, ALTER, DESCRIBE, CLUSTER_ACTION, DESCRIBE_CONFIGS, ALTER_CONFIGS, IDEMPOTENT_WRITE)
  • permission_type: Permission type (ANY, DENY, ALLOW)
  • pattern_type: Pattern type (ANY, MATCH, LITERAL, PREFIXED)

Validate configuration

kafka-ease apply -f acl.yaml --only-validate

Expected output:

Validating file acl.yaml...
File: acl.yaml
YAML file detected
2 topics found.
2 ACLs found.
File acl.yaml is valid.

Apply configuration

kafka-ease apply -f acl.yaml \
  --kafka-brokers kafka:9093 \
  --security-protocol SASL_PLAINTEXT \ 
  --sasl-mechanism SCRAM-SHA-256 \
  --sasl-username kafka-admin \
  --sasl-password SECRET

Expected output:

Applying file...
Kafka brokers: kafka:9093
Security protocol: SASL_PLAINTEXT
SASL mechanism: SCRAM-SHA-256
SASL username: kafka-admin
SASL password: ***************
File: acl.yaml
YAML file detected
1 topics found.
1 ACLs found.
Topic topic.name updated
Removing old ACLs 3
ACL User:kafka-user synced
File synced successfully.

Help

kafka-ease apply --help

Expected output:

Usage: kafka-ease apply [OPTIONS]

  Apply configuration file with topics and ACL to Kafka.

Options:
  --kafka-brokers TEXT            Kafka server to connect to use.
  --security-protocol [PLAINTEXT|SASL_PLAINTEXT]
                                  Kafka Security protocol to use.
  --sasl-mechanism [PLAIN|SCRAM-SHA-256|SCRAM-SHA-512]
                                  SASL mechanism to use.
  --sasl-username TEXT            SASL username to use.
  --sasl-password TEXT            SASL password to use.
  -f, --file TEXT                 File path to validate
  --only-validate                 Only validate the file
  --help                          Show this message and exit.

Changelog

• 1.0.0 (2023-09-20)

  Initial release of Kafka Ease

• 1.0.1 (2023-09-20)

  Fix packaging error

• 1.0.2 (2023-09-20)

  Fix minor bugs