Lacuna - a Pub/Sub Docker Operator

Lacuna is a Kubernetes-style operator that runs locally on your machine and manages Google Cloud Pub/Sub topics and push subscriptions for your docker containers. It is designed to run alongside a local Pub/Sub emulator and manages topics and subscriptions automatically by observing the docker containers running on your machine.

Overview

While testing locally, having a local Pub/Sub emulator helps to replicate the behavior of the production environment. However, it can be tedious to manually create topics and subscriptions for each service that needs to interact with Pub/Sub. This is especially true for Pub/Sub because topics and subscriptions can not be created using the gcloud CLI, but must be created using a proper Pub/Sub API client, which causes maintainance overhead for push subscriptions. Lacuna aims to solve this by creating topics and subscriptions for each container that needs to interact with Pub/Sub just by using docker labels.

Limitations

Currently, Lacuna only supports Google Cloud Pub/Sub, but it can be extended to support other messaging systems. Also, only push subscriptions are supported, as pull subscriptions have to be implemented in the consuming service anyways. However, Lacuna can still be used to create the topics pull subscriptions can subscribe to.

Usage

Use Lacuna by running it as a docker container alongside your Pub/Sub emulator. Lacuna will automatically create topics and push subscriptions for each container that has the lacuna.enabled label set to true. Lacuna will also take care of deleting topics and subscriptions when containers are stopped.

Make sure to mount the docker socket into the Lacuna container so it can observe container events.

Example

version: "3.9"

services:
    json-server:
        image: codfish/json-server:latest
        restart: unless-stopped
        ports:
            - "8080:80"
        labels:
            lacuna.enabled: true
            lacuna.subscription.test.topic: test
            lacuna.subscription.test.endpoint: http://json-server/messages
            lacuna.subscription.test.ack-deadline: "30s"
    pubsub:
        image: gcr.io/google.com/cloudsdktool/google-cloud-cli:emulators
        command: gcloud beta emulators pubsub start --host-port=0.0.0.0:8085 --project=pubsub
        restart: unless-stopped
        ports:
            - "8085:8085"
    lacuna:
        build: .
        restart: unless-stopped
        environment:
            LACUNA_PUBSUB_PROJECT_ID: pubsub
            # Configure the pubsub client to use the emulator
            # See https://cloud.google.com/pubsub/docs/emulator#env
            PUBSUB_EMULATOR_HOST: pubsub:8085 
        volumes:
            - /var/run/docker.sock:/var/run/docker.sock

Configuration

Lacuna is configured using docker labels. The following labels are supported:

Label	Description	Required
`lacuna.enabled`	Enables Lacuna for the container.	Yes
`lacuna.subscription.<name>.topic`	The name of the topic to subscribe to.	Yes
`lacuna.subscription.<name>.endpoint`	The endpoint to send messages to.	Yes
`lacuna.subscription.<name>.<option>`	See options below.	No

Subscription Options

For each subscription, the following options can be set. For a detailed description of each option, see the Pub/Sub API documentation.

Option	Description
`ack-deadline`	The number of seconds the subscriber has to acknowledge a message.
`retain-acked-messages`	Whether to retain acknowledged messages.
`retention-duration`	The number of seconds to retain acknowledged messages.
`enable-ordering`	Whether to enable message ordering.
`expiration-ttl`	The number of seconds a message can be retained.
`filter`	A filter expression.
`deliver-exactly-once`	Whether to deliver messages exactly once.
`dead-letter-topic`	The name of the dead letter topic.
`max-dead-letter-delivery-attempts`	The maximum number of delivery attempts for a message.
`retry-minimum-backoff`	The minimum backoff time for retrying a message.
`retry-maximum-backoff`	The maximum backoff time for retrying a message.

Acknowledgements

Lacuna's label-based configuration is inspired by Ofelia, a job scheduler for docker containers.

aplr / lacuna