matehat / weddell

A Google Pub/Sub library for Elixir

Weddell

Weddell is an Elixir client for Google Pub/Sub.

Documentation

Installation

1. Add Weddell and Goth to your list of dependencies in mix.exs:

def deps do
  [
    {:weddell, "~> 0.4"},
    {:goth, "~> 0.11"},
  ]
end

2. Configure Weddell and Goth with your GCP service account credentials:

config :weddell,
  project: "gcp-project-name"
config :goth,
  json: {:system, "GCP_CREDENTIALS_JSON"}

By default Weddell will start a client and connect on application start. This can be disabled by setting :no_connect_on_start in the application config. Clients can then be started with Weddell.Client.start_link/3.

Getting Started

Creating a topic and subscription

Weddell.create_topic("topic-name")
Weddell.create_subscription("subscription-name", "topic-name")

Creating a consumer

# In your application code
defmodule MyApp.Consumer do
  use Weddell.Consumer

  def handle_messages(messages) do
    %{true => processed_messages, false => failed_messages} =
      Enum.group_by(messages, &process_message/1)
    # Delay failed messages for at least 60 seconds
    delay_messages =
      Enum.map(failed_messages, fn msg -> {msg, 60} end)
    {:ok, ack: processed_messages, delay: delay_messages}
  end

  def process_message(message) do
    ...
  end
end

defmodule MyApp do
  use Application

  def start(_type, _args) do
    import Supervisor.Spec

    children = [
      {MyApp.Consumer, "subscription-name"}
    ]

    opts = [strategy: :one_for_one, name: MyApp.Supervisor]
    Supervisor.start_link(children, opts)
  end
end

Publishing a message

With data

"data"
|> Weddell.publish("topic-name")

With json data

%{foo: "bar"}
|> Poison.encode!()
|> Weddell.publish("topic-name")

With data and attributes

{"data", %{attr1: "value1"}}
|> Weddell.publish("topic-name")

With only attributes

%{attr1: "value1"}
|> Weddell.publish("topic-name")

Multiple messages

["message1", "message2", "message3"]
|> Weddell.publish("topic-name")

[{"message1", %{attr1: "value1"}},
 {"message2", %{attr2: "value2"}},
 {"message3", %{attr3: "value3"}}]
|> Weddell.publish("topic-name")

TODO

• Integration tests
• Update topics
• Update subscriptions
• Modify ack deadline (non-streaming)
• GRPC stream error handling
• Snapshots?

Alternatives

Weddell uses Pub/Sub's GRPC API which is still in beta. It also makes use of streaming APIs that are considered experimental. If the beta/experimental status of Weddell worries you Kane may be a better choice. It uses the more mature Pub/Sub REST API.