fsanggang / honeydew-ecto-notify-queue

A honeydew queue backed by Postgres via Ecto, using Postgres notifications to drive job reservation

HoneydewEctoNotifyQueue

Table of Contents

• HoneydewEctoNotifyQueue
  • Description
  • Notes
  • Setting it up
    • Installing
    • Generating the postgres migration
    • Starting the queue
  • Running the tests
  • Custom job configuration persistence

Description

HoneydewEctoNotifyQueue is a queue built for Honeydew that uses postgres notifications instead of polling. It was originally built before honeydew offered the ecto polling solution.

Contrary to the honeydew ecto polling solution, this package sets up two independent tables for managing the queue of jobs, and managing configuration over multiple instances;

A jobs table tracks jobs and job_configs table tracks configurations.

Notes

Honeydew.yield is not supported by this adapter.

Jobs are reserved using postgres' FOR UPDATE NOWAIT locking.

It is possible to suspend all job processing across instances by updating the suspended job config;

{:ok, _config} = HoneydewEctoNotifyQueue.Config.update_config(MyApp.Repo, "suspended", "true")

See more about configuration handling here

Setting it up

Installing

The package is available on hex.pm here.

You can add it to your mix.exs,

defp deps do
  [
    # ..,
    {:honeydew, "~> 1.1.5"},
    {:honeydew_ecto_notify_queue, "~> 0.1"}
  ]
end

Generating the postgres migration

You can generate a migration to set up the required db tables with

$ mix honeydew_ecto_notify_queue.db.gen.migration

Starting the queue

Note: You should read how to install honeydew here first

This queue takes some additional options. An example below,

def background_job_processes do
  [
    notifier_process(),
    {
      Honeydew.Queues,
      [
        :process, # queue name
        queue: {
          HoneydewEctoNotifyQueue, [
            repo: YourApp.Repo,
            max_job_time: 3_600, # seconds
            retry_seconds: 15, # seconds,
            notifier: YourApp.Notifier # this should match the `name` in `notifier_process` below
          ]
        },
        failure_mode: {Honeydew.FailureMode.Retry, times: 3}
      ]
    },
    {
      Honeydew.Workers,
      [:process, YourApp.Workers.ProcessWorker, num: 1]
    }
  ]
end

def notifier_process do
  %{
    id: Postgrex.Notifications,
    start: {Postgrex.Notifications, :start_link, [YourApp.Repo.config() ++ [name: YourApp.Notifier]]}
  }
end

def start(_type, _args) do
  children = [
    # ... The rest of your app's supervision tree
  ] ++ background_job_processes
  
  Supervisor.start_link(children, opts)
end

Running the tests

$ MIX_ENV=test mix do ecto.create, ecto.migrate
$ mix test

Custom job configuration persistence

This queue also adds support for persisting job configuration state via postgres. By default, this is how queue suspension is managed across multiple instances.

You can leverage the existing notification setup to synchronise other configurations across instances.

An example of this may be the disabling of automatic queuing of a job when an API is hit.

You can see an example of how to listen for configuration changes in examples/configuration_listener.ex