bensheldon / good_job

Multithreaded, Postgres-based, Active Job backend for Ruby on Rails.

Home Page:https://goodjob-demo.herokuapp.com/

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Whitespace in `queues` configuration can cause issues.

TAGraves opened this issue · comments

We had the following queues config for a worker:

"queues": "+workload_5s: 3; +workload_15s,workload_5s:3; +workload_1m,workload_15s,workload_5s:2; +workload_2m,workload_1m,workload_15s,workload_5s:2; +workload_5m,workload_2m,workload_1m,workload_15s,workload_5s:2",

We observed that with this configuration, workload_5m jobs were never being worked. I removed all the spaces from the above configuration:

"queues": "+workload_5s:3;+workload_15s,workload_5s:3;+workload_1m,workload_15s,workload_5s:2;+workload_2m,workload_1m,workload_15s,workload_5s:2;+workload_5m,workload_2m,workload_1m,workload_15s,workload_5s:2",

Then the workload_5m jobs started getting picked up. So it seems like the space between the ; and the +workload_5m, was causing an issue. I don't see it documented anywhere that whitespace is relevant, and indeed the readme even has several examples of whitespace being used in the queue configuration, so I suspect this is a bug.

GoodJob should be whitespace tolerant. I'm probably missing some strip's 🤔 Here's where the string gets broken down into a configuration hash:

  • Where the configuration constant is broken down:

    queue_string, max_threads = queue_string_and_max_threads.split(':')
    max_threads = (max_threads || configuration.max_threads).to_i
    job_performer = GoodJob::JobPerformer.new(queue_string)

  • Where the comma-separated list of queues gets broken down:

    def self.queue_parser(string)
    string = string.presence || '*'
    case string.first
    when '-'
    exclude_queues = true
    string = string[1..]
    when '+'
    ordered_queues = true
    string = string[1..]
    end
    queues = string.split(',').map(&:strip)
    if queues.include?('*')
    { all: true }
    elsif exclude_queues
    { exclude: queues }
    elsif ordered_queues
    {
    include: queues,
    ordered_queues: true,
    }
    else
    { include: queues }
    end
    end

Also, I love how you've set up your queues 😍

Let me make a test case, but I think it needs more striping

Also, I love how you've set up your queues 😍

Unrelated: my one piece of feedback would be not to use the +-queue ordering. That has performance downsides and you likely want to clear faster-SLO jobs first rather than the slower-SLO ones.