Sneakers number of processes

[Stijn V]

We use passenger docker and have a file 02_sneakers_start.sh copied in /etc/my_init.d/ which starts our sneakers workers.

The content of the 02_sneakers_start.sh is

#!/bin/sh

su app -c "bundle exec rake sneakers:run"

Our sneakers.rb in initializers looks like this:

Sneakers.configure :heartbeat => 2,
                   :daemonize => true,
                   :log => "log/sneakers.log",
                   :amqp => Rails.application.config.x.secrets.amqp,
                   :vhost => '/',
                   :exchange => 'backend',
                   :exchange_type => :direct, #fanout, #direct
                   :durable => true,
                   :workers => 1

What we notice is that over time the number of sneakers running keeps increasing:

In total we have 10 workers defined in app/jobs so I wouldn't expect any more processes.

What is causing this? Is this not a correct way to start sneakers?

[Stijn V]

Update: after changing the heartbeat value into 30 and setting log for acceptance to INFO instead of DEBUG it seems to stabilise and not increase.

[Stijn V]

Unfortunately it still keeps increasing, just takes a lot longer.

[Michael Klishin]

2 is an unreasonably aggressive value.

There can be any number of reasons, including something in your environment starting new instances. Adding extra logging to the Spawner might help. I'm not aware of previous reports like this and we don't have any information about your environment.

[Michael Klishin]

One interesting scenario where I see the potential for extra workers to be started is when there are missing workers, which Sneakers determines via Kernel.get_const. If class names or module nesting were to change at runtime, that would result in a "missing workers" condition.

[Stijn V]

Hi @michaelklishin,

As I wrote earlier, it improved much after changing the value from 2 to 30. However it still keeps increasing.
There are no class name changes at runtime. There is no sneaker_worker_groups file or ENV variable. I only have 10 worker files in the jobs dir that include Sneakers::Worker

I'll try to get some logging in the Spawner.

[Stijn V]

One more thing that could be useful. I also use Sneakers for background processing which also uses delayed job execution with this: https://pastebin.com/kuYfE0eD

The config to use it is this:

    config.active_job.queue_adapter = :sneakers
    config.active_job.queue_name = 'backend.default'
    config.action_mailer.deliver_later_queue_name = 'backend.mailer'

When I run Sneakers::Worker::Classes I see this:

[ActiveJob::QueueAdapters::SneakersAdapter::JobWrapper, ActivityEventWorker, BootWorker, ConnectWorker, EscalationEventWorker, MailWorker, ParameterWorker, PersonPositionEventWorker, PrimaryWorker, QrScanEventWorker, RabbitEventsWorker]