Sidekiq Render Autoscale plugin

This Sidekiq plugin allows Render dynos to be started, stopped, and scaled based on job workload. Why? Because running non-stop Sidekiq dynos on Render can rack up unnecessary costs for apps with modest background processing needs.

This is a adaptation of the sidekiq heroku autoscale project to work with render.

Tested with Sidekiq 6, but should be compatible with other recent Sidekiq versions.

How it works

This plugin operates by tapping into Sidekiq startup hooks and middleware.

Whenever a server is started or a job is queued, the appropriate process manager is called on to adjust its scale. Adjustments are throttled across process instances (dynos) so that the Render API is only called once every N seconds – 10 by default.
When workload demands more dynos, scale will adjust directly upward to target capacity.
As workload diminishes, scale will adjust downward one dyno at a time. When downscaling a process, the highest numbered dyno (ex: worker.1, worker.2, etc...) will be quieted and then removed from the formation. This combines Render's autoscaling logic with Sidekiq's quieting strategy.

Gem installation

gem 'sidekiq'
gem 'sidekiq-render-autoscale'

If you're not using Rails, you'll need to require sidekiq-render-autoscale after sidekiq.

Environment config

You'll need to generate a Render platform API token that enables your app to adjust its own dyno formation. This can be done through the Render CLI with:

render authorizations:create

Copy the Token value and add it along with your app's name as environment variables in your app:

SIDEKIQ_RENDER_AUTOSCALE_API_TOKEN=<token>
SIDEKIQ_RENDER_AUTOSCALE_APP=<app-name>

The Render Autoscaler plugin will automatically check for these two environment variables. You'll also find some setup suggestions in Sidekiq's Render deployment docs. Specifically, you'll want to include the -t 25 option in your Procfile's Sidekiq command to maximize process quietdown time:

web: bundle exec rails start
worker: bundle exec sidekiq -t 25

Plugin config

Add a configuration file for the Render Autoscale plugin. YAML works well. A simple configuration with one worker process (named in your Procfile) that monitors all Sidekiq queues and starts/stops in the presence of work looks like this:

config/sidekiq_render_autoscale.yml

app_name: test-app
processes:
  worker:
    system:
      watch_queues: *
      include_retrying: true
      include_scheduled: false
    scale:
      mode: binary
      max_dynos: 1

Then, add an initializer to hand your configuration off to the plugin:

config/initializers/sidekiq.rb

config = YAML.load_file('<path/to/config.yml>')
Sidekiq::RenderAutoscale.init(config)

A more advanced configuration with multiple process types that watch specific queues would look like this – where first and second are two Render process types:

api_token: <optional - for dynamic insertion only!>
app_name: test-app
throttle: 20
history: 7200
processes:
  first:
    system:
      watch_queues:
        - default
        - low
      include_retrying: false
      include_scheduled: false
    scale:
      mode: binary
      max_dynos: 2
    quiet_buffer: 15

  second:
    system:
      watch_queues:
        - high
      include_retrying: false
      include_scheduled: false
    scale:
      mode: linear
      max_dynos: 5
      workers_per_dyno: 20
      min_factor: 1

Config Options

api_token: optional, same as SIDEKIQ_RENDER_AUTOSCALE_API_TOKEN. Always prefer the ENV variable, or dynamically insert this.
app_name: optional, same as SIDEKIQ_RENDER_AUTOSCALE_APP.
throttle: number of seconds to throttle between scale adjustments. The default is 10, so the Render API will only be hit once every ten seconds regardless of how many time the process is pinged during that timeframe. This value also dictates the tick frequency on the web UI history graph.
history: number of seconds to track history in the web UI. The default is 3600, or 1 hour. The history graph renders ticks using the history duration divided by throttle time – so 3600 seconds of history on a 10 second throttle produce 360 data points. Therefore, it's best to keep these settings in modest proportions to one another. You'll probably be sad if you try to display days or weeks of history.
processes: a list of Render process types named in your Procfile. For example, worker or sidekiq.
process.system.watch_queues: a list of Sidekiq queues to watch for work, or * for all queues. Queue names must be mutually exclusive to avoid collisions. That means a queue name may only appear once across all processes, and that * (all) may not be combined with other queue names.
process.system.include_retrying: specifies if the Sidekiq retry set should be included while assessing workload. Watching retries may cause undesirable levels of uptime.
process.system.include_scheduled: specifies if the Sidekiq scheduled set should be included while assessing workload. Watching scheduled jobs may cause undesirable levels of idle uptime. Also, no new jobs will be scheduled unless Sidekiq is running.
process.scale.mode: accepts "binary" (on/off) or "linear" (scaled to workload).
process.scale.max_dynos: maximum allowed concurrent dynos. In binary mode, this will be the fixed operating capacity. In linear mode, this will be the maximum extent that dynos may scale up to.
process.scale.workers_per_dyno: Linear mode only. This specifies the anticipated workforce per dyno to calculate scale around. This should generally align with Sidekiq's concurrency setting.
process.quiet_buffer: number of seconds to quiet a dyno (stopping it from taking on new work) before downscaling its process. This buffer occurs before reducing the number of dynos for a given process type. After downscale, you may configure an additional quietdown threshold. Note: during this quiet buffer your formation includes a decomissioned dyno, which is awkward – thus no other scale adjustments (up or down) are allowed until the quieted dyno has been dropped. Be accordingly judicious with this buffer.

Web UI

The web UI is an optional extension of Sidekiq's web UI. To activate it, just require sidekiq/render_autoscale/web after the base sidekiq/web, and then mount Sidekiq::Web as normal:

require 'sidekiq/web'
require 'sidekiq/render_autoscale/web'

Rails.application.routes.draw do
  mount Sidekiq::Web, at: '/sidekiq'
end

Tests

Nothing fancy...

# start a redis server
redis-server test/redis_test.conf

# then run tests in another terminal window
bundle exec rake test

Contributors

Justin Love @wondible, https://github.com/JustinLove
Benjamin Kudria https://github.com/bkudria
claudiofullscreen https://github.com/claudiofullscreen
Fix Peña https://github.com/fixr
Gabriel Givigier Guimarães https://github.com/givigier
Matt Anderson https://github.com/tonkapark
Thibaud Guillaume-Gentil https://github.com/jilion

Licence

Sidekiq Render Autoscale plugin is released under the MIT license.

jlneto / sidekiq-heroku-autoscale