Setting up a dedicated Sidekiq queue

One of our lovely clients, Niice, had an issue where they have some very small tasks that need to be prioritised anytime they hit Sidekiq, and some fairly slow, longer running tasks that are less important.

The implementation that served them well up until this point was to have dedicated queues for each type of task, with “realtime” queues getting the highest priority.

With this setup, you tell Sidekiq, normally in a config/sidekiq.yml file, which queues to process and in which order.

#...
:queues:
- ["realtime", 5]
- ["mailers", 5]
- ["high_priority", 3]
- ["default", 2]
- ["low_priority", 1]
#...

One issue that I didn’t appreciate with this setup was the numbers. I mistakenly thought that when you used numbers, Sidekiq would still take the order of the queue into account. In the above example “realtime” should come before “mailers”.

However, as the Sidekiq wiki on queues states;

Note: Sidekiq does not support mixing ordered and weighted queue modes.

So all we were really doing here was saying “realtime” and “mailers” are the most important, but they are equally important. Which isn’t actually what we intended.

If we removed the numbers, so it is just positional, we get something a bit more sensible, a config like the following tells Sidekiq “only process high_priority queues when there is nothing to process in realtime, and nothing to process in mailers”

#...
:queues:
- realtime
- mailers
- high_priority
- default
- low_priority
#...

This is better, but can still lead to slow “low_priority” items taking up all available threads. If all the threads are in use when a “realtime” job comes in, Sidekiq will still need to wait for a thread to come available.

Enter Reserved Queues

The solution to ensure “realtime” is always given the time of day is to give it its own Sidekiq process.

Just in the same way that Sidekiq can handle multiple threads at once, it can also handle multiple processes at once. So long as you’re pointing Sidekiq at the same Redis and Postgres instances, you can have as many as you want! With the caveat that your Redis and Postgres instances have upper limits to how many connections they can accept at a time.

This client deploys to Heroku, which uses a Procfile to specify what processes to run. That means we could do something like;

web: ...
release: ...
worker: bundle exec sidekiq -C config/sidekiq_default.yml
realtime_worker: bundle exec sidekiq -C config/sidekiq_realtime.yml

The default location for Sidekiq’s config file is config/sidekiq.yml and if you use that convention you don’t need to tell Sidekiq where to go looking for it. Because we want to split out which queues are being used, we can create two configs, one for each Sidekiq instance.

The sidekiq_realtime.yml can specify just realtime queues, and the sidekiq_default.yml can either exclude realtime, or also process them ahead of any other.