Replacing Sidekiq with GoodJob

We recently moved a client’s ActiveJob processor from Sidekiq to GoodJob. In this article I want to share why we did it, how we did it, and our immediate first impressions having done it.

First of all, some background!

Sidekiq is a well-known and well-used Job processor used a lot in the Rails ecosystem. It is very solid and incredibly performant, with excellent documentation and a huge community of users.

Why on earth would we move then? There are a few reasons but before we get into that lets talk about GoodJob.

GoodJob does the same thing Sidekiq does, but instead of being Redis based, it uses Postgres. As such, it can piggy-back on your existing database.

Now, I’m vastly underselling what both can do here, so apologies to the respective creators.

Why we decided to move

The primary reason was money. Sidekiq needs Redis, and our client was paying about $30 a month across different environments to handle jobs.

This particular client was handling maybe 20 jobs a day, at the absolute max. So as you can imagine, they weren’t really getting their money’s worth from the hosting setup.

This lack of massive throughput meant we weren’t worried about what would happen to the database if it suddenly had way more reads and writes happening. It won’t.

Getting rid of Redis didn’t just save some money, it also removed additional complexity from the stack, which is always a worthy goal. Aside from Sidekiq, there were two other gems we were able to remove, making our Gemfile easier to read, and reducing the memory overhead of the app.

Sidekiq doesn’t look horrible, but GoodJob’s UI looks lovely, and there are some nice features within GoodJob such as history and search that you don’t get for free with Sidekiq.

We were already using ActiveJob to handle our jobs in Rails, so in theory this should be a fairly easy swap. No application code should need to change.

Steps we took to move from Sidekiq to GoodJob

These are the steps we were able to use on a very small and simple project, for more complex projects, you should certainly lean heavily on GoodJob’s documentation.

Add GoodJob to your Gemfile

We added GoodJob into our Gemfile;

gem 'good_job'

Then we installed it;

bundle install

Tell Rails to use GoodJob

Depending on how you set up Sidekiq, you will either need to update one place, config/application.rb, or several for each environment, e.g. config/environments/production.rb. Either way, look for config.active_job.queue_adapter and replace with;

config.active_job.queue_adapter = :good_job

Create some GoodJob migrations

Let your database know about GoodJob by generating and then running the relevant migrations;

bin/rails g good_job:install
bin/rails db:migrate

Get access to GoodJobs UI

Add a route in config/routes.rb to the GoodJob UI, and make sure it is locked down.

authenticate :user do
  mount GoodJob::Engine => 'jobs'
end

The above assumes something like Devise is being used, and that you want to visit mysite.com/jobs to access the UI.

Tell your server to run GoodJob

This client runs on Heroku, which uses Procfiles, you want to change whatever was calling bundle exec sidekiq to not do that anymore. For us it meant updating the Procfile to call GoodJob;

worker: bundle exec good_job start

Remove Sidekiq

With the above steps complete, we’re in a position to remove Sidekiq. For us that meant removing the Sidekiq and Redis gems, a config file specifically for Sidekiq, replacing a call to Sidekiq in the routes file, and some README lines talking about installing Redis locally.

That was it? It was way less painful than I imagined it might be. Because of the nature of this project, we were able to deploy without having to worry about if one job got lost in Sidekiq.

First impressions

This was the first time we’ve moved something between two ActiveJob adapters, and it was surprisingly straightforward. As I’ve said, this was the most simple use case, but I’m already thinking of other projects we could slim down by doing the swap.

I’ve already made use of the improved UI to see where we might make an improvement to a job in the future.

Being able to search jobs is very powerful, especially on staging when you have easy to know test data.

The ability to search does bring in some data-access concerns. Make sure that only people who should have access to live data have access to see jobs, depending on the nature of the jobs, you might see sensitive information.

GoodJob seems to process jobs quickly, certainly at least as quickly as Sidekiq appeared to. We didn’t run benchmarks for this particular client because of how small and infrequent the jobs are.

We’ve been able to shave $30 a month off our client’s hosting bill! Which was a nice way to kick off the new year with them! “Hello, you’re paying less”.

All in all I’m very happy we did this. We have a smaller application, taking up less resource, doing the exact same work.