Delayed Job vs. Sidekiq - Which One Should You Choose?

A few weeks ago I posed a question to Reddit asking the contributors to tell me which background job system I should use for a new project, Delayed Job or Sidekiq.    I took everyone's feedback and wrote this handy little guide.   

Choosing the right framework to process long running tasks (like sending email or running reports against Amazon Redshift) the background is crucial if you want your Ruby on Rails-based web applications to run smoothly.   When you're trying to choose  which framework to use,  comparing the pros and cons of each helps clear the picture.  In this post,  I'll help you do just that by comparing two popular Rails background job frameworks, Delayed Job and Sidekiq.   Keep reading for the goodies.

Delayed Job (DJ)

This Ruby background job framework helps you maintain a job table in the database. This helps you to track tasks and know where each task is in the lifecycle of the job. Here are some important pros and cons to consider when using Delayed Job:

Advantages

• Easy Integration with Rails. Whether you're using a relational database or non-relational database, you can easily integrate DJ with Rails.

• Custom Data Store. One of the main benefits of using DJ is that you can use your main data store. This is helpful as it gives you control over the offline processing and allows you to customize your data store to fit your needs.

• Minimal Dependencies. Delayed Job allows you to keep dependencies to a minimum. That means you don't have to run multiple dependencies at once. For example, if you only need three services to run your program, DJ outshines Sidekiq. Instead of requiring multiple services and dependencies to run your program efficiently, you can limit the dependencies that you need. This makes using the framework more efficient in regards to resource use in comparison to Sidekiq.

• Reliable Processes. Thanks to its longevity, DJ offers stability and reliability when it comes to processing background jobs. It's helped Shopify handle several tasks, including search engine updates and newsletter distribution.

Disadvantages

• Performance. While some have claimed to run DJ per second at fast speeds even on retail hardware, the general consensus is that DJ lacks in performance when you compare it to how well and fast Sidekiq performs.

• Database Dependent. Because DJ heavily relies on databases, you may run into an issue if you have a queued jobs backlog that may be too much for your database to handle. That's because having an application and job table that rely on the same database can put a heavy load on the database, which can hamper processing.

• Lacks a Dashboard. Without a dashboard that shows the status of your jobs in real-time, it can be challenging to know which task to prioritize when you need to debug jobs that get stuck or fail during processing.  You can use the delayed_job_web gem, but it lacks some features of a full-fledged dashboard.

Sidekiq

Sidekiq is a background job scheduler originally authored by Mike Perham and is open source. It's popular among enterprises, such as Comcast and Everlane, and ideal for those who have queries that require fast speed. Here are some important pros and cons to consider when using Sidekiq:

Advantages

• Super fast speed. Speed is one of the top reasons developers opt for Sidekiq. That's because Sidekiq is backed by Redis, which is a data store that's mostly memory and is used as a database. This makes retrieving and creating data faster than if the framework relied completely on a database. Sidekiq takes just 22 seconds to process as much as 100,000 jobs compared to DJ's 465 second-timeframe.

• Multithreading. When you need high concurrency, then Sidekiq is the ideal database to choose. Sidekiq provides higher concurrency than DJ thanks to its multithreading capabilities. This means that multiple jobs can be queued in the background before being processed without impacting the typical synchronous workflow of your application. It's also on-demand and available whenever you need it.

• Automatically runs Redis. With Sidekiq, you can rely on it to automatically run Redis. This makes it easy to set up and ideal if you don't want live queues in your main cache. You can also easily add it as an app or use a Redis instance on different cloud platforms, such as Heroku.

• Scalable. The great part about using Sidekiq is that you can scale your project if your client base grows thanks to Sidekiq's premium upgrades at the Pro- and Enterprise-level subscriptions. This makes it ideal when you're ready to scale as you grow from a small business to an enterprise.

• Offers a streamlined and simple API. Sidekiq's API is clean and offers versatility since you can script the web user interface (UI) with the API. Because Sidekiq's API is public, you can view queues and task in real-time. It also has a real-time dashboard, which provides insight into the status of jobs and queues.

• Enhanced support. While Sidekiq supports Ruby clients, Java clients can also use Sidekiq. Also, you can enjoy dedicated support via email when subscribing to its Pro- and Enterprise-level services.

Disadvantages

• Depends heavily on Redis. If you don't have Redis in your developer environment, then Sidekiq may not be the best framework to use. Instead of using a database to store your upcoming jobs, Sidekiq uses Redis. That means there may be a learning curve if you're not familiar with Redis.

• Data loss. If your Redis instance crashes while you are de-queueing or enqueueing a task, you could lose data. This is because Redis is an in-memory data structured data store.

• Requires several dependencies. In order to successfully run Sidekiq, it requires several services. This may mean running Redis simultaneously while using Foreman to launch your app.

In Conclusion...

Both Sidekiq and Delayed Job offer great solutions for processing background jobs. Ultimately, the framework you choose should be based on your needs. Just consider your need for support, simplicity, performance,  stability and resiliency when you're making your decision.  Good luck!

Try SQLBot - It's Free!

Sign up for a free account on SQLBot to see how easy it is to get SQL reports into your Slack.  What are you waiting for, it's free!