A Quick Primer on Background Processing
Sidekiq is a background processing library for Ruby, which simply means that it allows you to schedule certain jobs that run behind the scenes. For example, say you’re writing a marketplace app that allows users to buy and sell merchandise. When a transaction is completed you want to notify the seller that their listing was purchased.
def create transaction = Transaction.new() Item.transaction do item = Item.lock.find(params[:item_id]) if transaction.valid? transaction.save item.update_attributes(available: false) send_email(transaction, item) redirect_to action: "show", item_id: item.id, id: transaction.id else @presenter = NewTransactionPresenter(transaction, item) render "items/index", status: :unprocessable_entity end end end private def send_email(transaction, item) Mailer.confirmation_email(transaction, item).deliver_now end
Sending emails is one of the slowest tasks that a web application can do. It requires making a network connection to an external service and then waiting for a response. You do not want the users of your app to have to wait on an email being delivered successfully before they can go off and do something else.
Fortunately, by using Sidekiq and similar services you can schedule long-running tasks like emailing to
be handled asynchronously. To get a sense for how this works in Sidekiq, let’s take a closer look at the
send_email method in the example above.
Sidekiq in Practice
In our new implementation using Sidekiq let’s say that the method definition for
send_email now looks something
def send_email(transaction_id, item_id) MailerWorker.perform_async(transaction_id, item_id) end
where MailerWorker is
class MailerWorker include Sidekiq::Worker def perform(transaction_id, item_id) transaction = transaction.find(transaction_id) item = Item.find(item_id) Mailer.confirmation_email(transaction, item).deliver_now end end
All that we’ve done, is wrap the
Mailer.confirmation_email method in the mailer worker’s perform task, which
is necessary for a Sidekiq worker to be processed. Another thing you may have noticed is that instead of passing
in the transaction and item objects directly to the
perform method we pass in a their ids and then query the
database for the actual record. This is due to the in-memory data-structure-store that Sidekiq uses called Redis.
Redis and Serialization
In the example above, The database I am querying with
Item.find(item_id) is Postgres, a relational database.
You can think of Postgres and other relational databases as just a bunch of tables that represent the type of object
you are trying to store. The rows of the tables represent individual records for the object the table represents and
and the columns represent that object’s attributes. Redis, is a key-value database so it works a bit differently. Instead
of having nicely laid out structure in the form of tables, a key-value database like Redis relies on storing information
in hashes/dictionaries. This makes it much more flexible as each record can have different fields, unlike a relational database.
You don’t want to send things to Redis that can change state. Furthermore the Sidekiq API uses JSON.dump to send things to Redis, and JSON.load to get them back. both of these only work with simple JSON datatypes like strings, integers, arrays, etc.
Apologies and More Resources
This deep dive has taken longer than anticipated, and I’m not really sure where I’m going with this anymore. Here is a great article on getting started with Sidekiq from someone much more eloquent and articulated than me.