Don't call us, we'll call you: Sending webhooks with Rails

Integrating with other services is on the roadmap of almost any SaaS application. There’s only so much your application can do itself. Sometimes it’s best to leave tasks up to others who specialize on them.

Building specialized integrations with other tools is one way to approach this. However, wouldn’t it be nice if your application had a generic way for others to be notified about things that happen on your side so they can work with that information?

When it comes to passing data around in (near) realtime, webhooks are the way to go. There is no real standard for them, but most services are working with simple POST requests that send JSON as their request body. Webhooks are triggered for different events and transmit relevant data to other parties.

In the following, we’ll be talking about how to allow other services to integrate with your application, by allowing them to register webhook endpoints and notifying them when something interesting happens.

Registering webhook endpoints

In order to send webhooks, you have to know where to send them to. You have to provide your users with a way to set up new webhook endpoints. This can be done via user interface. In it’s simplest form, it would just be a textfield where users can enter a URL. In more advanced scenarios, you might allow them to specify multiple different endpoints and the types of events that should send a webhook to these endpoints.

To make things easier for your users, you can also decide to allow webhook endpoint registration via your API. This way, other services can just setup the endpoints they need, making things a lot easier for your users. Add authentication via OAuth on top, and it can be as easy as a single click of a button.

We start off by implementing a Webhook::Endpoint model. Each endpoint needs a target url, a set of events that it cares about, and an account this endpoint belongs to. To make things a little easier for ourselves, we use PostgreSQL’s array columns to store the events as a simple list.

class CreateWebhookEndpoints < ActiveRecord::Migration[5.1]
  def change
    create_table :webhook_endpoints do |t|
      t.string :target_url, null: false
      t.string :events, null: false, array: true
      t.references :account, foreign_key: true, null: false
      t.timestamps
      t.index :events
    end
  end
end

This example uses namespaces to keep things a bit more organized. We put all webhooks related code into a webhook directory. In the database we prefix all webhook related tables with webhook_.

# app/models/webhook/endpoint.rb
module Webhook
  class Endpoint < ApplicationRecord
    def self.table_name_prefix
      'webhook_'
    end

    # ...
end

To make the events array column work, we have to tell Rails about it by adding an attribute macro to the model.

attribute :events, :string, array: true, default: []

On the validation side, we only want valid http urls as target url. Luckily Ruby comes with a handy URI regular expression in it’s standard library, ready for us to use. We also add a presence validation for the events, so every endpoint is triggered by at least one event.

validates :target_url,
  presence: true,
  format: URI.regexp(%w(http https))

validates :events,
  presence: true

In order to quickly retrieve endpoints for one (or more) events later on, we implement a custom scope. It uses PostgreSQL’s contains operator (@>) to just return the endpoints interested in a particular event.

def self.for_event(events)
  where('events @> ARRAY[?]::varchar[]', Array(events))
end

Notice that we’re using Array(events) to convert the methods arguments into an array, or just leave it untouched if it already is one. This allows us to call the scope with one event name, or an array of multiple ones.

Next, we add some normalization and sanitization logic to the events setter method. This can be done, by simply overwriting the method, doing the necessary changes, and then calling super.

def events=(events)
  events = Array(events).map { |event| event.to_s.underscore }
  super(Webhook::Event::EVENT_TYPES & events)
end

In this case we’re normalizing the events into underscore strings. Afterwards we intersect the resulting array with a predefined array of event names (more on that later), to make sure all the events name are actually valid.

Finally, we add a method to deliver an event to this endpoint. It’s empty on purpose for now.

def deliver(event)
end

With the Webhook::Endpoint model in place, we need a controller, the view layer and most likely an API representation for it. There’s nothing special about the implementation of all of this, so we won’t cover it here.

Representing events

As mentioned before there is no standard on how webhook payloads should look like. In this example we use them to represent events that are triggered within your application. Assuming the application somehow works with a Project model, here’s an example of an event for a newly created project.

{
  "event_name": "project_created",
  "project": {
    "name": "Example Project",
  }
}

The webhook payload follows a simple structure. It always has an event_name and a set of embedded models with relevant data. You’re of course free to settle on a different format. Instead of embedding the event name in the payload itself, you might choose to send an X-Event-Name header with your request. Whatever you do, just stick to a pattern for all the webhooks you’re sending and document it. All the developers building integrations with your application will thank you for it.

Let’s implement a Webhook::Event model to represent a single event. This can be done as “plain old ruby object”. If you want to keep track of the state of every single event, you can store all events in the database as well. For the purpose of this example, we’ll keep it simple.

The implementation of Webhook::Event is straight-forward. It’s an object with two instance variables. One for the event’s name, and another one for its (optional) payload. To keep track of valid events, we add the EVENT_TYPES constant used earlier. Whenever you add a new event, add it to this list.

# app/models/webhook/event.rb
module Webhook
  class Event
    EVENT_TYPES = %w(
      project_created
      project_updated
      project_destroyed
    ).freeze

    attr_reader :event_name, :payload

    def initialize(event_name, payload = {})
      @event_name = event_name
      @payload = payload
    end

    # ...
end

As we want events to be delivered as JSON in the HTTP request’s body, we have to implement serialization for this event model. This can be as simple as taking a copy of the payload and adding the event_name to it.

def as_json(*args)
  hash = payload.dup || {}
  hash[:event_name] = event_name
  hash
end

However, we want to keep our code clean and not come up with the entire payload for every single event. If you’re using ActiveModel::Serializers to handle serialization of your API resources, you can leverage it to properly serialize the payload for you.

def as_json(*args)
  hash = payload.transform_values do |value|
    serialize_resource(value).as_json(*args)
  end

  hash[:event_name] = event_name
  hash
end

private

def serialize_resource(resource)
  ActiveModelSerializers::SerializableResource.new(resource, {})
end

This way, you can create events and simply pass your model instances as payload.

Webhook::Event.new(:project_created, { project: project })

Triggering events

With both Webhook::Endpoint and Webhook::Event in place, it’s time to actually trigger events. To keep things organized and DRY, we use a module to implement this. Whenever you want to trigger events, you just include that module in your model and call one of the helper methods.

We start off by adding a webhook_scope method. It’s just a stub, that has to be overwritten by each model to return the scope of the webhooks. If you’re getting a bit confused by that, don’t worry. It’ll get clear once you see the module used in an actual example in a few moments.

# app/models/webhook/delivery.rb
module Webhook
  module Delivery
    extend ActiveSupport::Concern

    def webhook_scope
      raise NotImplementedError
    end

    # ...
end

Next, we add a method to trigger events. It just takes two arguments. One for the event name, and one for its payload. It then constructs an Webhook::Event object from those and asks every endpoint interested in the event to deliver it.

def deliver_webhook_event(event_name, payload)
  event = Webhook::Event.new(event_name, payload || {})
  webhook_scope.webhook_endpoints.for_event(event_name).each do |endpoint|
    endpoint.deliver(event)
  end
end

To make things even simpler, we implement a default webhook_payload method, as well as a deliver_webhook method. The latter relies on a naming convention to generate an event for the current model based on an action name.

def webhook_payload
 {}
end

def deliver_webhook(action)
  event_name = "#{self.class.name.underscore}_#{action}"
  deliver_webhook_event(event_name, webhook_payload)
end

Coming back to our projects example from earlier, we can now trigger events like this:

# app/models/project.rb
class Project < ApplicationRecord
  include Webhook::Delivery

  after_commit on: :create do
    deliver_webhook(:created)
  end

  belongs_to :account

private

  def webhook_scope
    account
  end

  def webhook_payload
    { project: self }
  end
end

As most events revolve around creating, updating, and removing models in your database, we can also build a simple module to handle those events automatically.

# app/models/webhook/observable.rb
module Webhook
  module Observable
    extend ActiveSupport::Concern
    include Webhook::Delivery

    included do
      after_commit on: :create do
        deliver_webhook(:created)
      end

      after_commit on: :update do
        deliver_webhook(:updated)
      end

      after_commit on: :destroy do
        deliver_webhook(:destroyed)
      end
    end
  end
end

Please notice that the module is using after_commit callbacks. It’s important to only send webhooks after the data is actually present in your database. This is to prevent weird “not found” errors in the occasions where webhooks are delivered faster than your database takes to commit the transactions or it’s part of a larger transaction that fails shortly afterwards.

Using the Webhook::Observable module, adding webhook events to a model becomes as simple as this:

class Project < ApplicationRecord
  include Webhook::Observable

  belongs_to :account

private

  def webhook_scope
    account
  end

  def webhook_payload
    { project: self }
  end
end

Delivering webhooks

To deliver webhooks, I strongly recommend using a background job queue. As delivery depends on 3rd party services, you should expect them to be slow or even unreachable. As your application might send webhooks to multiple endpoints per event, you definitely don’t want your users having to wait until all of them are delivered. Using a queue also allows for retries when delivery fails for some reason.

In this example, we’re using Sidekiq as background job queue. This is not a requirement, though. ActiveJob with any backend you like will to this job as well.

# app/workers/webhook/delivery_worker.rb
require 'net/http'

module Webhook
  class DeliveryWorker
    include Sidekiq::Worker

    def perform(endpoint_id, payload)
      return unless endpoint = Webhook::Endpoint.find(endpoint_id)
      response = request(endpoint.target_url, payload)

      case response.code
      when 410
        endpoint.destroy
      when 400..599
        raise response.to_s
      end
    end

    # ...
end

The implementation will raise an exception for all HTTP status codes that indicate some sort of error. Sidekiq will catch the exception and just retry the delivery a few moments later. Please know that HTTP status code 410 Gone is an exception. It’s a small idea picked up from Zapier. Whenever the endpoint returns a 410, we consider it no longer valid and just remove it from the database.

The actual HTTP request is nothing fancy. As a result, we just use Net::HTTP from Ruby’s standard library.

def request(endpoint, payload)
  uri = URI.parse(endpoint)

  request = Net::HTTP::Post.new(uri.request_uri)
  request['Content-Type'] = 'application/json'
  request.body = payload

  http = Net::HTTP.new(uri.host, uri.port)
  http.use_ssl = (uri.scheme == 'https')

  http.request(request)
end

Finally, we add the implementation of Webhook::Endpoint#deliver we skipped earlier. It enqueues a new Webhook::DeliveryWorker for the given event.

def deliver(event)
  Webhook::DeliveryWorker.perform_async(id, event.to_json)
end

It’s important to serialize the event model in the very moment the delivery is triggered. Otherwise the payload might have changed until the delivery takes place, resulting in weird data and confused developers.

Taking things one step further

While the implementation shown here covers the most important parts to get webhooks implemented in your application, there are a few more things you should consider.

Allowing for authenticity checks

As webhooks are usually delivered to more or less public endpoints it’s best to add some way that allows the receiving application to ensure the data is actually coming from you. One common way to do this is to add a signature somewhere in the payload. This can either be done in the JSON body or in a custom HTTP header. The signature should be based on the payload itself and a secret only known to the sending and the receiving application.

Accidental Denial of Service attacks

Based on the details of your application, it might happen that you send a lot of requests to the endpoints. Some of them might not be able to handle the load. With retries in place, the number of requests sent to the endpoint will only increase, making things worse. It’s a good idea to have some sort of rate limiting in place or even temporarily disable an endpoint that is unable to process requests right now.

Wrapping up

Webhooks are a great way to allow applications to integrate with yours. At their core, they’re simple and easy to understand HTTP requests. With just a little bit of code, they allow your users to use your application in ways that you didn’t think of before.

You can get the source code of everything explained in this article at GitHub’s Gist. You should be able to use it in your existing applications, with just minor modifications.