Add Webhooks to Your API the Right Way

In the last 10 years, APIs have gone from nice-to-have to must-have, certainly for Software-as-a-Service (SaaS) applications. At the same time, users today are more impatient and expect more instant and interconnected apps. To answer user needs, developers are now adding webhooks to their APIs. Webhooks are the mechanism through which data from an API can be delivered proactively, without receiving a request for each response. We’ve found that given loose constraints, there are many ways developers might implement webhooks.

This post shares some high-level insights behind why you should include webhooks and how to add them to your API the right way.

Why you want to create webhooks

Webhooks are an architectural pattern that enable developers to receive updates to data as they happen rather than polling for the latest updates. The investment you put in up front building a webhook-enabled API can save your system resources and delight developers and end users alike.

A typical API lets developers read the latest data from your application. Perhaps they can query for specific data and may even be able to write new data if they have appropriate permissions. These sort of CRUD operations (Create, Read, Update, and Delete) provide the basic elements needed to emulate the functionality of your application.

However, your application exists within a network of other applications. Since changes to your user data affects many others, you need a robust model that takes this reality into account. An event-driven approach ensures you can send the right notification to the right application at the time it occurs.

An event-driven approach ensures you can send the right notification to the right application at the time it occurs.

Let’s say one feature of your application includes contacts, each of which has a name, email address, and phone number. The moment a new contact is added to your application, some other application may want to use that data to send a text message, look up their email address in another system, or backup the contact data. Without a way to alert other applications when a new contact is added, you force them to constantly poll for new data.

Polling is bad for everyone. It’s taxing to servers on both sides, which spend most of their time and resources with requesting responses that contain nothing new. When there is data, it’s only as new as the polling interval, which means it might not even satisfy a user’s thirst for instant data access. Other than webhooks, the only way to have up-to-the-second access to new data is to poll every second. While some developers may try that, it’s not likely to be sustainable.

Webhooks will make your end users happy, delight developers who use your API, and give your servers a break. So how do you get yourself some of this triple-win?

Create your subscription endpoint

Before sending notifications as webhooks, you need to know what to send notifications about. That’s where a subscription endpoint comes in handy. Developers will “subscribe” to certain actions and tell you where to send notifications.

Note: You may be familiar with adding a single webhook URL into an admin interface when working with other APIs. That is called a static webhook and is not recommended for robust subscription notifications. Static webhooks can only notify one URL and are limited to a single set of criteria. Plus, a user needs to manually add every notification they want. By comparison, subscription webhooks allow a single application to receive multiple types of notifications. Each can be added via API so a user is not required to interact with each subscription set up.

For most apps implementing webhooks, they already have an API. Even if you’re starting fresh, you’re likely planning out the endpoints for your API, which map to resources in your app. Sticking with the example in the previous section, your app might store contacts. For your API, the endpoint would be something like /contacts. You’ll make another endpoint for webhooks, such as /webhook, and treat it like any resource for your API.

Your /webhook endpoint would likely have the actions listed above, which would allow you to list, create, read, update, and delete webhooks.

Important: You would need to store and retrieve the data with the same security that you use to access the other resources in your application. That is, an external developer shouldn’t be able to work with a webhook {id} that is not registered to the account of the authenticated user.

When creating or updating a webhook, developers will need to pass details in the body of the request. At a minimum, you’ll need to accept a resource or events to watch for changes and a target URL (where to send the notifications). A common convention is to use noun.verb notation for describing webhooks, such as contact.create and contact.update. Whatever you choose, be consistent, and remember to send it along with each notification.

You’ll want to think about what criteria you need to determine what notifications to send. For example, you might have notifications for all new contacts or for updates to a specific contact id. Start with just the most common events. It’s easy to add more events later, but painful to remove them, especially once developers are depending on them.

Of course, with your webhook endpoint set up, all this does is track the webhooks. You’ll need another process to actually identify the times when we need to send notifications.

Implement your webhook queue

Now that you have a way for developers to subscribe to webhooks and update settings, you're ready to start sending notifications when the criteria matches. But when should you check the criteria and send the notifications? There are a few approaches you can take, each with its own advantages, complexity, and scalability.

• Do it inline: This is the simplest and most naive approach and should be reserved for prototypes or test environments. When an event occurs that might activate a webhook (such as a user adds a new contact), check whether a webhook fits the criteria. For all webhooks that do match, fire off a connection to the URL on file. The questionable part, for scaling, is making the request inline while the end user waits. A better solution is to create a queue so you can better deal with timeouts and retries.
• Create a DB queue: When you send webhooks inline, all your code is in one place. With this option, you’ll be creating a record in your existing database for each notification you need to send. Rather than initiating a connection immediately, you’ll need a separate process to frequently look for new notifications and send them. Since the end user isn’t waiting, timeouts are not as big of a concern. And if you need to retry a notification, just keep it marked active until it succeeds or is called the maximum number of times.
• Use a proper queue: If you know you’ll need to scale your solution, use a tool specifically designed for that. You can use open source scalable queueing solutions like RabbitMQ or a service like Amazon Simple Queuing Service. This way, your interaction is limited to adding and removing “messages,” which tell you what webhooks to call. Like the DB queue, you need a separate process to consume items from the queue and send notifications. In addition to using a tool designed for this purpose, a proper queue also saves database resources for what it does best–providing data to your primary application.

Tip: Whichever choice you use for queuing your webhook notifications, you’ll want to send them as quickly as you can. It may be tempting to run a cron job every minute, but a minute is a long time to wait for someone who wants to see if your webhooks work.

This post has covered a few of the high-level architectural decisions you’ll need to make as you implement webhooks. In the future, we’ll look at methods of retrying webhooks, authenticating payloads, and other factors of successful webhooks.