Some thoughts on building webhooks

Introduction

On a quiet Wednesday, I picked up a ticket about webhooks. That was my first time sending webhooks, not receiving them. Because of that, I needed to pay attention to a few concepts and, of course, learn them.

I'm gonna try to cover some good practices and I hope you'll understand all of them. I'm not an expert in this topic, I just learned a little bit and I want to share what I learned.

Store it

First of all, you have to know WHAT and WHEN you're sending it, it's always good to log everything that goes out.

That'll be really useful for logs, retries and metrics.

I'm not just recommending that you use a logger (such as Winston or Pino). Store the event (payload, destination, status) not a new record for every HTTP call. One event, many delivery attempts. Keep the response code and error on each attempt so you can inspect, retry and replay.

You must have control.

Store that event before you try to send it, not after you get a response. If your process dies right after creating the member and right before the request goes out, the event simply disappears and there's nothing left to retry.

Sooner or later

Webhooks are complex, you don't have control over the receiver server, for example.

Because of that, you must make sure the other server gets the event, it doesn't matter if it's now or in two hours.

With that in mind, your webhook setup needs a retry mechanism.

If the other side returns an error, don't just send it once, retry until it succeeds, with limits, of course :).

Take your time

You'll try again, but don't retry 50 times at the same time, alright? You don't want to fry the other side with a lot of requests.

Take advantage of Exponential Backoff.

This strategy waits longer between each attempt. The delay grows with every retry, it doesn't have to double exactly, just increase enough to space out the requests.

That way, you give the receiver time to recover instead of hammering it with too many requests. For example:

There's one exception worth respecting: if the receiver answers with a Retry-After header, follow it. They're telling you exactly when to come back, and that beats any schedule you came up with.

What should trigger a retry?

As we already know, there are a lot of types of errors (statuses, to be more specific).

Take Too Many Requests (429) and Unauthorized (401), for example. Both are errors, but they're not the same kind of problem: the first one probably will go away on its own, the second one won't, no matter how many times you try.

You must be smart about what triggers a retry.

These types of errors are temporary, the server can recover later:

On the other hand, these kinds of errors below are NOT temporary. If the server is returning 400, probably the payload is wrong or badly formatted and resending it 10 times won't fix it. Mark them failed and stop.

And don't forget the failures that have no status code at all: a DNS error, a refused connection, or your own timeout kicking in.

Idempotency

Try to imagine this scenario with me:

• Your system sends an event (a member creation, for example)

• The receiver gets the event and handles it
• Before returning a success response, the receiver fails

Probably your app will treat it as an error and, since you should retry, you'll send it again.

It's bad... The receiver might create the member again.

Sending an id doesn't make you idempotent. It's a contract: you put a unique identifier on the event, the consumer stores it and ignores duplicates :D

or

The id belongs to the event, not to the request. Every retry of the same event has to carry the same id, otherwise the consumer has no way to tell it's a duplicate.

Explicit timeout

Never let your fetcher wait forever, you must add a timeout to your function.

10s USUALLY sounds good, but analyze your case, that's the key.

If the timeout fires, treat it like any other transport error: the delivery failed, and it should be retried (with backoff).

Here you can see a traditional way to implement a timeout in JavaScript:

Be safe

The attack here is not against your server. Someone can POST to your customer's webhook URL and pretend the request came from you.

To avoid this kind of situation, come up with a strategy: you should sign every webhook event and send the result in a header, Signature, for example.

This feature will make the consumer able to verify the events.

• Send the Signature through the request
• Make the verification approach available (through your SDK, for example)
• Recommend your users to verify before processing those events, and to reject signatures that are too old or invalid.

I like how Resend and Stripe do, I'd follow these examples.