Webhook Sentry

Webhook Sentry is a proxy that helps you send webhooks to your customers securely.

Why?

Security

Sending webhooks appears simple on the surface -- they're just HTTP requests after all. But sending them securely is hard. If your application sends webhooks, does your implementation

Prevent SSRF attacks?
Protect against DNS rebinding attacks?
Support mutual TLS?
Validate SSL certificate chains correctly?
Use an updated CA certificate bundle?
Specify reasonable idle socket and connection timeouts?

By proxying webhooks through Webhook Sentry, you get all of these for free.

Auditability

Sending webhooks involves making connections to untrusted and possibly malicious servers on the public internet. Maintaining an audit trail is essential for forensics and compliance. Limiting the set of instances that send such requests to a single proxy layer makes auditing simpler and more manageable.

Static Egress IPs

Many customers require webhook requests to be sent from a list or range of static IPs in order to configure their firewalls. In a cloud environment with autoscaling, you may not want to allocate static IPs to your application instances. In other situations, like serverless applications, it may be impossible to assign static IPs. With a centralized egress proxy layer, you only need to assign static IPs to your proxy instances.

Getting Started

Download the latest release for your platform
Run the downloaded binary:

whsentry

Webhook Sentry runs on port 9090 by default. You can configure the address and port in the listeners section of the config.

Usage

HTTP target

curl -x http://localhost:9090 http://www.google.com

HTTPS target

HTTP clients create a CONNECT tunnel when a proxy is configured and the target is a https URL. This does not give us the benefits of initiating TLS from the proxy. To get around this behavior, Webhook Sentry supports a unique way of proxying to HTTPS targets. Pass a X-WhSentry-TLS header and change the protocol to http:

curl -v -x http://localhost:9090 --header 'X-WhSentry-TLS: true' http://www.google.com

Although CONNECT is supported, I strongly recommend using the header approach to take advantage of the TLS capabilities of Webhook Sentry, like mutual TLS and robust certificate validation.

Mutual TLS

Specify clientCertFile and clientKeyFile in the YAML configuration to enable mutual TLS:

clientCertFile: /path/to/client.pem
clientKeyFile: /path/to/key.pem

Protections

SSRF attack protection

Webhook Sentry blocks access to private/internal IPs to prevent SSRF attacks:

$ curl -i -x http://localhost:9090 http://127.0.0.1:3000

HTTP/1.1 403 Forbidden
Content-Type: text/plain; charset=utf-8
X-Content-Type-Options: nosniff
X-Whsentry-Reason: IP 127.0.0.1 is blocked
X-Whsentry-Reasoncode: 1000
Date: Fri, 18 Sep 2020 07:15:20 GMT
Content-Length: 24

IP 127.0.0.1 is blocked

Unlike naive implementations, it also correctly checks the IP after DNS resolution. This example makes use of the 1u.ms service which can serve up DNS records using any IP we want:

$ curl -i -x http://localhost:9090 http://make-127-0-0-1-rr.1u.ms

HTTP/1.1 403 Forbidden
Content-Type: text/plain; charset=utf-8
X-Content-Type-Options: nosniff
X-Whsentry-Reason: IP 127.0.0.1 is blocked
X-Whsentry-Reasoncode: 1000
Date: Fri, 18 Sep 2020 07:21:58 GMT
Content-Length: 24

IP 127.0.0.1 is blocked

DNS rebinding attack prevention

A malicious attacker can set up their DNS such that it first resolves to a valid public IP adddress, but subsequent resolutions point to private/internal IP addresses. This can be used to exploit webhook implementations that validate the resolved IP using getaddrinfo() or equivalent, then pass the original URL to a HTTP client library which resolves the host a second time. Again, let's use 1u.ms to first return a valid public IP and then the loopback IP:

$ curl -i -x http://localhost:9090 http://make-3-221-81-55-rebind-127-0-0-1-rr.1u.ms/get

HTTP/1.1 200 OK
Access-Control-Allow-Credentials: true
Access-Control-Allow-Origin: *
Content-Length: 324
Content-Type: application/json
Date: Wed, 30 Sep 2020 07:38:47 GMT
Server: gunicorn/19.9.0

{
  "args": {},
  "headers": {
    "Accept": "*/*",
    "Host": "make-3-221-81-55-rebind-127-0-0-1-rr.1u.ms",
    "User-Agent": "Webhook Sentry/0.1",
    "X-Amzn-Trace-Id": "Root=1-5f743607-afdf257ca619f90a14fc92b8"
  },
  "origin": "73.189.176.226",
  "url": "http://make-3-221-81-55-rebind-127-0-0-1-rr.1u.ms/get"
}

Mozilla CA certificate bundle

Webhook Sentry uses the latest Mozilla CA certificate bundle instead of relying on CA certificates bundled with the OS. This avoids the problem of out-of-date root CA certificates on older OS versions. See this blog post for why this is important. Notably, Stripe's webhooks were affected by this issue and took hours to fix.

On startup, Webhook Sentry checks if there is a newer version of the Mozilla CA certificate bundle than on disk, and if so, downloads it.

Additionally, by virtue of being written in Go, Webhook Sentry does not rely on OpenSSL or GnuTLS for certificate validation.

Configuration

You can configure webhook-sentry with a YAML file.

listeners: A list of HTTP/HTTPS endpoints the proxy listens on. For HTTPS endpoints, also specify certFile and keyFile.

Example:

listeners:
  - type: https
    address: 127.0.0.1:9091
    certFile: /path/to/cert
    keyFile: /path/to/key

connectTimeout: Timeout for the TCP connection to the destination host.

Default: 10s

connectionLifetime: Maximum time a connection to the destination can be alive.

Default: 60s

readTimeout: Maximum time a connection to the destination can remain idle.