bparli / convey

Layer 4 load balancer with dynamic configuration loading

Convey

Layer 4 load balancer with dynamic configuration loading featuring proxy, passthrough and direct server return modes

Features

• Stats page (at /stats) with basic connection/bytes counters and backend server pool statuses
• Dynamic configuration re-loading of backend servers and associated weights. Configuration is loaded via a .toml file (see sample.toml for a full example).
• Tcp-based health checking of backend servers at a configured interval. If a server fails its health check it will be automatically removed from selection and added back once its health checks are successful.

Proxy Features

• Event-driven TCP load balancer built on tokio.
• Weighted round-robin load balancing. For uniform round robin simply leave out the weights or set them to be equal.
• TCP connection termination

Passthrough and Direct Server Return (DSR) Features

• Packet forwarding (no TCP termination)
• Minimal internal connection tracking
• NAT

Usage

Convey 0.3.5

Usage:
  convey
  convey --config=<config_file>
  convey (-p | --passthrough) --config=<config_file>
  convey (-d | --dsr) --config=<config_file>
  convey (-p | --passthrough)
  convey (-d | --dsr)
  convey (-h | --help)
  convey (-v | --version)

Options:
  -h, --help               Show this screen.s
  -p, --passthrough        Run load balancer in passthrough mode (instead of default proxy mode)
  -d, --dsr                Run load balancer in direct server mode (instead of default proxy mode)
  --config=<config_file>   Config file location [default config.toml].
  -v, --version            Show version.

Passthrough mode

For passthrough mode we need a couple iptables rules on the convey load balancer to handle ingress packets from the client and responses from the backend load balanced servers. Since TCP is not terminating we need to ensure the OS does not send a RST in response to any packets destined for a port that does not have a process bound to it. We need to do the same for any packets came back through from a backend server. Convey internally assigns ephemeral ports 32768-61000 to map connections to clients.

For passthrough mode on the convey load balancer

sudo iptables -t raw -A PREROUTING -p tcp --dport <LOAD_BALANCER_PORT> -j DROP
sudo iptables -t raw -A PREROUTING -p tcp --sport <BACKEND_SERVER_PORT> --dport 33768:61000 -j DROP

To run

sudo ./target/release/convey --passthrough --config=sample-passthrough.toml

DSR Mode

For dsr mode we need the same iptables rule for ingress packets. Responses from the backend load balanced servers will be going directly to the clients. The "listening" port on the convey load balancer must match the backend load balanced servers listening ports in this mode.

For dsr mode on the convey load balancer

sudo iptables -t raw -A PREROUTING -p tcp --dport <LOAD_BALANCER_PORT> -j DROP

In dsr mode the backend servers "participate" in that their response packets must be sent directly to the client. Convey does not do any encapsulation so, for example, a gre tunnel is not an option. Instead, Traffic Control can be used as an egress nat.

For dsr mode on backend servers

sudo tc qdisc add dev <LOCAL_INTERFACE> root handle 10: htb

sudo tc filter add dev <LOCAL_INTERFACE> parent 10: protocol ip prio 1 u32 match ip src <LOCAL_SERVER_IP> match ip sport <LISTEN_PORT> 0xffff match ip dst <LOAD_BALANCER_IP> action ok

sudo tc filter add dev <LOCAL_INTERFACE> parent 10: protocol ip prio 10 u32 match ip src <LOCAL_SERVER_IP> match ip sport <LISTEN_PORT> 0xffff action nat egress <LOAD_BALANCER_IP>

To run

sudo ./target/release/convey --dsr --config=sample-passthrough.toml

Proxy

No special setup neccessary

To run

sudo ./target/release/convey --config=sample-proxy.toml

Tests

The easiest way to run tests is to run them as superuser. This is because some of the tests spin up test servers as well as a convey load balancer instance.

sudo ~/.cargo/bin/cargo test

AF_XDP

The feature/xdp branch is a WIP using AF_XDP to loadbalance in passthrough and DSR modes. For now it will be maintained in a separate branch since it requires kernel versions 5.4 or greater.

Build

cargo build --release