airbnb / omniduct

A toolkit providing a uniform interface for connecting to and extracting data from a wide variety of (potentially remote) data stores (including HDFS, Hive, Presto, MySQL, etc).

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Handling connection failover

s3bw opened this issue · comments

@matthewwardrop are there plans to support connection failover?

For example I have two postgres instances one of which is a replica, when connection to the replica fails, I'd like to connect to the other instance. How would you recommend going about handling this case?

For me I see the option of having library try to connection on ip:host after failing, attempt the connection on a different ip:host combination.

This would also result in us having to think about the connection yaml.

@Foxyblue Hi Sebastien! Omniduct already supports a "naive load balancer", in that if host is provided as a list, one is chosen at random until it finds a host that has the required port bound.

This should therefore be as simple as:

Duct(.... host=['host:port', 'host2:port2', ....])

or in YAML:

name:
    protocol: ....
    host:
       - host:port
       - host2:port2

Note that if the connection drops after a host has been selected, the naive load balancing won't occur again until .reset() has been called on the Duct instance, and it will otherwise attempt to connect to the same endpoint. If a Duct subclass requires a reset operation to be called automatically, you should call it in your implementation of _disconnect().

Does that help?

Yes, this is perfect. Thanks. You may close this Issue.