Elasticache client support

[Priyatam Mudi]

How do you integrate elasticache client provider with Spyglass for dynamic clients and auto discovery? In particular, what would be a minimal bin-connection-factory for this setup?

[Michael Klishin]

We don't. If you are curious about how you would do it, take a look at the code. We'd perhaps need to extract a common protocol for clients first, although this was never a goal for Spyglass.

However, if ElasticCache can be considered a Memcached implementation, I'm all for it.

[Ryan Wilson]

To be clear, ElastiCache is not a Memcached implementation. It's a set of AWS tools for provisioning and managing either a memcached or redis cluster. From an application standpoint, it's memcached.

What @priyatam was getting at is that part of those AWS tools include propagating the list of nodes throughout the cluster, such that a client can "discover" them.

Conveniently, the AWS ElastiCache client is a fork of spymemcached. So, this might be as simple as:

• On spyglass dependency, exclude net.spy/spymemcached and spy/spymemcached
• Add dependency on [com.amazonaws/elasticache-java-cluster-client "1.1.0"]
• Configure connection factories correctly

A cursory local test confirms that swapping the dependency works, at least:

user> (import net.spy.memcached.BinaryConnectionFactory)
net.spy.memcached.BinaryConnectionFactory
user> (import net.spy.memcached.ClientMode)
net.spy.memcached.ClientMode
user> (def cf (BinaryConnectionFactory. ClientMode/Static))
#'user/cf
user> (def c (mc/bin-connection "localhost:11211" cf))
#'user/c
user> @(mc/set c "foo" 0 "bar")
true
user> (mc/get c "foo")
"bar"

Note: the ClientMode nonsense is something AWS introduced to specify to the BinaryConnectionFactory

I haven't tried it yet, but I suspect using ClientMode/Dynamic and providing the ElastiCache configuration endpoint as the server-list will probably do the trick.

[Michael Klishin]

@rwilson thank you. It would be interesting to learn about how Spyglass can be used with ElastiCache, even though we currently don't have plans to support it. That said, if it turns out to be easy for us to maintain, I'd be willing to give it a shot in a new namespace. But it's not yet clear.

[Ryan Wilson]

Swapping dependencies seems to work, some futzing with the ConnectionFactory to specify ClientMode required. I have a branch to track this work, here's the diff.

Given the requirement of swapping a dependency, I suppose this could be a permanent fork, ie.g. clojurewerkz/spyglass-elasticache or whatnot. I'm still testing, will report back.

[Michael Klishin]

@rwilson if this is a mostly compatible change I'd consider depending on the Amazon client in the next major version.

[Priyatam Mudi]

Thanks @rwilson. I don't need an aws elastic load balancer client now (happy with the default one shipped with splyglass), and is running fine in production for seven months. The addition would be useful to someone, as an optional dependency with the connection factory helpers.

[Ryan Wilson]

OK, the changes in the above linked branch work. As you can see in the diff, the changes default to using connection factories initialized with ClientMode/Static, which is required whenever the servers list is not an ElastiCache dynamic configuration endpoint (e.g. non-ElastiCache environment, or even within ElastiCache if connecting directly to specific nodes).

To use it with ElastiCache's dynamic configuration endpoint, the connection just has to be created with a connection factory set to ClientMode/Dynamic.

For comparison, here's an example:

Normal Memcached (non-ElastiCache)

(def server-list "localhost:11211")

(def bc (mc/bin-connection server-list))
(def tc (mc/text-connection server-list))

;; ... use client connections as normal

ElastiCache Dynamic Configuration

(def server-list "[your.mc.endpoint].cfg.usw2.cache.amazonaws.com:11211")

(def bcf (mc/bin-connection-factory :client-mode :dynamic))
(def bc (mc/bin-connection server-list bcf))

(def tcf (mc/text-connection-factory :client-mode :dynamic))
(def tc (mc/text-connection server-list tcf))

;; ... use as normally

There's an alpha version on clojars if anybody wants to try it. Include: [rwilson/spyglass-elasticache "1.2.0-alpha1"]

How do you want to proceed? If you want to use the AWS ElastiCache client as a substitute for spymemcached in spyglass, I can create a PR for that. I'm a little skeptical of that, since I don't know whether to expect the AWS ElastiCache client to stay up to date with spymemcached, or whether to expect spymemcached to be fairly static and thus it wouldn't matter.

The safest play is probably to create a permanent fork, e.g. clojurewerkz/spyglass-elasticache or I could host/maintain it. There's the downside that maybe spyglass and spyglass-elasticache fall out of sync, but it prevents spyglass from getting bottlenecked on the AWS ElastiCache client release cycle.

Any preferences?