core/config: unconfigured cores don't know when they've been configured

[Tess Rinearson]

If you run three processes (listening on, say, 1999, 1998 and 1997) on an unconfigured core, and then configure the core via the process listening on 1999, process 1999 will exec itself but processes 1998 and 1997 will not. In fact, they won't realize that they are now part of a configured core until they get restarted, even though this change is reflected in sinkDB.

We have two options for resolving this:

1. We could have unconfigured core processes poll sinkDB://core/config* until something appears at that key, at which point they exec themselves.
2. We could block processes from joining a raft cluster until that cluster contains configuration state.

Polling kind of sucks, but the ux for the first solution seems strictly better to me. Also, polling shouldn't be that big of a deal because, seriously, what else is an unconfigured core going to do other than sit around polling sinkDB? (We may need to be careful about using stale vs linearizable reads when polling sinkDB.)

* totally just made this notation up

[kr]

I like option 1. We might also consider some sort of "watch" or "notify" functionality in sinkdb so that the config package doesn't have to explicitly poll.

[Tess Rinearson]

Vicki has started on option 1. We could add a TODO to consider adding watch functionality to sinkdb--it's something etcd has, and something we used in iago.