Separate critical system metadata from other non-critical ones
jzhou77 opened this issue · comments
Critical system metadata, e.g., server list, should always be available. For instance, if RateKeeper can't read server list, it throttles the whole cluster traffic and can cause unavailability.
On the other hand, other non-critical data, e.g., backup, txn samples, can be large. If not carefully configured, e.g., setting transaction sampling ratio to 1, they can hot shard the system keyspace. If the critical metadata happens to be on the same shard, then the database can become unavailable.
The solution to this problem can be solved if we can separate critical system metadata from other non-critical ones, i.e., storing them on different shards, thus reducing the chance of unavailability.