Optimize cassandra routing runtime

[Lucas Kent]

Cassandra routing in CassandraSinkCluster has shown up as a hotspot in profiling.

NodePool::get_replica_node_in_dc shows up in the samply profiler as taking up 2.6% of execution time in the cassandra,compression=none,connection_count=100,driver=scylla,operation=read_i64,protocol=v4,shotover=standard,topology=cluster3 windsock bench.

Here are some areas to explore for improving it:

• #1409
• #1406
• Avoid allocating vec in iter_replica_nodes
  • maybe we can store the vec in self and clear + reuse it?
• Explore ways to implement iter_replica_nodes without a BTreeMap, our current BTreeMap approach seems to be quite expensive, lets check how scylla driver implements it
  • #1428
• Avoid allocating vec in get_replica_node_in_dc
  • My first thought is to keep things as iterator instead of collecting into a vec but that seems to hit borrow check issues.
  • Lets try refactoring get_replica_node_in_dc so that a closure is passed that makes use of a single a connection and returns Err on failure to use connection. This should let us use simple iteration to avoid having to create collections.

[Lucas Kent]

Closing this. We made some of the wins described here, but probably best to start a performance analysis from scratch if we need to improve cassandra performance further, since so much has changed since then.