Update Kafka to add more partitions
dcode opened this issue · comments
Kafka scales by adding partitions. By default, we currently only create a single partition per topic. The number of partitions should be a multiple of the number of consumers. In the rocknsm stack alone, that's equivalent to the number of logstash consumer threads per topic which defaults to 1. We should increase the number of partitions to maybe a dozen by default. It doesn't hurt anything and only provides the capability to scale as needed.