Q1:How did changing values on the SparkSession property parameters affect the throughput and latency of the data?
Playing around with different config values impacted inputRowsPerSecond and processedRowsPerSecond. Based on different parameters sometimes both throughput and delay increased
Q2: What were the 2-3 most efficient SparkSession property key/value pairs? Through testing multiple variations on values, how can you tell these were the most optimal?
In my experimentation, when using the default values I get about 60 processedRowsPerSecond, but when especially increased maxOffsetPerTrigger to 5K then I got a better throughput of processedRowsPerSecond : 352.85815102328866
I also played with other params such as
maxRatePerPartitionspark.sql.inMemoryColumnarStorage.batchSizespark.sql.shuffle.partitions
However they were not impacting the throghput/delay drastically