martin-helmich / prometheus-nginxlog-exporter

prometheus-nginxlog-exporter/tail/tailer.go

Line 44 in f09c9dc

Poll: true,

add MaxLineSize: 65534 to improve Throughput of reading file?

prometheus-nginxlog-exporter/main.go

Line 372 in f09c9dc

for line := range t.Lines() {

to Concurrent Execute ParseString(line) in this loop?

over 25000 QPS access.log, exporter unable to update metrics accurately.

Tks.

25k qps is a respectable workload 😳 -- admittedly, this is the heaviest workload that I've seen this exporter been subjected to. I'd be happy to include optimizations for such heavy workloads, however I can't make any promises as to when I'll actually find the time to get to that. In the meantime, I'm happy to accept any PRs in this matter. 🙂

Do you have a rule of thumb for the max qps that you know the exporter can handle without pegging the one core that it uses and falling behind on log processing?

Improve Performance For "MaxLineSize" and Concurrency of ParseString(line)