Replace `sirupsen/logrus` with a more performant logger
tzaffi opened this issue · comments
Problem
In working on #128 , I ran some performance tests and noticed that log-level had a significant impact. logrus has been observed to be slow in go-algorand and there is an internal issue (2479) to replace it and a poc branch using zerolog.
Problematic experiment
Using the Justfile command
❯ just conduit-bootstrap-and-go 300to bootstrap testnet and run a postgresql exporter against it for 300 seconds. I ran it a number of times against both the original pipeline and the new one. Here are the experimental results:
| Log Level | Reps | Original rounds/300 sec (logs/round) | Pipelining rounds/300 sec (logs/round) | Pipelining v Original (%) |
|---|---|---|---|---|
| TRACE | 3 | 3718 (7.0) | 3509 (14.0) | -5.6% |
| INFO | 2 | 4578.5 (3.0) | 4423.5 (3.0) | -3.4% |
So comparing the results within each column we can see:
- 19% performance decline in the Original algorithm when going from 3 → 7 logs/round
- 21% performance decline in the Pipelining algorithm when going from 3 → 14 logs/round
The sample was very noisy but it looks like each log per round is costing around 1-5 % hit in terms of performance.
Action Items
- A POC PR which let's us validate the performance implications of swapping out logrus for zerolog
- Decide if we want to allow "plug and play" logger capability (we have heard one community member make this suggestion)
- Switch all logs in conduit to the new logger (possibly via an interface if we opt for the "plug and play" approach)
More links
- cce's noop logger
- Will's epic on go-algorand
- Google's intro of go 1.21 structured logging and a nice overview of logging in go
Dependencies
None
Urgency
Medium - as we're currently working on improving Conduit's performance, this seems like a useful avenue to persue.