sensor manager: allow concurrent listing with loading.
kkourt opened this issue · comments
5ce0f71 introduced a 0.9sec timeout to list the tracing policies from the sensor manager. The intention was to not block the metrics update for a long time.
However, users seem to be hitting the following warning in production, probably due to the sensor manager loading sensors that take a long time.
fix "error listing tracing policies to collect policies state" error="context deadline exceeded" warning messages
A solution would be to allow the listing of tracing policies to happen concurrently with loading. This can either be done via introducing a rw-lock or a new goroutine just for the loading.
Having statistics about how long a policy did take to load would also be useful.