Race condition in batching leading to lost logs
guifel opened this issue · comments
Guifel commented
In batching mode, clearBatch
is being called after the post request:
Line 198 in 67e5fdf
Logs that came in while the post request is pending will be lost. clearBatch
needs to be called before.
Guifel commented
A year later the issue is still around. This is the patch I have been using ever since:
diff --git a/node_modules/winston-loki/src/batcher.js b/node_modules/winston-loki/src/batcher.js
index 5659320..879ccae 100644
--- a/node_modules/winston-loki/src/batcher.js
+++ b/node_modules/winston-loki/src/batcher.js
@@ -250,17 +250,18 @@ class Batcher {
}
}
+ const savedBatchStreams = this.batch.streams
+ logEntry === undefined && this.clearBatch()
+
// Send the data to Grafana Loki
req.post(this.url, this.contentType, this.options.headers, reqBody, this.options.timeout)
.then(() => {
- // No need to clear the batch if batching is disabled
- logEntry === undefined && this.clearBatch()
this.batchSent()
resolve()
})
.catch(err => {
- // Clear the batch on error if enabled
- this.options.clearOnError && this.clearBatch()
+ // Revert clear batch for the logs to be sent on the next iteration
+ this.batch.streams = savedBatchStreams
this.options.onConnectionError !== undefined && this.options.onConnectionError(err)