Impossible to disable compression by default

[digriz33]

Hello!

First of all, thank you very much for such a compact and fast logging library.

It would be nice to have a way to disable compression of logs by default. This will make the library more useful because right now it's impossible to use it if you want such features like:

• logs-based monitoring (filebeat, logstash,...) usually requires the log files to be parsable line by line.
• logging with huge amounts of data. Right now it's impossible to rotate log files (as far as I understand the library). Disabled compression will allow users to split files manually or with some help of additional tools.

Question: are you planning to add a parameter to disable compression by default? If not, please, at least give us some hints on how to do this properly - we'll try to do this on our own.

[syang0]

Hi digriz33,

It seems like you're looking for two features within NanoLog:

1. Having the log files be in a human-readable format for log-based monitoring to parse line by line and
2. Performing log rotations.

For the first request, while one can disable compression in the library, this will not make the log file human-readable; the NanoLog system will still output a binary log file at runtime. To obtain a human-readable file that's line-delineated, one must use the decompressor to post-process the binary log files.

Outputting the log messages in a human-readable format at runtime would defeat most of the performance benefits of NanoLog, so it's not supported out of the box. Instead, I would recommend you attempt to incorporate the decompressor into your log ingest pipeline.

For the second request, while there are no automatic log rotation mechanisms implemented into NanoLog itself, one can perform a rotation manually by invoking NanoLog::setLogFile(const char* filename) as documented in the NanoLog Header. This will cleanly split the log file such that each split is independently interpretable by the decompressor application.

[digriz33]

Thank you very much for extensive feedback!

Am I right that setLogFile call is thread-safe?

Instead, I would recommend you attempt to incorporate the decompressor into your log ingest pipeline.

Does this mean that the logger compresses its entries in binary format, but it's still possible to somehow parse file's content gradually without using decompressor on the whole file?

[syang0]

Hi digriz33,

That is correct, setLogFile() should be thread safe.

For your second question, yes and no. See Issue #10 for more information about the structure of the log file and how you may parse it gradually.

But at a high level, the NanoLog log file consists of three entities: a header, dictionary entries, and bufferExtents (i.e. a chunk log messages from a single thread). The header and dictionary entries are needed to interpret the bufferExtents, so if you had a mechanism of keeping this state readily available, the bufferExtents can be interpreted independently.

What setLogFile() does is it duplicates the header + dictionary entries into the beginning of the new file, and this is what allows the decompressor to read the new log messages.

Hope that helps.

[digriz33]

Got it!
Thank you very much for your efforts in developing the library and for the help.
Have a nice weekend!