Improve GZIP compression when using local ouput
puffyCid opened this issue · comments
What new feature do you think would be cool to add to artemis?
Currently when artemsis compresses data that is being stored locally, it currently does this in two steps:
- Write the uncompressed output
- Read the file and then compress
- Save the compressed data
This only happens when storing files locally. When uploading files, artemis compresses the raw bytes and then uploads
Describe the solution you'd like
Instead of writing then reading and then writing again. Artemis should just compress the raw bytes and then write to disk.
Additional context
This was discovered when benchmarking the shimcache artifact. Artemis spend 30% of the time read and compressing the data. Hopefully by just compressing the raw bytes this will speed it up