Trying to use `-o -` results in lrzip writing compressed data to the console, not stdout
intelfx opened this issue · comments
problem
I'm trying to use lrzip in a pipeline with an external compressor. For purposes of the example, just redirecting stdout to a file also causes this problem:
lrzip -n windows.img -o - > windows.img.lrz
When doing this, lrzip spends some time printing progress messages on the console with zero output, and then starts dumping the compressed stream onto the console disregarding the stdout redirection.
version
$ lrzip --version
lrzip version 0.641
$ cat /etc/os-release
NAME="Arch Linux"
PRETTY_NAME="Arch Linux"
ID=arch
BUILD_ID=rolling
ANSI_COLOR="38;2;23;147;209"
HOME_URL="https://www.archlinux.org/"
DOCUMENTATION_URL="https://wiki.archlinux.org/"
SUPPORT_URL="https://bbs.archlinux.org/"
BUG_REPORT_URL="https://bugs.archlinux.org/"
LOGO=archlinux
$ uname -a
Linux able 5.11.10-arch1pf6-1 #16 SMP PREEMPT Thu, 01 Jan 1970 00:00:00 +0000 x86_64 GNU/Linux
Why would you do this? -o - >file.... -o was designed to output to a specific file, >file... will direct STDOUT to a file.
And, your command line is broken. The file to be compressed is always last.
lrzip -n windows.img -o - > windows.img.lrz
should be
lrzip -n windows.img
Nothing more.
Why would you do this?
Because it is a minimal example. The actual usage is lrzip -n INPUT -o - | COMPRESSOR | OTHER-PROGRAMS.
And, your command line is broken. The file to be compressed is always last.
The bug still exists if I invoke lrzip as lrzip -n -o - windows.img.
I'm sorry. Try this...
lrz -cn linux-5.10.tar |bzip2 -c >file.bz2
lrz is a gzip compatible command wrapper. -c will output to STDOUT.
Although this is not an invalid complaint. I'll look into making it a feature on my git.
Unfortunately lrz -nc | ... exhibits the same behavior.
IDK Works for me. Please see
$ lrz -nc file.tar | bzip2 -c >file.tar.lrz.bz2
LZ4 Threshold testing disabled due to Filtering and/or Compression type (gzip, lzo, rzip).
$ bzip2 -dkc file.tar.lrz.bz2 | lrz -dfc >file.tar
$ ll file* linux-5.10.tar
-rw-r--r-- 1 peter users 1158307840 Apr 8 16:41 file.tar
-rw-r--r-- 1 peter users 897427042 Apr 8 16:25 file.tar.lrz
-rw-r--r-- 1 peter users 169784946 Apr 8 16:40 file.tar.lrz.bz2
-rw-r--r-- 1 peter users 1158307840 Jan 15 09:43 linux-5.10.tar
$ cmp file.tar linux-5.10.tar
$
Piping is a little convoluted, but as you can see, it works as expected. Make sure you're using a current version. NB. This was tested using lrzip-next.
I'm using lrzip 0.641 as published in this project. My invocation:
lrz -cn windows.img | zstd -T0 -1 > windows.img.lrz.zst
Interestingly, this does not happen on a small test file!
lrzip -n INPUT -o - form also works as expected in this case.
Interestingly, this does not happen on a small test file!
lrzip -n INPUT -o -form also works as expected in this case.
Options can't come after filename. lrzip uses the last argv[] as filename. I would use zstd alone. But I don't know why. You get better compression with lrzip alone. Someone else requested adding zstd to lrzip. Personally anything other than lzma and zpaq are not useful. Good luck.
Options can't come after filename. lrzip uses the last argv[] as filename.
Looking at the code, lrzip uses getopt which will shuffle arguments as needed, so no, it doesn't matter.
I would use zstd alone. But I don't know why.
Again, this is only a usage example. It doesn't matter what comes after lrzip in the invocation. The bug is still there.
Patches are welcome.
I've instrumented the problem here (having inadequate ram to compress entirely in ram) but the solution is proving non-trivial. Still working on it.
Fixed in 09ceb85 .