Is there a way to enable compression?
tartavull opened this issue · comments
btw, Eris is amazing! thank you.
Offering compression is possible but it's worth nothing that it's relatively CPU heavy; e.g. nix-serve
actually stopped serving .nar.bz2
files because it was too CPU intensive, even with a single client. I don't know if xz
is quite as bad, admittedly. gzip
would be ideal (it's very cheap) but I don't think Nix supports it (correct me if I'm wrong.)
The only change needed to support this is to slightly alter the .narinfo
responses given by Eris; if Nix downloads a .narinfo
file from Eris which says to look for <hash>.nar
, then Nix will ask for that. If the .narinfo
says to use .nar.bz2
, Nix asks for that instead. So we would only need to slightly modify the .narinfo
sent to the client based on e.g. a configuration file option to support this (compression => 'xz'
option, or something.)
That is because there in fact does exist some code to handle bz2 and xz paths, however, but this has never been enabled explicitly. Rather, it's a fallback to ensure that if a client requests a .nar.bz2
then it can be served. This is because tools (that are not nix
) might not request those URLs specifically. So the code is there, we just don't serve the right .narinfo
files. This is, again, a conscious implementation choice.
I'm not opposed to offering an option for xz
perhaps, provided that some benchmarking showing it was sufficiently fast (e.g. what's the CPU use/wall clock time for downloading 500 small .nar.xz
files vs the same 500 files in .nar
format?) Even if the performance got really bad, I still wouldn't necessarily opposed to offering an option. But I would really want to know more about your use case before doing that: having an option that just effectively says "enable this to cripple the server and bottleneck on CPU" isn't really so useful in practice and also implies some level of support for it...
Also, I'm glad Eris is useful for you! :)
This is because tools (that are not nix) might not request those URLs specifically. So the code is there, we just don't serve the right .narinfo files. This is, again, a conscious implementation choice.
For the record, writing this out I realized this feature should also be gated behind an option anyway, because otherwise a client just requesting a single valid .nar.bz2
a million times will basically cause a denial of service...