Is there a way to enable compression?
tartavull opened this issue · comments
btw, Eris is amazing! thank you.
Offering compression is possible but it's worth nothing that it's relatively CPU heavy; e.g. nix-serve actually stopped serving .nar.bz2 files because it was too CPU intensive, even with a single client. I don't know if xz is quite as bad, admittedly. gzip would be ideal (it's very cheap) but I don't think Nix supports it (correct me if I'm wrong.)
The only change needed to support this is to slightly alter the .narinfo responses given by Eris; if Nix downloads a .narinfo file from Eris which says to look for <hash>.nar, then Nix will ask for that. If the .narinfo says to use .nar.bz2, Nix asks for that instead. So we would only need to slightly modify the .narinfo sent to the client based on e.g. a configuration file option to support this (compression => 'xz' option, or something.)
That is because there in fact does exist some code to handle bz2 and xz paths, however, but this has never been enabled explicitly. Rather, it's a fallback to ensure that if a client requests a .nar.bz2 then it can be served. This is because tools (that are not nix) might not request those URLs specifically. So the code is there, we just don't serve the right .narinfo files. This is, again, a conscious implementation choice.
I'm not opposed to offering an option for xz perhaps, provided that some benchmarking showing it was sufficiently fast (e.g. what's the CPU use/wall clock time for downloading 500 small .nar.xz files vs the same 500 files in .nar format?) Even if the performance got really bad, I still wouldn't necessarily opposed to offering an option. But I would really want to know more about your use case before doing that: having an option that just effectively says "enable this to cripple the server and bottleneck on CPU" isn't really so useful in practice and also implies some level of support for it...
Also, I'm glad Eris is useful for you! :)
This is because tools (that are not nix) might not request those URLs specifically. So the code is there, we just don't serve the right .narinfo files. This is, again, a conscious implementation choice.
For the record, writing this out I realized this feature should also be gated behind an option anyway, because otherwise a client just requesting a single valid .nar.bz2 a million times will basically cause a denial of service...