nix-env uses a ton of memory

Question

nix-env uses a ton of memory

kstenerud opened this issue 6 years ago · comments

Issue description

nix-env uses so much memory that it fails on allocation when querying packages; far more than should be for a simple package manager.

Steps to reproduce

On a system with 2gb or less:

[demo@nixos:~]$ nix-env -qa
GC Warning: Out of memory - trying to allocate requested amount (336 bytes)...
GC Warning: Header allocation failed: dropping block
GC Warning: Failed to expand heap by 8388608 bytes
GC Warning: Failed to expand heap by 65536 bytes
GC Warning: Out of Memory! Heap size: 736 MiB. Returning NULL!
error: out of memory

Technical details

Please run nix-shell -p nix-info --run "nix-info -m" and paste the
results.

system: "x86_64-linux"
host os: Linux 4.14.32, NixOS, 18.03.131807.489a14add9a (Impala)
multi-user?: yes
sandbox: no
version: nix-env (Nix) 2.0
channels(demo): "nixpkgs-18.09pre134800.e42ae4e586c"
channels(root): "nixos-18.03.131807.489a14add9a"
nixpkgs: /nix/var/nix/profiles/per-user/root/channels/nixos/nixpkgs

Eelco Dolstra · Answer 1 · Mon Apr 09 2018 16:41:41 GMT+0800 (China Standard Time)

Unfortunately Nix is not a simple package manager: it implements a lazy purely functional language, the evaluation of which uses a lot of memory. It has some design issues that make it hard to reduce memory usage (in particular the use of a monolithic pkgs set combined with the .override feature makes garbage collection largely ineffective).

Also, Nixpkgs has seen a lot of growth such that evaluation now takes > 3 times as much memory as two years ago (https://hydra.nixos.org/job/nixpkgs/trunk/metrics/metric/nix-env.qa.allocations). (This may also be a result of the use of certain new abstractions in Nixpkgs, e.g. stdenv.mkDerivation is a lot more complicated than two years ago.)

Kayo Phoenix · Answer 2 · Wed May 30 2018 21:30:03 GMT+0800 (China Standard Time)

When upgrading to NixOS 18.03 I had to add 6Gb swap on my laptop with 4Gb RAM to be able install some packages like KiCAD for example :(
Installation takes about 1.5 hours because using swap.
Of couse I also configured zram, because without zram I wont be able to complete installation.
The future of Nix on the middle/low-end systems in the mist...

Vladimír Čunát · Answer 3 · Thu May 31 2018 03:32:12 GMT+0800 (China Standard Time)

@katyo: I suspect your case is because there were no binaries for it, nix tried to build it locally, and /tmp/ in now in RAM by default.

Kayo Phoenix · Answer 4 · Thu May 31 2018 22:56:16 GMT+0800 (China Standard Time)

@vcunat I already use distributed build a some time ago because my laptop not so much powerful. Also I relocated /tmp on a disk But it don't helps to reduce memory usage.

Vladimír Čunát · Answer 5 · Fri Jun 01 2018 03:50:27 GMT+0800 (China Standard Time)

@katyo: what kind of command do you use? If it's nix-env, be sure to use one with -A selectors, to save on performance (and it's saner in other ways, too).

Elias Probst · Answer 6 · Tue Jul 17 2018 03:45:06 GMT+0800 (China Standard Time)

Ran into the same issue on a device with 8GB memory:

nonroot@box $ nix search mkpasswd
GC Warning: Failed to expand heap by 402653184 bytes
warning: using cached results; pass '-u' to update the cache
Attribute name: nixpkgs.perlPackages.StringMkPasswd
Package name: perl-String-MkPasswd
Version: 0.05
Description:

Attribute name: nixpkgs.mkpasswd
Package name: mkpasswd
Version: 5.3.0
Description: Overfeatured front-end to crypt, from the Debian whois package
nonroot@box $ nix-env -i mkpasswd
GC Warning: Failed to expand heap by 402653184 bytes

I also happened occasionally that my system simply started hanging out of the blue, which I suspect was caused by system.autoUpgrade.enable = true; which might suffer from a related issue - disabled it for now to see whether it still happens.

Vladimír Čunát · Answer 7 · Tue Jul 17 2018 04:04:17 GMT+0800 (China Standard Time)

@eliasp: almost all people really want to prefer -A for nix-env. You got

Attribute name: nixpkgs.mkpasswd

so you just nix-env -iA nixpkgs.mkpasswd. I haven't seen nixos-rebuild suffering from this, as it uses attributes "inside".

Jan Malakhovski · Answer 8 · Sat Aug 18 2018 22:44:24 GMT+0800 (China Standard Time)

@edolstra

Also, Nixpkgs has seen a lot of growth such that evaluation now takes > 3 times as much memory as two years ago This may also be a result of the use of certain new abstractions in Nixpkgs, e.g. `stdenv.mkDerivation` is a lot more complicated than two years ago.

I read changes of the significant bumps in that graph that were not immediately fixed several revisions later and all but the very last one were either changes to haskell packages or melpa. So my guess would be that we now have ~3x more packages in release.nix that two years ago, not that `stdenv.mkDerivation` became 3x slower. The very latest bump is interesting, though, I'm not sure what causes it, it's either #44423 or #44517, bisecting now. I think a much more useful metric would be "nix-env -qa avg allocations per result produced", not the absolute value.

davidak · Answer 9 · Thu Aug 01 2019 03:25:45 GMT+0800 (China Standard Time)

@edolstra can you give a status update? are there plans to fix the design problems? Maybe is the effort with flakes related? This seem to be a big issue considering that the project should grow in terms of package count.

Chris Stryczynski · Answer 10 · Sat Oct 12 2019 18:04:07 GMT+0800 (China Standard Time)

@vcunat many thanks for the suggestion to use nix-env -iA. Perhaps we can mention as a message when nix-env runs out of memory?

Newskooler · Answer 11 · Wed May 06 2020 06:47:04 GMT+0800 (China Standard Time)

I am just getting started with nix and following the Nix manual.
On chapter 9, one of the suggested lines to execute is this nix-env -qa, so when I run it I get the same error:

some_user@some_server:~$ nix-env -qa
GC Warning: Failed to expand heap by 16777216 bytes
...
GC Warning: Failed to expand heap by 16777216 bytes
GC Warning: Failed to expand heap by 262144 bytes
GC Warning: Out of Memory! Heap size: 500 MiB. Returning NULL!
error: out of memory

I am experiencing the same issue on a freshly installed Nix (I did not install any packages, just followed the manual linked above).

For more info:

some_user@some_server::~$ nix-shell -p nix-info --run "nix-info -m"
 - system: `"x86_64-linux"`
 - host os: `Linux 4.15.0-66-generic, Ubuntu, 18.04.3 LTS (Bionic Beaver)`
 - multi-user?: `yes`
 - sandbox: `yes`
 - version: `nix-env (Nix) 2.3.4`
 - channels(root): `"nixpkgs-20.09pre221814.10100a97c89"`
 - nixpkgs: `/nix/var/nix/profiles/per-user/root/channels/nixpkgs`

I am running a 1GB RAM 1VCPU server.

Does nix have minimum server requirements? If yes, what are they?

P.S. I also run out of memory with commands for specific packages as well such as nix-env -qa firefox or even package installation like nix-env -i subversion

AleXoundOS · Answer 12 · Wed May 06 2020 19:29:08 GMT+0800 (China Standard Time)

I am running a 1GB RAM 1VCPU server.

Does nix have minimum server requirements? If yes, what are they?

I'm not very experienced, but I think it's 3 GiB RAM nowadays (if not evaluating/building nix expressions remotely, i.e. run nixos-rebuild --target-host HOST from another machine).

P.S. I also run out of memory with commands for specific packages as well such as nix-env -qa firefox or even package installation like nix-env -i subversion

P.S.
It may sound quite contradictory (and regardless of the RAM usage) but nix-env is very unlikely the tool you want to provision your server with. As it does not comprise the best of declarative features NixOS can provide. Perhaps, it's one of the reasons nix-env can receive less care and performance fixes.

h.ono · Answer 13 · Tue Sep 15 2020 19:06:35 GMT+0800 (China Standard Time)

Package managers shouldn't eat all the RAM there is. No other package managers eat as much trying to install packages.

Not everyone wants to start from NixOS but try out Nix package management system and try to replace the current one but right now it get's OOM'ed trying to install several packages with 1.5GB of memory left (including swap).

Kayo Phoenix · Answer 14 · Tue Sep 15 2020 21:14:40 GMT+0800 (China Standard Time)

Package managers shouldn't eat all the RAM there is. No other package managers eat as much trying to install packages.

Not everyone wants to start from NixOS but try out Nix package management system but try to replace the current one but right now it get's OOM'ed trying to install several packages with 1.5GB of memory left (including swap).

It because nix isn't an ordinary package manager. It's configuration philosophy. Of course it eats too much RAM in spite of lazy evaluation of nix expressions. This is serious issue which can stop nix usage specifically in virtual environments and clouds where available RAM usually restricted.

I hope this problem can be (and should be) solved. May be special methods of caching can help to work around it (for example by splitting result of evaluation to hot/cold parts in RAM/disk respectively).

Ivan Tham · Answer 15 · Wed Oct 14 2020 22:13:53 GMT+0800 (China Standard Time)

Same goes for me when trying out nix with 2GB RAM, nix eats up half the memory while g++ taking only 20% of the memory, I wonder why it does not frees up memory for the compiler but takes it up itself, up to the point I have to quit firefox to fully build the application, I think this may be an issue in older raspberry pis.

Thomas Lonneman · Answer 16 · Wed Nov 25 2020 10:09:08 GMT+0800 (China Standard Time)

I am having this same issue with an 8GB laptop. When I run nix-index it stops at + querying available packages and a nix-env process slowly eats up all the memory until the OOM killer gets it. I am able to install packages and do upgrades just fine, so I don't know why nix-index needs over 7GB.

stale · Answer 17 · Fri Jun 04 2021 04:36:00 GMT+0800 (China Standard Time)

I marked this as stale due to inactivity. → More info

samrose · Answer 18 · Tue Aug 10 2021 21:15:08 GMT+0800 (China Standard Time)

I could be wrong, but from my experience if you every package pre-built and cached, and if you have 3GB or more of memory, you do not tend to run into this issue.

samrose · Answer 19 · Tue Aug 10 2021 21:16:24 GMT+0800 (China Standard Time)

Also, avoiding using nix-env is a good practice too. nixos-rebuild is much better

Deleted user · Answer 20 · Sat Jan 22 2022 18:57:29 GMT+0800 (China Standard Time)

running nix-env -qaP on new installation

KlarkC · Answer 21 · Thu Mar 10 2022 22:12:48 GMT+0800 (China Standard Time)

Unfortunately Nix is not a simple package manager: it implements a lazy purely functional language, the evaluation of which uses a lot of memory. It has some design issues that make it hard to reduce memory usage (in particular the use of a monolithic pkgs set combined with the .override feature makes garbage collection largely ineffective).

@edolstra I'm curious about this, has Nix language garbage collection and its inefficient, or it was not implemented because it's inefficient? I mean, Haskell GHC have garbage collection and Haskell is also a lazy purely functional language, and I normally don't see this kind of problem with Haskell compiled programs.

Eelco Dolstra · Answer 22 · Fri Mar 11 2022 18:51:30 GMT+0800 (China Standard Time)

I said ineffective, not inefficient. Large circular data structures like pkgs mean that there isn't much that the garbage collector can do.

Jan Ph. H. · Answer 23 · Tue Apr 12 2022 00:09:07 GMT+0800 (China Standard Time)

@edolstra Just curious: Is there analysis of "the most evil packages" yet or any tooling to analyze dependencies semi-automatically? I have not seen links in the issue yet that go into direction of "fixing the cause of the problem".

Nikolay Artamonov · Answer 24 · Fri Jan 20 2023 01:28:57 GMT+0800 (China Standard Time)

I am just getting started with nix and following the Nix manual. On chapter 9, one of the suggested lines to execute is this nix-env -qa, so when I run it I get the same error:

Got the same behaviour, tried nix-env -qaP firefox from the chapter 4 of Nix Manual and got out of memory. Virtual machine has 2GB of RAM.

I'm new to NixOS and learning it now. Honestly, the emergence of such problem with basic command after an hour of study is not what I expected. Otherwise, I very like the whole idea of NixOS but my confidence was shaken.

KlarkC · Answer 25 · Fri Jan 20 2023 02:19:08 GMT+0800 (China Standard Time)

Don't worry @nartamonov, I've been using Nix since for a while now and haven't had any major issues except for some rare out of memory events when building on my modest laptop. However, there are new features currently being developed that aim to improve performance, such as file deduplication, ca-derivations, and RFCs.

Doron Behar · Answer 26 · Fri Jan 27 2023 03:04:47 GMT+0800 (China Standard Time)

Got the same behaviour, tried nix-env -qaP firefox from the chapter 4 of Nix Manual

Although officially considered unstable, I recommend nix search.

Aroun · Answer 27 · Mon Mar 20 2023 06:21:30 GMT+0800 (China Standard Time)

I don't know if this is the right place, but I just noticed nixos-rebuild is taking near to 7Go on my system

Is there any perspective to reduce that?

Marco Rebhan · Answer 28 · Mon Mar 20 2023 07:41:27 GMT+0800 (China Standard Time)

I'm not familiar with Nix's memory model so I have no idea if this is theoretically possible and, if so, how much would have to change in Nix's code, but since Nix is a functional language, none of the evaluated expressions that are stuck in memory actually have to be kept around since they can always be recalculated, right? So, in case of low memory, is it possible to drop the least frequently referenced (or other heuristic) expressions from memory to free up space for what is currently needing the memory, and next time they are used, evaluate them again from scratch? Of course, this would come with decreased performance, but it would be better than running out of memory altogether.

Jan Ph. H. · Answer 29 · Mon Mar 20 2023 08:37:43 GMT+0800 (China Standard Time)

Zig master has now a declarative build system and handles user-provided memory limits (but not yet cgroups/lockdown for strict enforcing and graceful retries etc).
I'm quite surprised to hear that nobody implemented user-providable memory quotas with graceful fallbacks yet.

Tolki · Answer 30 · Mon Apr 17 2023 19:57:59 GMT+0800 (China Standard Time)

I was thinking of using nix-env to install some dependencies on my machines as I often juggle between Linux and Mac OS and nixpkgs is by far the most up-to-date packages repository as well as making clean installs compared to brew.

But like many people here nix-env crashes my 1Gb servers and forces me to fallback to other package managers with outdated/missing packages.

Is there any way to limit the RAM usage of nix-env somehow, particularly when only installing binary packages with specific versions?

Chris Stryczynski · Answer 31 · Mon Apr 17 2023 20:01:56 GMT+0800 (China Standard Time)

I was thinking of using nix-env to install some dependencies on my machines as I often juggle between Linux and Mac OS and nixpkgs is by far the most up-to-date packages repository as well as making clean installs compared to brew.

But like many people here nix-env crashes my 1Gb servers and forces me to fallback to other package managers with outdated/missing packages.

Is there any way to limit the RAM usage of nix-env somehow, particularly when only installing binary packages with specific versions?

A simple work around is to just use a swap. Which can be done with a few commands. I do this all the time on a server with 0 issues.

Tolki · Answer 32 · Mon Apr 17 2023 20:10:03 GMT+0800 (China Standard Time)

Yes, I could make it work with the -A option or using a swap, but nix-env --install xxx crashing any machine without access to 4Gb+ of RAM + swap still feels like a bug more than a feature.

I definitely have very little knowledge about the nix system as a whole so I'm unsure as to where the RAM usage is coming from. If it's linked to data structures within the nix programming language taking up too much RAM, could using sqlite and querying it instead of storing packages information in RAM help? Sorry if that's a stupid idea.

7c6f434c · Answer 33 · Mon Apr 17 2023 20:24:07 GMT+0800 (China Standard Time)

-i without -A is something that has not aged well… but even -iA or building a profile with buildEnv is still expensive.

On my VPS I define a toolset using an expression with buildEnv, evaluate it locally on my laptop, and then do closure-copying.

For some things I actually fetch the build data from Hydra, and then do stuff by output path (not attribute or name), which won't work except via substituters, but as I am asking Hydra for what is the latest stuff available in the cache…

Brian Ealdwine · Answer 34 · Mon Jul 10 2023 06:36:59 GMT+0800 (China Standard Time)

Is it planned to fix this?

Mattias de Hollander · Answer 35 · Sun Aug 27 2023 03:32:49 GMT+0800 (China Standard Time)

I have been also running into this memory issue on a opensuse tumbleweed machine with nix installed. Installing packages worked using the -iA flag, but how to update them?
Running nix-channel --update and nix-env --upgrade has the same memory problems. And there is no nixos-rebuild or home-manager available, as those are the other suggested upgrade options.

Marco Rebhan · Answer 36 · Mon Aug 28 2023 08:11:43 GMT+0800 (China Standard Time)

@mdehollander nix-env is pretty much the worst in terms of anything, including memory usage, I think. If you want to use imperative package management, you could try nix profile, because it implements updates in a way that doesn't have to evaluate the entire nixpkgs and also isn't very fragile (nix-env stores packages as their package names instead of attr path which is not good since those don't have an index and aren't even necessarily unique).

Otherwise, if you want to do declarative package management and don't want to use home manager, I suppose you could make an env yourself with pkgs.buildEnv along with a rebuild script to build the expression for it and link it to a known path. Then add its bin to the PATH environment. The expression might look something like this:

let
  pkgs = import <nixpkgs> {};
in pkgs.buildEnv {
  name = "path";
  paths = with pkgs; [
    nix
    zsh
    # more packages
  ];
  pathsToLink = ["/bin"]; # plus anything else you might want
}

nixos-discourse · Answer 37 · Mon Jan 08 2024 22:26:42 GMT+0800 (China Standard Time)

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/reduce-nix-env-u-memory-usage/37954/2

Tobias Bora · Answer 38 · Sat Feb 24 2024 06:35:45 GMT+0800 (China Standard Time)

I'm also hitting this issue with nix-review, that uses nix-env under the hood… 11Gb of RAM used for a really basic package, even before compilation starts. Is there any alternative here or plans to fix this?