clairctl: export-updaters OOM issues
BirgerM opened this issue · comments
Description of Problem
Im trying to follow the steps as documented in the clair documentation, but i have the following issue:
Running clairctl export-updaters updates.json.gz to export updaters to a file ends up with the process consuming all available system memory before the process gets killed by the OOM killer or the machine enters a frozen state. I find it hard to believe that this command would require more than 16GB of available memory.
Expected Outcome
clairctl runs configured exporters and exports the results to a file.
Actual Outcome
Process gets killed by the OOM killer resulting in an empty updates.json.gz file.
or
Process hangs indefinitely waiting for more available system resources.
Environment
Freshly installed virtual machine in Azure
- clairctl version: v4.7.2 (claircore v1.5.19)
- Host OS: Red Hat Enterprise Linux 8.8 (Ootpa)
- Kernel (e.g.
uname -a): 4.18.0-477.27.1.el8_8.x86_64 - Total online memory: 16G
I've opened a PR at quay/claircore#1120 in an attempt to address this. If you can build a patched clairctl and take it for a spin, I'd greatly appreciate it!
I built a way to test this a bit easier here, and build this version of clairctl (for amd64/linux). Please try it if you get a chance.
Tried running the new clairctl on Red Hat Enterprise Linux 8.8 (Ootpa), but had some issues with the available version of glibc.
./clairctl: /lib64/libc.so.6: version `GLIBC_2.34' not found (required by ./clairctl)
Since GLIBC_2.34 is available on RHEL 9 i installed a new virtual machine with Red Hat Enterprise Linux 9.2 (Plow) and ran clairctl. The memory consumption stays around 3-4GB, but i keep getting this error:
./clairctl export-updaters updates.json.gz
< omitted output >
2023-10-27T23:05:05+02:00 INF successful update component=libvuln/updates/Manager.driveUpdater ref=c8045fa9-139d-4fea-86c5-2e332c0a98ba updater=debian/updater
2023-10-27T23:05:05+02:00 INF finished update component=libvuln/updates/Manager.driveUpdater updater=debian/updater
json: error calling MarshalJSON for type *jsonblob.bufShim: unexpected EOF
okay, thanks for trying. I'll keep hacking at it.
Current version of that PR runs and seems to peak (just eyeballing it) at ~2GB of memory usage.
I'll also note that setting GOMAXPROCS to something like 1 seems to pretty aggressively constrain memory with the linked PR.
Should be completely fixed in v4.7.3