-m, unexpected behavior?
hakavlad opened this issue · comments
-m PERCENT[,KILL_PERCENT] set available memory minimum to PERCENT of total (default 10 %).
earlyoom sends SIGTERM once below PERCENT, then SIGKILL once below
KILL_PERCENT (default PERCENT/2).
I do not understand how this option works.
earlyoom -m 22[,20]
earlyoom v1.2-2-gb52cc56
mem total: 5875 MiB, sending SIGTERM at 22 %, SIGKILL at 11 %
swap total: 0 MiB, sending SIGTERM at 10 %, SIGKILL at 5 %
Could not lock memory - continuing anyway: Cannot allocate memory
mem avail: 3697 of 5875 MiB (62 %), swap free: 0 of 0 MiB ( 0 %)
mem avail: 3693 of 5875 MiB (62 %), swap free: 0 of 0 MiB ( 0 %)
Is it normal?
How to set 22% for SIGTERM and 20% for SIGKILL?
Explain this in more detail, please.
Another example:
earlyoom -m 22[,16] -s 18[,28]
earlyoom v1.2-2-gb52cc56
mem total: 5875 MiB, sending SIGTERM at 22 %, SIGKILL at 11 %
swap total: 0 MiB, sending SIGTERM at 18 %, SIGKILL at 9 %
Could not lock memory - continuing anyway: Cannot allocate memory
mem avail: 3659 of 5875 MiB (62 %), swap free: 0 of 0 MiB ( 0 %)
earlyoom -M 220[,160] -S 180[,170]
earlyoom v1.2-2-gb52cc56
fatal: -S: sigterm value in '180[,170]' exceeds limit 0
WTF?
Remove the "[ ]", for example,
earlyoom -m 22,20
earlyoom v1.1-54-g20e4a01-dirty
mem total: 7835 MiB, sending SIGTERM at 22 %, SIGKILL at 20 %
swap total: 0 MiB, sending SIGTERM at 10 %, SIGKILL at 5 %
The "[ ]" just mean that the value is optional. I will add examples to make that clear
Ok, thank you.
What about
earlyoom -M 220000,160000 -S 180000,170000
earlyoom v1.2-2-gb52cc56
fatal: -S: sigterm value in '180000,170000' exceeds limit 0
?
earlyoom -S 200000,100000
earlyoom v1.2-2-gb52cc56
fatal: -S: sigterm value in '200000,100000' exceeds limit 0
It is not normal.
Strange. How much swap do you have? (output of free -k ?)
swap = 0 MiB
Well, if you have 0 swap, you cannot set -S to anything above zero.
with swap works normal
earlyoom -S 200000,100000
earlyoom v1.2-2-gb52cc56
mem total: 5875 MiB, sending SIGTERM at 10 %, SIGKILL at 5 %
swap total: 5875 MiB, sending SIGTERM at 3 %, SIGKILL at 1 %
mem avail: 3621 of 5875 MiB (61 %), swap free: 5875 of 5875 MiB (100 %)
mem avail: 3618 of 5875 MiB (61 %), swap free: 5875 of 5875 MiB (100 %)
cannot set -S to anything above zero
Maybe It is not well.
Hmm, earlyoom v1.1 had a better message, but it would still exit with an error:
earlyoom -S 1000
earlyoom v1.1-26-g867d0e8-dirty
-S: the value you passed (1000 kiB) is above total swap (0 kiB)
The error message is different in v1.2 because the parsing of the "X,Y" values is more complicated. I'll try to improve it.
I can start earlyoom with swap:
earlyoom -M 220000,160000 -S 180000,170000
and all is well. Next, I turn off the swap and all is well again, earlyoom works without swap with options earlyoom -M 220000,160000 -S 180000,170000
But I can not start earlyoom without swap with this options. Why?
Next, If I set earlyoom with -M 220000,160000 -S 180000,170000 In /etc, it will works fine. Next I will disable swap and restart earlyoom: Here the daemon will fall!
Hmm, should earlyoom ignore the swap settings when there is zero swap? What do you think?
PS: If you restart with less RAM, the daemon will fail as well
should earlyoom ignore the swap settings when there is zero swap?
Only if SwapTotal = 0.
It seems obvious to me that swap settings should only be applied when SwapTotal>0.
@rfjakob Do you agree with me?
How it can works if swap disabled.
Config:
MemAvailable levels.
mem_min_sigterm = 10 %
mem_min_sigkill = 5 %
SwapFree levels.
swap_min_sigterm = 200M
swap_min_sigkill = 100M
Output:
sudo nohang
The path to the config: /etc/nohang/nohang.conf
Monitoring started!
TRIGGERED!
MemAvailable (577 MiB, 9.8 %) < mem_min_sigterm (588 MiB, 10.0 %)
SwapFree (0 MiB, 0.0 %) < swap_min_sigterm (200 MiB, - %)
Finding the process with the highest badness
Victim is python3, pid: 13654, badness: 0, VmRSS: 4141 MiB, VmSwap: 0 MiB
Sending SIGTERM to the victim; signal received; response time: 18 ms
earlyoom -S 200000,100000
earlyoom v1.2-2-gb52cc56
fatal: -S: sigterm value in '200000,100000' exceeds limit 0
It is not normal, it should be fixed, IMHO.
mem avail: 1110 of 5875 MiB (18 %), swap free: 0 of 0 MiB ( 0 %)
mem avail: 989 of 5875 MiB (16 %), swap free: 0 of 0 MiB ( 0 %)
mem avail: 867 of 5875 MiB (14 %), swap free: 0 of 0 MiB ( 0 %)
mem avail: 744 of 5875 MiB (12 %), swap free: 0 of 0 MiB ( 0 %)
Low Memory! At or below sigterm limits (mem: 10 %, swap: 3 %)
mem avail: 635 of 5875 MiB (10 %), swap free: 0 of 0 MiB ( 0 %)
Killing process 'python3' with signal 15, pid: 14069, badness: 583, VmRSS: 3422 MiB
mem avail: 2336 of 5875 MiB (39 %), swap free: 0 of 0 MiB ( 0 %)
mem avail: 4059 of 5875 MiB (69 %), swap free: 0 of 0 MiB ( 0 %)
mem avail: 4059 of 5875 MiB (69 %), swap free: 0 of 0 MiB ( 0 %)
Earlyoom can works with -S 200000,100000 if swap will be disabled after earlyoom starts.
Why it is forbidden to start with -S 200000,100000 without swap?
How about this
-m SIGTERM_PERCENT,SIGKILL_PERCENT set available memory minimum to SIGTERM_PERCENT of total (default 10 %).
Second value is optional and by default SIGKILL_PERCENT = 1/2 SIGTERM_PERCENT
earlyoom sends SIGTERM once below SIGTERM_PERCENT, then SIGKILL once below SIGKILL_PERCENT.
-s SIGTERM_PERCENT,SIGKILL_PERCENT set free swap minimum to SIGTERM_PERCENT of total (default 10 %),
second value is optional and by default equals 1/2 SIGTERM_PERCENT
-M SIGTERM_SIZE,SIGKILL_SIZE set available memory minimum to SIGTERM_SIZE KiB (second value is optional)
-S SIGTERM_SIZE,SIGKILL_SIZE set free swap minimum to SIGTERM_SIZE KiB (second value is optional)
or this
-m SIGTERM_PERCENT,SIGKILL_PERCENT set available memory minimum to SIGTERM_PERCENT of total (default 10 %).
Second value is optional and by default SIGKILL_PERCENT = 1/2 SIGTERM_PERCENT
earlyoom sends SIGTERM once below SIGTERM_PERCENT, then SIGKILL once below SIGKILL_PERCENT.
-s SIGTERM_PERCENT,SIGKILL_PERCENT set free swap minimum to SIGTERM_PERCENT of total (default 10 %),
second value is optional and by default equals 1/2 SIGTERM_PERCENT
-M SIGTERM_KiB,SIGKILL_KiB set available memory minimum to SIGTERM_KiB KiB (second value is optional)
-S SIGTERM_KiB,SIGKILL_KiB set free swap minimum to SIGTERM_KiB KiB (second value is optional)
?
earlyoom -m 5,9
earlyoom v1.2-2-gb52cc56
fatal: -m: sigkill value exceeds sigterm value in '5,9'
In this case I expect 9% for SIGKILL and ignore SIGTERM, not fatal error.
earlyoom -m 5,0
earlyoom v1.2-2-gb52cc56
fatal: -m: zero sigkill value in '5,0'
In this case I expect 5% SIGTERM and ignore SIGKILL
SIGTERM_PERCENT should be ignored if SIGKILL_PERCENT > SIGTERM_PERCENT, not daemon fails.
0 value shouldn't cause err. It should cause only ignoring the signal.
Also, the documentation does not indicate what values SIGTERM_PERCENT and SIGKILL_SIZE can take (forbidden zero value and KLL can't be higher than TERM size)
These two work now:
earlyoom -m 5,0
earlyoom -m 5,9
Startup succeeds now with 0 swap even if you pass -S. It cannot calculate the percentages, so it uses defaults.
About the help message, I like it how it is. I think the square brackets are the usual way to show that a value is optional. Thanks for the report, I think this is done!
this is done!
Sorry, seems like No!
earlyoom -M 220000,160000 -S 180000,170000
earlyoom v1.2-12-ge600459
warning: -S: total swap is zero, using default percentages
mem total: 5875 MiB, swap total: 0 MiB
Sending SIGTERM when mem <= 3 % and swap <= 10 %,
SIGKILL when mem <= 2 % and swap <= 5 %
Could not lock memory - continuing anyway: Cannot allocate memory
mem avail: 4000 of 5875 MiB (68 %), swap free: 0 of 0 MiB ( 0 %)
mem avail: 3996 of 5875 MiB (68 %), swap free: 0 of 0 MiB ( 0 %)
mem avail: 3997 of 5875 MiB (68 %), swap free: 0 of 0 MiB ( 0 %)
...
mem avail: 0 of 5875 MiB ( 0 %), swap free: 3053 of 5875 MiB (51 %)
mem avail: 0 of 5875 MiB ( 0 %), swap free: 2283 of 5875 MiB (38 %)
mem avail: 0 of 5875 MiB ( 0 %), swap free: 1521 of 5875 MiB (25 %)
mem avail: 0 of 5875 MiB ( 0 %), swap free: 804 of 5875 MiB (13 %)
Low Memory! At or below SIGTERM limits (mem: 3 %, swap: 10 %)
mem avail: 0 of 5875 MiB ( 0 %), swap free: 576 of 5875 MiB ( 9 %)
Killing process 'tail' with signal 15, pid: 5355, badness: 760, VmRSS: 4830 MiB
mem avail: 4933 of 5875 MiB (83 %), swap free: 4523 of 5875 MiB (76 %)
mem avail: 4917 of 5875 MiB (83 %), swap free: 4529 of 5875 MiB (77 %)
mem avail: 4914 of 5875 MiB (83 %), swap free: 4531 of 5875 MiB (77 %)
In this case I expect -S 180000,170000, not 10% (after enable swap)
Using 10% is also unexpected behavior. User set exact -S 180000,170000!
After mount swapspace earlyoom should use -S 180000,170000, not 10% that was set at startup.
Good practice:
earlyoom -M 220000,160000 -S 180000,170000
earlyoom v1.2...
warning: -S: total swap is zero, that's why swap will be ignored now. After mounting swapspace earlyoom will use -S 18000,17000
Internally, earlyoom always uses percentages.
The -S and -M options are there for user friendlyness, but are converted to percentages at startup. If there is no swap, we cannot calculate the percentages, and we use default values.
I think this is good enough, and i do not want to change the internal design for this.
You just don't want to fix the bug.
Yes.
And now I have even documented it: https://github.com/rfjakob/earlyoom/blob/master/MANPAGE.md#bugs
Thanks