Log2ram runs out of space

Question

Log2ram runs out of space

anderl78 opened this issue 3 years ago · comments

Hello! Log2ram runs out of space after a few days. If i check the size of /var/log, it shows the expected size: ways smaller than the configured size of the ramdisk. Fir example /var/log is around 15 MB, the size of the ramdisk is 100 MB - but gets more and more full. Logrotate is already optimized. If i manually write the ramdisks content to disk, there is no difference in the usage of the ramdisk. The only way to fix it, is to reboot the system. System is a raspi 3b

azlux · Answer 1 · Fri Oct 08 2021 20:47:34 GMT+0800 (China Standard Time)

Did you get an error message on the systemd service or the mount point is full ?
Second question, do you have rsync installed ?

Az

anderl78 · Answer 2 · Sat Oct 09 2021 15:27:33 GMT+0800 (China Standard Time)

Thank you for your answer! Rsync is installed and i get no errors on systemd service. The mount-point runs full. But if i look through it, a du gives me only a few mb used...

Just now (did areboot around 24 hours), i have a usage of 13MB of /var/log (du /var/log).
But a df shows me 29 mb used. And this 29 mb will increase day by day, till it reaches the maximum space available

xtvdata · Answer 3 · Sat Oct 09 2021 21:55:01 GMT+0800 (China Standard Time)

Hi anderl78,
this is a a bit strange, however I doubt it is something related to log2ram.
log2ram just performs a synchronization of the contents in the RAM disk and in the persisted disk (e.g. SD card), thus it is not responsible of the the amount of data in /var/log, it just periodically ensures that what you have in RAM is also stored on the physical disk.

Typically if you have an ever growing /var/log it means that you have an issue either with:

one or more application logging in debug mode (even tho this is not a reason for ever growing, since the logs should be rotated, but if the amount of debug is really high I might be that rotating and compressing log once per day is not enough) => if this is the case (see if the /var/log you have huge files) I'd suggest check if you really need debug level info
rotation does not take into account part of the /var/log contents or rotation rules need to be fine-tuned (see previous point)

If the growing behavior is just inside a single 24hours the issue is most probably related to the first point, but if the growing goes on with the same trend after rotation it means it's a mix of the two.

However, I find strange that a reboot fixes the issue, as a clean reboot should not change the size of the logs... possibly there is something weird in /var/log. What happens if you just restart log2ram with the following command?

sudo systemctl restart log2ram.service

Does it produce any error?
All of the contents of /var/log are indeed synced with /var/hdd.log/?

Another strange thing is the fact that you have such a significant difference running du and df.
There could be a few reasons for this, including:

df works in the same way with any user, but du ignores all the parts of the filesystem to which the current user is not authorized (ensure to run both commands as root)
possibly (but I'm not sure) the difference is due to the fact you have a huge number of small files in some subdirectory of /var/log. du checks the size of each file ad sums it up dir by dir, however df checks the used space on a filesystem; every file is written to the filesystem in blocks and the minimum block size on ext4 is 1024, thus every file space occupancy is always rounded up to an integer number of kb... and if you have a large number of small files this could justify the difference.
it might be (but this would be quite unusual) that you have some sparse files in /var/log. Sparse files' size is not directly correlated to actual disk usage, and in some cases my also lead to a strange behavior of rsync.

Just to give an idea on raspberries.
In all my regular RPIs, all of them with logging granularity and rotation optimized, /var/log contains between 200 and 400 files, the total size is about 5-6MB and difference between du and df is about 0.5%.
On the RPI I use to collect all logs from all devices (including routers, switches and other stuff), the total size of /var/log is about 50MB and the difference between du and df is about 5-6%. Obviously in this case the file count is much higher, including a number of very small files, thus the difference between du and df is justified.
To estimate the number of files in /var/log try to run:

sudo ls -lRA /var/log | wc -l

(I know it's an overestimation far from being precise, as it takes into account directories twice, there are empty lines, and bla bla..., but who cares, it gives you a very quick and dirty idea - just know that this file count is a bit overestimated...)

P.S.: having a limited amount of logs due to optimizations of logging granularity and rotation, might lead to the idea that using log2ram is an overkill.
Well it's not.
Indeed log2ram has a very strong point concerning SD preservation, but also it has a secondary benefit (which in my view is not secondary at all): it does significantly reduce general I/O operations on physical discs, and I/O is generally the most limited resource on a Raspberry Pi (with an SSD is much better, yet not stellar, due to USB3 limitations) and concurrency kills them when you are running on SDcard.

anderl78 · Answer 4 · Wed Nov 03 2021 04:54:57 GMT+0800 (China Standard Time)

Hey, sorry for my late answer. I had corona and was not able to do anything about this. Thank you very much for your answer!

Now i tried, as you suggested, a sudo systemctl restart log2ram.service. And after this, the usage seems to go back to normal... Dont know why!

Lucas Verney · Answer 5 · Thu Jan 06 2022 05:25:48 GMT+0800 (China Standard Time)

Hi,

It seems I had a similar issue. For some reason (buggy mail loop), my /var/log filled and log2ram was running out of space. I cleared up some space, resulting in ncdu indicating a total size of 100M for /var/log (300M configured in log2ram).

Still, this extra space was not reflected in df:

$ df -h
[...]
log2ram            300M    300M     0 100% /var/log

Looking at the systemd unit, it seems there is something missing and when the main process exits, the unit does not fall back to failed / stopped state nor systemd does automatically restart it:

$ sudo systemctl status log2ram
● log2ram.service - Log2Ram
     Loaded: loaded (/etc/systemd/system/log2ram.service; enabled; vendor preset: enabled)
     Active: active (exited) since Tue 2021-12-28 16:40:17 CET; 1 weeks 1 days ago
    Process: 24666 ExecReload=/usr/local/bin/log2ram write (code=exited, status=1/FAILURE)
   Main PID: 287 (code=exited, status=0/SUCCESS)
        CPU: 8.850s

A manual restart of log2ram did the trick though:

$ sudo systemctl restart log2ram
$ df -h
log2ram            300M    101M  200M  34% /var/log

Best

JP · Answer 6 · Sun Mar 06 2022 03:45:38 GMT+0800 (China Standard Time)

I also just experienced this. only 3M used on /var/log doing sudo du -hs but df shows it full and services fail due to being unable to write their logs...

restarting the log2ram service made df show the correct usage.

azlux · Answer 7 · Sun Mar 06 2022 05:45:13 GMT+0800 (China Standard Time)

@ferrouswheel do you use rsync on log2ram ? Something weird about this issue.
And what's the output of :

du --apparent-size -s -h /var/log
du -s -h /var/log
du --apparent-size -s -h /var/hdd.log
du -s -h /var/hdd.log

JP · Answer 8 · Sun Mar 06 2022 05:57:26 GMT+0800 (China Standard Time)

My /etc/log2ram.conf has USE_RSYNC=false, but rsync is installed. Should I be using that?

Output of those commands is:

pi@raspberrypi:~ $ sudo du --apparent-size -s -h /var/log
2.8M	/var/log
pi@raspberrypi:~ $ sudo du -s -h /var/log
3.0M	/var/log
pi@raspberrypi:~ $ sudo du --apparent-size -s -h /var/hdd.log
2.7M	/var/hdd.log
pi@raspberrypi:~ $ sudo du -s -h /var/hdd.log
2.7M	/var/hdd.log

(I'll also run these commands if I experience the issue again)

azlux · Answer 9 · Sun Mar 06 2022 06:01:26 GMT+0800 (China Standard Time)

The only thing I suppose to appear here is somehow you have sparse files ( amounts of space preallocated), and log2ram don't see it by default.
You can try to use rsync, but if it's because of some sparse file, this will change nothing.

Michael Goldstein · Answer 10 · Fri Mar 11 2022 06:54:07 GMT+0800 (China Standard Time)

I'm also seeing this issue. I had rsync disabled, just enabled it. Restarting log2ram or the device cleared out the "ghost" space that was filling up the log2ram filesystem. Somewhat odd.

Let me know if there are any logs I can pull that would be helpful.

Julian · Answer 11 · Mon Mar 14 2022 20:00:49 GMT+0800 (China Standard Time)

I have the problem that log2ram runs out of space aswell. The big problem with this in my case is that if gets to 100% my raspberry pi loses its ethernet connection so I have to manually power it off/on to get it working again. I had this problem a while back and never got to the roots of this but now I installed log2ram a few weeks ago on a new pi and after a couple of days it started happening again.

Manually restarting the log2ram service doesn't do anything for me. I even increased the partition size via the log2ram config file to 80MB but it still got to 100% usage.
Running sudo du -hs /var/log/* | sort -h | tail -n 3 gives me this:

16M	/var/log/daemon.log.1
16M	/var/log/syslog.1
25M	/var/log/journal

I don't know why journal gets over 25MB because I set it to 20MB max like shown in the readme.
For now I have to remove log2ram from my pi because these outages are really impractical for me.
Let me know if you need any more information from me to maybe get this fixed.

Thanks! :)

Yuri Vidal · Answer 12 · Sun Jul 24 2022 04:47:17 GMT+0800 (China Standard Time)

I have the problem that log2ram runs out of space aswell. The big problem with this in my case is that if gets to 100% my raspberry pi loses its ethernet connection so I have to manually power it off/on to get it working again. I had this problem a while back and never got to the roots of this but now I installed log2ram a few weeks ago on a new pi and after a couple of days it started happening again.

Manually restarting the log2ram service doesn't do anything for me. I even increased the partition size via the log2ram config file to 80MB but it still got to 100% usage. Running sudo du -hs /var/log/* | sort -h | tail -n 3 gives me this:
16M	/var/log/daemon.log.1
16M	/var/log/syslog.1
25M	/var/log/journal
I don't know why journal gets over 25MB because I set it to 20MB max like shown in the readme. For now I have to remove log2ram from my pi because these outages are really impractical for me. Let me know if you need any more information from me to maybe get this fixed.

Thanks! :)

I have the exact same problem. Did you ever find a solution to this?

Julian · Answer 13 · Wed Jul 27 2022 03:56:06 GMT+0800 (China Standard Time)

I have the problem that log2ram runs out of space aswell. The big problem with this in my case is that if gets to 100% my raspberry pi loses its ethernet connection so I have to manually power it off/on to get it working again. I had this problem a while back and never got to the roots of this but now I installed log2ram a few weeks ago on a new pi and after a couple of days it started happening again.

Manually restarting the log2ram service doesn't do anything for me. I even increased the partition size via the log2ram config file to 80MB but it still got to 100% usage. Running sudo du -hs /var/log/* | sort -h | tail -n 3 gives me this:
16M /var/log/daemon.log.1

16M /var/log/syslog.1

25M /var/log/journal
I don't know why journal gets over 25MB because I set it to 20MB max like shown in the readme. For now I have to remove log2ram from my pi because these outages are really impractical for me. Let me know if you need any more information from me to maybe get this fixed.

Thanks! :)

I have the exact same problem. Did you ever find a solution to this?

Unfortunately I did not...

Yuri Vidal · Answer 14 · Wed Jul 27 2022 04:02:10 GMT+0800 (China Standard Time)

Seems like the logs are not getting properly recycled and journal logs get bigger than what they are supposed to.
My workaround was to add journal vacuum commands to run every hour to my crontab. I set vacuum anything older than 8hours

Trung Lam · Answer 15 · Sat Jul 30 2022 18:18:14 GMT+0800 (China Standard Time)

I ran into this issue and managed to find the solution:

❯ df -h
Filesystem Size Used Avail Use% Mounted on
tmpfs 187M 15M 173M 8% /run
/dev/mmcblk0p2 59G 4.7G 52G 9% /
tmpfs 934M 0 934M 0% /dev/shm
tmpfs 5.0M 0 5.0M 0% /run/lock
/dev/mmcblk0p1 253M 134M 120M 53% /boot/firmware
tmpfs 187M 4.0K 187M 1% /run/user/1000
log2ram 40M 40M 0 100% /var/log

After having a look on it ,the issue is the log of journald getting too big and it quickly fill up all the log2ram space. Solution ? Just clean it a bit:
sudo journalctl --vacuum-time=7d
sudo systemctl restart log2ram
The above cli will clean all the log older than 7 days. This issue can be prevented by set the max size for journald log and also increase the size for log2ram space. But imo juts clean journald logs regularly is good enough.

Result:

❯ df -h
Filesystem Size Used Avail Use% Mounted on
tmpfs 187M 15M 173M 8% /run
/dev/mmcblk0p2 59G 4.1G 52G 8% /
tmpfs 934M 0 934M 0% /dev/shm
tmpfs 5.0M 0 5.0M 0% /run/lock
/dev/mmcblk0p1 253M 134M 120M 53% /boot/firmware
tmpfs 187M 4.0K 187M 1% /run/user/1000
log2ram 40M 8.8M 32M 22% /var/log

Yuri Vidal · Answer 16 · Sat Jul 30 2022 18:36:41 GMT+0800 (China Standard Time)

I ran into this issue and managed to find the solution:

❯ df -h
Filesystem Size Used Avail Use% Mounted on
tmpfs 187M 15M 173M 8% /run
/dev/mmcblk0p2 59G 4.7G 52G 9% /
tmpfs 934M 0 934M 0% /dev/shm
tmpfs 5.0M 0 5.0M 0% /run/lock
/dev/mmcblk0p1 253M 134M 120M 53% /boot/firmware
tmpfs 187M 4.0K 187M 1% /run/user/1000
log2ram 40M 40M 0 100% /var/log

After having a look on it ,the issue is the log of journald getting too big and it quickly fill up all the log2ram space. Solution ? Just clean it a bit: sudo journalctl --vacuum-time=7d sudo systemctl restart log2ram The above cli will clean all the log older than 7 days. This issue can be prevented by set the max size for journald log and also increase the size for log2ram space. But imo juts clean journald logs regularly is good enough.

Result:

❯ df -h
Filesystem Size Used Avail Use% Mounted on
tmpfs 187M 15M 173M 8% /run
/dev/mmcblk0p2 59G 4.1G 52G 8% /
tmpfs 934M 0 934M 0% /dev/shm
tmpfs 5.0M 0 5.0M 0% /run/lock
/dev/mmcblk0p1 253M 134M 120M 53% /boot/firmware
tmpfs 187M 4.0K 187M 1% /run/user/1000
log2ram 40M 8.8M 32M 22% /var/log

This is not a fix for the issue, but rather a workaround. The root of the problem is that journal logs are getting bigger than what they should (they get bigger than what is specified on the journal config file), so, somehow the journal logs are not being properly rotated.

Benjamin Lupton · Answer 17 · Mon Oct 03 2022 15:10:21 GMT+0800 (China Standard Time)

I fixed this error by doing sudo vim /etc/systemd/journald.conf with:

# https://www.freedesktop.org/software/systemd/man/journald.conf.html
SystemMaxUse=20M
RuntimeMaxUse=20M
MaxRetentionSec=1day
MaxFileSec=1day

Actually, a few days later and it still ran out of log space.

camelcamro · Answer 18 · Thu Dec 15 2022 14:57:34 GMT+0800 (China Standard Time)

df -h
log2ram 50M 50M 0M 100% /var/log

hmm, for now, I'm doing it via a simple daily cronjob

#cleanup workaround for lo2ram
11 11 * * * sudo journalctl --vacuum-time=1d;sudo systemctl restart log2ram

still have no lcue what is really the problem, but a restart fix the issue for me.

and after: sudo systemctl restart log2ram
df -h
log2ram 50M 2,1M 48M 5% /var/log

coaxk · Answer 19 · Tue Feb 21 2023 21:11:56 GMT+0800 (China Standard Time)

also seeing the same issue. jumping on the thread for updates

thickconfusion · Answer 20 · Fri Mar 24 2023 10:34:44 GMT+0800 (China Standard Time)

Pi Zero W here. Running Pi Hole and log2ram. I have been fighting this a while, thinking more and more it's a Systemd journal bloating issue tangled with log2ram. Anyway, I found this as well. home-assistant/operating-system#2226

ollie-campbell · Answer 21 · Wed Jun 21 2023 16:35:16 GMT+0800 (China Standard Time)

+1 here:

df -h
Filesystem      Size  Used Avail Use% Mounted on
....
log2ram          40M   40M     0 100% /var/log

sudo journalctl --vacuum-files=1
Vacuuming done, freed 0B of archived journals from /run/log/journal/11316449f0af4dd893e46267e666acbe.

sudo systemctl status log2ram.service

sudo systemctl status log2ram.service
● log2ram.service - Log2Ram
   Loaded: loaded (/etc/systemd/system/log2ram.service; enabled; vendor preset: enabled)
   Active: active (exited) since Wed 2023-06-21 09:38:47 BST; 4s ago
  Process: 734 ExecStart=/usr/local/bin/log2ram start (code=exited, status=0/SUCCESS)
 Main PID: 734 (code=exited, status=0/SUCCESS)

Jun 21 09:38:47 HOSTNAME log2ram[734]: apt/term.log.3.gz
Jun 21 09:38:47 HOSTNAME log2ram[734]: apt/term.log.4.gz
Jun 21 09:38:47 HOSTNAME log2ram[734]: apt/term.log.5.gz
Jun 21 09:38:47 HOSTNAME log2ram[734]: apt/term.log.6.gz
Jun 21 09:38:47 HOSTNAME log2ram[734]: apt/term.log.7.gz
Jun 21 09:38:47 HOSTNAME log2ram[734]: apt/term.log.8.gz
Jun 21 09:38:47 HOSTNAME log2ram[734]: private/
Jun 21 09:38:47 HOSTNAME log2ram[734]: sent 41,703,080 bytes  received 1,626 bytes  83,409,412.00 bytes/sec
Jun 21 09:38:47 HOSTNAME log2ram[734]: total size is 41,687,168  speedup is 1.00
Jun 21 09:38:47 HOSTNAME systemd[1]: Started Log2Ram.

df -h
Filesystem      Size  Used Avail Use% Mounted on
...
log2ram          40M   40M   52K 100% /var/log

journalctl -e

Jun 21 09:40:26 HOSTNAME rsyslogd[362]: action 'action-1-builtin:omfile' (module 'builtin:omfile') message lost, could not be processed. Check for additional error messages before this one. [v8.1901.0 try https://www.rsyslog.com/e/2027 ]
Jun 21 09:40:26 HOSTNAME rsyslogd[362]: file '8' write error: No space left on device [v8.1901.0 try https://www.rsyslog.com/e/2027 ]
Jun 21 09:40:26 HOSTNAME rsyslogd[362]: action 'action-1-builtin:omfile' (module 'builtin:omfile') message lost, could not be processed. Check for additional error messages before this one. [v8.1901.0 try https://www.rsyslog.com/e/2027 ]
Jun 21 09:40:26 HOSTNAME rsyslogd[362]: file '8' write error: No space left on device [v8.1901.0 try https://www.rsyslog.com/e/2027 ]
Jun 21 09:40:26 HOSTNAME rsyslogd[362]: action 'action-1-builtin:omfile' (module 'builtin:omfile') message lost, could not be processed. Check for additional error messages before this one. [v8.1901.0 try https://www.rsyslog.com/e/2027 ]
Jun 21 09:40:26 HOSTNAME rsyslogd[362]: file '8' write error: No space left on device [v8.1901.0 try https://www.rsyslog.com/e/2027 ]
Jun 21 09:40:26 HOSTNAME rsyslogd[362]: action 'action-1-builtin:omfile' (module 'builtin:omfile') message lost, could not be processed. Check for additional error messages before this one. [v8.1901.0 try https://www.rsyslog.com/e/2027 ]
Jun 21 09:40:26 HOSTNAME rsyslogd[362]: file '8' write error: No space left on device [v8.1901.0 try https://www.rsyslog.com/e/2027 ]