mtail fails to recover after log rotation

[John Rando]

i'm running a modified configuration, so this might be a "me" problem and not an issue with the project...

here's my setup:

Synology DS2419+ with Docker v18.09.8 (latest via Package Center)

• chiamon_mtail
• chiamon_chia_exporter
• chiamon_node_exporter
• chia (full node, farmer, and harvester)

PhotonOS with Docker v19.03.15

• chiamon_prometheus
• chiamon_grafana

The Issue:

when chia's debug.log rotates, mtail stops updating with new information. the docker image itself stays running, but no updates related to Search Time or new Plots are seen until i restart the chiamon_mtail container.

log rotated at :26, container manually restarted at :36 -

$ ls -l /volume1/chia/appdata/mainnet/log/  
total 123540  
-rwxrwxrwx+ 1 root root   643928 May  6 17:36 debug.log  
-rwxrwxrwx+ 1 root root 20971568 May  6 17:26 debug.log.1  
-rwxrwxrwx+ 1 root root 20971585 May  6 13:01 debug.log.2  
-rwxrwxrwx+ 1 root root 20971565 May  6 09:07 debug.log.3  
-rwxrwxrwx+ 1 root root 20971717 May  6 05:58 debug.log.4  
-rwxrwxrwx+ 1 root root 20971521 May  6 02:24 debug.log.5  
-rwxrwxrwx+ 1 root root 20971583 May  5 23:10 debug.log.6  

no entry in mtail's container log around :26, but does have entries when i restarted the container -

$ sudo docker logs chiamon_mtail_1  
I0506 21:12:21.649568       1 mtail.go:193] Listening on [::]:3903  
I0506 22:12:21.465474       1 store.go:153] Running Store.Expire()  
I0506 23:12:21.444691       1 store.go:153] Running Store.Expire()  
I0507 00:12:21.443927       1 store.go:153] Running Store.Expire()  
I0507 01:12:21.446024       1 store.go:153] Running Store.Expire()  
I0507 01:35:12.859262       1 main.go:155] Received terminated, exiting...  
I0507 01:35:12.859578       1 loader.go:428] END OF LINE  
I0507 01:35:12.859576       1 mtail.go:206] Shutdown requested.  
I0507 01:35:12.859602       1 vm.go:1025] VM "chialog.mtail" finished  
I0507 01:35:56.682599       1 main.go:113] mtail version v3.0.0-rc45-31-g12162ff3 git revision 12162ff338afebf71536cef709f15b0bf5c29b8b go version go1.16.3 go arch amd64 go os linux  
I0507 01:35:56.682685       1 main.go:114] Commandline: ["/usr/bin/mtail" "-progs" "/etc/mtail" "-logs" "/var/log/chia/debug.log" "-logtostderr"]  
I0507 01:35:56.683161       1 store.go:178] Starting metric store expiry loop every 1h0m0s  
I0507 01:35:56.825930       1 loader.go:242] Loaded program chialog.mtail  
I0507 01:35:56.826086       1 tail.go:259] Tailing /var/log/chia/debug.log  
I0507 01:35:56.826239       1 mtail.go:193] Listening on [::]:3903  

[Kevin Retzke]

Ah, what's happening is that bind-mounts work off inode, not filename, so when the log is rotated (typically done by renaming the file) the mount stays with the old log. The solution is simple, mount the log directory instead of the file:

    volumes:
      - ${HOME}/.chia/mainnet/log/:/var/log/chia/

Thanks for letting me know, I'll push a fix.

[John Rando]

i made this change and can confirm that it correctly survives a log rotation. thanks!

$ ls -l /volume1/chia/appdata/mainnet/log
total 147708
-rwxrwxrwx+ 1 root root  4415677 May  6 21:21 debug.log
-rwxrwxrwx+ 1 root root 20971605 May  6 20:36 debug.log.1
-rwxrwxrwx+ 1 root root 20971568 May  6 17:26 debug.log.2
-rwxrwxrwx+ 1 root root 20971585 May  6 13:01 debug.log.3
-rwxrwxrwx+ 1 root root 20971565 May  6 09:07 debug.log.4
-rwxrwxrwx+ 1 root root 20971717 May  6 05:58 debug.log.5
-rwxrwxrwx+ 1 root root 20971521 May  6 02:24 debug.log.6
-rwxrwxrwx+ 1 root root 20971583 May  5 23:10 debug.log.7