Handling deletion of nested directories
LeKovr opened this issue · comments
Filesystem would issue only one event for deleted directory, so it's needed to add logic to delete files by matching their path prefix.
Originally posted by @gordonbondon in #6 (comment)
Tests shows that separate event generated for every deleted file. How can I reploduce the issue problem?
I was able to reproduce this with latest code on MacOS. Given dir structure:
○ /tmp/logs
→ tree
.
├── file_in_root.log
└── many
├── four.log
├── one.log
├── three.log
└── two.log
1 directory, 5 files
and deleting directory via rm -rf ./many i see this logs in debug + trace mode (replaced ev with ev.Op.String() to see event names ):
2021-02-11T20:16:13.944+0800 INFO webtail/indexer.go:97 Handling file event {"event": "Name: /tmp/logs/many/one.log, Op: REMOVE"}
2021-02-11T20:16:13.944+0800 INFO webtail/hub.go:232 Trace from indexer {"message": {"mtime":"0001-01-01T00:00:00Z","size":0,"name":"many/one.log","deleted":true}}
2021-02-11T20:16:13.944+0800 INFO webtail/indexer.go:79 Deleting path from index {"path": "many/one.log"}
2021-02-11T20:16:13.944+0800 INFO webtail/hub.go:302 Send reply {"message": "{\"type\":\"index\",\"data\":{\"mtime\":\"0001-01-01T00:00:00Z\",\"size\":0,\"name\":\"many/one.log\",\"deleted\":true}}"}
2021-02-11T20:16:13.944+0800 INFO webtail/indexer.go:97 Handling file event {"event": "Name: /tmp/logs/many/two.log, Op: REMOVE"}
2021-02-11T20:16:13.944+0800 INFO webtail/indexer.go:97 Handling file event {"event": "Name: /tmp/logs/many, Op: REMOVE|WRITE"}
2021-02-11T20:16:13.944+0800 INFO webtail/hub.go:232 Trace from indexer {"message": {"mtime":"0001-01-01T00:00:00Z","size":0,"name":"many/two.log","deleted":true}}
2021-02-11T20:16:13.944+0800 INFO webtail/indexer.go:97 Handling file event {"event": "Name: ., Op: CREATE"}
2021-02-11T20:16:13.944+0800 INFO webtail/indexer.go:79 Deleting path from index {"path": "many/two.log"}
2021-02-11T20:16:13.944+0800 INFO webtail/hub.go:302 Send reply {"message": "{\"type\":\"index\",\"data\":{\"mtime\":\"0001-01-01T00:00:00Z\",\"size\":0,\"name\":\"many/two.log\",\"deleted\":true}}"}
2021-02-11T20:16:13.944+0800 INFO webtail/hub.go:232 Trace from indexer {"message": {"mtime":"0001-01-01T00:00:00Z","size":0,"name":"many","deleted":true}}
2021-02-11T20:16:13.944+0800 INFO webtail/hub.go:302 Send reply {"message": "{\"type\":\"index\",\"data\":{\"mtime\":\"0001-01-01T00:00:00Z\",\"size\":0,\"name\":\"many\",\"deleted\":true}}"}
2021-02-11T20:16:13.944+0800 INFO webtail/indexer.go:97 Handling file event {"event": "Name: , Op: REMOVE"}
2021-02-11T20:16:13.944+0800 INFO webtail/hub.go:232 Trace from indexer {"message": {"mtime":"0001-01-01T00:00:00Z","size":0,"name":"","deleted":true}}
2021-02-11T20:16:13.945+0800 INFO webtail/hub.go:302 Send reply {"message": "{\"type\":\"index\",\"data\":{\"mtime\":\"0001-01-01T00:00:00Z\",\"size\":0,\"name\":\"\",\"deleted\":true}}"}
2021-02-11T20:16:13.955+0800 INFO webtail/indexer.go:97 Handling file event {"event": "Name: , Op: REMOVE"}
2021-02-11T20:16:13.955+0800 INFO webtail/hub.go:232 Trace from indexer {"message": {"mtime":"0001-01-01T00:00:00Z","size":0,"name":"","deleted":true}}
2021-02-11T20:16:13.955+0800 INFO webtail/hub.go:302 Send reply {"message": "{\"type\":\"index\",\"data\":{\"mtime\":\"0001-01-01T00:00:00Z\",\"size\":0,\"name\":\"\",\"deleted\":true}}"}
We can see that it properly receives two remove events but then gets some events with an empty name and a REMOVE|WRITE for the directory, and some random CREATE event for ..
I used fswatch to observe fsevents from macos at the same time and I can see all events are properly sent by the OS:
→ fswatch -x /tmp/logs
/private/tmp/logs/many/one.log Removed IsFile
/private/tmp/logs/many/two.log Removed IsFile
/private/tmp/logs/many/four.log Removed IsFile
/private/tmp/logs/many/three.log Removed IsFile
/private/tmp/logs/many Removed IsDir
I was not able to reproduce on linux, so I guess this could be some bug in fsnotify package related to how it handles events from macos fsevents.
I was able to reproduce this with latest code on MacOS.
Great work, @gordonbondon ! Thank you a lot.
Should we make a workaround for this or just wait for appropriate fsnotify fixes?
I guess this could be some bug in fsnotify package
So there is nothing to fix in webtail code