Why is the 'WATCHDOG_TRAVERSE_MOVED_DIR_DELAY' constant so large?
matkuki opened this issue · comments
Hi,
watchdog works wonderfully, but when using it to watch a a C-project's build directory, there are a lot of times when the WATCHDOG_TRAVERSE_MOVED_DIR_DELAY kicks-in, and the program waits for a second before continuing, which happens about 5 to 8 times during a compilation.
So when I compile and then create a new file 2 second after pressing build, watchdog needs more than 5 seconds for the file-created callback to fire and tell me that the file was created.
Can I decrease this WATCHDOG_TRAVERSE_MOVED_DIR_DELAY value or should I do something else?
Thanks and keep up the great work
Matic
Hm it's old code (back in 2010 with 3551cc4). I am not sure what we can do with that, honestly. I'm always open to improvements though.
You can tweak the value, and see if it improves your workflow:
from watchdog.observers import read_directory_changes
read_directory_changes.WATCHDOG_TRAVERSE_MOVED_DIR_DELAY = 0.1 # or anything elseExcellent, thanks 👍
It works with 0.01 for me, but I was just hesitant to change it as I wasn't sure if it would break something.
Actually, if you set the value to 0, do you see any regression?
It's a hack introduced at the same time we added Windows support. I guess it's not a problem for your use case. Try keeping it at 0, and keep us updated if you see any problem.
I'm not sure we can remove it completely. We would need good tests from other projects like synchronisation engines. I've such a project, so will try something.
@samschott did you ever think about that hack? Do you think you could try on your project to see if you have regressions?
I don't think I was ever involved in the WinAPI implementation - and my project only supports Linux and macOS. That being said, happy to give my opinion:
It looks like WATCHDOG_TRAVERSE_MOVED_DIR_DELAY was present already in the first version of the winapi implementation, introduced in 3551cc4 about 13 years ago. It is likely that the original author was in fact seeing issues without this hack. However, things may have changed since then.
My recommendation would be have a test case for moving a directory tree, maybe even containing files that have a slightly larger size, and make sure that it passes on Windows. You can then try to reduce or remove the delay and check if things break. This is of course no guarantee that the change is safe, Windows supports different file systems and hard-drive types. But its at least a first indication that its a safe change:)
Actually, I just checked an old project I used to work on, and for years now I removed that delay: https://github.com/nuxeo/nuxeo-drive/blob/f7289e29771d575df053cd754d44c2dab72c6dec/nxdrive/engine/watcher/local_watcher.py#L36-L40
It was made specially to not miss events... I'll deprecate-remove that hack from the code. FTR that software is used by thousands of users, and they are syncing thousands/millions of files/folders of all kind of sizes. If any issue with the hack being removed would arise, it would have been catched a long time ago.
Doing so would close/fix:
Finally, a full search on GitHub returns only 2 occurences (the one I linked above), and another one in the same software, but at a different "age". See the search.