Updates to the workflow repository may not be noticed by all nodes
mikub opened this issue · comments
It seems that on ECS Fargate/EFS cluster setup, the Directory watcher used in titanoboa.repo/RepoWatcherComponent does not get triggered every time a new workflow (aka job definition) revision is saved.
This leads to a situation where some nodes may have an obsolete view of repo:
e.g. node A sees that the latest :head revision of workflow X is 5
while node B sees that the latest :head revision of workflow X is 2
This is interesting since with EFS this DOES work on EC2/EFS setup. So it may be related to some idiosyncrasy of Fargate.
We can fix this by periodically refreshing the workflow repo atom.