Rerun of workflows took significantly longer time.
click2cloud-sagarB opened this issue · comments
Dear Farmvibes Team,
When I attempted to rerun the workflow in timelapse_visulization notebook, it took more than an hour to complete.
Earlier same entire workflow only takes less than 5 minutes to rerun.
Attached to this issue, you will find the following files for your reference:
log files for:
- terravibes-worker
- terravibes-orchestrator
You might also need these details:
- file system used : Root filesystem is used and is formatted with the "ext4" filesystem type.
- No of entries under the ~/.cache/farmvibes-ai is 205735.
Hello, @click2cloud-sagarB. Thank you for raising this issue and sharing the logs. We have investigated the increase in timeouts since last release's migration to async.io. We have improved the stability of the cluster and haven't experienced similar issues internally. The fix will be available on the next release.
Thank you @rafaspadilha, I'm glad to hear that the stability of the cluster has been improved, and you haven't encountered similar issues internally. Looking forward to the next release with the fix.
Hi, @click2cloud-sagarB. Are you still experiencing this issue?
Hi @rafaspadilha. We are unable to estimate the time required for a complete rerun of workflows since rerun of complete workflow fails due to Cache Retrieval Errors in Workflows since the most recent commit (issue #121).
Thanks for letting us know. We are working on fixing #121 and will keep you updated.
Hi, @click2cloud-sagarB. We have just released a new version of FarmVibes fixing the caching issues reported on #121 . Please, could you verify if the problem was solved?
Closing this issue, as @click2cloud-sagarB reported offline that it was fixed after the release.