reimagined / resolve

Full stack CQRS, DDD, Event Sourcing framework for Node.js

Home Page:https://reimagined.github.io/resolve/

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Creating new deployment and reset sagas leads to high load and accessibility problems with independent deployments

pavsenin opened this issue · comments

Describe the bug
We have a task to automate our release. It does the following steps:

  1. Create new deployment (resolve-cloud deploy)
  2. Clone old event store to backup it (resolve-cloud clone)
  3. Link new deployment old event store (resolve-cloud link)
  4. Reset read models and sagas for new deployment (resolve-cloud read-models reset all & sagas reset-all)
  5. Manually create about 5 projects, 1 team, remove 3 projects (about 20 events).

Yesterday we run this script twice in the afternoon.
After that our independent main production deployment had the following issues:

  • 20-40 sec delays for any api requests
  • a lot of 503 errors

In new deployments event store adds about 5000 events in hour without any activity.
In a 5 hours the event store had 25000 events

After two hours the situation became better and we tried to reproduce the issue and run the steps above one more time, exactly after that the issue with production deployment returned.
After that we decided to delete all three new deployments and in a minute the load became zero and any issues were gone.

  • Three new deployment ids: tthtfa cyixfh menhr6, event store id l5pcpg
  • ReSolve Version [e.g. 0.32.0]