[BUG] Rocket.Chat Apps data is not synchronized across Rocket.Chat instances

[Cuong Huu Nguyen]

We are using a production design that uses more than one host which multiple instances per host.

When deploying a Rocket.Chat App, we expect that the Rocket.Chat Apps data includes status and configurations are synchronized across all instances.

There is currently a bug in Rocket.Chat App Engine that makes Apps data cached and cannot be synced when we change the data in the other instances of Rocket.Chat.

In src/server/AppManager.ts, you load all of the data from the database in function load into field app. But then, in the function get you did not get the data from the database but from the app field. This makes the AppManager cannot get the updated information from the database as well as from the other instance.

We should fix this problem to let apps synchronize their data with the database and to provide consistency

[Nghia Tran]

I got the same issue with my Rocket.Chat instances. Whenever I changed the App config, I have to restart all Rocket.Chat instances to let them retrieve configuration from database.

[Bradley Hilton]

@d-gubert @lolimay what changed to cause this to no longer work? Changing a setting on one instance used to change it across all other instances due to the oplog handler...this is a major breaking change without a doubt. 😢

[Douglas Gubert]

Hey folks! This is definitely a non-trivial bug - there was no intentional change that would cause this behavior. We'll need further investigation to pinpoint what is happening.

[Nghia Tran]

Are there any updates on this issue?

[Cuong Huu Nguyen]

For your information, the data in the database is still correct, somehow the app cannot get the updated data from the database. Restarting all instances makes the Apps data return to the correct value. However, every time we change the data, we still have to restart all the instances again 😭

[Cuong Huu Nguyen]

This issue is caused by the firewall between VMs.
As my investigation, the server settings are synchronized by the mongo event handler but the Apps settings are only can be synced by the Stream Broadcast.
Using a firewall between VMs blocks the connection therefore the settings cannot be synced
Anw, it still seems to be a bug
@d-gubert @lolimay @graywolf336

[Shiqi Mei]

Thanks @meomay503 for your effort! It helped us a lot! @d-gubert , do we have plans to fix it?

[Cuong Huu Nguyen]

Thanks for having a look into this issue, I've been looking for this fix for a very long time

[Cuong Huu Nguyen]

Hi, could you please inform me if you have any plan to move this sync to DB change stream or something?
The information could help us a lot in planning the next steps. Thank you.