Serious CPU usage with large UNPUBLISHED collections

[0o-de-lally]

Cross posted on: http://stackoverflow.com/questions/34844705/why-is-meteor-using-so-much-bandwidth-is-it-houston-admin-subscriptions

With Houston installed there is a strange scaling issue with Meteor. UNPUBLISHED collections seem to be affecting server usage. Houston seems to publish subscriptions even when the should be hidden to the meteor app.

REPRODUCE:
Here is the reproducible code: https://github.com/keyscores/houston_admin_performance_issue

SCENARIO A:
I started with a prototype app, which was aggregating/reducing 5 docs in 'db.originalData', into 1 doc in 'db.reducedResult'. The only collection published is the tiny 'reducedResult'.
All this looks very efficient:

• 1% CPU usage
• 20mb ram allocated

SCENARIO B:
However, when I test real data with 15,000 records in originalData, not published:

• 100% CPU is used
• 100% ram is used (1GB)
• Memory swapping to Disk
• 100% Bandwidth is used 10mb/S

This is strange because only the small collection * 1 record * is published. The large collection, originalCollection, is unpublished.

---

Infrastructure:
The mongo server is on a separate server (compose.io) than the Meteor front end (Digital Ocean via MUP)

[0o-de-lally]

@AlexeyMK @gterrono Hi Guys, just want to surface this issue to you. For now I've had to uninstall Houston from a production application which was generating this issue. I'm not familiar enough with the code to see where subscriptions are being created.

[Tim]

Also had to remove Houston for same reason. Was spiking CPU and bringing down app.