Account following/followers scalability issue

[techcoderx]

Currently, the full list of accounts followed and followers are stored in a single document in accounts collection. As MongoDB limits document sizes to 16MB, this could become an issue when a channel has millions of subscribers, where storing the entire list of account names will exceed this limit.

Also, the /account/:name API returns the full account details, which is all the data stored in the document. If the full list of accounts following/followers can be returned, the response could be several gigabytes in size for a large channel.

[heimindanger]

@techcoderx Thank you for adding this issue.

In short term, we definitely need to add some protection in transactions verification to avoid going over 16MB in a single document (if JSON.stringify(new_doc).length > 16000000 -> invalid transaction), because it clearly would cause a node crash. Even though even if all accounts followed a single account right now, we would still be far from it, it's clearly possible in the future.

In practise, this means there will be a hard-cap to how many followers you can get (probably around 1-2M). In order to lift this limit in the future I suppose there are two solutions:

1- Remove the 'followers' field on accounts data, and let the API nodes manage it based on the 'follows' field alone which has a max length of 2000. This solution has a negative side though, as 100% of accounts would need to be loaded in memory when the API nodes start.

2- Remove both followers and follows fields in account data. And instead use a new collection, and then each follow will become a single document, lifting the limit completely (we could even imagine getting rid of the maxFollows per account)

Also I'm pretty sure the same scalability issue generally applies to all existing arrays in documents. So it also applies to votes on contents and custom keys on accounts. So probably creating a more generic 'list' collection is better. Then we can use it for future features too (like the 'verified content creator' badge, or playlists).

[Hightouch]

I definitely agree with the solution number 2. This would improve the scalability of avalon and as you said permit to get rid of many limits. Followers and followings are also not necessarily loaded for an user in most of cases/pages and now each account call is clearly getting them. And it probably doesn't need for those collection to be loaded in the nodes cache.