feedv2: DB-ify
zhehaowang opened this issue · comments
Updating / reading 20K files in 1.6K folders and each time decoding and encoding the whole file might be inefficient at some point (!). We should look into transitioning to a DB-based (data warehouse / columnar) solution.
The hope is that the barrier was strong enough to ease this transition.
Should we do this before or after cloud deployment?
Probably the best approach will be two / three csvs for each?