Use a separate table to ingest data, then merge
jacopofar opened this issue · comments
The current ingestion process populates the card table from scratch. This has two problems:
- it takes a long time (20 minutes to many hours), during which the app is unusable
- it doesn't account for updates, only insertion
- if the table is truncated and recreated and the process is interrupted, it's left like that
it will get worse when adding lemmatization, which will increase the size and time.
So, a solution is to first create a staging table with the data and only after that merge it with the current one using SQL transactions.
Done, together with the new partitioning it makes it a bit faster but most importantly allows incremental update while the app is running