Documented, safe mechanism for running bulk update transformations against our data
simonw opened this issue · comments
We often need to run a bulk update against all of our data.
Our tools for doing this are ad-hoc and under-documented:
- Sometimes we use Django data migrations (though these perform poorly if they have to do a lot of SQL queries due to the restricted network speed between Cloud Build and Cloud SQL)
- Often we use various API methods such as /api/updateLocations and /api/updateLocationConcordances - in conjunction with data exported from a Django SQL Dashboard query and maybe some ad-hoc Python code, e.g. #370 (comment)
Having a documented, safe mechanism for doing these in a more structured way would be very valuable.