Detect and deal with changes in structure to current datasets
levyj opened this issue · comments
From time to time, the structure of a dataset will change. How can Open Grid find out about and deal with these changes? This is related to the same issue in Plenario -- UrbanCCD-UChicago/plenario#291.
@HeyZoos - you mentioned this was deployed on Plenario now, right?
@tomschenkjr - Yep, during the ETL process Plenario completely replaces the existing dataset instead of trying to perform upserts. This has been enough to ingest datasets where the number of columns changes.
@tomschenkjr - However this does not address communicating the changes to Open Grid, the metadata about the dataset will change but that would have to actively be checked for.
@HeyZoos - great, and that does happen so closing the issue.