Create a backend mysql script to drop partitions not linked to any hdfs directory
mckempanna opened this issue · comments
When hive-sre generates partition related alter recommendations, for some environments, this list gets too huge in the range of 1-1.5 million statements.
Executing this hql from the beeline takes a lot of splitting and parallelizing and takes more than 10-12 hours in the best case and days in the worst case.
It would be helpful to have a script that can be directly run on the backend Database mysql that drops the partitions with no hdfs path.
This will save a lot of time for the upgrade window in cases where admins will be asked to do it only just before the upgrade in the green zone.
'u3e' process created for Metastore Direct Adjustments. See docs for u3e.