Global Table Support
er1c opened this issue · comments
When syncing to/from dynamodb global tables, the diffing should ignore the additional global-table related fields.
I technically haven't tested this, simply looking at the code trying to figure out if this would work for global table replication, I think this would fail. I think the diffing would need to ignore any of the aws: prefixed fields. For reference: https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/globaltables_HowItWorks.html
Example
{
"aws:rep:deleting": {
"BOOL": false
},
"aws:rep:updateregion": {
"S": "us-west-2"
},
"aws:rep:updatetime": {
"N": "1542916362.440001"
},
"created_on": {
"N": "1539334691"
},
"id": {
"S": "CatalogId-B2ZyEvviYV5CdgBwjbAN6"
},
"key": {
"S": "catalog.autocare_webservice_enabled"
},
"updated_on": {
"N": "1539334691"
},
"value": {
"BOOL": false
}
}The work around would simply be to specifically use your --ignore-att flags, but the new "official" attributes should probably be added to the default list. (I'm not sure if this is best handled in the cli.js or part of the Synchronizer.js in the omit function?
Haha, when I saw your issue I thought "wait, I already fixed that". I had a local test branch where I'd tested the changes, but I never pushed it. 🤦♂️I'll get that done soon and cut a new release.
@er1c could you possibly test the repl-fields branch to make sure it works for you? Unfortunately we don't have good automated testing on this project, and I'm pretty sure when I made this commit back in May I did the manual testing, but I won't have time this week to do any manual testing of it, and it sounds like you have a usecase readily available.
@er1c also, could you give a brief description of your usecase? I know when I wrote this patch I was simply using the library to validate that AWS was keeping the data sync'd after I first changed a batch of tables to their global tables. But I wasn't actually using it to make data changes. If you're using it to make changes, I'm curious why? Would you mind elaborating?
@jthomerson I'm using it to create new "environment-specific" tables (staging, test, personal dev, etc). DynamoDB Global Tables currently don't support the native backup/restore (which is ghetto). So the need to sync data if we want to have separate environments is the current requirement.
The idea would be to copy our production tables into our stage table.
Working Example:
dynamodb-table-sync \
--master us-west-2:attribute_values_development \
--slave us-west-2:attribute_values_eric \
--ignore-attr "aws:rep:deleting" \
--ignore-attr "aws:rep:updateregion" \
--ignore-attr "aws:rep:updatetime" \
--write-missing \
--write-differing \
--delete-extra
Source Table
Source Item
{
"aws:rep:deleting": {
"BOOL": false
},
"aws:rep:updateregion": {
"S": "us-west-2"
},
"aws:rep:updatetime": {
"N": "1542916362.440001"
},
"created_on": {
"N": "1539334691"
},
"id": {
"S": "CatalogId-B2ZyEvviYV5CdgBwjbAN6"
},
"key": {
"S": "catalog.autocare_webservice_enabled"
},
"updated_on": {
"N": "1539334691"
},
"value": {
"BOOL": false
}
}Destination Table
Destination Item
{
"aws:rep:deleting": {
"BOOL": false
},
"aws:rep:updateregion": {
"S": "us-west-2"
},
"aws:rep:updatetime": {
"N": "1542916362.440001"
},
"created_on": {
"N": "1539334691"
},
"id": {
"S": "CatalogId-B2ZyEvviYV5CdgBwjbAN6"
},
"key": {
"S": "catalog.autocare_webservice_enabled"
},
"updated_on": {
"N": "1539334691"
},
"value": {
"BOOL": false
}
}Result Success
It seems to be syncing successfully, and properly ignoring these replication-specific attributes.
Yeah, that makes sense. Actually, in thinking back to why I originally authored the commit in question, it wasn't for comparison as I originally thought - it was exactly what you're doing - provisioning a new environment and seeding it with data from PRD.
👍 Confirmed this is working in master
dynamodb-table-sync \
--master us-west-2:attribute_values_development \
--slave us-west-2:attribute_values_eric \
--write-missing \
--write-differing \
--delete-extra