Possible migration issue with latest version of DynamoDB local

[Darren]

While onboarding some new devs, they ran into problems while running migrations against DDB local - specifically, migrations that were adding indexes to an "on-demand" table that did not specify provisioned throughput were failing with an unknown error.

For example

defmodule MyApp.Repo.Migrations.AddIndextoFoo do
  use Ecto.Migration

  def up do
    alter table(:foo,
      options: [
        global_indexes: [
          [index_name: "foo_bar",
            keys: [:foo, :bar],
            create_if_not_exists: true]
        ]
      ]) do

      add :foo, :string, hash_key: true
      add :bar, :string, range_key: true
    end
  end

  def down do
    alter table(:foo,
      options: [
        global_indexes: [
          [index_name: "foo_bar",
            drop_if_exists: true]]
      ]
    ) do
      remove :foo_bar
    end
  end
end

would fail with an error like

{"message":"2020-3-31 19:16:21 UTC [Ecto dynamo info] Ecto.Adapters.DynamoDB.Migration.update_table_recursive: DynamoDB/ExAws response","attributes":{"Ecto.Adapters.DynamoDB.Migration.update_table_recursive-result":"{:error, {:http_error, 500, \"{\\\"__type\\\":\\\"com.amazonaws.dynamodb.v20120810#InternalFailure\\\",\\\"message\\\":\\\"The request processing has failed because of an unknown error, exception or failure.\\\"}\"}}"}}
{"message":"2020-3-31 19:16:21 UTC [Ecto dynamo info] Ecto.Adapters.DynamoDB.Migration.update_table_recursive: error attempting to update table. Stopping...","attributes":{"Ecto.Adapters.DynamoDB.Migration.update_table_recursive-error":{"table_name":"foo","error_tuple":"{:http_error, 500, \"{\\\"__type\\\":\\\"com.amazonaws.dynamodb.v20120810#InternalFailure\\\",\\\"message\\\":\\\"The request processing has failed because of an unknown error, exception or failure.\\\"}\"}","data":"%{attribute_definitions: [%{attribute_name: :foo, attribute_type: \"S\"}, %{attribute_name: :bar, attribute_type: \"S\"}], global_secondary_index_updates: [%{create: %{index_name: \"foo_bar\", key_schema: [%{attribute_name: \"foo\", key_type: \"HASH\"}, %{attribute_name: \"bar\", key_type: \"RANGE\"}], projection: %{projection_type: \"ALL\"}}}]}"}}}

These devs were running the latest (as of 4/1/20) version of DynamoDB local, which may have introduced new behaviors that we're not accounting for (or fixed issues that we no longer need to account for). I (and other devs) have had no problem running migrations like this against DDB local 1.11.477, which was released 2/6/19.

While I strongly suspect that this is related to the version of DDB local, this requires further investigation, as the root cause of this problem is not yet understood.

[Darren]

If the local DDB version does turn out to be the issue (and the bug is resolved), be sure to make note of this in the README.md and upgrade_guides/version_2_upgrade_guide.md files.

[Darren]

Confirmed - after upgrading my local version of DDB to 1.11.478 (released 2020/1/16), I was able to reproduce this issue.

It seems to stem from the fact that the map returned for a request for table info (Info.table_info("table")) on a table with provisioned throughput no longer includes a BillingModeSummary key; however, a similar request on a pay-per-request table still includes that value as expected. This needs to be tested against production DDB to see if this issue only affects local versions, or if DDB's response has changed (though I see nothing in the DDB docs to suggest this).

While that difference is observable in a few tests where we were specifically checking for that value in a table_info request, it caused the bug initially noted in this issue based on the conditional at line 334 in Ecto.Adapters.DynamoDB.Migration -

if table_info["BillingModeSummary"]["BillingMode"] == "PROVISIONED" do

I replaced this with

if !Map.has_key?(table_info, "BillingModeSummary") do

and the migrations ran with no problem.

This requires some further investigation and discussion...

• confirm production DDB behavior
• assuming there's a discrepancy between production and local DDB - since we get table info via the ExAws.Dynamo API, should we aim to alter the data there, adding a "BillingModeSummary" key to the data if none is found?

[Darren]

Actually, this appears to mirror the current live DDB behaviors. Here are the results of two calls to ExAws.Dynamo.describe_table

Provisioned:

%{
  "Table" => %{
    "AttributeDefinitions" => [
      %{"AttributeName" => "id", "AttributeType" => "S"}
    ],
    "CreationDateTime" => 1586555408.882,
    "ItemCount" => 0,
    "KeySchema" => [%{"AttributeName" => "id", "KeyType" => "HASH"}],
    "ProvisionedThroughput" => %{
      "NumberOfDecreasesToday" => 0,
      "ReadCapacityUnits" => 1,
      "WriteCapacityUnits" => 1
    },
    "TableArn" => "arn:aws:dynamodb:us-east-1:206176965479:table/billing_test",
    "TableId" => "a4c4d9e7-5703-4b9e-917b-5c9b24132b5f",
    "TableName" => "billing_test",
    "TableSizeBytes" => 0,
    "TableStatus" => "ACTIVE"
  }
}

On-demand:

%{
  "Table" => %{
    "AttributeDefinitions" => [
      %{"AttributeName" => "id", "AttributeType" => "S"}
    ],
    "BillingModeSummary" => %{
      "BillingMode" => "PAY_PER_REQUEST",
      "LastUpdateToPayPerRequestDateTime" => 1586555576.668
    },
    "CreationDateTime" => 1586555576.668,
    "ItemCount" => 0,
    "KeySchema" => [%{"AttributeName" => "id", "KeyType" => "HASH"}],
    "ProvisionedThroughput" => %{
      "NumberOfDecreasesToday" => 0,
      "ReadCapacityUnits" => 0,
      "WriteCapacityUnits" => 0
    },
    "TableArn" => "arn:aws:dynamodb:us-east-1:206176965479:table/billing_test",
    "TableId" => "a2ac8df2-264d-406c-86e7-eddac069640d",
    "TableName" => "billing_test",
    "TableSizeBytes" => 0,
    "TableStatus" => "ACTIVE",
    "TableThroughputModeSummary" => %{
      "LastUpdateToPayPerRequestDateTime" => 1586555576.668,
      "TableThroughputMode" => "PAY_PER_REQUEST"
    }
  }
}

So it appears that DDB has dropped that key in the results when describing a provisioned table.

[Darren]

f8eb7e0