`UnmarshalItem` does not work with the auxiliary types

[Dmytrii Nagirniak]

Details

When implementing a dynamo.ItemUnmarshaler interface using a pretty standard technique in Go for maintaining backwards compatibility, the behaviour is very different to the likes of json.Unmarshal and the dynamodbattribute.UnmarshalMap.

The dynamo.UnmarshalItem implementation does not allocate any pointers and does not work with "aux" type declarations.

All these examples are expected to be unmarshalled correctly but most of those are not:

package main

import (
	"encoding/json"
	"fmt"
	"log"

	"github.com/aws/aws-sdk-go/aws"
	"github.com/aws/aws-sdk-go/service/dynamodb"
	"github.com/aws/aws-sdk-go/service/dynamodb/dynamodbattribute"
	"github.com/guregu/dynamo"
)

type Coffee struct {
	Name string
}

func main() {
	in := map[string]*dynamodb.AttributeValue{
		"ID":   {S: aws.String("intenso")},
		"Name": {S: aws.String("Intenso 12")},
	}

	// No custom unmrashalling.
	{
		type coffeeItemDefault struct {
			ID string
			Coffee
		}
		var out coffeeItemDefault
		if err := dynamo.UnmarshalItem(in, &out); err != nil {
			log.Fatal(err)
		}
		fmt.Printf("Default behaviour:\t%+v\n", out)
		// Default behaviour:	{ID:intenso Coffee:{Name:Intenso 12}}
	}

	// Embedded pointer using AWS SDK custom unmarshalling.
	{
		var out coffeeItemSDKEmbeddedPointer
		if err := dynamo.UnmarshalItem(in, &out); err != nil {
			log.Fatal(err)
		}
		fmt.Printf("AWS SDK pointer:\t%+v %+v\n", out, out.Coffee)
		// AWS SDK pointer:	{ID:intenso Coffee:0xc000099bb0} &{Name:Intenso 12}
	}

	// Flat unmarshalling.
	{
		var out coffeeItemFlat
		if err := dynamo.UnmarshalItem(in, &out); err != nil {
			log.Fatal(err)
		}
		fmt.Printf("Flat unmarshalling:\t%+v\n", out)
		// Flat unmarshalling:	{ID: Name:}
	}

	// Embedded struct example.
	{
		var out coffeeItemEmbedded
		if err := dynamo.UnmarshalItem(in, &out); err != nil {
			log.Fatal(err)
		}
		fmt.Printf("Embedded struct:\t%+v\n", out)
		// Embedded struct:	{ID: Coffee:{Name:}}
	}

	// Embedded pointer struct example.
	{
		var out coffeeItemEmbeddedPointer
		if err := dynamo.UnmarshalItem(in, &out); err != nil {
			log.Fatal(err)
		}
		fmt.Printf("Embedded pointer:\t%+v %+v\n", out, out.Coffee)
		// Embedded pointer:	{ID: Coffee:<nil>} <nil>
	}

	// The above is different to standard JSON behaviour
	// (as a point of comparison).
	{
		var out coffeeItemEmbeddedPointer
		if err := json.Unmarshal([]byte(`{"ID":"intenso","Name":"Intenso 12"}`), &out); err != nil {
			log.Fatal(err)
		}
		fmt.Printf("JSON Embedded pointer:\t%+v %+v\n", out, out.Coffee)
		// JSON Embedded pointer:	{ID:intenso Coffee:0xc0000a3c00} &{Name:Intenso 12}
	}
}

type coffeeItemFlat struct {
	ID   string
	Name string
}

func (c *coffeeItemFlat) UnmarshalDynamoItem(item map[string]*dynamodb.AttributeValue) error {
	type alias coffeeItemFlat
	aux := struct {
		*alias
	}{
		alias: (*alias)(c),
	}
	if err := dynamo.UnmarshalItem(item, &aux); err != nil {
		return err
	}
	// Custom unmarshalling logic omitted for brevity.
	return nil
}

type coffeeItemEmbedded struct {
	ID string
	Coffee
}

func (c *coffeeItemEmbedded) UnmarshalDynamoItem(item map[string]*dynamodb.AttributeValue) error {
	type alias coffeeItemEmbedded
	aux := struct {
		*alias
	}{
		alias: (*alias)(c),
	}
	if err := dynamo.UnmarshalItem(item, &aux); err != nil {
		return err
	}
	// Custom unmarshalling logic omitted for brevity.
	return nil
}

type coffeeItemEmbeddedPointer struct {
	ID string
	*Coffee
}

func (c *coffeeItemEmbeddedPointer) UnmarshalDynamoItem(item map[string]*dynamodb.AttributeValue) error {
	type alias coffeeItemEmbeddedPointer
	aux := struct {
		*alias
	}{
		alias: (*alias)(c),
	}
	if err := dynamo.UnmarshalItem(item, &aux); err != nil {
		return err
	}
	// Custom unmarshalling logic omitted for brevity.
	return nil
}

func (c *coffeeItemEmbeddedPointer) UnmarshalJSON(data []byte) error {
	type alias coffeeItemEmbeddedPointer
	aux := struct {
		*alias
	}{
		alias: (*alias)(c),
	}
	if err := json.Unmarshal(data, &aux); err != nil {
		return err
	}
	// Custom unmarshalling logic omitted for brevity.
	return nil
}

type coffeeItemSDKEmbeddedPointer struct {
	ID string
	*Coffee
}

func (c *coffeeItemSDKEmbeddedPointer) UnmarshalDynamoItem(item map[string]*dynamodb.AttributeValue) error {
	type alias coffeeItemEmbeddedPointer
	aux := struct {
		*alias
	}{
		alias: (*alias)(c),
	}
	if err := dynamodbattribute.UnmarshalMap(item, &aux); err != nil {
		return err
	}
	// Custom unmarshalling logic omitted for brevity.
	return nil
}

Expected

The above program to output:

Default behaviour:	{ID:intenso Coffee:{Name:Intenso 12}}
AWS SDK pointer:	{ID:intenso Coffee:0xc000022c82} &{Name:Intenso 12}
Flat unmarshalling:	{ID:intenso Name:Intenso 12}
Embedded struct:	{ID:intenso Coffee:{Name:Intenso 12}}
Embedded pointer:	{ID:intenso Coffee:0xc000033c83} &{Name:Intenso 12}
JSON Embedded pointer:	{ID:intenso Coffee:0xc000013c80} &{Name:Intenso 12}

Actual

The above program outputs:

Default behaviour:	{ID:intenso Coffee:{Name:Intenso 12}}
AWS SDK pointer:	{ID:intenso Coffee:0xc000013ba0} &{Name:Intenso 12}
Flat unmarshalling:	{ID: Name:}
Embedded struct:	{ID: Coffee:{Name:}}
Embedded pointer:	{ID: Coffee:<nil>} <nil>
JSON Embedded pointer:	{ID:intenso Coffee:0xc000013c80} &{Name:Intenso 12}

Workaround 1.

Implement dynamo.ItemUnmarshaler interface using dynamodbattribute.UnmarshalMap (see the coffeeItemSDKEmbeddedPointer type).

NOTE: this workaround gives up all of the dynamo tags and benefits and it would have to resort to the AWS SDK supported unmarshalling. For example, github.com/google/uuid.UUID type can no longer be marshalled because AWS SDK does not support encoding.TextUnmarshaler while dynamo does.

Workaround 2.

Copy all the field definitions into the aux structure.
Obvious disadvantage is that these fields need to be in sync.

Workaround 3.

See below #181 (comment)

[guregu]

Thank you for the detailed report. I'll take a look. If we can fix this without breaking compatibility I'll put it in v1, worst case scenario it can be something for v2.

A 3rd workaround would be to use an embedded struct without the aux stuff, something like:

type Outer struct {
	Inner
}

type Inner struct {
	Foo string
	Bar string
}

func (o *Outer) UnmarshalDynamoItem(item map[string]*dynamodb.AttributeValue) error {
	if err := dynamo.UnmarshalItem(item, &o.Inner); err != nil {
		return err
	}
	// custom stuff ...
	return nil
}

[guregu]

Oh yeah, if the fix is somehow incompatible with v1 we can also provide a way to unmarshal items that ignores the custom methods. Hacky but would be better than nothing.

[guregu]

Sorry for the long wait. I finally fixed this in the latest release: https://github.com/guregu/dynamo/releases/tag/v1.12.0 (see #188).

Thanks a bunch for the test cases, they all work now. I added them to the unit tests.