Can the Collection Upsert method take a GenericData?

[C Miller]

Hi,

We add/edit collections a lot and so far we've been using the Add and Update methods which both take GenericData objects. We're starting to get race conditions around that so we'd like to move to using Upsert, but that takes a CollectionObject instead of GenericData and I'm having trouble building the collection object. Is it possible to pass generic data to Upsert?

I'm trying to make the collection object like this:

var collectionObject = new CollectionObject(collectionEntryId);

var dataDict = new Dictionary<string, object>
{
	{ PropertyName1, PropertyValue1 },
	... etc
};

collectionObject.SetData(dataDict);

And the error I'm getting is

Internal Server Error, Can not add property id to Newtonsoft.Json.Linq.JObject. Property with the same name already exists on object.   at Newtonsoft.Json.Linq.JObject.ValidateToken(JToken o, JToken existing)
   at Newtonsoft.Json.Linq.JContainer.InsertItem(Int32 index, JToken item, Boolean skipParentCheck)
   at Newtonsoft.Json.Linq.JObject.InsertItem(Int32 index, JToken item, Boolean skipParentCheck)
   at Newtonsoft.Json.Linq.JContainer.Add(Object content)
   at Newtonsoft.Json.Linq.JObject.Add(String propertyName, JToken value)
   at Stream.GenericData.<>c__DisplayClass7_0.<AddToJObject>b__0(KeyValuePair`2 x)
   at Stream.Extensions.ForEach[T](IEnumerable`1 items, Action`1 action)
   at Stream.GenericData.AddToJObject(JObject& root)
   at Stream.CollectionObject.ToJObject()
   at Stream.Collections.<>c.<UpsertMany>b__3_0(CollectionObject x)
   at System.Linq.Enumerable.SelectArrayIterator`2.MoveNext()
   at Newtonsoft.Json.Linq.JContainer.TryAddInternal(Int32 index, Object content, Boolean skipParentCheck)
   at Newtonsoft.Json.Linq.JContainer.Add(Object content)
   at Newtonsoft.Json.Linq.JArray..ctor(Object content)
   at Newtonsoft.Json.Linq.JProperty..ctor(String name, Object content)
   at Stream.Collections.UpsertMany(String collectionName, IEnumerable`1 data)
   at Stream.Collections.Upsert(String collectionName, CollectionObject data)
   at [our code from here]

It's true that one of the property names is id but that was never a problem using the Add method so I'm not sure why it should affect Upsert specifically?

Thanks very much.

[ferhat elmas]

cc @peterdeme

[Peter Deme]

@chmchmchm
The error message is valid and honestly even Add() method supposed to throw it. Here's why.

The Data or _data property of CollectionObject is actually getting merged into the root CollectionObject. Example:

Collection object:

{
  "id": "id-1",
  "user_id": "user-1"
}

Custom data:

{
  "age": 28,
  "color": "blue"
}

When we send it to the backend, it ends up like this:

{
  "id": "id-1",
  "user_id": "user-1",
  "age": 28,
  "color": "blue"
}

Do you see why you can't have id the he custom data? I'm not sure why isn't Add() throwing an error for you without seeing an example but it should.

Side note: we actually released the v5 version of this library with some breaking changes, you might want to look into upgrading it. 🙂

[C Miller]

Thanks @peterdeme

We're trying to upgrade to v5 and switch to upsert and finding the data is in a different format between AddAsync and UpsertAsync. Can you explain how to use upsert without changing the data format? In both cases we're looking at the data on an enriched post with return (await client.Batch.GetEnrichedActivitiesAsync(new[] { activityId })).Results.Single(); and inspecting the Data property while debugging.

The main thing blocking us is the extra data property and level of nesting in the UpsertAsync version which breaks our UI code, which only expects one level of nesting. But I'm also not sure why the AddAsync data is formatted like Value={{...}}, key=keyname but the UpsertAsync data is formatted like {[keyname, {... as though these are different types.

With AddAsync

Sending the data

var data= new Dictionary<string, object> { { "key1", "value1" } };
await client.Collections.AddAsync(“collectionName”, data, “someid”);

Read data format

Value = {{
  "id": "someId",
  "collection": "collectionName",
  "foreign_id": "collectionName:someId",
  "data": {
    "id": "someId",
    "key1": "value1"
  },
  "created_at": "2022-03-10T22:59:59.469591Z",
  "updated_at": "2022-03-10T22:59:59.469591Z"
}}, Key = "collectionName"

With UpsertAsync

Sending the data

var data= new Dictionary<string, object> { { "key1", "value1" } };
var collectionObject = new CollectionObject(“someid”);
collectionObject.SetData(data);
await client.Collections.UpsertAsync(“collectionName”, collectionObject);

Read data format

{[collectionName, {{
  "id": "someid",
  "collection": "collectionName",
  "foreign_id": "collectionName:someid",
  "data": {
    "data": {
      "id": "someid",
      "key1": "value1"
    },
    "id": "someid"
  },
  "created_at": "2022-03-10T22:42:25.273369Z",
  "updated_at": "2022-03-10T22:42:25.273369Z"
}}]}

[C Miller]

Hey @peterdeme, any word on this?

[Peter Deme]

@chmchmchm the double nesting isn't intentional. let me take a 👀

[Peter Deme]

@chmchmchm
If you take a look at the tests, does this help?
#72

[C Miller]

Sorry I'm not sure without trying it and seeing what the read format looks like, but happy to try it if that gets merged in. Or can you send the format of the Data property e.g. from debugging?

[Peter Deme]

@chmchmchm just released v6.0.0 with flattened custom data. hopefully it solves your issue.

[C Miller]

@peterdeme thanks very much, this is working great and simplified our code a lot!