A slightly contrived issue looping over (and appending to) arrays

[Deleted user]

Hello,

I have been using fastjson for some time now and it really is very excellent.

Now I have hit a very specific issue that I am struggling to resolve and I would be most grateful for any helpful suggestions.

I have a .json file structured in this way (inserting this as a quote as code formatting was messed up):

{
"id": 1,
"display": 1,
"sync_state": 0,
"name": "Test",
"type": "Type",
"version": "v1",
"social_fields":[
{"type": "social",
"label": "Phone",
"field_value":"123456789"
},
{"type": "social",
"label": "Facebook",
"field_value":"www.facebook.com"
},
{"field_type": "social",
"label": "Twitter",
"ffield_value":"www.twitter.com"
},
{"field_type": "social",
"label": "Pinterest",
"field_value":"www.pinterest.com"
}
],
"col1_heading": "Column Heading",
"column_1_fields":[
{"field_type": "column_1",
"placeholder": "First Column Info",
"field_value":"First Column Info"
},
{"field_name": "column_1",
"placeholder": "First Column Info",
"field_value":"First Column Info"
}
],
"col2_heading": "Column Heading",
"column_2_fields":[
{"field_type": "column_2",
"description_1": "Second Column Info:",
"find_us_field_value_1":"Second Column Info",
"description_2": "Second Column Info:",
"find_us_field_value_2":"Second Column Info"
},
{"field_type": "column_2",
"description_1": "Second Column Info:",
"find_us_field_value_1":"Second Column Info",
"description_2": "Second Column Info:",
"find_us_field_value_2":"Second Column Info"
}
],
"col3_heading": "Column Heading",
"column_3_fields":[
{"field_type": "column_3",
"placeholder": "Third Column Info",
"field_value":"Third Column Info"
},
{"field_type": "column_3",
"placeholder": "Third Column Info",
"field_value":"Third Column Info"
}
]
}

This .json file is manipulated via golang based on a form submission with field names identical to the ones you see here.

So, here is the function which ingests this file and updates it (error handling and print statements omitted for brevity):

func processJSON(form map[string][]string, jsonContent *fastjson.Value) (*fastjson.Value, error) {
// first, delete the sections of json I want to update (it is easier to just delete the whole section, then start again, I think)
jsonContent.Del("social_fields")
jsonContent.Del("column_1_fields")
jsonContent.Del("column_2_fields")
jsonContent.Del("column_3_fields")

//second, make four arenas (as it seems arenas share memory somehow, so reusing the same arena won't work in the scenario below)
var arena1 fastjson.Arena
var arena2 fastjson.Arena
var arena3 fastjson.Arena
var arena4 fastjson.Arena

//third, make four separate fastjson values, so that we can call arena.Reset() after each for/range loop, while preserving what was stored in the arrays
var fjvalue1 *fastjson.Value
var fjvalue2 *fastjson.Value
var fjvalue3 *fastjson.Value
var fjvalue4 *fastjson.Value

//two counters to be iterated, due to the difference in field structure for fields in column_2
j := 0
i := 0

//four arrays, one for each field type, all using unique arenas due to previously mentioned memory allocation
social_field_array := arena1.NewArray()
column_1_field_array := arena2.NewArray()
column_2_field_array := arena3.NewArray()
column_3_field_array := arena4.NewArray()

//iterate over all fields submitted from the form (the form contains a number of inputs all named 'field_type', to make it possible to iterate over them all in one go

for _, field_type := range form["field_type"] {
if field_type == "social" {
// create a unique fastjson.Parser to avoid memory allocation issues
var p fastjson.Parser
v, err := p.Parse({ "field_type":" + field_type + ", "label":" + jsonEscape(form["field_label"][i]) + ", "find_us_field_value":" + jsonEscape(form["field_value"][i]) + " })
if err != nil {
//handle error
}
// add the parsed json to the array we created earlier, using the index i
social_field_array.SetArrayItem(i, v)
i += 1
}
}
//once the loop is done, copy this value into a new variable and reset the arena to avoid memory allocation issues
fjvalue1 = social_field_array
arena1.Reset()
for _, field_type := range form["field_type"] {
if field_type == "column_1" {
// here is another new parser again
var q fastjson.Parser
v, err := q.Parse({"field_type":" + field_type + ", "placeholder":"First Column Info", "field_value":" + jsonEscape(form["field_value"][i]) + "})
if err != nil {
//handle error
}
i += 1
}
}
// same as above
fjvalue2 = column_1_field_array
arena2.Reset()
for _, field_type := range form["field_type"] {
if field_type == "column_2" {
var r fastjson.Parser
v, err := r.Parse({ "field_type":" + field_type + ", "placeholder_1":"Second Column Info:", "field_value_1":" + jsonEscape(form["field_value_1"][j]) + ", "placeholder_2":"Second Column Info:", "field_value_2":" + jsonEscape(form["field_value_2"][j]) + " })
if err != nil {
//handle error
}
j += 1
}
}
fjvalue3 = column_2_field_array
arena3.Reset()

//technically there is another loop in here, but I am omitting this for brevity, as it does the same thing again

}
fjvalue4 = column_3_field_array
arena4.Reset()
jsonContent.Set("social_fields", fjvalue1)
jsonContent.Set("column_1_fields", fjvalue2)
jsonContent.Set("column_2_fields", fjvalue3)
jsonContent.Set("column_3_fields", fjvalue4)
return jsonContent, nil
}

Now, this approach generates a .json file that looks a bit like this:

{
"id": 1,
"display": 1,
"sync_state": 0,
"name": "Test",
"type": "Type",
"version": "v1",
"social_fields":[
{"type": "social",
"label": "Phone",
"field_value":"123456789"
},
{"type": "social",
"label": "Facebook",
"field_value":"www.facebook.com"
},
{"field_type": "social",
"label": "Twitter",
"ffield_value":"www.twitter.com"
},
{"field_type": "social",
"label": "Pinterest",
"field_value":"www.pinterest.com"
}
],
"col1_heading": "Column Heading",
"column_1_fields":[null, null, null, null,
{"field_type": "column_1",
"placeholder": "First Column Info",
"field_value":"First Column Info"
},
{"field_name": "column_1",
"placeholder": "First Column Info",
"field_value":"First Column Info"
}
],
"col2_heading": "Column Heading",
"column_2_fields":[
{"field_type": "column_2",
"description_1": "Second Column Info:",
"find_us_field_value_1":"Second Column Info",
"description_2": "Second Column Info:",
"find_us_field_value_2":"Second Column Info"
},
{"field_type": "column_2",
"description_1": "Second Column Info:",
"find_us_field_value_1":"Second Column Info",
"description_2": "Second Column Info:",
"find_us_field_value_2":"Second Column Info"
}
],
"col3_heading": "Column Heading",
"column_3_fields":[ null, null, null, null, null, null, null,
{"field_type": "column_3",
"placeholder": "Third Column Info",
"field_value":"Third Column Info"
},
{"field_type": "column_3",
"placeholder": "Third Column Info",
"field_value":"Third Column Info"
}
]
}

It would appear that no matter what I do, due to the looping I am doing, there are always these "null" fields inserted (see just above this "column_3_fields"), which, for the first column data coincidentally always seems to be just as many as there were social fields submitted.

It is worth noting that you are not looking at the first iteration of this issue. I originally had all of this packed into a single for loop, with a single arena, and arena.reset calls, trying to reuse the same parser and arena for efficiency. I experimented with different ways of trying to 'convince' fastjson to not create these additional "null" values, but by now I am sadly out of ideas.

I have used the methodology of deleting a whole section and then just adding it back in by parsing pre-written json strings in golang and appending them to a new array created from an arena via value.SetArrayItem(), which is then re-appended to the .json file via value.Set("arrayname", value) before, but it was always fine, as it only ever concerned just one array and not multiple arrays within the same .json file.

It is obvious that the above cannot be the correct approach, but I am illustrating it here to show you where I am at with my mental model of things.

You got any tips for me? :)