POST with JSON string not working

Question

POST with JSON string not working

bclewi opened this issue 2 years ago · comments

Brandon Lewis commented 2 years ago

Thank you for your work on this awesome library!

Should we be able to pass JSON strings in the Body of a POST request? Am I not using the Client correctly?

I get a 200 ResponseCode in the README example.
I get a 200 ResponseCode on POST when passing Body a variable path to a valid DUT.
I get a 400 ResponseCode on POST when passing Body an empty string (Body := '') because my endpoint validates the request.
I get a 200 ResponseCode when using curl with valid JSON data.
I do not get a ResponseCode on POST when passing Body a JSON string with valid JSON data.

Failing example with JSON string

MAIN.TcPOU

Client : HttpClient;
Execute : BOOL := TRUE;
HasError : BOOL;
ErrorId : UDINT;

Client(
	Execute := Execute,
	Address := 'https://www.fake-example.com/ExampleController/Post',
	CallMethod := 'POST',
	Body := '{"stringProperty": "string", "dintProperty": 1, "arrayProperty": ["string"]}',
	ResponseCode := 'GVL.ResponseCode', // This is not updated from the default value of 0.
	Response := '',
	HasError => HasError, // No error returned
	ErrorId => ErrorId);

IF Execute THEN
        Execute := FALSE;
END_IF

GVL.TcGVL

{attribute 'qualified_only'}
VAR_GLOBAL
    ResponseCode : INT;
END_VAR

header.json

{
  "Accept": "*/*",
  "Content-Type": "application/json"
}

The .NET/C# endpoint responds to POST with 200 if it receives a valid request body

ExampleController.cs

namespace ExampleProject.API.Controllers;

[ApiController, Route("[controller]/[action]")]
public class ExampleController : ControllerBase
{
  // "/ExampleController/Post"
  [HttpPost]
  [ProducesResponseType(StatusCodes.Status200OK)]
  [ProducesResponseType(StatusCodes.Status400BadRequest)]
  public async Task<IActionResult> Post([FromBody] Request request) => return Ok();
}

Request.cs

namespace ExampleProject.API.Models;

public class Request
{
    // JSON must include this property with a non-empty string value.
    [Required(AllowEmptyStrings = false)]
    public string StringProperty { get; set; } = string.Empty;

    // JSON must include this property with any int value.
    [Required]
    public int DintProperty { get; set; }

    // JSON must include this property with 0 or more strings in an array.
    [Required]
    public IEnumerable<string> ArrayProperty { get; set; } = Enumerable.Empty<string>();
}

Working example with variable path

MAIN.TcPOU

Client : HttpClient;
Execute : BOOL := TRUE;
HasError : BOOL;
ErrorId : UDINT;

Client(
	Execute := Execute,
	Address := 'https://www.fake-example.com/ExampleController/Post',
	CallMethod := 'POST',
	Body := 'GVL.Body',
	ResponseCode := 'GVL.ResponseCode', // This is 200 on success.
	Response := '',
	HasError => HasError,
	ErrorId => ErrorId);

IF Execute THEN
        Execute := FALSE;
END_IF

GVL.TcGVL

{attribute 'qualified_only'}
VAR_GLOBAL
    Body : Body;
    ResponseCode : INT;
END_VAR

Body.TcDUT

TYPE Body :
STRUCT
    {attribute 'json' := 'stringProperty'}
    StringProperty : STRING := 'string';

    {attribute 'json' := 'dintProperty'}
    DintProperty : INT := 1;

    {attribute 'json' := 'arrayProperty'}
    ArrayProperty : ARRAY[0..9] OF STRING := ['string'];
END_STRUCT
END_TYPE

header.json

{
  "Accept": "*/*",
  "Content-Type": "application/json"
}

Why do I care about using a JSON string if the variable path to a DUT is working ok?

In my working example above, the DUT contains a property of type ARRAY[0..9] OF STRING, a static array, but the REST API endpoint takes in a dynamic array of strings, expected to vary with each call. For this use case, it is preferred to send:

{"stringProperty": "string", "dintProperty": 1, "arrayProperty": ["string"]}

instead of what the DUT deserializes to:

{"stringProperty": "string", "dintProperty": 1, "arrayProperty": ["string","","","","","","","","",""]}

Passing a JSON string to the HttpClient Body would be a workaround for all requests with dynamic arrays that do not serialize appropriately.

Federico Barresi · Answer 1 · Fri Jan 20 2023 04:55:46 GMT+0800 (China Standard Time)

Hi!

Thank you for using this library!

I just summarize your questions here with my inline reply.

Should we be able to pass JSON strings in the Body of a POST request?

No, you are supposed to use a type and let the library do the serialization in json for you.

Am I not using the Client correctly?

You use it as I designed it as long as you use a variable path into your body field.

The key is into this line.

I understand your use case. I think in the meanwhile twincat supports array with variable length, but they will ever have a fixed size during the serialization and since there isn't nullable in twincat (yet?) the problem will probably ever persist.

I think that programmers life could be hard enough also without the need to work with strings operation under twincat, but I can imagine that sometimes might be easier to fill in or put together a json instead of handling with structures.

I think in this case the solution might be easy: whenever the software recognize a json in the body field (e.g. starting with { and ending with }) it would not try to read and serialize a variable, but will forward the content as it is.

How does it sound to you? Would that solve your problem?

Best regards,
FB

Brandon Lewis · Answer 2 · Sat Jan 21 2023 01:23:03 GMT+0800 (China Standard Time)

Thank you so much for thoughtfully considering my use case!

No, you are supposed to use a type and let the library do the serialization in json for you.

Ah, that clears things up. I may have had the wrong expectations based on this line of the README and the comment on this line of the HttpClient. I misread the README as allowing Json string data, and misread the comment as allowing (Variable path) OR (Json data string).

I understand your use case. I think in the meanwhile twincat supports array with variable length, but they will ever have a fixed size during the serialization and since there isn't nullable in twincat (yet?) the problem will probably ever persist.

I think you correctly pointed out the root issue here. My team is adding more complexity than is preferred to solve the problems that come with serializing/parsing JSON data with dynamic length arrays, working with that data in TwinCAT structures, and providing external REST APIs data in an expected format. It would be one thing to design a system around these TwinCAT limitations (letting the external API handle more of the complexity), but it is more difficult when the external REST API is not open to changes.

I think in this case the solution might be easy: whenever the software recognize a json in the body field (e.g. starting with { and ending with }) it would not try to read and serialize a variable, but will forward the content as it is.

How does it sound to you? Would that solve your problem?

In my opinion, this solution would provide a nice option! However, do you think the JSON string would still be limited in length due to the limitations with the HttpClient usage of NT_StartProcess.COMNDLINE? If so, I'm not sure if this would solve the majority of our problems, as the variable length arrays could get quite long, and the JSON string may get truncated unexpectedly.

Alternative Feature Suggestion 1

What might solve the problem is if a user could define a sufficiently sized STRING variable in TwinCAT, and if the user could pass a variable path to the STRING to the HttpClient Body along with a flag to tell HttpClient not to serialize the Body. If the underlying Extension/API can read/write very long string variables to/from the PLC, this would be a high value feature that would solve most of our problems!

Do you think this feature is possible? If so, would it be difficult to implement?

Example

MAIN.TcPOU

Client(
	Execute := Execute,
	Address := 'https://www.fake-example.com/ExampleController/Post',
	CallMethod := 'POST',
	Body := 'GVL.JsonString', // The software does not try to serialize this variable, it only read/writes from/to it.
	ResponseCode := 'GVL.ResponseCode',
	Response := 'GVL.Response', // The software still parses the response JSON to a structure as usual.
        SerializeBody := FALSE, // Optional BOOL flag that defaults to TRUE, and is added as a command line argument.
	HasError => HasError,
	ErrorId => ErrorId);

GVL.TcGVL

{attribute 'qualified_only'}
VAR_GLOBAL CONSTANT
    MAX_JSON_STRING_LENGTH : UDINT := 1000000000;
END_VAR
VAR_GLOBAL
    JsonString : STRING(MAX_JSON_STRING_LENGTH);
    ResponseCode : INT;
    Response : Response;
END_VAR

Alternative Feature Suggestion 2

My team has been experimenting with dynamic arrays on the heap as well. If support was added for serializing JSON array data from pointers to dynamic arrays, this could also be a solution. However, I suspect that this would be quite difficult.

Body.TcDUT

TYPE Body :
STRUCT
    {attribute 'json' := 'stringProperty'}
    StringProperty : STRING := 'string';

    {attribute 'json' := 'dintProperty'}
    DintProperty : INT := 1;

    {attribute 'json' := 'dynamicIntArrayProperty'}
    pDynamicIntArrayProperty : POINTER TO INT;

    pDynamicIntArrayProperty_FirstIndex : UINT := 0
    pDynamicIntArrayProperty_LastIndex : UINT := 2;
END_STRUCT
END_TYPE

MAIN.TcPOU

Body.pDynamicIntArray := __NEW(INT, Body.CurrentArraySize);

Client(
	Execute := Execute,
	Address := 'https://www.fake-example.com/ExampleController/Post',
	CallMethod := 'POST',
	(* The software gets the valid index range from structure properties that conform to a naming convention. *)
	Body := 'GVL.Body', 
	ResponseCode := 'GVL.ResponseCode',
	Response := 'GVL.Response',
	HasError => HasError,
	ErrorId => ErrorId);

// Whenever we are done with the Body structure...
// __DELETE(pDynamicIntArray);

GVL.TcGVL

{attribute 'qualified_only'}
VAR_GLOBAL
    Body : Body;
    ResponseCode : INT;
    Response : Response;
END_VAR