[dart:ffi] Inline arrays in Structs

[Daco Harkes]

Helpers for converting between ffi.Pointers and arrays.

Maybe use external typed data for efficient conversions.

[djpnewton]

also support for fixed size arrays.

its quite common to have a struct with fixed sized arrays to pass around in C

[Daco Harkes]

We have asTypedList for exposing the native memory as a Dart array.

For structs with inline arrays:

struct MyStruct {
  uint8_t arr[10];
  int64_t arr2[3];
};

We need a dart syntax of representing an inline array.

We could opt for exposing this as a pointer, which provides [] and []= access.

class MyStruct extends Struct {
  @InlineArray(10)
  Pointer<Uint8> arr;

  @InlineArray(3)
  Pointer<Int64> arr2;
}

Using a Pointer as the representing type does not allow us to distinguish pointers and inline arrays by the Dart type. However, this is not really a problem since C does not support passing fixed-size arrays by value in calls.

cc @mkustermann

edit: We do not want to expose it as a pointer, rather as a CArray which represents a fixed-size inlined array.

class MS extends Struct {}

MS foo(MS ms); // C signature for passing struct by value.
@size(4)CArray<MS> foo(@size(4)CArray<MS> mss); // C signature for passing inline array of structs.
Pointer<MS> foo(Pointer<MS> ms); // C signature for passing pointer.

class MS2 extends Struct {
  MS ms; // Nested struct.
}
class MS3 extends Struct {
  @size(4)
  CArray<MS> mss; // Nested array of structs.
}
class MS4 extends Struct {
  Pointer<MS> mss; // Pointer to struct.
}

CArray would be a view on top of the underlying pointer (kind of like struct).

If we expose the nested structs and the inline array as getter/setter pair, the setters would do a memcopy of the whole underlying structure.

(As sidenote, we should also introduce memcopy on structs:

Pointer<MS> p1, p2;
p1.value = p2.ref; // Copies over all memory.

)

[kakyoism]

@dcharkes

I got this error when trying your solution to getting static arrays on Dart side

Code

class StaticArray extends Struct {
  @InlineArray(10)
  Pointer<Int32> array;
  @Int32()
  int len;
}

Result

$ dart structs.dart 
structs.dart:48:4: Error: Method not found: 'InlineArray'.
  @InlineArray(10)
   ^^^^^^^^^^^
structs.dart:129:15: Error: A function expression can't have a name.
  staticArray.forEach(idx, elem) => print("array[$idx]: $elem");
              ^^^^^^^
structs.dart:129:15: Error: Expected an identifier, but got 'forEach'.
  staticArray.forEach(idx, elem) => print("array[$idx]: $elem");
              ^^^^^^^

Does it mean that this is just a proposal?
What would be the canonical way of getting array data from FFI C pointers?
I'm referring to my latest question.

Thanks in advance!

[Daco Harkes]

Yes, for now it's just a proposal.

See the workaround made by @timsneath here for now.

I'll get around to implementing this in the future, I'm working on other dart:ffi features at the moment.

[kakyoism]

@dcharkes Thanks! FFI has been great so far.
On a related matter, the struct example never frees the malloc'ated memory. Does this mean that the memory will be GC'ed on Dart side?

This would affect how I use the array from a returned pointer.

[Tim Sneath]

You need to use free() to release the memory. 
it'll happen automatically when the process exits, of course. 

[kakyoism]

@timsneath Thanks. I looked at the static array workaround but it seems difficult to scale when my array is say 500-1000 elements big.

Is it that the realistic workaround is to always use malloc'ated arrays?

[Arto Bendiken]

See the workaround made by @timsneath here for now.

I'll get around to implementing this in the future, I'm working on other dart:ffi features at the moment.

@dcharkes As I alluded to in the conversation in #36140, the lack of support for inline arrays in structs is a blocker for me in creating OpenXR bindings for Dart and Flutter.

The proposed workaround is infeasible for the structs I need to map, which include the following:

struct XrActionCreateInfo {
  XrStructureType type;
  const void* next;
  char actionName[64];
  XrActionType actionType;
  uint32_t countSubactionPaths;
  const XrPath* subactionPaths;
  char localizedActionName[128];
};

struct XrApplicationInfo {
  char applicationName[128];
  uint32_t applicationVersion;
  char engineName[128];
  uint32_t engineVersion;
  XrVersion apiVersion;
};

struct XrEventDataBuffer {
  XrStructureType type;
  const void* next;
  uint8_t varying[4000];
};

[Darel Bitsy]

As a workaround, would not creating a pointer to uint8 or utf8 with size matching the required, good enough?

For example:
class XrEventDataBuffer extends Struct {
....
Pointer varying;

factory XrEventDataBuffer.create(..., Uint8List varying) {
     final Pointer<Uint8> data = allocate<Uint8>(count: varying.length);
     // since pointer is indexable you can just
     // do a for loop and fill the pointer
     fillpointerWithIntlist(data, varying);
     
    // then initialize the struct
 }

}

Sorry for missing the auto formatting, I’m on the phone.

[Daco Harkes]

What about multi dimensional inline arrays, what should the API be for multidimensional arrays?
(ty @mannprerak2!)

[Tim Sneath]

@artob extending a little from the idea @bitsydarel approached, you might check out https://github.com/timsneath/win32/blob/bluetooth/lib/src/structs.dart#L1941 for an example of wrapping a similar Win32 struct that seems like a viable workaround for you. (In this scenario, I have a 250 character Utf-16 string that I need to wrap).

[Arto Bendiken]

@artob extending a little from the idea @bitsydarel approached, you might check out [...] an example of wrapping a similar Win32 struct that seems like a viable workaround for you.

@timsneath It's a good workaround for the BLUETOOTH_DEVICE_INFO struct you have there, but it won't quite suffice for my use case: the OpenXR API has large character arrays inline and intermixed with other struct fields, not only as the last trailing field in the struct.

So, the suggested approach seems like it could be a workaround for XrEventDataBuffer with its 4000-byte trailing array, but defining XrActionCreateInfo and XrApplicationInfo would still be a pain in the butt given the 64/128-byte inline arrays. (And there are more structs like these.) Any suggests for the likes of those?

As a workaround, would not creating a pointer to uint8 or utf8 with size matching the required, good enough?

@bitsydarel After @timsneath's code example I understand better what you were proposing for XrEventDataBuffer. Thanks, seems a viable approach for that struct 👍

[Tim Sneath]

It's not the prettiest code I've ever written, but wouldn't something like this work?
https://gist.github.com/timsneath/983764458690f89a5c8231b4ff2e2e64

Here we use the same technique as for the BLUETOOTH_DEVICE_INFO struct of using properties to index into the created struct, and treating as the struct just as a binary blob for the purposes of allocating space in the heap.

You could extend this still further by providing setters where appropriate, and adding clean String properties to convert from a C-style string into a Dart string, with guards to ensure a 128-byte maxlength.

Curious to know if this works for you?

[Daco Harkes]

For inline arrays of primitive values https://pub.dev/packages/ffigen has an option to generate 'n' fields and provide indexed access (with array-workaround: true).

[Daco Harkes]

Update^2 design:

class MyStruct extends Struct {
  @Array(8)
  external Array<Uint8> inlineArray;

  @Array(2, 2, 2)
  external Array<Uint8> threeDimensionalInlineArray;

  @Array.multi([2, 2, 2, 2, 2, 2, 2, 2])
  external Array<Uint8> eightDimensionalInlineArray;
}

source #45023 (comment)

[Daco Harkes]

This will be available on the first dev-release after Version 2.13.0-73.0.dev.

[Sunbreak]

Is it possible to have the address of the Array<Int8>, so that we could convert it asTypedList?

[Daco Harkes]

Is it possible to have the address of the Array<Int8>, so that we could convert it asTypedList?

We're tracking this in #45508.