How to use python protobuf example to show image data?

[Felix F Xu]

Hi,

I can run the python protobuf example and the Foxglove Studio can display it:

But I want it to be useful:

currently only the raw message panel can be used (as show above). What should I do to display an image data?

[Felix F Xu]

I have made some progress but now is stuck.

in protobuf-server.py, I modified the code a little bit:

    async with FoxgloveServer("0.0.0.0", 8765, "example server") as server:
        server.set_listener(Listener())
        chan_id = await server.add_channel(
            {
                "topic": "example_msg",
                "encoding": "protobuf",
                "schemaName": "foxglove.RawImage",  # Matches `message ExampleMsg` in ExampleMsg.proto
                "schema": schema_base64,  # Represents the parsed contents of ExampleMsg.proto
            }
        )

In the code above, I change the schemaName: "schemaName": "foxglove.RawImage" because I want to display a image in the pane of Image, by the document of foxglove.RawImage

        mat = cv2.imread("xgyparking.png")
        i = 0
        while True:
            i += 1
            await asyncio.sleep(0.2)
            await server.send_message(
                chan_id,
                time.time_ns(),
                ExampleMsg_pb2.RawImage(
                    timestamp=ExampleMsg_pb2.Timestamp(
                        seconds=int(time.time_ns() / 1000000000),
                        nanos= 2
                    ),
                    width=1818,
                    height=678,
                    encoding='8UC3',
                    step=1818,
                    data=bytes(mat.data)
                    ).SerializeToString(),  # type: ignore
            )

then I read a local image using opencv, prepare the conent of ExampleMsg_pb2.RawImage (I'm not sure if I have made any mistake here.)

at least the server code can run, and in foxglove studio, raw message pane is ok with the data:

but Image pane reports an error:

I don't want to (or not able to) debug foxglove studio's typescript code, so if any quick fix or a complete python example that can display some image is highly appreciated!

[Felix F Xu]

ExampleMsg.proto is also modified:

syntax = "proto3";

package foxglove;

message Timestamp {
  // Represents seconds of UTC time since Unix epoch
  // 1970-01-01T00:00:00Z. Must be from 0001-01-01T00:00:00Z to
  // 9999-12-31T23:59:59Z inclusive.
  int64 seconds = 1;

  // Non-negative fractions of a second at nanosecond resolution. Negative
  // second values with fractions must still have non-negative nanos values
  // that count forward in time. Must be from 0 to 999,999,999
  // inclusive.
  int32 nanos = 2;
}


// A raw image
message RawImage {
  // Timestamp of image
  Timestamp timestamp = 1;

  // Image width
  fixed32 width = 2;

  // Image height
  fixed32 height = 3;

  // Encoding of the raw image data
  string encoding = 4;

  // Byte length of a single row
  fixed32 step = 5;

  // Raw image data
  bytes data = 6;
}

I didn't quite follow the document, but copy all proto message difinitions into the existing ExampleMsg.proto file. I'm not sure If I can to it that way.

[Jacob Bandes-Storch]

Can you try using this proto file? https://github.com/foxglove/schemas/blob/main/schemas/proto/foxglove/RawImage.proto

[Felix F Xu]

Actually I have tried

1. copy the file of RawImage.proto to the ws-protocol example folder, which is same as existing ExampleMsg.proto

2. generate py2.py and .bin files

3. change the python code a little bit:

    # Load the FileDescriptorSet, which was generated via `protoc --descriptor_set_out`.
    with open(
        os.path.join(os.path.dirname(RawImage_pb2.__file__), "RawImage.bin"), "rb"
    ) as schema_bin:
        schema_base64 = standard_b64encode(schema_bin.read()).decode("ascii")

    async with FoxgloveServer("0.0.0.0", 8765, "example server") as server:
        server.set_listener(Listener())
        chan_id = await server.add_channel(
            {
                "topic": "rawimage",
                "encoding": "protobuf",
                "schemaName": "foxglove.RawImage",  # Matches `message RawImage` in RawImage.proto
                "schema": schema_base64,  # Represents the parsed contents of RawImage.proto
            }
        )

        i = 0
        while True:
            i += 1
            await asyncio.sleep(0.2)
            await server.send_message(
                chan_id,
                time.time_ns(),
                RawImage_pb2.RawImage().SerializeToString(),  # type: ignore
            )

Then in Foxglove Studio, there is an error:

Failed to parse channel schema on rawimage
Error: no such Type or Enum '.google.protobuf.Timestamp' in Type .foxglove.RawImage
at Type.lookupTypeOrEnum (file:///opt/Foxglove%20Studio/resources/app.asar/renderer/main.js:89507:17)
at Field.resolve (file:///opt/Foxglove%20Studio/resources/app.asar/renderer/main.js:89050:96)
at Type.resolveAll (file:///opt/Foxglove%20Studio/resources/app.asar/renderer/main.js:91254:23)
at Namespace.resolveAll (file:///opt/Foxglove%20Studio/resources/app.asar/renderer/main.js:89458:25)
at Root.resolveAll (file:///opt/Foxglove%20Studio/resources/app.asar/renderer/main.js:89458:25)
at Root.resolveAll (file:///opt/Foxglove%20Studio/resources/app.asar/renderer/main.js:90670:47)
at parseChannel (file:///opt/Foxglove%20Studio/resources/app.asar/renderer/main.js:150383:16)
at EventEmitter. (file:///opt/Foxglove%20Studio/resources/app.asar/renderer/main.js:155211:35)
at EventEmitter.emit (file:///opt/Foxglove%20Studio/resources/app.asar/renderer/main.js:68122:35)
at ws.onmessage (file:///opt/Foxglove%20Studio/resources/app.asar/renderer/main.js:154893:30)

Now even the RawMessage pane can't work.

[Felix F Xu]

I figured it out, I need to use --include_imports for protoc