foxglove / ws-protocol

Great job with including websockets as a datasource!

Many panels support / require standard messages formats. It would be great if these could be sent over the websocket json format as well. As an example, consider the 3D Panel that requires sensor_msgs/PointCloud2D.

What is they way to achieve this in Python? I've tried adapting the json server example with the following schema and message, but could not display any points:

schema = {
    'type': 'object',
    'properties': {
        'height': {'type': 'number'},
        'width': {'type': 'number'},
        'is_bigendian': {'type': 'boolean'},
        'point_step': {'type': 'number'},
        'row_step': {'type': 'number'},
        'fields': {
            'type': 'array', 
            'items': {
                'type': 'object',
                'properties': {
                    'name': {'type': 'string'},
                    'offset': {'type': 'number'},
                    'dataype': {'type': 'number'},
                    'count': {'type': 'number'}
                }
            }
        },
        'data': {
            'type': 'array',
            'items': {'type': 'number'}
        }
     }
}

I suppose there are standardized schemas for this, right?

message = {
    'height': 1,
    'width': N,
    'is_bigendian': False,
    'point_step': point_step,
    'row_step': N * point_step,
    'fields': [
        {
            'name': 'x',
            'offset': 0,
            'datatype': 7,
            'count': 1
        },
        {
            'name': 'y',
            'offset': 4,
            'datatype': 7,
            'count': 1
        },
    ],
    'data': list(X.astype(np.float32).tobytes(order='C'))
}

Comparing with the example rosbag, the datatype for data is probably incorrect and one has to send binary data somehow. Also, The 3D Panel asks for a frame to follow - how to specify that over the websocket protocol?

Thanks a lot for your help!

Thanks for your questions!

the datatype for data is probably incorrect and one has to send binary data somehow.

That's exactly right. You can use "data": { type: "string", contentEncoding: "base64" } and provide your data as a base64-encoded string. You can find an example in the image server here:

ws-protocol/typescript/ws-protocol-examples/src/examples/image-server.ts

Line 106 in 00fc632

data: { type: "string", contentEncoding: "base64" },

Also, The 3D Panel asks for a frame to follow - how to specify that over the websocket protocol?

The list of frames in the 3D panel comes from two sources: the /tf topic (or any topic using a tf2_msgs/TFMessage message type) and the .header.frame_id field of any Marker or MarkerArray messages.

Added some documentation on contentEncoding to https://foxglove.dev/docs/connection/foxglove-websocket

Excellent, thanks so much!

I managed to get it working with
'data': base64.b64encode(X.astype(np.float32).flatten(order='C')).decode('utf8') for the message and schema as you suggested. I also could post the /tf topic to get a coordinate system.

Two minor confusions remain:

when adding a channel add_channel we specify schemaName as parameter. Shouldn't that be better called messageType?
when calling send_message we pass a timestamp, but that is not automatically injected in the header of the message, is it? Shouldn't it be?

when adding a channel add_channel we specify schemaName as parameter. Shouldn't that be better called messageType?

It's an interesting idea. Here is some background on why we chose schemaName:

We decided the word "schema" is best to represent the structure of message data throughout the Foxglove Studio codebase. ROS uses the word "datatype" to mean our schema or schemaName, but we felt that "datatype" is pretty generic whereas "schema" evokes some information about the message structure.
The schemaName is not always required or important. Some panels like Raw Messages don't use the schemaName. In some contexts the name is meant to be a human-readable identifier for the schema, hence the word "name". (Although, when using encoding: "protobuf", the schemaName is required to match a protobuf message type defined in the schema.)
The same name is used in our new .mcap file format and we wanted to be consistent.

when calling send_message we pass a timestamp, but that is not automatically injected in the header of the message, is it? Shouldn't it be?

Messages can be any user data and not all of them contain a field called header with a timestamp. Often, even if the message can contain a timestamp, robotics engineers will use the timestamp as an identifier to link a message from a post-processing step (such as object detection / ML inference) with the timestamp of the original sensor data which it is derived from — in which case the timestamp may appear to be "old" relative to when the message itself was sent.

Regardless of the message data, our websocket protocol includes a timestamp with every message, which is interpreted by the app as the "receive timestamp" (the time when the message was received by the subscriber, in this case the websocket server). This is the timestamp you are passing to send_message. In the future we might expand this to include something like a "send timestamp" as well, since we have found this kind of detailed timing information to be useful when diagnosing and debugging system reliability issues.

Added some auto-generated JSON Schemas to https://github.com/foxglove/ros-message-schemas

Thanks for the detailed explanation. That all makes perfect sense. The JSON Schemas you generated are very helpful.

How to posting standard message types over Websocket, e.g. a point cloud?