kube-rs / k8s-pb

We need to handle the envelope wrapper application/vnd.kubernetes.protobuf (4 bytes prefix + Unknown message). This makes the message self describing.

It should be pretty similar to how we're handling JSON.

https://github.com/kubernetes/api/blob/master/README.md#recommended-use

If you want to store or interact with proto-formatted Kubernetes API objects, we recommend using the "official" serialization stack in k8s.io/apimachinery. Directly serializing these types to proto will not result in data that matches the wire format or is compatible with other kubernetes ecosystem tools. The reason is that the wire format includes a magic prefix and an envelope proto. Please see: kubernetes.io/docs/reference/using-api/api-concepts/#protobuf-encoding

For the same reason, we do not recommend embedding these proto objects within your own proto definitions. It is better to store Kubernetes objects as byte arrays, in the wire format, which is self-describing. This permits you to use either JSON or binary (proto) wire formats without code changes. It will be difficult for you to operate on both Custom Resources and built-in types otherwise.

https://kubernetes.io/docs/reference/using-api/api-concepts/#protobuf-encoding

Kubernetes uses an envelope wrapper to encode Protobuf responses. That wrapper starts with a 4 byte magic number to help identify content in disk or in etcd as Protobuf (as opposed to JSON), and then is followed by a Protobuf encoded wrapper message, which describes the encoding and type of the underlying object and then contains the object.

A four byte magic number prefix:

Bytes 0-3: "k8s\x00" [0x6b, 0x38, 0x73, 0x00]

An encoded Protobuf message with the following IDL:

message Unknown {
  // typeMeta should have the string values for "kind" and "apiVersion" as set on the JSON object
  optional TypeMeta typeMeta = 1;

  // raw will hold the complete serialized object in protobuf. See the protobuf definitions in the client libraries for a given kind.
  optional bytes raw = 2;

  // contentEncoding is encoding used for the raw data. Unspecified means no encoding.
  optional string contentEncoding = 3;

  // contentType is the serialization method used to serialize 'raw'. Unspecified means application/vnd.kubernetes.protobuf and is usually
  // omitted.
  optional string contentType = 4;
}

message TypeMeta {
  // apiVersion is the group/version for this type
  optional string apiVersion = 1;
  // kind is the name of the object schema. A protobuf definition should exist for this object.
  optional string kind = 2;
}

Generated Rust:

k8s-pb/k8s-pb-codegen/out/apimachinery.pkg.runtime.rs

Lines 87 to 104 in 7b4bf26

    
           #[derive(Clone, PartialEq, ::prost::Message)] 
        
           pub struct Unknown { 
        
               #[prost(message, optional, tag="1")] 
        
               pub type_meta: ::core::option::Option<TypeMeta>, 
        
               /// Raw will hold the complete serialized object which couldn't be matched 
        
               /// with a registered type. Most likely, nothing should be done with this 
        
               /// except for passing it through the system. 
        
               #[prost(bytes="vec", optional, tag="2")] 
        
               pub raw: ::core::option::Option<::prost::alloc::vec::Vec<u8>>, 
        
               /// ContentEncoding is encoding used to encode 'Raw' data. 
        
               /// Unspecified means no encoding. 
        
               #[prost(string, optional, tag="3")] 
        
               pub content_encoding: ::core::option::Option<::prost::alloc::string::String>, 
        
               /// ContentType  is serialization method used to serialize 'Raw'. 
        
               /// Unspecified means ContentTypeJSON. 
        
               #[prost(string, optional, tag="4")] 
        
               pub content_type: ::core::option::Option<::prost::alloc::string::String>, 
        
           }

k8s-pb/k8s-pb-codegen/out/apimachinery.pkg.runtime.rs

Lines 68 to 76 in 7b4bf26

    
           #[derive(Clone, PartialEq, ::prost::Message)] 
        
           pub struct TypeMeta { 
        
               /// +optional 
        
               #[prost(string, optional, tag="1")] 
        
               pub api_version: ::core::option::Option<::prost::alloc::string::String>, 
        
               /// +optional 
        
               #[prost(string, optional, tag="2")] 
        
               pub kind: ::core::option::Option<::prost::alloc::string::String>, 
        
           }

	#[derive(Clone, PartialEq, ::prost::Message)]
	pub struct Unknown {
	#[prost(message, optional, tag="1")]
	pub type_meta: ::core::option::Option<TypeMeta>,
	/// Raw will hold the complete serialized object which couldn't be matched
	/// with a registered type. Most likely, nothing should be done with this
	/// except for passing it through the system.
	#[prost(bytes="vec", optional, tag="2")]
	pub raw: ::core::option::Option<::prost::alloc::vec::Vec<u8>>,
	/// ContentEncoding is encoding used to encode 'Raw' data.
	/// Unspecified means no encoding.
	#[prost(string, optional, tag="3")]
	pub content_encoding: ::core::option::Option<::prost::alloc::string::String>,
	/// ContentType is serialization method used to serialize 'Raw'.
	/// Unspecified means ContentTypeJSON.
	#[prost(string, optional, tag="4")]
	pub content_type: ::core::option::Option<::prost::alloc::string::String>,
	}

	#[derive(Clone, PartialEq, ::prost::Message)]
	pub struct TypeMeta {
	/// +optional
	#[prost(string, optional, tag="1")]
	pub api_version: ::core::option::Option<::prost::alloc::string::String>,
	/// +optional
	#[prost(string, optional, tag="2")]
	pub kind: ::core::option::Option<::prost::alloc::string::String>,
	}

Handling Envelope Wrapper