Error in BroadwayCloudPubSub.GoogleApiClient.wrap_received_messages/2

[Adam Lancaster]

For some reason that I can't figure out, when querying Pub Sub in
production, line 71 in GoogleApiClient the pubsub_projects_subscriptions_pull/3
returns me something that looks like this:

 {:ok,
 %GoogleApi.PubSub.V1.Model.PullResponse{
   receivedMessages: [
     %{
       "ackId" => "ID",
       "message" => %{
         "attributes" => %{"space" => "vegans"},
         "data" => "VEhJUyBJUyBBIFRFU1Q=",
         "messageId" => "1234321",
         "publishTime" => "2019-09-05T17:41:31.520Z"
       }
     }
   ]
 }}

However I notice when I run the tests here the same function returns me:

{:ok,
 %GoogleApi.PubSub.V1.Model.PullResponse{
   receivedMessages: [
     %GoogleApi.PubSub.V1.Model.ReceivedMessage{
       ackId: "ACK ID",
       message: %GoogleApi.PubSub.V1.Model.PubsubMessage{
         attributes: %{"space" => "vegans"},
         data: "VEhJUyBJUyBBIFRFU1Q=",
         messageId: "708601755676243",
         publishTime: ~U[2019-09-05 17:41:31.520Z]
       }
     }
   ]
 }}

Any ideas?

[Michael Crumm]

@Adzz Are you using the latest version of this library? There are possible incompatibilities with Poison in previous releases. The 0.4.0 release ensures your project is using a compatible version.

This seems reminiscent of googleapis/elixir-google-api#1119

[Adam Lancaster]

@mcrumm yes am on the latest version of this lib 0.4.0 and of broadway. Does sound suspicious though. i did check the poison versions also, which is sub 4 (3.1.0 if memory serves). We use jason in our app if that makes a difference?

I’ll triple check the mix.lock version and paste here tomorrow if that’s okay.

[Adam Lancaster]

@mcrumm checked the mix.lock:

"poison": {:hex, :poison, "3.1.0", "d9eb636610e096f86f25d9a46f35a9facac35609a7591b3be3326e99a0484665", [:mix], [], "hexpm"},
"broadway": {:hex, :broadway, "0.4.0", "b8daf580baed44347a9690449f9d8d7a308c1ca086648397d1a88eea893d047e", [:mix], [{:gen_stage, "~> 0.14", [hex: :gen_stage, repo: "hexpm", optional: false]}], "hexpm"},
"broadway_cloud_pub_sub": {:hex, :broadway_cloud_pub_sub, "0.4.0", "4da2d145b80669c94a6c7b62a79d29c1be9d970bce88bb9d7200bdf7d249a9a9", [:mix], [{:broadway, "~> 0.4.0", [hex: :broadway, repo: "hexpm", optional: false]}, {:google_api_pub_sub, "~> 0.11", [hex: :google_api_pub_sub, repo: "hexpm", optional: false]}, {:goth, "~> 1.0", [hex: :goth, repo: "hexpm", optional: true]}, {:hackney, "~> 1.6", [hex: :hackney, repo: "hexpm", optional: false]}], "hexpm"},

Which I think is correct?

[Michael Crumm]

Can you please post at least partial output from running GoogleApi.PubSub.V1.Api.Projects.pubsub_projects_subscriptions_pull/5 for your subscription?

I'd like to verify that this is an issue with :google_api_pub_sub, and not with our connector.

[Adam Lancaster]

Sure thing, the output is what I put in the issue above:

 {:ok,
 %GoogleApi.PubSub.V1.Model.PullResponse{
   receivedMessages: [
     %{
       "ackId" => "ID",
       "message" => %{
         "attributes" => %{"space" => "vegans"},
         "data" => "VEhJUyBJUyBBIFRFU1Q=",
         "messageId" => "1234321",
         "publishTime" => "2019-09-05T17:41:31.520Z"
       }
     }
   ]
 }}

which makes me think it is a poison thing, but the versions are correct as far as I can tell.

[Adam Lancaster]

It looks like this line is not returning what I want, so I think you are right it must be an issue in the google client:
https://github.com/googleapis/elixir-google-api/blob/master/clients/gax/lib/google_api/gax/response.ex#L67

[Michael Crumm]

Sorry, I misspoke in the other issue. The struct for receivedMessages should be %GoogleApi.PubSub.V1.Model.ReceivedMessage{}, of which message is a PubsubMessage struct. So there are two layers of Poison decoding that aren't working as I would expect.

PullResponse defines nested ReceivedMessage list: https://github.com/googleapis/elixir-google-api/blob/master/clients/pub_sub/lib/google_api/pub_sub/v1/model/pull_response.ex#L36

ReceivedMessage defines nested PubsubMessage field: https://github.com/googleapis/elixir-google-api/blob/master/clients/pub_sub/lib/google_api/pub_sub/v1/model/received_message.ex#L36

Decoding for most, if not all, structs ultimately ends up in ModelBase: https://github.com/googleapis/elixir-google-api/blob/master/clients/gax/lib/google_api/gax/model_base.ex#L47

So somewhere between PullResponse and ReceivedMessage Poison stopped decoding based on the struct defined for the field 🤔

[Michael Crumm]

@Adzz Have you had any more luck resolving this issue? As it stands, I can't seem to reproduce it.

I built a sample application, mostly to help aid in testing this. Can we do some compare/contrast for dependency versions beyond what's already been listed, and any other differences that you might see? https://github.com/mcrumm/cloud_pubsub_samples/blob/master/mix.lock

/cc @wojtekmach

[Adam Lancaster]

@mcrumm Thanks this looks great, I will really try to get to it today. Stay tuned!

[Adam Lancaster]

@mcrumm Okay annoyingly, I tried it today and it just works. I tried rolling back the version of the google_api_pub_sub lib to 0.11.0, but that had no affect. So like am stumped. I guess it works so it's fine, 🤷‍♂ sorry for wasting your time, I'll close the issue.

[Michael Crumm]

I am stumped, too. I wish we could have determined the root cause of your issue, but I am glad that the latest changes are working for you.