Doesn't work with OTP 23 - TLS client: In state wait_cert_cr
michaelst opened this issue · comments
[error] GenServer Spendable.Broadway.SyncMember.Broadway.Producer_0 terminating
** (MatchError) no match of right hand side value: {:error, %HTTPoison.Error{id: nil, reason: {:tls_alert, {:internal_error, 'TLS client: In state wait_cert_cr at tls_handshake_1_3.erl:1384 generated CLIENT ALERT: Fatal - Internal Error\n {unexpected_error,undef}'}}}}
(broadway_cloud_pub_sub 0.6.0) lib/broadway_cloud_pub_sub/google_api_client.ex:38: BroadwayCloudPubSub.GoogleApiClient.conn!/2
(broadway_cloud_pub_sub 0.6.0) lib/broadway_cloud_pub_sub/google_api_client.ex:93: BroadwayCloudPubSub.GoogleApiClient.receive_messages/2
(broadway_cloud_pub_sub 0.6.0) lib/broadway_cloud_pub_sub/producer.ex:202: BroadwayCloudPubSub.Producer.handle_receive_messages/1
(broadway 0.6.2) lib/broadway/topology/producer_stage.ex:106: Broadway.Topology.ProducerStage.handle_demand/2
(gen_stage 1.0.0) lib/gen_stage.ex:2103: GenStage.noreply_callback/3
(stdlib 3.13) gen_server.erl:680: :gen_server.try_dispatch/4
(stdlib 3.13) gen_server.erl:756: :gen_server.handle_msg/6
(stdlib 3.13) proc_lib.erl:226: :proc_lib.init_p_do_apply/3
Last message: {:"$gen_producer", {#PID<0.694.0>, #Reference<0.572113041.2632974340.248001>}, {:ask, 10}}
State: %{consumers: [{#PID<0.694.0>, #Reference<0.572113041.2632974340.248001>}], module: BroadwayCloudPubSub.Producer, module_state: %{client: {BroadwayCloudPubSub.GoogleApiClient, %{ack_ref: #Reference<0.572113041.2632974337.247890>, adapter: Tesla.Adapter.Hackney, connection_pool: Spendable.Broadway.SyncMember.ConnectionPool, pull_request: %GoogleApi.PubSub.V1.Model.PullRequest{maxMessages: 10, returnImmediately: nil}, subscription: %{projects_id: "cloud-57", subscriptions_id: "spendable.sync-member-request"}, token_generator: {BroadwayCloudPubSub.GoogleApiClient, :generate_goth_token, ["https://www.googleapis.com/auth/pubsub"]}}}, demand: 0, receive_interval: 5000, receive_timer: nil}, rate_limiting: nil, transformer: nil}
Hi! Can you please include precise Erlang/OTP and Elixir versions? Does the error happen consistently or from time to time? Please include as much information as possible, the error + stacktrace is often not enough for us to act on it. Thanks!
I apologize I did not get a notification for the response to this issue. This was running off of the elixir:1.10.4-alpine Dockerfile built with erlang:23-alpine, which was either 23.0.2 or 23.0.3 at the time.
The error is happening every time, the application will not start on otp 23.
Thanks! I It is more likely then that it is a certificate issue indeed. Can you try updating to the latest hackney/httpoison and related deps? Such as certifi and friends.
All of those deps are currently showing as up to date for me
"castore": {:hex, :castore, "0.1.7", "1ca19eee705cde48c9e809e37fdd0730510752cc397745e550f6065a56a701e9", [:mix], [], "hexpm", "a2ae2c13d40e9c308387f1aceb14786dca019ebc2a11484fb2a9f797ea0aa0d8"},
"certifi": {:hex, :certifi, "2.5.2", "b7cfeae9d2ed395695dd8201c57a2d019c0c43ecaf8b8bcb9320b40d6662f340", [:rebar3], [{:parse_trans, "~>3.3", [hex: :parse_trans, repo: "hexpm", optional: false]}], "hexpm", "3b3b5f36493004ac3455966991eaf6e768ce9884693d9968055aeeeb1e575040"},
"hackney": {:hex, :hackney, "1.16.0", "5096ac8e823e3a441477b2d187e30dd3fff1a82991a806b2003845ce72ce2d84", [:rebar3], [{:certifi, "2.5.2", [hex: :certifi, repo: "hexpm", optional: false]}, {:idna, "6.0.1", [hex: :idna, repo: "hexpm", optional: false]}, {:metrics, "1.0.1", [hex: :metrics, repo: "hexpm", optional: false]}, {:mimerl, "~>1.1", [hex: :mimerl, repo: "hexpm", optional: false]}, {:parse_trans, "3.3.0", [hex: :parse_trans, repo: "hexpm", optional: false]}, {:ssl_verify_fun, "1.1.6", [hex: :ssl_verify_fun, repo: "hexpm", optional: false]}], "hexpm", "3bf0bebbd5d3092a3543b783bf065165fa5d3ad4b899b836810e513064134e18"},
"httpoison": {:hex, :httpoison, "1.7.0", "abba7d086233c2d8574726227b6c2c4f6e53c4deae7fe5f6de531162ce9929a0", [:mix], [{:hackney, "~> 1.16", [hex: :hackney, repo: "hexpm", optional: false]}], "hexpm", "975cc87c845a103d3d1ea1ccfd68a2700c211a434d8428b10c323dc95dc5b980"},
Dependency Current Latest Update possible
absinthe 1.5.3 1.5.3
absinthe_plug 1.5.0 1.5.0
bcrypt_elixir 2.2.0 2.2.0
broadway 0.6.2 0.6.2
broadway_cloud_pub_sub 0.6.0 0.6.0
castore 0.1.7 0.1.7
certifi 2.5.2 2.5.2
cors_plug 2.0.2 2.0.2
cowlib 2.9.1 2.9.1
credo 1.4.0 1.4.0
dataloader 1.0.8 1.0.8
ecto_enum 1.4.0 1.4.0
ecto_sql 3.4.5 3.4.5
ex_machina 2.4.0 2.4.0
excoveralls 0.13.1 0.13.1
faker 0.14.0 0.14.0
gettext 0.18.1 0.18.1
goth 1.2.0 1.2.0
guardian 2.1.1 2.1.1
hackney 1.16.0 1.16.0
jason 1.2.1 1.2.1
kadabra 0.4.5 0.4.5
mint 1.1.0 1.1.0
mock 0.3.5 0.3.5
phoenix 1.5.4 1.5.4
phoenix_ecto 4.2.0 4.2.0
phoenix_html 2.14.2 2.14.2
phoenix_live_reload 1.2.4 1.2.4
pigeon 1.5.1 1.5.1
plug_cowboy 2.3.0 2.3.0
postgrex 0.15.5 0.15.5
protobuf 0.7.1 0.7.1
sentry 8.0.1 8.0.2 Yes
tesla 1.3.3 1.3.3
weddell 0.3.0 0.3.0
Hi @michaelst, thanks for bearing with me.
I think the most obvious issues are out of the way, so the next step would be isolate this. The failure is actually coming from this call Goth.Token.for_scope("https://www.googleapis.com/auth/pubsub"). So you can try reproducing it only with the Goth library.
I just built everything from scratch again and this is no longer occurring. Potentially maybe it was the erlang:23-alpine docker image at the time.