Workbench for GCW session calls intermittently return `405 Method Not Found` errors
ianpittwood opened this issue · comments
Recently @jthomasmock noticed that calls to retrieve, create, or modify sessions in RStudio on Workbench for GCW were returning 405 Method Not Found errors. After further investigation, we found multiple calls similar to the following in rserver.log:
2023-07-07T15:36:00.089430Z [rserver] ERROR Invalid method OST requested for uri: /events/get_events; LOGGED FROM: bool rstudio::core::http::AsyncServerImpl<ProtocolType>::onHeadersParsed(rstudio_boost::shared_ptr<rstudio::core::http::AsyncConnectionImpl<typename ProtocolType::socket> >, rstudio::core::http::Request*) [with ProtocolType = rstudio_boost::asio::local::stream_protocol typename ProtocolType::socket = rstudio_boost::asio::basic_stream_socket<rstudio_boost::asio::local::stream_protocol>] src/cpp/server/ServerInitLocalStream.cpp:466
For some reason, requests reaching Workbench are partially cut off, yielding calls to methods such as OST, ET, or ST. Since these are not valid methods, the server does as expected and returns 405 Method Not Found errors.
We are unsure why client requests are being cut off as they are sent correctly from the client, but sometimes reach the Workbench server as fragments. This issue has been escalated to Google to further investigate the Workstation intermediary networking and servers for a cause.
There is more troubleshooting we can do on our side to ensure that the issue is with Google's upstream proxy, but their development team said they were looking into this.
If they don't get back with us soon and do not respond, we should look into running tshark within the development container to ensure that the incoming traffic is truncated - if it isn't, that would imply a Workbench issue.
However, we have seen this in the access logs on nginx that this is already truncated, so this is very likely an upstream issue.