Proxy support for OTel export
FuriousBottle opened this issue · comments
Is your feature request related to a problem? Please describe.
We currently have a self hosted instance of Pixie Cloud and the Pixie agents in an air-gapped environment. We have a proxy that provides limited access to endpoints outside of this environment. The endpoint that we would like to send our OTel data to is on the permitted list of the proxy.
We would like to be able to input a proxy to be used for OTel exports, and not impact any Vizier -> Vizier or Vizier -> Cloud communications.
Describe the solution you'd like
It would be nice to have a configmap or something that can set the proxy for script exports.
Describe alternatives you've considered
I have attempted to set various proxy environment variables in the Kelvin and vizier-query-broker deployments to provide this functionality, but while the proxy is being used, I haven't found the appropriate no_proxy settings to allow this to work.
If the appropriate no_proxy settings were defined, I believe this should work, but I don't have any previous experience with gRPC to be able to judge that further. The environment variables could then be mounted to the pods from a configmap.
Environment variables that I've tried setting. As you can see, it's a bit of a scattergun approach
- HTTPS_PROXY:
<proxy-endpoint> - HTTP_PROXY:
<proxy-endpoint> - https_proxy:
<proxy-endpoint> - http_proxy:
<proxy-endpoint> - grpc_proxy:
<proxy-endpoint> - NO_PROXY:
<local-pixie-endpoint>,*.<local-pixie-endpoint>,svc,local - no_proxy:
<local-pixie-endpoint>,*.<local-pixie-endpoint>,svc,local - no_grpc_proxy:
<local-pixie-endpoint>,*.<local-pixie-endpoint>,svc,local
Additional context
With no proxy variables set, the error we see in the kelvin and query-broker logs look like the below. Script outputs are able to be viewed at the cloud, but obviously don't get sent to the OTel endpoint.
E20230926 03:28:20.615234 47 exec.cc:59] Query 5b0cade0-be1f-43f7-b7af-d409b01a915f failed, reason: Internal : OTel export (carnot node_id=285) failed with error 'UNAVAILABLE'. Details: DNS resolution failed for <OTel-endpoint>:443: C-ares status is not ARES_SUCCESS qtype=A name=<OTel-endpoint> is_balancer=0: Could not contact DNS servers
With proxy variables set as above, the error we see in the kelvin and query-broker logs look like the below. The cluster shows as unhealthy in the cloud interface, as all script executions fail.
E20230926 03:31:15.549403 47 exec.cc:59] Query 107db284-107f-4faa-af20-6a46445cf383 failed, reason: Internal : Failed to call Finish on TransferResultChunk. Status: failed to connect to all addresses; last error: UNKNOWN: ipv4:<proxy-ip-address>:3128: HTTP proxy returned response code 403
I just realised that this is basically a duplicate of #599
I'll close this, as the info in that issue allowed me to export the data to my endpoint, and that issue has been addressed by the maintainers. Thanks!