Getting proxy_on_grpc_close error when attempting to use WasmPlugin in istio
collinlee opened this issue · comments
When attempting to use Istio WasmPlugin for a wasm plugin that makes GRPC calls (github.com:NomadXD/envoy-wasm-rust-grpc.git)) in my service mesh.
I am seeing this error in the istio ingressgateway logs:
│ 2023-02-24T21:40:36.044290Z error envoy wasm Function: proxy_on_grpc_close failed: Uncaught RuntimeError: unreachable
│ Proxy-Wasm plugin in-VM backtrace:
│ 0: 0x9c71c - rust_panic
│ 1: 0x1bb51 - _ZN3std9panicking20rust_panic_with_hook17h911ba172265603dfE
│ 2: 0x1afcd - ZN3std9panicking11begin_panic28
│ 5: 0x939a - ZN83$LT$envoy_wasm_rust_grpc..ExampleFilter$u20$as$u20$proxy_wasm..traits..Context$GT$21on_grpc_call_response17h67f3abf900b3e33
│ 6: 0x9b52e - proxy_on_grpc_close
│ 7: 0xa1982 - proxy_on_grpc_close.command_export
The docker-compose example provided that uses envoy doesn't give the same error.
I'm trying to understand if this is an issue with Istio or Envoy.
istioctl version
client version: 1.12.8
control plane version: 1.12.8
data plane version: 1.12.8 (2 proxies)
Has anyone else come across this before?
I think is actually caused by calling the panic function in the rust code.
@collinlee you're probably hitting #153, fixed in #154. You need upgrade to proxy-wasm = "0.2".
Thanks @PiotrSikora. A bit unrelated question, but I can't seem to figure out what the service parameter in the function dispatch_grpc_call really means. In the example from https://github.com/NomadXD/envoy-wasm-rust-grpc it's "api.ExampleService" (from the protoc results in rust that point to the proto file for the service). However, is this equivalent to an actual service name in kubernetes outside of this example? The envoy-wasm-rust-grpc's example works great out of the box - but that is assuming the usage of the docker compose environment. I can't seem to get the equivalent working porting this to istio and loading the generated rust wasm with WasmPlugin. I always get GRPC status 14 even after having defined a cluster name for the go grpc service. Is there more documentation on dispatch_grpc_call I could find somewhere?
A bit unrelated question, but I can't seem to figure out what the service parameter in the function dispatch_grpc_call really means. In the example from https://github.com/NomadXD/envoy-wasm-rust-grpc it's "api.ExampleService" (from the protoc results in rust that point to the proto file for the service). However, is this equivalent to an actual service name in kubernetes outside of this example? The envoy-wasm-rust-grpc's example works great out of the box - but that is assuming the usage of the docker compose environment. I can't seem to get the equivalent working porting this to istio and loading the generated rust wasm with WasmPlugin. I always get GRPC status 14 even after having defined a cluster name for the go grpc service. Is there more documentation on dispatch_grpc_call I could find somewhere?
From that example:
"grpc_service"is the name of the upstream/cluster/k8s service that you're trying to reach,"api.ExampleService"is theservicename from the gRPC proto definition,"GenerateHeader"is the RPC method from the gRPC proto definition.
You might need to add the upstream name via Istio's EnvoyFilter patching CLUSTER.
I have something like this to attempt to add the upstream service as a cluster to the ingressgateway:
apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
name: wasm-filter
namespace: istio-system
spec:
workloadSelector:
# select by label in the same namespace
labels:
istio: ingressgateway
configPatches:
# The Envoy config you want to modify
- applyTo: CLUSTER
match:
cluster:
service: "grpc-go"
patch:
operation: ADD
value:
name: grpc_cluster
type: STRICT_DNS
connect_timeout: 10s
lb_policy: ROUND_ROBIN
http2_protocol_options: {}
load_assignment:
cluster_name: grpc_cluster
endpoints:
- lb_endpoints:
- endpoint:
address:
socket_address:
address: "grpc-go.default.svc.cluster.local"
port_value: 50051
And then the wasm is loaded via WasmPlugin like this:
apiVersion: extensions.istio.io/v1alpha1
kind: WasmPlugin
metadata:
name: wasm-plugin
namespace: istio-system
spec:
selector:
matchLabels:
istio: ingressgateway
phase: STATS
pluginName: add_header
url: oci://docker.io/joekahuna/cors-wasm:http_headers
imagePullPolicy: Always
Service defined like this
apiVersion: v1
kind: Service
metadata:
name: "grpc-go"
labels:
app: "grpc-go"
service: "gprc-go"
spec:
ports:
- name: grpc
port: 50051
targetPort: 50051
selector:
app: "grpc-go"
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: "grpc-go-deployment"
labels:
app: "grpc-go"
spec:
replicas: 1
selector:
matchLabels:
app: "grpc-go"
template:
metadata:
labels:
app: "grpc-go"
spec:
containers:
- name: "grpc-go-container"
image: docker.io/joekahuna/grpc-go:latest
resources:
requests:
cpu: "100m"
imagePullPolicy: Always
ports:
- containerPort: 50051
command: ["/app/service"]
lib.rs calls grpc
match self.dispatch_grpc_call(
"grpc_cluster",
"api.ExampleService",
"GenerateHeader",
Vec::<(&str, &[u8])>::new(),
Some(message.as_slice()),
Duration::from_secs(5),
)
I'm not sure if it's the WasmPlugin usage that is incompatible and that perhaps that also needs to be defined via the EnvoyFilter method
I was able to get the grpc response in Rust wasmplugin after upgrading from istio 1.12 to 1.16. It's my best guess as to why, but I don't know for sure. Using proxy-wasm 0.2 or building against the latest SDK I now see this error:
2023-02-28T02:18:05.635296Z info xdsproxy connected to upstream XDS server: istiod.istio-system.svc:15012
2023-02-28T02:19:21.283833Z info wasm fetching image joekahuna/cors-wasm from registry index.docker.io with tag http_headers
2023-02-28T02:19:24.302395Z error envoy wasm wasm log istio-system.wasm-plugin add_header: Successfully dispatched gRPC unary call
2023-02-28T02:19:24.313725Z error envoy wasm wasm log istio-system.wasm-plugin add_header: gRPC response received
2023-02-28T02:19:24.313765Z error envoy wasm wasm log istio-system.wasm-plugin add_header: 5
2023-02-28T02:19:24.313777Z error envoy wasm wasm log istio-system.wasm-plugin add_header: 0
2023-02-28T02:19:24.313784Z error envoy wasm wasm log istio-system.wasm-plugin add_header: 55
2023-02-28T02:19:24.313939Z error envoy wasm wasm log istio-system.wasm-plugin add_header: response: path_type: "REQUEST_PATH" header: "REQ0e7d4d46-ee7f-4c48-b127-686b3d26968c"
2023-02-28T02:19:24.313976Z critical envoy wasm wasm log istio-system.wasm-plugin add_header: panicked at 'unexpected status: 2', /Users/collin.lee/.cargo/registry/src/github.com-1ecc6299db9ec823/proxy-wasm-0.2.1/src/hostcalls.rs:310:27
2023-02-28T02:19:24.314081Z error envoy wasm Function: proxy_on_grpc_receive failed: Uncaught RuntimeError: unreachable
Proxy-Wasm plugin in-VM backtrace:
0: 0x9b54d - rust_panic
1: 0x1d324 - std::panicking::rust_panic_with_hook::hbafe3e603d331223
2: 0x9b9a2 - std::panicking::begin_panic_handler::_$u7b$$u7b$closure$u7d$$u7d$::h8ab6ee68d5b4c391
3: 0x9b8d8 - std::sys_common::backtrace::__rust_end_short_backtrace::h008f69666d134159
4: 0xd154 - rust_begin_unwind
5: 0x1d2b - core::panicking::panic_fmt::h1d17fc068f528130
6: 0x30bf - proxy_wasm::hostcalls::set_map_value::h99823cd0419270c9
7: 0xae0c - _$LT$envoy_wasm_rust_grpc..ExampleFilter$u20$as$u20$proxy_wasm..traits..Context$GT$::on_grpc_call_response::hb35a3a0ba99f5bb5
8: 0x982f0 - proxy_on_grpc_receive
Ignore the "error" logs - I had to use error for now as info level logging wouldn't appear in the ingress logs. From the line
asm log istio-system.wasm-plugin add_header: 0
we can see that the grpc response code is 0; however the error then ensues.
any suggestions for resolving this?
I don't see the code, but from the errors it sounds like you might be mixing request and response maps (e.g. trying to set request headers in the response path).
It's the code from here: https://github.com/NomadXD/envoy-wasm-rust-grpc/blob/main/filter/src/lib.rs#L76-#L91. I tried commenting out the line self.resume_http_request but the issue remains. Will try using other callbacks