使用higress网关访问后端协议为grpc，无法访问。

Question

使用higress网关访问后端协议为grpc，无法访问。

ldl-home opened this issue 2 months ago · comments

If you are reporting any crash or any potential security issue, do not
open an issue in this repo. Please report the issue via ASRC(Alibaba Security Response Center) where the issue will be triaged appropriately.

I have searched the issues of this repository and believe that this is not a duplicate.

Ⅰ. Issue Description

基于k8s环境使用如下配置创建了一个higress访问的域名，后端服务提供grpc

---
apiVersion: v1
kind: Service
metadata:
  namespace: higress-system
  name: go-ping-s
spec:
  selector:
    app: go-ping-s
  ports:
    - name: grpc
      port: 80
      targetPort: 8080
      protocol: TCP
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  namespace: higress-system
  annotations:
    higress.io/backend-protocol: "GRPC"
  name: go-ping-s
spec:
  ingressClassName: higress
  rules:
    - host: grpc.test.com
      http:
        paths:
          - backend:
              service:
                name: go-ping-s
                port: 
                  number: 80
            path: /
            pathType: Exact

Ⅱ. Describe what happened

通过 postman的grpc协议调用grpc.test.com其中的一个ping方法。无法到后端服务

日志如下
{"authority":"grpc.test.com:80","bytes_received":"0","bytes_sent":"0","downstream_local_address":"192.168.201.50:80","downstream_remote_address":"192.168.201.103:56441","duration":"0","istio_policy_status":"-","method":"POST","path":"/ping.Ping/Ping","protocol":"HTTP/2","request_id":"dd357754-f453-40aa-be3a-eb5171ee649f","requested_server_name":"-","response_code":"501","response_flags":"NR","route_name":"-","start_time":"2024-04-08T05:47:46.917Z","trace_id":"-","upstream_cluster":"-","upstream_host":"-","upstream_local_address":"-","upstream_service_time":"-","upstream_transport_failure_reason":"-","user_agent":"grpc-node-js/1.8.10","x_forwarded_for":"192.168.201.103"}

Just paste your stack trace here!

Ⅲ. Describe what you expected to happen

Ⅳ. How to reproduce it (as minimally and precisely as possible)

xxx
xxx
xxx

Ⅴ. Anything else we need to know?

Ⅵ. Environment:

Higress version:
OS :
Others:

Kent Dong · Answer 1 · Mon Apr 08 2024 15:19:24 GMT+0800 (China Standard Time)

"response_flags":"NR" 是找不到路由的意思。能提供一下PostMan的测试请求配置吗？

Sweet_home-li · Answer 2 · Mon Apr 08 2024 16:18:40 GMT+0800 (China Standard Time)

syntax = "proto3";

package ping;
option go_package="./ping";

message Request {
string ping = 1;
}

message Response {
string pong = 1;
}

service Ping {
rpc Ping(Request) returns(Response);
}

Kent Dong · Answer 3 · Mon Apr 08 2024 16:29:28 GMT+0800 (China Standard Time)

好的。我本地验证一下。

澄潭 · Answer 4 · Mon Apr 08 2024 16:30:59 GMT+0800 (China Standard Time)

path: /
pathType: Exact

@ldl-home 精确匹配/肯定不行了，你看下日志里的path是/ping.Ping/Ping

Sweet_home-li · Answer 5 · Mon Apr 08 2024 17:07:45 GMT+0800 (China Standard Time)

不是这个原因

这个后面我该过了

澄潭 · Answer 6 · Mon Apr 08 2024 17:13:11 GMT+0800 (China Standard Time)

@ldl-home 改成新配置后，对应的日志发一下，也是501 NR？

Kent Dong · Answer 7 · Mon Apr 08 2024 17:14:32 GMT+0800 (China Standard Time)

我本地试了一下，是可以的。

syntax = "proto3";

option java_multiple_files = true;
option java_package = "com.realch3cho.grpctest.grpc";
option go_package = ".;common";

message HelloRequestType {
  string name = 1;
}

message HelloResponseType {
  string message = 1;
}

service HelloService {
  rpc SayHello(HelloRequestType) returns (HelloResponseType) {}
}

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    higress.io/backend-protocol: GRPC
    higress.io/destination: grpc.static
    higress.io/ignore-path-case: "false"
  creationTimestamp: "2024-03-22T02:08:05Z"
  generation: 2
  labels:
    higress.io/resource-definer: higress
  name: test
  namespace: higress-system
  resourceVersion: "571475"
  uid: 241ae809-8df4-43c1-a94e-edbd65bc0a9a
spec:
  ingressClassName: higress
  rules:
  - http:
      paths:
      - backend:
          resource:
            apiGroup: networking.higress.io
            kind: McpBridge
            name: default
        path: /
        pathType: Prefix

澄潭 · Answer 8 · Mon Apr 08 2024 17:14:35 GMT+0800 (China Standard Time)

另外看上去是console界面上配置的，ingress yaml也提供下吧，在higress-system命名空间下

Kent Dong · Answer 9 · Mon Apr 08 2024 17:14:50 GMT+0800 (China Standard Time)

我有一个问题，8080端口是gRPC的吗？

Sweet_home-li · Answer 10 · Mon Apr 08 2024 17:51:11 GMT+0800 (China Standard Time)

@johnlanni

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: go-ping
  namespace: higress-system
  uid: 32fda53a-b2b8-4485-8514-c9bbfad45edc
  resourceVersion: '10440242'
  generation: 1
  creationTimestamp: '2024-04-08T07:07:11Z'
  labels:
    higress.io/domain_grpc.iotbull.com: 'true'
    higress.io/resource-definer: higress
  annotations:
    higress.io/destination: go-ping-s.default.svc.cluster.local:8080
    higress.io/ignore-path-case: 'false'
    nginx.ingress.kubernetes.io/backend-protocol: GRPC
  managedFields:
    - manager: Kubernetes Java Client
      operation: Update
      apiVersion: networking.k8s.io/v1
      time: '2024-04-08T07:07:11Z'
      fieldsType: FieldsV1
      fieldsV1:
        f:metadata:
          f:annotations:
            .: {}
            f:higress.io/destination: {}
            f:higress.io/ignore-path-case: {}
          f:labels:
            .: {}
            f:higress.io/domain_grpc.iotbull.com: {}
            f:higress.io/resource-definer: {}
        f:spec:
          f:ingressClassName: {}
          f:rules: {}
          f:tls: {}
    - manager: node-fetch
      operation: Update
      apiVersion: networking.k8s.io/v1
      time: '2024-04-08T07:08:21Z'
      fieldsType: FieldsV1
      fieldsV1:
        f:metadata:
          f:annotations:
            f:nginx.ingress.kubernetes.io/backend-protocol: {}
  selfLink: /apis/networking.k8s.io/v1/namespaces/higress-system/ingresses/go-ping
status:
  loadBalancer: {}
spec:
  ingressClassName: higress
  tls:
    - hosts:
        - grpc.iotbull.com
      secretName: iotbull
  rules:
    - host: grpc.test.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              resource:
                apiGroup: networking.higress.io
                kind: McpBridge
                name: default

Sweet_home-li · Answer 11 · Mon Apr 08 2024 17:51:43 GMT+0800 (China Standard Time)

这个我也试过： higress.io/backend-protocol: GRPC

Sweet_home-li · Answer 12 · Mon Apr 08 2024 17:52:48 GMT+0800 (China Standard Time)

我有一个问题，8080端口是gRPC的吗？

就是自己测试启动的一个grpc服务。

Kent Dong · Answer 13 · Mon Apr 08 2024 17:55:56 GMT+0800 (China Standard Time)

ingress 修改成 prefix 之后的 access log 能发一下吗？还是之前的报错吗？

你可以试试 exec 到 higress-gateway 那个 pod 里，执行 curl -svk http://go-ping-s.default.svc.cluster.local:8080，把输出贴一下。

Sweet_home-li · Answer 14 · Mon Apr 08 2024 18:09:55 GMT+0800 (China Standard Time)

这样连接被拒绝了

connect to 10.68.5.183 port 8080 failed: Connection refused
Failed to connect to go-ping-s.default.svc.cluster.local port 8080 after 4 ms: Connection refused
Closing connection 0

Kent Dong · Answer 15 · Mon Apr 08 2024 18:10:32 GMT+0800 (China Standard Time)

这样连接被拒绝了

connect to 10.68.5.183 port 8080 failed: Connection refused

Failed to connect to go-ping-s.default.svc.cluster.local port 8080 after 4 ms: Connection refused

Closing connection 0

这就对应了我前面的问题，你的gRPC服务监听的什么端口，是8080吗？

Sweet_home-li · Answer 16 · Mon Apr 08 2024 18:14:12 GMT+0800 (China Standard Time)

我这边又部署了一个新的pod 服务a提供api的接口，通过api加上path方式访问grpc服务（服务b），直接访问pod IP是可以通的。通过clusterip地址访问不到
可能是server层有问题

Kent Dong · Answer 17 · Mon Apr 08 2024 18:16:17 GMT+0800 (China Standard Time)

go-ping-s.default.svc.cluster.local:8080

嗯，那你要查查kubeproxy什么的了。看一下Service和Endpoints呢，里面有Pod吗？

Sweet_home-li · Answer 18 · Mon Apr 08 2024 18:32:02 GMT+0800 (China Standard Time)

目前测试下来正常了，通过higress还是访问不到
root@k8s-node01:/# curl -svk http://go-ping-s.default.svc.cluster.local:8080

Trying 10.68.5.183:8080...
Connected to go-ping-s.default.svc.cluster.local (10.68.5.183) port 8080 (#0)

GET / HTTP/1.1
Host: go-ping-s.default.svc.cluster.local:8080
User-Agent: curl/7.81.0
Accept: /

Received HTTP/0.9 when not allowed

{"authority":"web.test.com","bytes_received":"0","bytes_sent":"15","downstream_local_address":"192.168.201.50:443","downstream_remote_address":"192.168.201.103:54807","duration":"9054","istio_policy_status":"-","method":"GET","path":"/ping?target=grpc.iotbull.com:443","protocol":"HTTP/1.1","request_id":"043d4be7-5aa9-474f-9bda-c5d58bda32ec","requested_server_name":"web.test.com","response_code":"503","response_flags":"URX","route_name":"web-test","start_time":"2024-04-08T10:29:24.954Z","trace_id":"f6c8e00c3196ec8cf5a37bfe645a6fa7","upstream_cluster":"outbound|80||go-web.default.svc.cluster.local","upstream_host":"172.20.135.157:8888","upstream_local_address":"172.20.85.192:52034","upstream_service_time":"9054","upstream_transport_failure_reason":"-","user_agent":"PostmanRuntime/7.37.0","x_forwarded_for":"192.168.201.103"}

Kent Dong · Answer 19 · Mon Apr 08 2024 18:54:25 GMT+0800 (China Standard Time)

URX: The request was rejected because the upstream retry limit (HTTP) or maximum connect attempts (TCP) was reached.
https://www.envoyproxy.io/docs/envoy/latest/configuration/observability/access_log/usage

目测是连不上。上面命令是在Gateway的Pod里做的吗？我看host变了。新的ingress发一下呢？

Sweet_home-li · Answer 20 · Mon Apr 08 2024 19:17:07 GMT+0800 (China Standard Time)

URX: The request was rejected because the upstream retry limit (HTTP) or maximum connect attempts (TCP) was reached.
https://www.envoyproxy.io/docs/envoy/latest/configuration/observability/access_log/usage

目测是连不上。上面命令是在Gateway的Pod里做的吗？我看host变了。新的ingress发一下呢？

通过postman 直接调用的服务A 的接口，然后通过target指定grpc服务地址。拿到返回值
类似如下

Kent Dong · Answer 21 · Mon Apr 08 2024 19:28:16 GMT+0800 (China Standard Time)

curl -svk http://go-ping-s.default.svc.cluster.local:8080

建议你 exec 到 higress-gateway 的 pod 里再 curl 一下

Sweet_home-li · Answer 22 · Mon Apr 08 2024 19:52:07 GMT+0800 (China Standard Time)

@CH3CHO 执行结果如下

Kent Dong · Answer 23 · Tue Apr 09 2024 10:13:23 GMT+0800 (China Standard Time)

@CH3CHO 执行结果如下

你这个是在node上执行吧？能kubectl exec到pod里面吗？

Sweet_home-li · Answer 24 · Tue Apr 09 2024 15:10:13 GMT+0800 (China Standard Time)

这个就是在higress-gateway的pod内执行的

Kent Dong · Answer 25 · Tue Apr 09 2024 15:10:59 GMT+0800 (China Standard Time)

这个就是在higress-gateway的pod内执行的

那这个前面的机器名有点迷惑性啊

澄潭 · Answer 26 · Tue Apr 09 2024 19:47:17 GMT+0800 (China Standard Time)

{"authority":"web.test.com","bytes_received":"0","bytes_sent":"15","downstream_local_address":"192.168.201.50:443","downstream_remote_address":"192.168.201.103:54807","duration":"9054","istio_policy_status":"-","method":"GET","path":"/ping?target=grpc.iotbull.com:443","protocol":"HTTP/1.1","request_id":"043d4be7-5aa9-474f-9bda-c5d58bda32ec","requested_server_name":"web.test.com","response_code":"503","response_flags":"URX","route_name":"web-test","start_time":"2024-04-08T10:29:24.954Z","trace_id":"f6c8e00c3196ec8cf5a37bfe645a6fa7","upstream_cluster":"outbound|80||go-web.default.svc.cluster.local","upstream_host":"172.20.135.157:8888","upstream_local_address":"172.20.85.192:52034","upstream_service_time":"9054","upstream_transport_failure_reason":"-","user_agent":"PostmanRuntime/7.37.0","x_forwarded_for":"192.168.201.103"}

看日志是HTTP/1.1呀，不是gRPC请求，而且upstream host字段是172.20.135.157:8888，没有请求8080

澄潭 · Answer 27 · Tue Apr 16 2024 10:09:20 GMT+0800 (China Standard Time)

@ldl-home 有调试通吗，看你上面发的日志不是gRPC请求