java control plane v.0.1.28 not working with envoy proxy v.1.18.2 + possibly because of v2 apis

Question

java control plane v.0.1.28 not working with envoy proxy v.1.18.2 + possibly because of v2 apis

madhukkc opened this issue 3 years ago · comments

Madhusudanan K K C (makk) commented 3 years ago

Hi i use v 0.1.28 of java control plane with slight modifications ( i load my routes from db), and i changed the code such that it uses all the V3 objects ( and not v2 backeard compatible fileds bare being used to best of my knowledge).

the java control plane is able to talk to data plane,using v3 apis, and i see http filters, listener, config etc being loaded correctly,
however i see that Route info is missing in my envoy config , and i get the following erro in enovy proxy log

DATA PLANE LOGS /ENVOY PROXY LOGS

[2021-05-25 08:36:15.379][10][warning][config] [source/common/config/grpc_subscription_impl.cc:127] gRPC config for type.googleapis.com/envoy.api.v2.RouteConfiguration rejected: The v2 xDS major version is deprecated and disabled by default. Support for v2 will be removed from Envoy at the start of Q1 2021. You may make use of v2 in Q4 2020 by following the advice in https://www.envoyproxy.io/docs/envoy/latest/faq/api/transition. (envoy.api.v2.RouteConfiguration) [2021-05-25 08:36:15.379][10][debug][config] [source/common/config/grpc_mux_impl.cc:129] Resuming discovery requests for type.googleapis.com/envoy.api.v2.RouteConfiguration (previous count 1) [2021-05-25 08:36:15.382][10][debug][config] [source/common/config/grpc_mux_impl.cc:158] Received gRPC message for type.googleapis.com/envoy.api.v2.RouteConfiguration at version 1621931747625 [2021-05-25 08:36:15.382][10][debug][config] [source/common/config/grpc_mux_impl.cc:122] Pausing discovery requests for type.googleapis.com/envoy.api.v2.RouteConfiguration (previous count 0)

java control plane logs
[2021-05-25 08:36:15.379][10][warning][config] [source/common/config/grpc_subscription_impl.cc:127] gRPC config for type.googleapis.com/envoy.api.v2.RouteConfiguration rejected: The v2 xDS major version is deprecated and disabled by default. Support for v2 will be removed from Envoy at the start of Q1 2021. You may make use of v2 in Q4 2020 by following the advice in https://www.envoyproxy.io/docs/envoy/latest/faq/api/transition. (envoy.api.v2.RouteConfiguration) [2021-05-25 08:36:15.379][10][debug][config] [source/common/config/grpc_mux_impl.cc:129] Resuming discovery requests for type.googleapis.com/envoy.api.v2.RouteConfiguration (previous count 1) [2021-05-25 08:36:15.382][10][debug][config] [source/common/config/grpc_mux_impl.cc:158] Received gRPC message for type.googleapis.com/envoy.api.v2.RouteConfiguration at version 1621931747625 [2021-05-25 08:36:15.382][10][debug][config] [source/common/config/grpc_mux_impl.cc:122] Pausing discovery requests for type.googleapis.com/envoy.api.v2.RouteConfiguration (previous count 0)

I am not sure why java contorl plane use v2 routeconfiguration when v3 object is there in the code.

My GRPC daemon look like
GRPCDamemon.java

ServerBuilder builder = NettyServerBuilder.forPort(config.getGrpcPort()) // removed all v2 stuff .addService(discoveryServer.getAggregatedDiscoveryServiceImpl()) // removed all v2 stuff .addService(discoveryServer.getClusterDiscoveryServiceImpl()) // removed all v2 stuff .addService(discoveryServer.getEndpointDiscoveryServiceImpl()) // removed all v2 stuff .addService(discoveryServer.getListenerDiscoveryServiceImpl()) // removed all v2 stuff .addService(discoveryServer.getRouteDiscoveryServiceImpl()) .addService(v3DiscoveryServer.getAggregatedDiscoveryServiceImpl()) .addService(v3DiscoveryServer.getClusterDiscoveryServiceImpl()) .addService(v3DiscoveryServer.getEndpointDiscoveryServiceImpl()) .addService(v3DiscoveryServer.getListenerDiscoveryServiceImpl()) .addService(v3DiscoveryServer.getRouteDiscoveryServiceImpl()); server = builder.build(); server.start();

I create my Simple as as follow

cache = new SimpleCache<>(new NodeGroup<String>() { @Override public String hash(Node node) { throw new IllegalStateException("not supported/disabled"); } @Override public String hash(io.envoyproxy.envoy.config.core.v3.Node node) { return routerConfiguration.getGroupName(); } });
and this is how initialize my cache manger

ConfigSource rdsSource = ConfigSource.newBuilder().setAds(AggregatedConfigSource.getDefaultInstance()) .setApiConfigSource(ApiConfigSource.newBuilder().setTransportApiVersion(ApiVersion.V3) .setApiType(ApiType.GRPC) .addGrpcServices(GrpcService.newBuilder().setEnvoyGrpc( EnvoyGrpc.newBuilder().setClusterName(routerConfiguration.getXdsClusterName())))) .build()

Not sure if iam missing something, this seems to be a bug to me.

Krzysztof Słonka · Answer 1 · Tue May 25 2021 20:02:24 GMT+0800 (China Standard Time)

Hi,

you have to manually set the api version for routes like this: https://github.com/allegro/envoy-control/blob/1d12931cb4b136d341da27c833528231d464984c/envoy-control-core/src/main/kotlin/pl/allegro/tech/servicemesh/envoycontrol/snapshot/resource/listeners/EnvoyListenersFactory.kt#L297 , did you do it?

Best regards,
KS

Madhusudanan K K C (makk) · Answer 2 · Tue May 25 2021 20:06:12 GMT+0800 (China Standard Time)

thanks for quick note @slonka , This is how my config source looks , but no luck

ConfigSource.newBuilder().setApiConfigSource(ApiConfigSource.newBuilder().setTransportApiVersion(ApiVersion.V3)
.setApiType(ApiType.GRPC)
.addGrpcServices(GrpcService.newBuilder().setEnvoyGrpc(
EnvoyGrpc.newBuilder().setClusterName("myclustername"))))
.build();

Krzysztof Słonka · Answer 3 · Tue May 25 2021 20:22:03 GMT+0800 (China Standard Time)

Do you have all of the resources set to v3 in the Envoy config (like this: https://github.com/allegro/envoy-control/blob/791cdc9493521a7e01a01970b2560b8e88a16105/envoy-control-tests/src/main/resources/envoy/config_ads.yaml#L5-L17)?

Madhusudanan K K C (makk) · Answer 4 · Tue May 25 2021 20:27:00 GMT+0800 (China Standard Time)

Do you have all of the resources set to v3 in the Envoy config (like this: https://github.com/allegro/envoy-control/blob/791cdc9493521a7e01a01970b2560b8e88a16105/envoy-control-tests/src/main/resources/envoy/config_ads.yaml#L5-L17)?

yup!
it looks somethign like this
`admin:
address:
socket_address: { address: 0.0.0.0, port_value: 8081 }
access_log_path: "/dev/stdout"
dynamic_resources:
ads_config:
api_type: GRPC
transport_api_version: V3
grpc_services:
- envoy_grpc:
cluster_name: xds
cds_config:
resource_api_version: V3
ads: {}
lds_config:
resource_api_version: V3
ads: {}
rds_config:
resource_api_version: V3
ads: {}
node:
cluster: xds-cluster
id: xds
static_resources:
clusters:

connect_timeout: 5s
type: STRICT_DNS
typed_extension_protocol_options:
envoy.extensions.upstreams.http.v3.HttpProtocolOptions:
"@type": type.googleapis.com/envoy.extensions.upstreams.http.v3.HttpProtocolOptions
explicit_http_config:
http2_protocol_options: {}
name: idm-router
load_assignment:
cluster_name: xds
endpoints:
- lb_endpoints:
  - endpoint:
    address:
    socket_address:
    address: 192.168.1.100
    `

Madhusudanan K K C (makk) · Answer 5 · Tue May 25 2021 21:14:02 GMT+0800 (China Standard Time)

Hi,

you have to manually set the api version for routes like this: https://github.com/allegro/envoy-control/blob/1d12931cb4b136d341da27c833528231d464984c/envoy-control-core/src/main/kotlin/pl/allegro/tech/servicemesh/envoycontrol/snapshot/resource/listeners/EnvoyListenersFactory.kt#L297 , did you do it?

Best regards,
KS

oh i c, you are suggesting to resource api version to v3
i had actualy set setApiConfigSource(ApiConfigSource.newBuilder().setTransportApiVersion(ApiVersion.V3)

whats the differnece?

Krzysztof Słonka · Answer 6 · Tue May 25 2021 21:39:26 GMT+0800 (China Standard Time)

The transport API version indicates the API endpoint and version of DiscoveryRequest/DiscoveryResponse messages used.

The resource API version indicates whether a v2 or v3 resource, e.g. v2 RouteConfiguration or v3 RouteConfiguration, is delivered.

https://www.envoyproxy.io/docs/envoy/latest/faq/api/envoy_v3

You need to set the resource version.

Also I checked that Envoy 1.18.2 fails because of some old proto definitions. Feel free to update them and open up a PR.

Madhusudanan K K C (makk) · Answer 7 · Tue May 25 2021 23:14:15 GMT+0800 (China Standard Time)

thanks it worked after i changed both tranport and resource api version to v3.

like this

ConfigSource baseRdsResource  = ConfigSource.newBuilder()
			.setResourceApiVersion(ApiVersion.V3)
			.setApiConfigSource(ApiConfigSource.newBuilder().setTransportApiVersion(ApiVersion.V3)
			.setApiType(ApiType.GRPC)
			.addGrpcServices(GrpcService.newBuilder().setEnvoyGrpc(
					EnvoyGrpc.newBuilder().setClusterName(routerConfiguration.getXdsClusterName())))).setAds(AggregatedConfigSource.getDefaultInstance()).build();