Help connecting to AKS cluster using exec
PixelRobots opened this issue · comments
We are trying to use the official C API for Kubernetes: https://github.com/kubernetes-client/c
What we want is to connect to an AKS cluster from a local workstation, and use the API to list pods in a cluster.
We have already installed kubectl/az/kubelogin/etc on this workstation, and listing pods via kubectl works just fine.
Our AKS is on version 1.24
We are following the example here.
However, after we load the K8s config and get the API client:
int rc = load_kube_config(&basePath, &sslConfig, &apiKeys, NULL); /* NULL means loading configuration from $HOME/.kube/config */
if (rc != 0) {
printf("Cannot load kubernetes configuration.\n");
return -1;
}
apiClient_t *apiClient = apiClient_create_with_base_path(basePath, sslConfig, apiKeys);
if (!apiClient) {
printf("Cannot create a kubernetes client.\n");
return -1;
}
list_pod(apiClient);
we receive a 401 error when we try to list pods:
pod_list = CoreV1API_listNamespacedPod(apiClient, "default", /*namespace */
....
Our Azure Infrastructure partner has suggested that this may be due to some authentication changes that were introduced recently.
Can you please send us a sample c/c++ code that would work with AKS clusters version 1.24 or higher or point us into the right direction.
Many Thanks
The exec
provider here:
https://github.com/kubernetes-client/c/blob/master/kubernetes/config/exec_provider.c
should work.
You need the kubelogin
binary, and a config as described here:
https://github.com/Azure/kubelogin#web-browser-flow-default
If you have a config that uses exec
and it works in kubectl
but doesn't work in this client, that's a bug, please let us know about it.
Thanks Brendan. I will pass the information on to the team and will update here after.
There is an example using the exec
provider:
https://github.com/kubernetes-client/c/tree/master/examples/exec_provider
Hope it helps
Hello @ityuhui ,
Thank you for providing us a sample. We tried to use this approach but we have trouble figuring out how to retrieve token.
We tried to generate a token using the following command:
get-token --environment AzurePublicCloud --server-id --client-id --tenant-id --login devicecode (we also tried interactive)
Then we passed the generated token as the last argument to my_exec_provider.
The result is as the following:
kube_exec_and_get_result(): The buffer for exec args is not sufficient.
kubeconfig_exec(): The kubeconfig exec failed.
load_kube_config(): Cannot exec command in kubeconfig.
Cannot load kubernetes configuration.
We also tried to use refresh-token from ~/.kube/config but we receive HTTP 401.
Also, from my_exec_provider code, I see there is another option to use client private key and client certificate instead of token. May I ask how to retrieve the values for these two parameters?
Hello @brendandburns ,
Thank you for the suggestion you made yesterday.
If I understood correctly, you suggested to use the web browser flow. We did that using the following kube config:
apiVersion: v1
clusters:
- cluster:
certificate-authority-data: <our certificate-authority-data>
server: [SERVER-ADDRESS]
name: aks-aimms-dev
contexts:
- context:
cluster: aks-aimms-dev
user: clusterUser_rg-aks-aimms-dev_aks-aimms-dev
name: aks-aimms-dev
current-context: aks-aimms-dev
kind: Config
preferences: {}
users:
- name: clusterUser_rg-aks-aimms-dev_aks-aimms-dev
user:
exec:
apiVersion: client.authentication.k8s.io/v1beta1
args:
- get-token
- --environment
- AzurePublicCloud
- --server-id
- 6dae42f8-4368-4678-94ff-3960e28e3630
- --client-id
- 80faf920-1908-4b52-b5ef-a8e7bedfc67a
- --tenant-id
- b6b9252d-06ac-4ea3-8c02-f52b5c7dc792
- --login
- interactive
command: kubelogin
We tried to load it using this: load_kube_config(&basePath, &sslConfig, &apiKeys, NULL);
The rest of the code is the same as the sample. This also result in HTTP 401.
Does this Kubeconfig file work correctly when you use kubectl
?
Yes, kubectl works fine.
kube_exec_and_get_result(): The buffer for exec args is not sufficient.
This is a defect. The reason is insufficient buffer memory. Please enlarge the value:
KUBECONFIG_EXEC_ARGS_BUFFER_SIZE and KUBECONFIG_EXEC_COMMAND_BUFFER_SIZE
https://github.com/kubernetes-client/c/blob/master/kubernetes/config/exec_provider.c#L8-L9
And take another try.
BTW, my_exec_provider
is just an example of an exec
binary that I think cloud platform providers should provide.
@shayan-eftekhari can you try increasing the buffer size per @ityuhui 's suggestion and see if that works with the exec
provider config from kubelogin
.
If that doesn't work, I will try to reproduce this locally and see what is happening.
Thank you for your suggestions @brendandburns and @ityuhui.
I increased the buffer size so I don't see the buffer size problem anymore when I use the generated token of kubelogin, yet I still get HTTP 401.
I tried these two approaches:
- Using my_exec_provider and the following config file:
apiVersion: v1
clusters:
- cluster:
certificate-authority-data: <my certificate-authority-data>
server: [SERVER-ADDRESS]
name: kubernetes
contexts:
- context:
cluster: kubernetes
user: kubernetes-admin
name: kubernetes-admin@kubernetes
current-context: kubernetes-admin@kubernetes
kind: Config
preferences: {}
users:
- name: kubernetes-admin
user:
exec:
command: "./my_exec_provider"
apiVersion: "client.authentication.k8s.io/v1beta1"
args:
- "arg1"
- "arg2"
- "Token generated by `kubelogin get-token --environment AzurePublicCloud --server-id 6dae42f8-4368-4678-94ff-3960e28e3630 --client-id 80faf920-1908-4b52-b5ef-a8e7bedfc67a --tenant-id b6b9252d-06ac-4ea3-8c02-f52b5c7dc792 --login devicecode|azurecli|interactive`"
- Using default kube config in ~/.kube/config which is:
apiVersion: v1
clusters:
- cluster:
certificate-authority-data: <my certificate-authority-data>
server: [SERVER-ADDRESS]
name: aks-aimms-dev
contexts:
- context:
cluster: aks-aimms-dev
user: clusterUser_rg-aks-aimms-dev_aks-aimms-dev
name: aks-aimms-dev
current-context: aks-aimms-dev
kind: Config
preferences: {}
users:
- name: clusterUser_rg-aks-aimms-dev_aks-aimms-dev
user:
exec:
apiVersion: client.authentication.k8s.io/v1beta1
args:
- get-token
- --login
- azurecli
- --server-id
- 6dae42f8-4368-4678-94ff-3960e28e3630
command: kubelogin
env: null
provideClusterInfo: false
For the first approach: Using my_exec_provider and the following config file
Can you try with cURL to verify your token ?
curl -X GET $APISERVER/api --header "Authorization: Bearer $TOKEN" --insecure
Hi Hui Yu,
Here is the output:
{ "kind": "APIVersions", "versions": [ "v1" ], "serverAddressByClientCIDRs": [ { "clientCIDR": "0.0.0.0/0", "serverAddress": "\"[SERVER-ADDRESS]" } ] }
The token looks fine.
If the token works fine, the exec
provider example should work.
Did you pass in your kubeconfig ?
Hi Hui Yu,
Yes, I passed the custom kube config. Let me give you more update.
In my previous setup, I created a conan package for kubernetes and I was trying to connect to AKS in my project.
Now I am running a fresh Ubuntu 22.04 docker container. I increased buffer size to 4096 and I followed the instructions listed here: https://github.com/kubernetes-client/c to compile kubernetes and exec_provider example.
Then I used the following command to generate the token:
kubelogin get-token --environment AzurePublicCloud --server-id 6dae42f8-4368-4678-94ff-3960e28e3630 --client-id 80faf920-1908-4b52-b5ef-a8e7bedfc67a --tenant-id b6b9252d-06ac-4ea3-8c02-f52b5c7dc792 --login devicecode
Then I modified the config_with_exec_provider like this:
apiVersion: v1
clusters:
- cluster:
certificate-authority-data: [MY CERTIFICATE AUTHORITY DATA]
server: [SERVER-ADDRESS]
name: kubernetes
contexts:
- context:
cluster: kubernetes
user: kubernetes-admin
name: kubernetes-admin@kubernetes
current-context: kubernetes-admin@kubernetes
kind: Config
preferences: {}
users:
- name: kubernetes-admin
user:
exec:
command: "./my_exec_provider_bin"
apiVersion: "client.authentication.k8s.io/v1beta1"
env:
- name: "exec_client_certificate_data"
value: "-----BEGIN CERTIFICATE-----\n\n-----END CERTIFICATE-----"
- name: "exec_client_private_key"
value: "-----BEGIN RSA PRIVATE KEY-----\n\n-----END RSA PRIVATE KEY-----"
args:
- "arg1"
- "arg2"
- "[GENERATED TOKEN]"
Everything else is exactly the same as the sample. Here is the output:
The return code of HTTP request=401 Cannot get any pod.
Let's go back to the 2nd approach: Using default kube config in ~/.kube/config which is:
Can you update the command with an absolute path in your ~/.kube/config
command: /path/to/kubelogin
And try to debug the token getting from kubelogin at
c/kubernetes/config/exec_provider.c
Line 90 in 859fc3f
You can enable debugging when buiding the c client library:
mkdir build
cd build
cmake -DCMAKE_BUILD_TYPE=Debug -DCMAKE_INSTALL_PREFIX=/usr/local ..
make
sudo make install
Hi Hui Yu,
Thank you Hui Yu for your suggestion.
We logged the token in exec_provider and it was fine. Then we followed the code logic until we found this function:
https://github.com/kubernetes-client/c/blob/master/kubernetes/config/kube_config.c#L86
This function truncates the token because BEARER_TOKEN_BUFFER_SIZE is only 2048 bytes.
We increased the buffer size here as well and the problem is solved.
You may want to increase this buffer size as well as those two you mentioned earlier in the library.
Thank you for all your help,
Shayan
Glad you got this fixed, we should update the code to have a larger buffer size (and we should probably also print better error messages if the token is too long :)
Yes. the errors of insufficient buffer are caught and printed in the function kube_exec_and_get_result
, but not in setApiKeys
. I'm going to add this.