canal_etcd_tls.yaml doesnt work with K8s 1.7.x and RBAC enabled

Question

canal_etcd_tls.yaml doesnt work with K8s 1.7.x and RBAC enabled

FaKod opened this issue 7 years ago · comments

Christopher Schmidt commented 7 years ago

Expected Behavior

I tried to use Canal with tls secured etcd and RBAC enabled using canal_etcd_tls.yaml

I expected a working demo at http://docs.projectcalico.org/v2.4/getting-started/kubernetes/tutorials/simple-policy

Current Behavior

Networking didn't work at all and calico-policy-controller throws the following exception:

> kubectl -n kube-system logs -f calico-policy-controller-718627407-mxh28 | more
2017-08-09 05:13:39,735 7 INFO Configuring /etc/hosts
2017-08-09 05:13:39,736 7 INFO Appended 'kubernetes.default  -> 10.100.0.1' to /etc/hosts
2017-08-09 05:13:39,737 7 INFO Beginning execution
2017-08-09 05:13:39,738 7 DEBUG Getting ServiceAccount token from: /var/run/secrets/kubernetes.io/serviceaccount/token
2017-08-09 05:13:39,739 7 DEBUG Found ServiceAccount token: eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJrdWJlLXN5c3RlbSIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJkZWZhdWx0LXRva2VuLWR4bWI2Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQubmFtZSI6ImRlZmF1bHQiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC51aWQiOiJkZTljYTVmNC03YzBkLTExZTctYjEzNi0wNjc1ZGEwMDA1OWYiLCJzdWIiOiJzeXN0ZW06c2VydmljZWFjY291bnQ6a3ViZS1zeXN0ZW06ZGVmYXVsdCJ9.qLcmpY3z2GDjVi_3RmE7CbGkinxGWgE7edU_k8wy1tuz-6Cy-HoVo4yL_5KpIbYJ8vVb1ERpP4FWnyQJH6MxLYxNPn2Auqj2lWTTe2D7ficYjJOVXrZ__gZV6KZh-BXKpXzIiPhNbk-caS5LMwLG-K-x21IGW0iC9N_HuBFFQXIniHvnUfDfp8qoAfIe8a_fcIhSdG233_xtqjGw-3W57iFjVwS3p6jmmJr4k82P31q3R5jd47vzYDpYy9tcvo-qoalqz1G-9hB8FSgQbWwv5S5o0bhjyVDZ1846Lq4s8NiqqUp10QLh222YI2XzqDV9up54qSyBqk2VVOpyXT63lg
2017-08-09 05:13:39,739 7 DEBUG Using auth token: eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJrdWJlLXN5c3RlbSIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJkZWZhdWx0LXRva2VuLWR4bWI2Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQubmFtZSI6ImRlZmF1bHQiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC51aWQiOiJkZTljYTVmNC03YzBkLTExZTctYjEzNi0wNjc1ZGEwMDA1OWYiLCJzdWIiOiJzeXN0ZW06c2VydmljZWFjY291bnQ6a3ViZS1zeXN0ZW06ZGVmYXVsdCJ9.qLcmpY3z2GDjVi_3RmE7CbGkinxGWgE7edU_k8wy1tuz-6Cy-HoVo4yL_5KpIbYJ8vVb1ERpP4FWnyQJH6MxLYxNPn2Auqj2lWTTe2D7ficYjJOVXrZ__gZV6KZh-BXKpXzIiPhNbk-caS5LMwLG-K-x21IGW0iC9N_HuBFFQXIniHvnUfDfp8qoAfIe8a_fcIhSdG233_xtqjGw-3W57iFjVwS3p6jmmJr4k82P31q3R5jd47vzYDpYy9tcvo-qoalqz1G-9hB8FSgQbWwv5S5o0bhjyVDZ1846Lq4s8NiqqUp10QLh222YI2XzqDV9up54qSyBqk2VVOpyXT63lg
2017-08-09 05:13:39,750 7 DEBUG Setting NetworkPolicy ADDED handler: <function add_update_network_policy at 0x7fd6489bf8c0>
2017-08-09 05:13:39,750 7 DEBUG Setting NetworkPolicy MODIFIED handler: <function add_update_network_policy at 0x7fd6489bf8c0>
2017-08-09 05:13:39,751 7 DEBUG Setting NetworkPolicy DELETED handler: <function delete_network_policy at 0x7fd64a1cbaa0>
2017-08-09 05:13:39,751 7 DEBUG Setting Namespace ADDED handler: <function add_update_namespace at 0x7fd6489c5ed8>
2017-08-09 05:13:39,751 7 DEBUG Setting Namespace MODIFIED handler: <function add_update_namespace at 0x7fd6489c5ed8>
2017-08-09 05:13:39,752 7 DEBUG Setting Namespace DELETED handler: <function delete_namespace at 0x7fd6489d2488>
2017-08-09 05:13:39,752 7 DEBUG Setting Pod ADDED handler: <function add_pod at 0x7fd6489d2758>
2017-08-09 05:13:39,752 7 DEBUG Setting Pod MODIFIED handler: <function update_pod at 0x7fd6489d27d0>
2017-08-09 05:13:39,753 7 DEBUG Setting Pod DELETED handler: <function delete_pod at 0x7fd6489d2578>
2017-08-09 05:13:39,753 7 INFO Leader election enabled? False
2017-08-09 05:13:39,753 7 DEBUG Attempting to remove old tier k8s-network-policy
2017-08-09 05:13:39,774 7 INFO Syncing 'NetworkPolicy' objects
2017-08-09 05:13:39,775 7 INFO Started worker thread for: NetworkPolicy
2017-08-09 05:13:39,775 7 DEBUG Getting API resources 'https://159.100.243.108:443/apis/extensions/v1beta1/networkpolicies' at version 'None'. stream=False
2017-08-09 05:13:39,776 7 INFO Started worker thread for: Namespace
2017-08-09 05:13:39,778 7 INFO Syncing 'Namespace' objects
2017-08-09 05:13:39,779 7 INFO Started worker thread for: Pod
2017-08-09 05:13:39,779 7 INFO Syncing 'Pod' objects
2017-08-09 05:13:39,779 7 DEBUG Getting API resources 'https://159.100.243.108:443/api/v1/namespaces' at version 'None'. stream=False
2017-08-09 05:13:39,780 7 DEBUG Reading from event queue
2017-08-09 05:13:39,780 7 DEBUG Getting API resources 'https://159.100.243.108:443/api/v1/pods' at version 'None'. stream=False
2017-08-09 05:13:39,801 7 DEBUG Response: <Response [403]>
2017-08-09 05:13:39,804 7 DEBUG Response: <Response [403]>
2017-08-09 05:13:39,805 7 DEBUG Response: <Response [403]>
2017-08-09 05:13:39,806 7 ERROR Unhandled exception killed Pod manager
Traceback (most recent call last):
  File "<string>", line 320, in _manage_resource
  File "<string>", line 437, in _sync_resources
  File "site-packages/requests/models.py", line 866, in json
  File "site-packages/simplejson/__init__.py", line 516, in loads
  File "site-packages/simplejson/decoder.py", line 370, in decode
  File "site-packages/simplejson/decoder.py", line 400, in raw_decode
JSONDecodeError: Expecting value: line 1 column 1 (char 0)
2017-08-09 05:13:39,806 7 ERROR Unhandled exception killed NetworkPolicy manager
Traceback (most recent call last):
  File "<string>", line 320, in _manage_resource
  File "<string>", line 437, in _sync_resources
  File "site-packages/requests/models.py", line 866, in json
  File "site-packages/simplejson/__init__.py", line 516, in loads
  File "site-packages/simplejson/decoder.py", line 370, in decode
  File "site-packages/simplejson/decoder.py", line 400, in raw_decode
JSONDecodeError: Expecting value: line 1 column 1 (char 0)
2017-08-09 05:13:39,807 7 ERROR Unhandled exception killed Namespace manager
Traceback (most recent call last):
  File "<string>", line 320, in _manage_resource
  File "<string>", line 437, in _sync_resources
  File "site-packages/requests/models.py", line 866, in json
  File "site-packages/simplejson/__init__.py", line 516, in loads
  File "site-packages/simplejson/decoder.py", line 370, in decode
  File "site-packages/simplejson/decoder.py", line 400, in raw_decode
JSONDecodeError: Expecting value: line 1 column 1 (char 0)
2017-08-09 05:13:39,807 7 INFO Restarting watch on resource: Pod
2017-08-09 05:13:39,807 7 INFO Restarting watch on resource: NetworkPolicy
2017-08-09 05:13:39,808 7 INFO Restarting watch on resource: Namespace
2017-08-09 05:13:40,809 7 INFO Syncing 'Pod' objects
2017-08-09 05:13:40,809 7 INFO Syncing 'Namespace' objects
2017-08-09 05:13:40,810 7 DEBUG Getting API resources 'https://159.100.243.108:443/api/v1/pods' at version 'None'. stream=False
2017-08-09 05:13:40,810 7 INFO Syncing 'NetworkPolicy' objects
2017-08-09 05:13:40,810 7 DEBUG Getting API resources 'https://159.100.243.108:443/api/v1/namespaces' at version 'None'. stream=False
2017-08-09 05:13:40,813 7 DEBUG Getting API resources 'https://159.100.243.108:443/apis/extensions/v1beta1/networkpolicies' at version 'None'. stream=False
2017-08-09 05:13:40,830 7 DEBUG Response: <Response [403]>
2017-08-09 05:13:40,830 7 ERROR Unhandled exception killed Pod manager
Traceback (most recent call last):
  File "<string>", line 320, in _manage_resource
  File "<string>", line 437, in _sync_resources
  File "site-packages/requests/models.py", line 866, in json
  File "site-packages/simplejson/__init__.py", line 516, in loads
  File "site-packages/simplejson/decoder.py", line 370, in decode
  File "site-packages/simplejson/decoder.py", line 400, in raw_decode
JSONDecodeError: Expecting value: line 1 column 1 (char 0)
2017-08-09 05:13:40,831 7 INFO Restarting watch on resource: Pod
2017-08-09 05:13:40,834 7 DEBUG Response: <Response [403]>
2017-08-09 05:13:40,835 7 ERROR Unhandled exception killed Namespace manager
Traceback (most recent call last):
  File "<string>", line 320, in _manage_resource
  File "<string>", line 437, in _sync_resources
  File "site-packages/requests/models.py", line 866, in json
  File "site-packages/simplejson/__init__.py", line 516, in loads
  File "site-packages/simplejson/decoder.py", line 370, in decode
  File "site-packages/simplejson/decoder.py", line 400, in raw_decode

Possible Solution

More permissions (if that's the issue)?

Your Environment

Calico version: v2.4.0
Flannel version: v0.8.0
Orchestrator version: Kubernetes v1.7.3
Operating System and version: Container Linux (CoreOS)
Link to your project (optional): n/a

Casey Davenport · Answer 1 · Thu Aug 10 2017 02:11:47 GMT+0800 (China Standard Time)

Sounds like it might be a problem with RBAC, but off the top of my head I wouldn't have expected anything to have changed.

Is this a v1.6 vs v1.7 issue? i.e. does the same manifest work on 1.6?

Erik Stidham · Answer 2 · Thu Aug 10 2017 02:23:16 GMT+0800 (China Standard Time)

I don't think this manifest was updated for K8s 1.6 (or RBAC). It still contains the tolerations for K8s 1.5:

        scheduler.alpha.kubernetes.io/tolerations: |
          [{"key": "dedicated", "value": "master", "effect": "NoSchedule" },
           {"key": "CriticalAddonsOnly", "operator": "Exists"}]

Casey Davenport · Answer 3 · Sat Sep 16 2017 01:39:45 GMT+0800 (China Standard Time)

@FaKod did you ever find a fix for this?

If so would you be open to contributing it upstream?

Christopher Schmidt · Answer 4 · Mon Sep 18 2017 12:44:38 GMT+0800 (China Standard Time)

No, sorry, I switched to the API Server "version" and K8s 1.7.x. That works fine for me.

Casey Davenport · Answer 5 · Fri Mar 02 2018 03:49:58 GMT+0800 (China Standard Time)

Going to close since this is for an old version of calico/canal.