always get sleep

Question

always get sleep

kulong0105 opened this issue 4 years ago · comments

Describe the bug
always get sleep, cannot work as expected

To Reproduce
Steps to reproduce the behavior:

$powerfulseal autonomous --no-cloud  --inventory-kubernetes --headless  --remote-user root --ssh-path-to-private-key ~/.ssh/id_rsa --policy-file ./policy.yml
2020-07-28 21:05:46 Allen-MBP.local powerfulseal[41173] INFO modules powerfulseal.cli.__main__ : verbosity None : log level DEBUG : handler level ERROR
2020-07-28 21:05:46 Allen-MBP.local powerfulseal.cli.__main__[41173] INFO Creating kubernetes client with config /Users/yilong/.kube/config (path found for backwards compatibility)
2020-07-28 21:05:46 Allen-MBP.local powerfulseal.k8s.k8s_client[41173] INFO Initializing with config: /Users/yilong/.kube/config
2020-07-28 21:05:46 Allen-MBP.local powerfulseal[41173] INFO No driver - some functionality disabled
2020-07-28 21:05:46 Allen-MBP.local powerfulseal[41173] INFO Attempting to read the inventory from kubernetes
2020-07-28 21:05:46 Allen-MBP.local powerfulseal.node.node_inventory[41173] INFO Sync Nodes
2020-07-28 21:05:46 Allen-MBP.local powerfulseal.clouddrivers.no_cloud_driver[41173] INFO Trying to sync things while using a no-cloud driver. If you don't expect to be seeing this, you might want to rethink some of your choices
2020-07-28 21:05:46 Allen-MBP.local powerfulseal[41173] INFO Using stdout metrics collector
2020-07-28 21:05:46 Allen-MBP.local powerfulseal[41173] INFO NOT starting the UI server
2020-07-28 21:05:46 Allen-MBP.local powerfulseal[41173] INFO STARTING AUTONOMOUS MODE
2020-07-28 21:05:46 Allen-MBP.local powerfulseal.policy.policy_runner[41173] INFO Sleeping for 165 seconds
^CTraceback (most recent call last):
  File "/usr/local/bin/powerfulseal", line 8, in <module>
    sys.exit(start())
  File "/usr/local/lib/python3.7/site-packages/powerfulseal/cli/__main__.py", line 684, in start
    main(sys.argv[1:])
  File "/usr/local/lib/python3.7/site-packages/powerfulseal/cli/__main__.py", line 637, in main
    metric_collector=metric_collector
  File "/usr/local/lib/python3.7/site-packages/powerfulseal/policy/policy_runner.py", line 94, in run
    time.sleep(sleep_time)
KeyboardInterrupt
...
$ cat policy.yml
scenarios:
- name: Hello chaos!
  description: >
    Verifies that after a pod is killed,
    it's succesfully rescheduled after 30 seconds.
  steps:
  # kill a kube-system pod
  - podAction:
      matches:
        - namespace: kube-system
      filters:
        - randomSample:
            size: 1
      actions:
        - kill:
            probability: 1
  - wait:
      seconds: 30
  # make sure all pods are running in the namespace
  - podAction:
      matches:
        - namespace: kube-system
      actions:
        - checkPodState:
            state: Running

Expected behavior
start to kill pod

github-actions · Answer 1 · Tue Jul 28 2020 21:11:17 GMT+0800 (China Standard Time)

Thanks for the first issue!

Allen Ren · Answer 2 · Tue Jul 28 2020 21:12:33 GMT+0800 (China Standard Time)

pls help take a look

Mikolaj Pawlikowski · Answer 3 · Tue Jul 28 2020 21:13:41 GMT+0800 (China Standard Time)

Are you using the code from master branch?

Mikolaj Pawlikowski · Answer 4 · Tue Jul 28 2020 21:32:37 GMT+0800 (China Standard Time)

I just ran the exact same command and the config and it seems to work fine. Could you re-try with the latest commit at the master branch?

Allen Ren · Answer 5 · Wed Jul 29 2020 13:51:18 GMT+0800 (China Standard Time)

Are you using the code from master branch?

i am not sure, i use this command to install: pip3 install powerfulseal

Allen Ren · Answer 6 · Wed Jul 29 2020 13:54:48 GMT+0800 (China Standard Time)

# powerfulseal  -V
powerfulseal 2.9.0

Allen Ren · Answer 7 · Wed Jul 29 2020 14:41:01 GMT+0800 (China Standard Time)

i try to use docker image to run, get error:

# docker run -ti --entrypoint=/bin/bash -v ~/.kube:/root/.kube -v ./policy.yaml:/root/policy.yaml ./bloomberg/powerfulseal:3.0.0rc12
root@dd0b4c3416ad:~# powerfulseal autonomous --policy-file ./policy.yaml
2020-07-29 06:38:23 INFO __main__ verbosity: None; log level: INFO; handler level: INFO
2020-07-29 06:38:23 INFO __main__ Creating kubernetes client with config /root/.kube/config (path found for backwards compatibility)
2020-07-29 06:38:23 INFO k8s_client Initializing with config: /root/.kube/config
2020-07-29 06:38:23 INFO __main__ No cloud driver - some functionality disabled
2020-07-29 06:38:23 WARNING urllib3.connectionpool Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLError(399, '[SSL: EE_KEY_TOO_SMALL] ee key too small (_ssl.c:3911)'))': /api/v1/nodes
2020-07-29 06:38:23 WARNING urllib3.connectionpool Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLError(399, '[SSL: EE_KEY_TOO_SMALL] ee key too small (_ssl.c:3911)'))': /api/v1/nodes
2020-07-29 06:38:23 WARNING urllib3.connectionpool Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLError(399, '[SSL: EE_KEY_TOO_SMALL] ee key too small (_ssl.c:3911)'))': /api/v1/nodes
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 677, in urlopen
    chunked=chunked,
  File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 381, in _make_request
    self._validate_conn(conn)
  File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 976, in _validate_conn
    conn.connect()
  File "/usr/local/lib/python3.7/site-packages/urllib3/connection.py", line 370, in connect
    ssl_context=context,
  File "/usr/local/lib/python3.7/site-packages/urllib3/util/ssl_.py", line 365, in ssl_wrap_socket
    context.load_cert_chain(certfile, keyfile)
ssl.SSLError: [SSL: EE_KEY_TOO_SMALL] ee key too small (_ssl.c:3911)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/bin/powerfulseal", line 8, in <module>
    sys.exit(start())
  File "/usr/local/lib/python3.7/site-packages/powerfulseal/cli/__main__.py", line 656, in start
    main(sys.argv[1:])
  File "/usr/local/lib/python3.7/site-packages/powerfulseal/cli/__main__.py", line 525, in main
    groups_to_restrict_to = k8s_client.get_nodes_groups()
  File "/usr/local/lib/python3.7/site-packages/powerfulseal/k8s/k8s_client.py", line 65, in get_nodes_groups
    nodes = self.list_nodes()
  File "/usr/local/lib/python3.7/site-packages/powerfulseal/k8s/k8s_client.py", line 87, in list_nodes
    resp = self.client_corev1api.list_node()
  File "/usr/local/lib/python3.7/site-packages/kubernetes/client/api/core_v1_api.py", line 13726, in list_node
    (data) = self.list_node_with_http_info(**kwargs)  # noqa: E501
  File "/usr/local/lib/python3.7/site-packages/kubernetes/client/api/core_v1_api.py", line 13821, in list_node_with_http_info
    collection_formats=collection_formats)
  File "/usr/local/lib/python3.7/site-packages/kubernetes/client/api_client.py", line 345, in call_api
    _preload_content, _request_timeout)
  File "/usr/local/lib/python3.7/site-packages/kubernetes/client/api_client.py", line 176, in __call_api
    _request_timeout=_request_timeout)
  File "/usr/local/lib/python3.7/site-packages/kubernetes/client/api_client.py", line 366, in request
    headers=headers)
  File "/usr/local/lib/python3.7/site-packages/kubernetes/client/rest.py", line 241, in GET
    query_params=query_params)
  File "/usr/local/lib/python3.7/site-packages/kubernetes/client/rest.py", line 214, in request
    headers=headers)
  File "/usr/local/lib/python3.7/site-packages/urllib3/request.py", line 76, in request
    method, url, fields=fields, headers=headers, **urlopen_kw
  File "/usr/local/lib/python3.7/site-packages/urllib3/request.py", line 97, in request_encode_url
    return self.urlopen(method, url, **extra_kw)
  File "/usr/local/lib/python3.7/site-packages/urllib3/poolmanager.py", line 336, in urlopen
    response = conn.urlopen(method, u.request_uri, **kw)
  File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 765, in urlopen
    **response_kw
  File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 765, in urlopen
    **response_kw
  File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 765, in urlopen
    **response_kw
  File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 725, in urlopen
    method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
  File "/usr/local/lib/python3.7/site-packages/urllib3/util/retry.py", line 439, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='39.106.120.151', port=6443): Max retries exceeded with url: /api/v1/nodes (Caused by SSLError(SSLError(399, '[SSL: EE_KEY_TOO_SMALL] ee key too small (_ssl.c:3911)')))
root@dd0b4c3416ad:~#

Allen Ren · Answer 8 · Wed Jul 29 2020 14:42:37 GMT+0800 (China Standard Time)

remove below lines in etc/ssl/openssl.cnf will avoid the ssl error:

[system_default_sect]
MinProtocol = TLSv1.2
CipherString = DEFAULT@SECLEVEL=2

https://wiki.debian.org/ContinuousIntegration/TriagingTips/openssl-1.1.1

Mikolaj Pawlikowski · Answer 9 · Thu Jul 30 2020 05:42:56 GMT+0800 (China Standard Time)

Hi @kulong0105 - did you manage to make it work?

Mikolaj Pawlikowski · Answer 10 · Sun Aug 30 2020 16:45:56 GMT+0800 (China Standard Time)

Closing for now. Please use a recent version of PowerfulSeal, which doesn't seem to have this problem.