Periodic check fails

[Michael Ingold]

Hi,

I tool the liberty of testing this service on K8S v1.9.6, running the mongodb-operator:upgrade-to-crds image as the operator. I discovered, that the cluster gets started but, the periodic check in the operator process fails:

2018-04-16 10:00:09,163 ERROR PeriodicCheck (0)
Reason: Handshake status 500
Traceback (most recent call last):
  File "/root/.local/share/virtualenvs/mongodb_operator-VPESk89U/lib/python3.5/site-packages/kubernetes/stream/ws_client.py", line 260, in websocket_call
    client = WSClient(configuration, get_websocket_url(url), headers)
  File "/root/.local/share/virtualenvs/mongodb_operator-VPESk89U/lib/python3.5/site-packages/kubernetes/stream/ws_client.py", line 72, in __init__
    self.sock.connect(url, header=header)
  File "/root/.local/share/virtualenvs/mongodb_operator-VPESk89U/lib/python3.5/site-packages/websocket/_core.py", line 214, in connect
    self.handshake_response = handshake(self.sock, *addrs, **options)
  File "/root/.local/share/virtualenvs/mongodb_operator-VPESk89U/lib/python3.5/site-packages/websocket/_handshake.py", line 65, in handshake
    status, resp = _get_resp_headers(sock)
  File "/root/.local/share/virtualenvs/mongodb_operator-VPESk89U/lib/python3.5/site-packages/websocket/_handshake.py", line 122, in _get_resp_headers
    raise WebSocketBadStatusException("Handshake status %d", status)
websocket._exceptions.WebSocketBadStatusException: Handshake status 500

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/mongodb_operator/mongodb_operator/periodical.py", line 21, in periodical_check
    check_existing()
  File "/opt/mongodb_operator/mongodb_operator/periodical.py", line 111, in check_existing
    check_if_replicaset_needs_setup(cluster_object)
  File "/opt/mongodb_operator/mongodb_operator/mongodb_helpers.py", line 40, in check_if_replicaset_needs_setup
    tty=False)
  File "/root/.local/share/virtualenvs/mongodb_operator-VPESk89U/lib/python3.5/site-packages/kubernetes/stream/stream.py", line 32, in stream
    return func(*args, **kwargs)
  File "/root/.local/share/virtualenvs/mongodb_operator-VPESk89U/lib/python3.5/site-packages/kubernetes/client/apis/core_v1_api.py", line 835, in connect_get_namespaced_pod_exec
    (data) = self.connect_get_namespaced_pod_exec_with_http_info(name, namespace, **kwargs)
  File "/root/.local/share/virtualenvs/mongodb_operator-VPESk89U/lib/python3.5/site-packages/kubernetes/client/apis/core_v1_api.py", line 935, in connect_get_namespaced_pod_exec_with_http_info
    collection_formats=collection_formats)
  File "/root/.local/share/virtualenvs/mongodb_operator-VPESk89U/lib/python3.5/site-packages/kubernetes/client/api_client.py", line 321, in call_api
    _return_http_data_only, collection_formats, _preload_content, _request_timeout)
  File "/root/.local/share/virtualenvs/mongodb_operator-VPESk89U/lib/python3.5/site-packages/kubernetes/client/api_client.py", line 155, in __call_api
    _request_timeout=_request_timeout)
  File "/root/.local/share/virtualenvs/mongodb_operator-VPESk89U/lib/python3.5/site-packages/kubernetes/stream/stream.py", line 27, in _intercept_request_call
    return ws_client.websocket_call(config, *args, **kwargs)
  File "/root/.local/share/virtualenvs/mongodb_operator-VPESk89U/lib/python3.5/site-packages/kubernetes/stream/ws_client.py", line 266, in websocket_call
    raise ApiException(status=0, reason=str(e))
kubernetes.client.rest.ApiException: (0)
Reason: Handshake status 500

There are corresponding errors in the logs from mongod:

2018-04-16T10:02:24.827+0000 I NETWORK  [listener] connection accepted from 127.0.0.1:56100 #7 (4 connections now open)
2018-04-16T10:02:24.923+0000 I NETWORK  [conn7] received client metadata from 127.0.0.1:56100 conn: { application: { name: "MongoDB Shell" }, driver: { name: "MongoDB Internal Client", version: "3.6.2" }, os: { type: "Linux", name: "PRETTY_NAME="Debian GNU/Linux 8 (jessie)"", architecture: "x86_64", version: "Kernel 4.13.0-1012-azure" } }
2018-04-16T10:02:24.925+0000 I ACCESS   [conn7] Unauthorized: not authorized on admin to execute command { replSetGetStatus: 1.0, $clusterTime: { clusterTime: Timestamp(1523872941, 1), signature: { hash: BinData(0, BEC6749B2D41777268237586C11027A1C94DAAC1), keyId: 6544978079712804865 } }, $db: "admin" }
2018-04-16T10:02:24.929+0000 I NETWORK  [conn7] end connection 127.0.0.1:56100 (3 connections now open)

This looks like the operator is trying to access the mongod replicas but does not have sufficent permissions to get the replicas state!

Am I doing something wrong and if yes - what?

[Philipp Strube]

I believe the WebSocketBadStatusException is due to the operator trying to exec into the pod to run the replica set init command, but the pod isn't up yet. I need to look into that and catch the exception properly.

While ugly, it doesn't prevent the replica set from coming up though. It takes a couple of tries, but eventually you'll have a replica set.

I've merged CRD support into master now #8, and added some happy path integration tests #9 and #10. Here's an example build https://travis-ci.org/kbst/mongodb/builds/378412340

You should be able to use the kubestack/mongodb:latest image. I want to do a bit more testing before I cut the first release.

Let me know if above is true for your setup too.