Giters

Substra / substra

Low-level Python library used to interact with a Substra network

Home Page:https://docs.substra.org

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Worker pod cannot resolve substra-backend.node-1.com

chrisalexandrepena opened this issue · comments

commented

I've tried to run the titanic example on a clean install of substra on a ubuntu AWS server, but get an error when trying to run the python scripts/add_train_algo_random_forest.py command:

Adding algo...
Registering traintuple...
Registering testtuple...
Requests error status 400: {"message":"could not register this testtuple, the traintuple 3ff431efc764dbbb34c46ccf42b27a31fd44e17fc22339e24fa05d0b464f449b has a status failed","pkhash":"887599f8513ae358eaecbee85b834b7eda30f58dc83b711ec268352e5671d7f1"}
Traceback (most recent call last):
  File "/home/chrisalexandre/dev/substra/substra/examples/titanic/.venv/lib/python3.6/site-packages/substra/sdk/rest_client.py", line 121, in __request
    r.raise_for_status()
  File "/home/chrisalexandre/dev/substra/substra/examples/titanic/.venv/lib/python3.6/site-packages/requests/models.py", line 941, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 400 Client Error: Bad Request for url: http://substra-backend.node-1.com/testtuple/

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "scripts/add_train_algo_random_forest.py", line 87, in <module>
    }, exist_ok=True)
  File "/home/chrisalexandre/dev/substra/substra/examples/titanic/.venv/lib/python3.6/site-packages/substra/sdk/client.py", line 40, in wrapper
    return f(*args, **kwargs)
  File "/home/chrisalexandre/dev/substra/substra/examples/titanic/.venv/lib/python3.6/site-packages/substra/sdk/client.py", line 527, in add_testtuple
    res = self._add(assets.TESTTUPLE, data, exist_ok=exist_ok)
  File "/home/chrisalexandre/dev/substra/substra/examples/titanic/.venv/lib/python3.6/site-packages/substra/sdk/client.py", line 185, in _add
    **requests_kwargs)
  File "/home/chrisalexandre/dev/substra/substra/examples/titanic/.venv/lib/python3.6/site-packages/substra/sdk/rest_client.py", line 256, in add
    return self._add(name, exist_ok=exist_ok, **request_kwargs)
  File "/home/chrisalexandre/dev/substra/substra/examples/titanic/.venv/lib/python3.6/site-packages/substra/sdk/rest_client.py", line 232, in _add
    return self.request('post', name, **request_kwargs)
  File "/home/chrisalexandre/dev/substra/substra/examples/titanic/.venv/lib/python3.6/site-packages/substra/sdk/utils.py", line 170, in wrapper
    return f(*args, **kwargs)
  File "/home/chrisalexandre/dev/substra/substra/examples/titanic/.venv/lib/python3.6/site-packages/substra/sdk/rest_client.py", line 187, in request
    **request_kwargs,
  File "/home/chrisalexandre/dev/substra/substra/examples/titanic/.venv/lib/python3.6/site-packages/substra/sdk/rest_client.py", line 165, in _request
    return self.__request(request_name, url, **request_kwargs)
  File "/home/chrisalexandre/dev/substra/substra/examples/titanic/.venv/lib/python3.6/site-packages/substra/sdk/rest_client.py", line 133, in __request
    raise exceptions.InvalidRequest.from_request_exception(e)
substra.sdk.exceptions.InvalidRequest: 400 Client Error: Bad Request for url: http://substra-backend.node-1.com/testtuple/: could not register this testtuple, the traintuple 3ff431efc764dbbb34c46ccf42b27a31fd44e17fc22339e24fa05d0b464f449b has a status failed

The traintuple fails. Here's what the worker logs show:
image

When logging into the worker pod and running a curl http://substra-backend.node-1.com/algo/bc0701d4d9733779ba92f3ab9bdb4308410bf81e9f30edc90b51bbb703ab57bb/file/ and get the following response:
image

It seems the node-1.com url doesn't point to the substra cluster.

commented

@chrisalexandrepena Thanks for reporting these errors.

There are two sub-issues here:

commented

Closing as the main underlying issue has been fixed..