_Rendezvous of RPC that terminated with (StatusCode.UNAVAILABLE, Endpoint read failed

[zoeylee02]

Sometimes will get this error from gRPC as below.
When executing GetLatestSignedLogRootRequest,

File "/home/docker/code/services/core/trillianclient/client.py", line 83, in get_and_verify_root
GetLatestSignedLogRootRequest(log_id=self.log.tree_id))
File "/usr/local/lib/python3.6/site-packages/grpc/_channel.py", line 492, in call
return _end_unary_response_blocking(state, call, False, deadline)
File "/usr/local/lib/python3.6/site-packages/grpc/_channel.py", line 440, in _end_unary_response_blocking
raise _Rendezvous(state, None, None, deadline)
grpc._channel._Rendezvous: <_Rendezvous of RPC that terminated with (StatusCode.UNAVAILABLE, Endpoint read failed)>

just wondering what kind of behavior will cause this error? or how to avoid this?

[Al Cutter]

Hi Zoey,

could you provide a bit more info to help us reproduce this?

It sounds like maybe the dockerized Trillian server isn't running, or the database might be unavailable - but it's impossible to tell for sure with the info here.

Thanks,
Al.

[zoeylee02]

Hi,

Here is what we run in our instances.

CONTAINER ID        IMAGE                              COMMAND                  CREATED             STATUS              PORTS                              NAMES
3e705bc7865d        python-api-service                 "sh /home/docker/cod…"   5 weeks ago         Up 5 weeks          0.0.0.0:8000->8000/tcp             python-api-service_1
313c4088e88d        deployment_trillian-log-signer     "/trillian_log_signe…"   2 months ago        Up 2 months         0.0.0.0:8092->8091/tcp             deployment_trillian-log-signer_1
94736e8b4ed1        deployment_trillian-log-server     "/trillian_log_serve…"   2 months ago        Up 2 months         0.0.0.0:8090-8091->8090-8091/tcp   deployment_trillian-log-server_1
a6aaacd03360        deployment_mysql                   "docker-entrypoint.s…"   2 months ago        Up 2 months         3306/tcp, 33060/tcp         

python-api-service will access 127.0.0.1:8090 to Trillian-service through gRpc, but sometimes will get failed which I mentioned above.

Just wondering why will get an error sometimes, but sometimes won't?
And how to fix it?

[zoeylee02]

Hi @AlCutter

Still will get an error as I mentioned before, any ideas for helping us to solve it?
Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/flask/app.py", line 1982, in wsgi_app
response = self.full_dispatch_request()
File "/usr/local/lib/python3.6/site-packages/flask/app.py", line 1614, in full_dispatch_request
rv = self.handle_user_exception(e)
File "/usr/local/lib/python3.6/site-packages/flask/app.py", line 1517, in handle_user_exception
reraise(exc_type, exc_value, tb)
File "/usr/local/lib/python3.6/site-packages/flask/_compat.py", line 33, in reraise
raise value
File "/usr/local/lib/python3.6/site-packages/flask/app.py", line 1612, in full_dispatch_request
rv = self.dispatch_request()
File "/usr/local/lib/python3.6/site-packages/flask/app.py", line 1598, in dispatch_request
return self.view_functionsrule.endpoint
File "./main/core/apn_api.py", line 93, in validate
root = blueprint.trillian_client.get_and_verify_root()
File "./main/libs/trillianclient/client.py", line 87, in get_and_verify_root
GetLatestSignedLogRootRequest(log_id=self.log.tree_id))
File "/usr/local/lib/python3.6/site-packages/grpc/_channel.py", line 492, in call
return _end_unary_response_blocking(state, call, False, deadline)
File "/usr/local/lib/python3.6/site-packages/grpc/_channel.py", line 440, in _end_unary_response_blocking
raise _Rendezvous(state, None, None, deadline)
grpc._channel._Rendezvous: <_Rendezvous of RPC that terminated with (StatusCode.UNAVAILABLE, Endpoint read failed)>

How I did is creating flask API + Nginx to access Trillian by gRPC

Thanks,
Zoey

[Rob Percival]

Are you using our docker-compose config? If so, could you provide the server logs when this next occurs please? You can get them by running the following command from the "trillian" directory:

docker-compose -f "examples/deployment/docker-compose.yml" logs

[zoeylee02]

After upgrading Trillian to version 1.3.1, it seems not to happen again