couldn't start a new transaction: could not create new transaction write: broken pipe
mstrYoda opened this issue · comments
Preflight checklist
- I could not find a solution in the existing issues, docs, nor discussions.
- I agree to follow this project's Code of Conduct.
- I have read and am following this repository's Contribution Guidelines.
- This issue affects my Ory Cloud project.
- I have joined the Ory Community Slack.
- I am signed up to the Ory Security Patch Newsletter.
Describe the bug
We are running 10 pods of keto instances on kubernetes.
When we send create request to Keto after a while Keto prints the following logs and our write requests fail. In the meantime when I execute keto check/expand commands inside that pod it works.
error=couldn't start a new transaction: could not create new transaction: write failed: write tcp ...:55274->...:5432: write: broken pipe
Reproducing the bug
helm install ory/keto
Use acl WriteService
Relevant log output
error=couldn't start a new transaction: could not create new transaction: write failed: write tcp 10.233.68.125:55274->10.84.90.178:5432: write: broken pipe grpc.code=Unknown grpc.method=TransactRelationTuples grpc.service=ory.keto.acl.v1alpha1.WriteServiceRelevant configuration
No response
Version
0.8.0-alpha.0
On which operating system are you observing this issue?
No response
In which environment are you deploying?
Kubernetes with Helm
Additional Context
No response
I think this is related to your database closing connections. You can configure various timeouts and other connection options according to https://www.ory.sh/docs/ecosystem/deployment#postgresql
I think this is related to your database closing connections. You can configure various timeouts and other connection options according to https://www.ory.sh/docs/ecosystem/deployment#postgresql
Hi @zepatrik, thank you for your response.
Even though db closes the one session, should not that keto create new session/connection?
Even though db closes the one session, should not that keto create new session/connection?
No, we don't do that. You should instead set the connection parameters in a way that the connection works as you need it. Or is there any other recommendation @aeneasr?
Exactly, this is heavily dependent on your environment and database!
time=2023-03-21T20:49:20Z level=error msg=failed to look up direct access in db audience=application error=map[message:write failed: write tcp {IP}:49656->{IP}:5432: write: broken pipe] method=checkDirect service_name=Ory Keto service_version=master
time=2023-03-21T20:49:20Z level=error msg=An error occurred while handling a request audience=application error=map[message:unable to fetch records: write failed: write tcp 10.23.25.105:46334->172.20.213.42:5432: write: broken pipe]
I am also facing the same issue with ory sometimes when I am calling the check api I am getting the same error, is it related to my max connection timout as it is getting timed out or do I have to retry with db if I face any error once and add the retry logic to the code.
I am using Postgres with default configurations, haven't added any new parameters in the configuration.
Can you please suggest some configuration change as what might be causing this and what needs to be done?