OathKeeper Default Helm Chart Issue | Pod throwing 503.
sabinayakc opened this issue · comments
Preflight checklist
- I could not find a solution in the existing issues, docs, nor discussions.
- I agree to follow this project's Code of Conduct.
- I have read and am following this repository's Contribution Guidelines.
- This issue affects my Ory Network project.
- I have joined the Ory Community Slack.
- I am signed up to the Ory Security Patch Newsletter.
Describe the bug
Installed helm chart using helm install oathkeeper ory/oathkeeper
.
oathkeeper pod has an error with 503 code.
Reproducing the bug
- Install helm chart
helm install oathkeeper ory/oathkeeper
Relevant log output
time=2023-04-21T17:39:53Z level=info msg=TLS has not been configured for api, skipping audience=application service_name=ORY Oathkeeper service_version=v0.40.2
time=2023-04-21T17:39:53Z level=info msg=Listening on http://:4456 audience=application service_name=ORY Oathkeeper service_version=v0.40.2
time=2023-04-21T17:39:53Z level=info msg=TLS has not been configured for proxy, skipping audience=application service_name=ORY Oathkeeper service_version=v0.40.2
time=2023-04-21T17:39:53Z level=info msg=Listening on http://:4455 audience=application service_name=ORY Oathkeeper service_version=v0.40.2
time=2023-04-21T17:39:55Z level=error msg=An error occurred while handling a request audience=application error=map[message:The requested resource could not be found] http_request=map[headers:map[accept:*/* connection:close user-agent:kube-probe/1.23+] host:127.0.0.1 method:GET path:/health/ready query:<nil> remote:172.26.6.14:33658 scheme:http] http_response=map[status_code:503] service_name=ORY Oathkeeper service_version=v0.40.2
time=2023-04-21T17:39:56Z level=error msg=An error occurred while handling a request audience=application error=map[message:The requested resource could not be found] http_request=map[headers:map[accept:*/* connection:close user-agent:kube-probe/1.23+] host:127.0.0.1 method:GET path:/health/ready query:<nil> remote:172.26.6.14:33662 scheme:http] http_response=map[status_code:503] service_name=ORY Oathkeeper service_version=v0.40.2
time=2023-04-21T17:39:57Z level=error msg=An error occurred while handling a request audience=application error=map[message:The requested resource could not be found] http_request=map[headers:map[accept:*/* connection:close user-agent:kube-probe/1.23+] host:127.0.0.1 method:GET path:/health/ready query:<nil> remote:172.26.6.14:42292 scheme:http] http_response=map[status_code:503] service_name=ORY Oathkeeper service_version=v0.40.2
Relevant configuration
- Default Helm Values
Version
Helm Version: 0.31.0
On which operating system are you observing this issue?
Linux
In which environment are you deploying?
Kubernetes with Helm
Additional Context
- AWS EKS 1.24
- When I set
demo: true
it works - I also tried creating a Rule using CRDs.
Hello there,
This behaviour is due to the k8s readiness probe failing and restarting the pod. The probe is failing because no rules are present in the system, that is why the demo mode works, as it deploys sample data.
As Oathkeeper is not k8s native, it expects the rules to be present on start, and treats an an empty rule array as an error state.
How can I instantiate it with a basic rule? Do I have to provide it a basic rule via Helm Values always?
The default helm has the following access rule which might be blank.
config:
access_rules:
repositories:
- file:///etc/rules/access-rules.json
I also created a rule using the CRD to see if it picks it up.
apiVersion: oathkeeper.ory.sh/v1alpha1
kind: Rule
metadata:
name: test-rule
namespace: default
spec:
match:
url: http://http-bin.example/echo
methods:
- GET
authenticators:
- handler: anonymous
authorizer:
handler: allow
mutators:
- handler: noop
To clear up some confusion :)
- The first code snippet you posted configures where Oathkeeper will look for rules, as that can be defined as a local file, or an url. Please refer to https://www.ory.sh/docs/oathkeeper/reference/configuration for the full config :)
- Rules can be supplied to the chart during installation using this parameter: https://github.com/ory/k8s/blob/master/hacks/values/oathkeeper.yaml#L50 The values in hacks are used for CI testing and represent ready to use examples
- The CRDs will be picked up, but you need to enable the k8s controller to do that. The extra controller is configurable via this option https://github.com/ory/k8s/blob/master/hacks/values/oathkeeper.yaml#L3-L4
Once the CRD is picked up, it will be transformed into an updated cm for the oathkeeper to read
I see your version is 0.40.2
try to downgrade in 0.40.1
Closing as this is an user error, please reopen if you need more guidance :)
I think the bug is not fixed in the actual version (0.40.6)
This is a problem created by PR ory/oathkeeper#1061 and so Release v0.40.2, we should have a flag or option to enable the check performed on PR 1061.
@zepatrik @hperl In our case, if we use Oathkeeper Maester, we get a 503 error when launching Oathkeeper Readiness Probe because there are no rules, although we create them later from the controller.
👋 @Demonsthere can you please reopen the issue ?
I see, so this is a upstream issue from oathkeeper itself 🤔 I will talk with the devs, maybe adding a --allow-empty-rules
flag could be added to disable that check, which would be the default option for maester enabled charts
Would be fine by me.
Is there any progress on this issue?