Unable to install on k3d cluster
matty1979 opened this issue · comments
Following the instructions on the page on a local k3d cluster.
Operator installs correctly and able to put in correct secret.
helm install orchestra tremolo/openunison-k8s-login-oidc --namespace=openunison -f values.yaml
It starts installing however the pods are not created correctl with the following errors
[2021-06-23 22:04:11,417][main] INFO OpenUnisonOnUndertow - Adding property : 'K8S_IMPERSONATION'
[2021-06-23 22:04:11,417][main] INFO OpenUnisonOnUndertow - Adding property : 'K8S_DASHBOARD_HOST'
[2021-06-23 22:04:11,417][main] INFO OpenUnisonOnUndertow - Adding property : 'GIVEN_NAME_CLAIM'
[2021-06-23 22:04:11,417][main] INFO OpenUnisonOnUndertow - Adding property : 'FAMILY_NAME_CLAIM'
[2021-06-23 22:04:11,417][main] INFO OpenUnisonOnUndertow - Adding property : 'OIDC_CLIENT_ID'
[2021-06-23 22:04:11,418][main] INFO OpenUnisonOnUndertow - Adding property : 'MYVD_CONFIG_PATH'
[2021-06-23 22:04:11,418][main] INFO OpenUnisonOnUndertow - Adding property : 'OIDC_IDP_USER_URL'
[2021-06-23 22:04:11,418][main] INFO OpenUnisonOnUndertow - Adding property : 'DISPLAY_NAME_CLAIM'
[2021-06-23 22:04:11,418][main] INFO OpenUnisonOnUndertow - Adding property : 'K8S_DASHBOARD_NAMESPACE'
[2021-06-23 22:04:11,418][main] INFO OpenUnisonOnUndertow - Adding property : 'OU_QUARTZ_MASK'
[2021-06-23 22:04:11,418][main] INFO OpenUnisonOnUndertow - Adding property : 'GROUPS_CLAIM'
[2021-06-23 22:04:11,418][main] INFO OpenUnisonOnUndertow - Adding property : 'K8S_DASHBOARD_SERVICE'
[2021-06-23 22:04:11,418][main] INFO OpenUnisonOnUndertow - Adding property : 'OIDC_IDP_LIMIT_DOMAIN'
[2021-06-23 22:04:11,419][main] INFO OpenUnisonOnUndertow - Adding property : 'PROMETHEUS_SERVICE_ACCOUNT'
[2021-06-23 22:04:11,419][main] INFO OpenUnisonOnUndertow - Adding property : 'K8S_URL'
[2021-06-23 22:04:11,419][main] INFO OpenUnisonOnUndertow - Adding property : 'SESSION_INACTIVITY_TIMEOUT_SECONDS'
[2021-06-23 22:04:11,419][main] INFO OpenUnisonOnUndertow - Adding property : 'K8S_CLUSTER_NAME'
[2021-06-23 22:04:11,420][main] INFO OpenUnisonOnUndertow - Loading keystore for Undertow
[2021-06-23 22:04:11,421][main] INFO OpenUnisonOnUndertow - OpenUnison XML File : '/usr/local/openunison/work/webapp/WEB-INF/unison.xml'
[2021-06-23 22:04:11,456][main] INFO OpenUnisonConfigLoader - No config from include files, using original
[2021-06-23 22:04:12,418][main] INFO OpenUnisonOnUndertow - Loading keystore : '/etc/openunison/unisonKeyStore.p12'
[2021-06-23 22:04:12,419][main] INFO OpenUnisonOnUndertow - Building Undertow
[2021-06-23 22:04:12,460][main] INFO OpenUnisonOnUndertow - Check if enabling HTTP2 - false
[2021-06-23 22:04:12,460][main] INFO OpenUnisonOnUndertow - Enabling HTTP2
[2021-06-23 22:04:12,463][main] INFO OpenUnisonOnUndertow - Adding open port : '8080'
Exception in thread "main" java.io.IOException: Invalid keystore format
at java.base/com.sun.crypto.provider.JceKeyStore.engineLoad(JceKeyStore.java:725)
at java.base/java.security.KeyStore.load(KeyStore.java:1479)
at com.tremolosecurity.openunison.undertow.OpenUnisonOnUndertow.setupTlsListener(OpenUnisonOnUndertow.java:533)
at com.tremolosecurity.openunison.undertow.OpenUnisonOnUndertow.main(OpenUnisonOnUndertow.java:280)
The dashboard ingress and orchestra ingress are not being created either.
value.yaml
network:
openunison_host: "ou.ou-test.runshiftup.local"
dashboard_host: "dashboard.ou-test.runshiftup.local"
api_server_host: "http://127.0.0.1:49202/856a8c21-4dad-48d9-b82a-c9ba7dab23cc"
session_inactivity_timeout_seconds: 900
k8s_url: https://kubernetes.default.svc:6443
createIngressCertificate: true
ingress_type: nginx
ingress_annotations:
kubernetes.io/ingress.class: nginx
force_redirect_to_tls: true
ingress_certificate: ou-tls-certificate
cert_template:
ou: "Kubernetes"
o: "Dev"
l: "My Cluster"
st: "VA"
c: "US"
image: "docker.io/tremolosecurity/openunison-k8s-login-oidc:latest"
myvd_config_path: "WEB-INF/myvd.conf"
k8s_cluster_name: k3d-dev
enable_impersonation: true
dashboard:
namespace: "kubernetes-dashboard"
cert_name: "kubernetes-dashboard-certs"
label: "k8s-app=kubernetes-dashboard"
service_name: kubernetes-dashboard
certs:
use_k8s_cm: false
trusted_certs: []
monitoring:
prometheus_service_account: system:serviceaccount:monitoring:prometheus-k8s
oidc:
client_id: 2523f4fb-005d-4b8d-99f3-f61a444bd55a
auth_url: https://login.prod-a.runshiftup.com/auth/realms/ShiftUp/protocol/openid-connect/auth
token_url: https://login.prod-a.runshiftup.com/auth/realms/ShiftUp/protocol/openid-connect/token
user_in_idtoken: false
userinfo_url: https://login.prod-a.runshiftup.com/auth/realms/ShiftUp/protocol/openid-connect/userinfo
domain: ou.ou-test.runshiftup.local
scopes: openid email profile
claims:
email: email
profile: profile
roles: roles
sub: sub
web-origins: web-origins
impersonation:
use_jetstack: false
jetstack_oidc_proxy_image: quay.io/jetstack/kube-oidc-proxy:v0.3.0
explicit_certificate_trust: false
ca_secret_name: ou-tls-secret
network_policies:
enabled: false
ingress:
enabled: true
labels:
app.kubernetes.io/name: ingress-nginx
monitoring:
enabled: true
labels:
app.kubernetes.io/name: monitoring
apiserver:
enabled: true
labels:
app.kubernetes.io/name: kube-system
services:
enable_tokenrequest: false
token_request_audience: api
token_request_expiration_seconds: 600
node_selectors: []
pullSecret: ""
openunison:
replicas: 1
non_secret_data: {}
secrets: []
openunison-orchestra in namespace openunison isn't creating an endpoint
I see two issues:
api_server_host: "http://127.0.0.1:49202/856a8c21-4dad-48d9-b82a-c9ba7dab23cc"
combined with
enable_impersonation: true
The network.api_server_host gets used as a host in the Ingress object that is created. It's also used in the internal certificate created for OpenUnison so it must be a valid host name. This is likely why you're not seeing any Endpoint or Ingress objects.
https://openunison.github.io/deployauth.html#host-names-and-networking details how network.*_host settings relate to your Ingress and LoadBalancer. If you want to enable impersonation support, create a host name for the api requests or disable impersonation if you want k3s to interact with OpenUnison directly using openid connect.
If you're not using impersonation, set network.k8s_url to http://127.0.0.1:49202/856a8c21-4dad-48d9-b82a-c9ba7dab23cc
Delete the orchestra helm deployment (helm delete orchestra -n openunison) to clear out the generated Secret objects. then redeploy with your fixed values.yaml.`
Setting impersonation to false and k8s_url: http://127.0.0.1:49202/856a8c21-4dad-48d9-b82a-c9ba7dab23cc
No difference in it.
No endpoints because the endpoint points to the pod, no active pod no endpoint makes sense. No ingress however is being created either however.
No endpoints because the endpoint points to the pod, no active pod no endpoint makes sense. No ingress however is being created either however.
IF the Ingress isn't being created it's likely an issue with the host configuration. the operator logs displays the results when an object fails to get created. You should be able to look for the Ingress being created to see what the failure is. I think whatever is causing the Ingress to not be created is the same issue with the keystore. I'll get openunison running on k3s to check if there's something specific to k3s.
Issue was within the oidc_client_secret and caused it not connect. This is /closed