NACK Configuration issue

[Prakash I V T]

Hi,

i have installed the nats in the K8 using helm charts and configured the account using this link https://github.com/nats-io/k8s/blob/main/setup/nsc-setup.sh.

After i installed nacks, i am not able to create stream using Yaml. it is providing below error ( NACK is using the sys.creds via the K8 secret)

E0301 09:56:15.775121 1 controller.go:416] failed to process stream: failed to check if stream exists: context deadline exceeded

For sys account, even it is not listing the Streams

nats stream ls  --creds ./nsc/nkeys/creds/DEMO/SYS/sys.creds
nats: error: could not list streams: context deadline exceeded, try --help. 

For others account it is able to list the streams,

How to resolve this system-account , Nack issues ?

[Waldemar Quevedo]

Following up from slack seems like a permissions issue? What are the permissions of the credentials? Is it across accounts or all in the same account?

[Nicholas Frush]

I'm experiencing a similar issue.

[Jasper]

I'm facing the same issue (at least the symptoms are identical). I used mTLS for the repro, it's documented here: jasper-d@bc5d472
Each directory is essentially one step and after applying repro/08_accounts_streams the following errors are logged by jetstream-controller:

E0419 13:44:23.334341 1 controller.go:416] failed to process consumer: failed to check if consumer exists: context deadline exceeded
E0419 13:44:23.335567 1 controller.go:416] failed to process stream: failed to check if stream exists: context deadline exceeded
E0419 13:44:28.344475 1 controller.go:416] failed to process consumer: failed to check if consumer exists: context deadline exceeded
E0419 13:44:28.344578 1 controller.go:416] failed to process stream: failed to check if stream exists: context deadline exceeded

I checked the certificates from nats-box, they look alright. All certificates ( CN = nats.default.svc.cluster.local, CN = nats-sys-user, CN = nack-a) use the same CA. When running nats stream ls from within nats-box, I have different results, depending on the TLS cert used for authentication

# Using nats-sys-user cert (fails):
/etc/nats-certs/clients/nats-sys-tls # nats stream ls --tlsca ca.crt --tlscert tls.crt --tlskey tls.key
nats: error: could not list streams: context deadline exceeded, try --help
/etc/nats-certs/clients/nats-sys-tls # nats consumer ls --tlsca ca.crt --tlscert tls.crt --tlskey tls.key
nats: error: could not select Stream: context deadline exceeded
# Using account cert (works):
/etc/nats-certs/clients/nats-sys-tls # cd ../nats-account-tls/
/etc/nats-certs/clients/nats-account-tls # nats consumer ls --tlsca ca.crt --tlscert tls.crt --tlskey tls.key
nats: error: could not select Stream: no Streams are defined
/etc/nats-certs/clients/nats-account-tls # nats stream ls --tlsca ca.crt --tlscert tls.crt --tlskey tls.key
No Streams defined

I do not understand much about Nats Accounts and NACK yet, but it looks like jetstream-controller is using the SYS accounts cert and that does not have the required permission. Now I'm wondering if this is a setup issue and SYS account should have these permissions or is NACK using the wrong client cert?

After taking a brief look at stream.go, it looks like the controller should use specific account certs and not just SYS. Is that correct?

nack/controllers/jetstream/stream.go

Lines 70 to 101 in ccb5796

 if spec.Account != "" && c.opts.CRDConnect { 
 	// Lookup the account. 
 	acc, err := c.accLister.Accounts(ns).Get(spec.Account) 
 	if err != nil { 
 		return err 
 	} 
  
 	// Lookup the TLS secrets 
 	if acc.Spec.TLS != nil && acc.Spec.TLS.Secret != nil { 
 		secretName := acc.Spec.TLS.Secret.Name 
 		secret, err := c.ki.Secrets(ns).Get(c.ctx, secretName, k8smeta.GetOptions{}) 
 		if err != nil { 
 			return err 
 		} 
  
 		// Write this to the cacheDir 
 		accDir := filepath.Join(c.cacheDir, ns, spec.Account) 
 		if err := os.MkdirAll(accDir, 0755); err != nil { 
 			return err 
 		} 
  
 		remoteClientCert = filepath.Join(accDir, acc.Spec.TLS.ClientCert) 
 		remoteClientKey = filepath.Join(accDir, acc.Spec.TLS.ClientKey) 
 		remoteRootCA = filepath.Join(accDir, acc.Spec.TLS.RootCAs) 
 		accServers = acc.Spec.Servers 
  
 		for k, v := range secret.Data { 
 			if err := os.WriteFile(filepath.Join(accDir, k), v, 0644); err != nil { 
 				return err 
 			} 
 		} 
 	} 

[Jasper]

After debugging the controller, turned out I was missing the --crd-connect flag when starting the controller. After adding it, streams and consumers are created as expected. :)
Maybe that something that could be fixed in https://github.com/nats-io/k8s/tree/main/helm/charts/nack?
Edit: No need to change anything, the helm charts works just fine when removing jetstream.nats.url https://github.com/nats-io/k8s/blob/b3af2e05396b077e0e878d17f5bfbf5e385ff059/helm/charts/nack/templates/deployment-jetstream-controller.yml#L82-L86.

[Pepe Barbe]

After debugging the controller, turned out I was missing the --crd-connect flag when starting the controller. After adding it, streams and consumers are created as expected. :) Maybe that something that could be fixed in nats-io/k8s@main/helm/charts/nack? Edit: No need to change anything, the helm charts works just fine when removing jetstream.nats.url nats-io/k8s@b3af2e0/helm/charts/nack/templates/deployment-jetstream-controller.yml#L82-L86.

I think this need to be documented/publicized more. I've wasted a lot of trying trying to debug this problem until I got here.