KubeCon 2020: NATS Streams and Services: From Zero to Hero
Talk: https://sched.co/ekHe
NATS is high performance cloud native messaging system that allows you to build globally available and secure applications based on streams and services that are both fast and simple to operate. In this talk you will learn: how to get started with NATS streams and services, about the NATS Adaptive Edge Architecture to span clusters across different networks and clusters using leafnodes and gateway connections, followed by a deep dive into the latest NATS features and how to deploy in Kubernetes with Helm.
You can watch the tutorial by clicking on the image below:
Follow along with this repo: https://github.com/nats-io/kubecon2020
git clone https://github.com/nats-io/kubecon2020
curl -LO https://raw.githubusercontent.com/nats-io/nsc/master/install.sh less install.sh sh ./install.sh
curl -fSl https://nats-io.github.io/k8s/setup/nsc-setup.sh | sh
source .nsc.env
tree nsc/ | less
nsc describe jwt -f nsc/accounts/nats/KO/KO.jwt
- We will separate the system into two accounts:
- An ADMIN account for management of users (create/revoke)
- An CHAT account for the users to interact with each other.
- We need 3 users at least:
- Admin User
- Credentials Provisioner and revocation
- Chat User
- Dynamically generated
- Bootstrap User
- Chat Credentials Requestor
nsc add account --name CHAT
nsc add account --name ADMIN
nsc list accounts
nsc describe jwt -f ./nsc/accounts/nats/KO/accounts/CHAT/CHAT.jwt
nsc describe jwt -f ./nsc/accounts/nats/KO/accounts/ADMIN/ADMIN.jwt
This is needed to be able to create users dynamically by the credentials provisioner.
$ nsc generate nkey --account CHAT --store
ADCQ6MQXSOSLMFIW7J6EDLJO2NCQIH4C44YR3MOWS6QLQYQFDV2UXBHH
account key stored /Users/wallyqs/repos/nats-dev/src/github.com/nats-io/kubecon2020/nsc/nkeys/keys/A/DC/ADCQ6MQXSOSLMFIW7J6EDLJO2NCQIH4C44YR3MOWS6QLQYQFDV2UXBHH.nk
#
# This signing key will be used to create users
#
$ nsc edit account -n CHAT --sk AAT6ZEPUN4QIP75K572IRETRJQJP2NJ7LU3FPHTNYSHDBUA4U5NOXNPQ
[ OK ] added signing key "ADCQ6MQXSOSLMFIW7J6EDLJO2NCQIH4C44YR3MOWS6QLQYQFDV2UXBHH"
[ OK ] edited account "CHAT"
$ nsc generate nkey --account ADMIN --store
AAEYWYQI7LICX2ZRIGVDKJYEAV7V3YK6BEDVDOBOVSGBJT3NB24QMIYN
account key stored /Users/wallyqs/repos/nats-dev/src/github.com/nats-io/kubecon2020/nsc/nkeys/keys/A/AE/AAEYWYQI7LICX2ZRIGVDKJYEAV7V3YK6BEDVDOBOVSGBJT3NB24QMIYN.nk
$ nsc edit account -n ADMIN --sk AD6Q2YIB5YETBUHTB72IMB4KCQI2YPRENU4A6LD6WGITKWK6BBSBV6UT
[ OK ] added signing key "AAEYWYQI7LICX2ZRIGVDKJYEAV7V3YK6BEDVDOBOVSGBJT3NB24QMIYN"
[ OK ] edited account "ADMIN"
$ nsc describe jwt -f ./nsc/accounts/nats/KO/accounts/CHAT/CHAT.jwt | grep Signing
│ Signing Keys │ ADCQ6MQXSOSLMFIW7J6EDLJO2NCQIH4C44YR3MOWS6QLQYQFDV2UXBHH │
$ nsc describe jwt -f ./nsc/accounts/nats/KO/accounts/ADMIN/ADMIN.jwt | grep Signing
│ Signing Keys │ AAEYWYQI7LICX2ZRIGVDKJYEAV7V3YK6BEDVDOBOVSGBJT3NB24QMIYN │
This is in the ADMIN account, a user that is able to provision credentials.
$ nsc add user -a ADMIN chat-access \ -K $NKEYS_PATH/keys/A/D6/AD6Q2YIB5YETBUHTB72IMB4KCQI2YPRENU4A6LD6WGITKWK6BBSBV6UT.nk \ --allow-sub 'chat.req.access' \ --allow-sub 'chat.KUBECON.online' \ --allow-pubsub 'chat.req.provisioned' \ --allow-pubsub 'chat.req.provisioned.updates' \ --allow-pubsub 'chat.req.revoke' \ --allow-pubsub '_INBOX.>' \ --allow-pubsub '_R_.>' \ --allow-pub-response [ OK ] generated and stored user key "UALGE5RGSDC3DC4DWIO2KU6Y2R35F6UWKD4GGDBL3TTMXROPKKAPRZBQ" [ OK ] generated user creds file `~/repos/nats-dev/src/github.com/nats-io/kubecon2020/nsc/nkeys/creds/KO/ADMIN/chat-access.creds` [ OK ] added user "chat-access" to account "ADMIN" $ nsc describe jwt -f $NKEYS_PATH/creds/KO/ADMIN/chat-access.creds
This is a bootstrapping user shared by everyone, like a guest user.
$ nsc add user -a ADMIN chat-creds-request \ -K $NKEYS_PATH/keys/A/D6/AD6Q2YIB5YETBUHTB72IMB4KCQI2YPRENU4A6LD6WGITKWK6BBSBV6UT.nk \ --allow-pubsub '_INBOX.>' \ --allow-pubsub '_R_.>' \ --allow-pub 'chat.req.access' $ nsc describe jwt -f $NKEYS_PATH/creds/KO/ADMIN/chat-creds-request.creds
nsc add export -a ADMIN --service -n chat-access -s chat.req.access nsc add export -a CHAT -n chat-online -s chat.KUBECON.online rm ./nsc/accounts/nsc.json nsc add import -a CHAT --service --src-account $(nsc list accounts 2>&1 | grep ADMIN | awk '{print $4}') -n chat-access --remote-subject chat.req.access -s chat.req.access nsc add import -a ADMIN -n chat-online \ --src-account $(nsc list accounts 2>&1 | grep CHAT | awk '{print $4}') \ --remote-subject chat.KUBECON.online
Generate the NATS configuration.
source .nsc.env
nsc list accounts
mkdir conf jwt
nsc generate config --sys-account SYS --nats-resolver > conf/resolver.conf
cat conf/resolver.conf
Start the NATS Server:
nats-server -c conf/resolver.conf
Create a mock admin responder:
nats-rply -creds $NKEYS_PATH/creds/KO/ADMIN/chat-access.creds chat.req.access example
Try to make a request:
nats-req -creds $NKEYS_PATH/creds/KO/ADMIN/chat-creds-request.creds chat.req.access example
You can find info here:
https://docs.nats.io/nats-on-kubernetes/super-cluster-on-digital-ocean
Let’s create a cluster in Digital Ocean:
doctl kubernetes cluster create nats-k8s-sfo2 --count 3 --region sfo2
- 4222 is the client port
- 7422 is the port for leafnodes
- 7522 is the port for gateway connections (cluster of clusters)
- 443 is the websocket port
for firewall in `doctl compute firewall list | tail -n 3 | awk '{print $1}'`; do
doctl compute firewall add-rules $firewall --inbound-rules protocol:tcp,ports:4222,address:0.0.0.0/0
doctl compute firewall add-rules $firewall --inbound-rules protocol:tcp,ports:7422,address:0.0.0.0/0
doctl compute firewall add-rules $firewall --inbound-rules protocol:tcp,ports:7522,address:0.0.0.0/0
doctl compute firewall add-rules $firewall --inbound-rules protocol:tcp,ports:443,address:0.0.0.0/0
done
First deploy the external-dns component using the DigitalOcean provider:
echo '
rbac:
create: true
provider: digitalocean
digitalocean:
apiToken: himitsu
interval: "1m"
policy: sync # or upsert-only
' > k8s/external-dns.yaml
helm repo add bitnami https://charts.bitnami.com/bitnami
helm install external-dns bitnami/external-dns -f k8s/external-dns.yaml
This NodePort
is required in order to be able to expose the host ports.
apiVersion: v1
kind: Service
metadata:
name: nats-nodeport
labels:
app: nats
annotations:
external-dns.alpha.kubernetes.io/hostname: sfo.nats.chat
spec:
type: NodePort
selector:
app: nats
externalTrafficPolicy: Local
ports:
- name: client
port: 4222
nodePort: 30222
targetPort: 4222
- name: websocket
port: 443
nodePort: 30223
targetPort: 443
kubectl apply -f k8s/node-port.yaml
This will make it possible to reach out to the NATS cluster by using the sfo.nats.chat
domain:
nslookup sfo.nats.chat
apiVersion: v1
kind: Service
metadata:
name: nats-lb
spec:
type: LoadBalancer
selector:
app: nats-chat-frontend
ports:
# - protocol: TCP
# port: 4222
# targetPort: 4222
# name: client
- protocol: TCP
port: 80
targetPort: 8080
name: websocket
brew install helm helm repo add nats https://nats-io.github.io/k8s/helm/charts/ helm repo update
kubectl --context do-sfo2-nats-k8s-sfo2 create cm nats-accounts --from-file conf/resolver.conf
We will need TLS for the websockets:
kubectl create secret generic nats-tls \ --from-file=letsencrypt/live/sfo.nats.chat/fullchain.pem \ --from-file=letsencrypt/live/sfo.nats.chat/privkey.pem kubectl create secret generic nats-frontend-tls \ --from-file=letsencrypt/live/nats.chat/fullchain.pem \ --from-file=letsencrypt/live/nats.chat/privkey.pem
Secrets for the provisioner:
mkdir creds cp ./nsc/nkeys/keys/A/AT/AAT6ZEPUN4QIP75K572IRETRJQJP2NJ7LU3FPHTNYSHDBUA4U5NOXNPQ.nk creds/sk.nk cp ./nsc/nkeys/keys/O/C2/OC22KWMRNWFF4RAENUOU6DAH4LBEZSKEXMVA57ZQLTP3HJ37IELNV3FO.nk creds/osk.nk kubectl create secret generic nats-admin-creds \ --from-file=./nsc/accounts/nats/KO/accounts/CHAT/CHAT.jwt \ --from-file=./creds/osk.nk \ --from-file=./creds/sk.nk \ --from-file=./nsc/nkeys/creds/KO/SYS/sys.creds \ --from-file=./nsc/nkeys/creds/KO/ADMIN/chat-access.creds
Generic bootstrap credentials for users:
kubectl create secret generic nats-bootstrap-creds --from-file=bootstrap-creds=./nsc/nkeys/creds/KO/ADMIN/chat-creds-request.creds
# helm install nats nats/nats -f k8s/sfo-nats-server.yaml
helm install nats ./helm/helm/charts/nats -f k8s/sfo-nats-server.yaml
Upload the accounts:
nats-req -s tls://sfo.nats.chat:4222 -creds $NKEYS_PATH/creds/KO/SYS/sys.creds "\$SYS.REQ.ACCOUNT.$(nsc list accounts 2>&1 | grep CHAT | awk '{print $4}').CLAIMS.UPDATE" $(cat ./nsc/accounts/nats/KO/accounts/CHAT/CHAT.jwt)
# FIXME: workaround to prevent colors matching in the line below.
rm ./nsc/accounts/nsc.json
nats-req -s tls://sfo.nats.chat:4222 -creds $NKEYS_PATH/creds/KO/SYS/sys.creds "\$SYS.REQ.ACCOUNT.$(nsc list accounts 2>&1 | grep ADMIN | awk '{print $4}').CLAIMS.UPDATE" $(cat ./nsc/accounts/nats/KO/accounts/ADMIN/ADMIN.jwt)
kubectl apply -f k8s/creds-provisioner.yaml
kubectl apply -f k8s/chat-frontend-deploy.yaml
nats-req -s tls://sfo.nats.chat:4222 --creds ./nsc/nkeys/creds/KO/ADMIN/chat-creds-request.creds chat.req.access wallyqs 2> my.creds
nats-req -s tls://sfo.nats.chat:4222 --creds ../nsc/nkeys/creds/KO/ADMIN/chat-creds-request.creds chat.req.access wallyqs3 2> chat.creds