LabServer with k8s

This repo integrate all the serivce related to toolmen.

Service that integrated:

Installation of all
Nextcloud(like google drive)
traefik(load balancer)
network policy(k8s/policy.yml: cannot access internal ip from user's instance)
Certification
harbor(docker registry)
labbox: The main service for Toolmen lab.

Related repo

Dockerfile Monitor labbox

Use with dockercompsoe

It has not been maintained now.

How to Use with k8s

0. Make sure

ssh
git
nvidia-driver
You have nameserver: [10.96.0.10, 8.8.8.8] in local(Use netplan)

1. Clone from Github

git clone https://github.com/linnil1/LabServer
cd LabServer

2. Install k8s and build k8s cluster

Install k8s for master machine

cd install-k8s/
./k8s-install-master.sh
./helm_install.sh
cd ..

Install k8s for worker machine.

(Note: Worker servers can be logined via sshkey from master)

cd install-k8s/
./k8s-install-slave.sh server1 server2
cd ..

3. Setting and Init Setup

Set your own secret data, e.g. ip, path, nas configuration

Rename and change the setting in config.example.yaml to config.yaml

./setup.sh will

Replace the variable
Build image from dockerfile for nextcloud
Setup harbor setting

4. Set Certification

Use Lets Encrypt it! You should open port 80 and 443 for verification

docker pull certbot/certbot
docker run --rm -it -p 80:80 -v $PWD/letsencrypt:/etc/letsencrypt certbot/certbot certonly --standalone

1.1. Renew it

docker run --rm -it -p 80:80 -p 443:443 -v $PWD/letsencrypt:/etc/letsencrypt certbot/certbot renew

Then copy to certs/

cp letsencrypt/live/my.domain.ntu.edu.tw/* certs/

Self-signed

cd certs
openssl genrsa 1024 > privkey.pem
chmod 400 privkey.pem
openssl req -new -x509 -nodes -sha1 -days 365 -key privkey.pem -out fullchain.pem
cd ..

Finally, restart the pods with new certification ./renewcert.sh

5. Start

./start_service.sh

6. Setup harbor

Login

Default User: admin
Default Password: Harbor12345

Add regstries

Provider: docker-registry
Endpoint: http://harbor-harbor-registry.default.svc.cluster.local:5000
SSL: no

Test

docker login harbor.default.svc.cluster.local
docker push harbor.default.svc.cluster.local/linnil1/nextcloudfpm:19

Finally, Install labbox(Main interface)

git clone https://github.com/NTU-ToolmenLab/labbox.git
cd labbox

and follow the guide in https://github.com/NTU-ToolmenLab/labbox

Some note

Network Policy

https://hackmd.io/dZEPlsD0S22ZKBPe53iFXg

Nextcloud Enable Oauth

Go to oauth setting web https://my.domain.ntu.edu.tw:443/oauth/client (If you are admin)

Add client

{
    "client_id": "",
    "client_secret": "",
    "client_name": "testapp",
    "client_uri": "https://my.domain.ntu.edu.tw:443/drive/",
    "grant_types": ["authorization_code"],
    "redirect_uris": ["https://my.domain.ntu.edu.tw:443/drive/apps/sociallogin/custom_oidc/testapp"],
    "response_types": ["code"],
    "scope": "profile",
    "token_endpoint_auth_method": "client_secret_post"
}

Modify code in Nextcloud/nextcloud/custom_apps/sociallogin/lib/Service/ProviderService.php to make login with their name $uid = $profileId;

Go to setting web https://my.domain.ntu.edu.tw:443/drive/settings/admin/sociallogin

Add client configuration (Custom Oauth2)

Internal_name: testapp
API_Base_URL:  https://my.domain.ntu.edu.tw:443
Authorize_url: https://my.domain.ntu.edu.tw:443/oauth/authorize
Token_url:     https://my.domain.ntu.edu.tw:443/oauth/token
Profile_url:   https://my.domain.ntu.edu.tw:443/oauth/profile
Clinet_id:
Clinet_Secret:
Scope:         profile
Groups claim (optional): groups
Group mapping: fill it

You can substitude testapp to any you want(Should be consistent between oauth client and server).

NTU-ToolmenLab / LabServer