knivey/codesand

codesand is a rest server using amphp to run code inside containers

It requires LXD to be setup

Here is the process I've gone through to get set up. My system was running Debian GNU/Linux 10 (buster) and later Debian GNU/Linux 11 (bullseye)

You can skip the initial lxd setup if you already have lxc & lxd setup, just head to the commands to make the container

Install lxc and lxd

Do the following commands as root:

apt install lxc debian-archive-keyring snapd
snap install core
snap install lxd
# this may be different for non debian, supposedly you could relogin too
export PATH=$PATH:/snap/bin

And now init your lxd system, its best to use a dedicated disk for the storage pool. Response time will be MUCH faster and you can then do proper blkio limits.

Read more in the manual about loop vs dedicated disk under "Where to store LXD data" https://linuxcontainers.org/lxd/docs/master/storage

lxd init

Make Container

Now you can create the container and enter it:

lxc launch images:debian/11 codesand
lxc exec codesand -- /bin/bash

Inside the container:

Optional add sury repo for php8:

apt -y install apt-transport-https lsb-release ca-certificates curl wget
wget -O /etc/apt/trusted.gpg.d/php.gpg https://packages.sury.org/php/apt.gpg
sh -c 'echo "deb https://packages.sury.org/php/ $(lsb_release -sc) main" > /etc/apt/sources.list.d/php.list'
apt update

apt install php-cli  build-essential python python3 golang tcc tcl time fish zsh toilet figlet toilet-fonts cowsay default-jdk
#some php extensions
apt install php-bcmath php-curl php-mbstring php-sqlite3 php-tokenizer php-zip php-xml php-bz2 php-gmp
adduser codesand
adduser codesand games

Install anything else you might need for running scripts etc

If you install kotlin you need to do it manually because it seems snap can't run inside lxc

If you need to update or install things on all the containers later there are instructions to do that.

exit

Adjust container settings profile

lxc config set codesand boot.autostart=true
lxc profile copy default codesand
lxc profile edit codesand

Make it look like this adjust values to your liking. The limits help stop abusive code from hogging machine.

The config options are explained here: https://linuxcontainers.org/lxd/docs/master/instances#keyvalue-configuration

Notice I commented the network device

The disk IO limit is rather untrustworthy and probably won't work at all with zfs. Due to caching you can write zeros way past the limit.. cat /dev/zero > ~/zero will still cause my whole server to grind down with this config.

config:
  limits.cpu.allowance: 30%
  limits.memory: 120MiB
  limits.processes: "50"
  limits.cpu.priority: "0"
description: Default codesand LXD profile
devices:
# Even lxc will not save your comments so write this down to save
#  eth0:
#    name: eth0
#    network: lxdbr0
#    type: nic
  root:
    path: /
    pool: default
    size: 3000MB
    limits.max: 200MB
    type: disk
name: codesand
used_by: []

Assign the modified profile to the container and make a snapshot so that it can be reset to this state after code runs.

lxc profile assign codesand codesand
lxc restart codesand
# if restart fails try bigger disk size above
lxc snapshot codesand default

Add the user running bots to lxd group

usermod -a -G lxd USERNAME

Setup "pool" of containers

Make at least a few containers, when one is busy we go down the list to find one to use. It takes several seconds for a container to reset and restart and become non-busy.

./makeContainers.php 10
./startAllContaiers.php

Setup keys for access

Edit keys.yaml the file is simple

keyname: key
keyname2: key2

etc...

Running server

Make sure config.yaml is setup with where you would like to listen for connections

then just run main.php

LXC and cgroup v2

if you get this error:

WARNING: cgroup v2 is not fully supported yet, proceeding with partial confinement

Then you can get rid of it by editing /etc/default/grub

add systemd.unified_cgroup_hierarchy=0 to GRUB_CMDLINE_LINUX

run update-grub

reboot

This will make your system use cgroups v1 you should remove this when lxc fully supports cgroups v2

Installing packages and updates later

Switch the codesand container to the default profile (we shouldn't be using it for any api calls so no need to stop server yet)

lxc profile assign codesand default
lxc restart codesand

Enter the container to install packages or updates then exit

lxc exec codesand -- /bin/bash

When done, reassign the limited codesand profile and update snapshot

lxc profile assign codesand codesand
lxc restart codesand
lxc snapshot codesand default --reuse

You will need to shutdown the api server during the next part.

Delete all containers in the pool and then remake them from the updated codesand

lxc delete $(cat container.list) --force
rm container.list
./makeContainers.php 10

Then you may start the server again

knivey / codesand