kaushikcfd / machine-shop-maintenance

Scripts and Issues for the birds and the beers

Bill's, Luke's, Edgar's, and Andreas's Machine Shop

• Admin notes
• User notes
• Machine Inventory

Issues

See current issues with the cluster.

Monitoring (only on campus or via VPN/SSH port forward)

• GPU temperatures
• Node loads

Making configuration changes

Whenever possible, changes should be made by editing update.sh and then running this on all machines in the cluster. Todo this, proceed as follows:

• Make the change (ideally as a PR, continue here when merged)
• Do git pull in /shared/config (on any of the machines, this is shared via NFS)
• Open a tmux
• Run shoprun /shared/config/update.sh. This will spawn one tmux window per machine, each running the script. On some machines, you will need to enter a password for sudo. (If you don't have direct-to-root ssh, this may require changes to shoprun). See shoprun.
• Babysit all the shell sessions, hitting Enter and confirming as necessary. Make sure every session says COMPLETED SUCCESSFULLY at the end. If it didn't, troubleshoot.

Note that update.sh is written to be idempotent, i.e. it should be possible to rerun it as often as needed, and no changes should be made on the second run.