Info Now also works with SLURM < 21.08
For templating differences between slurm>21.08 and slurm<21.08 see templating guide
- message delivery to: msteams, telegram, e-mail
- gathering of job statistics
- generating hints for users on how to tune their job scripts (see examples below)
- templateable messages (readme)
- message throttling
Goslmailer (GoSlurmMailer) is a drop-in replacement MailProg for slurm.
With goslmailer configured as as the slurm mailer,
MailProg = /usr/bin/goslmailer
it provides users with the ability to specify a comma-separated list of receivers [connector:]target
in the sbatch --mail-user
switch to select where the messages will be sent out (similar to URI scheme).
e.g.
sbatch --mail-type=ALL --mail-user="mailto:useremailA,msteams:usernameB,telegram:NNNNNNN,usernameC"
To support future additional receiver schemes, a connector package has to be developed and its configuration block present in configuration file.
If you would like to contribute to this project by developing a new connector, here is a heavily annotated connector boilerplate (fully functional) to help you get started.
git clone https://github.com/CLIP-HPC/goslmailer.git
make test
make build
make install
Prerequisites:
- generated RSA keypair (passwordless) (
ssh-keygen -t rsa
) ssh $USER@localhost
must work without password
Known caveats:
- redhat/centos: must have lsb_release binary installed, package:
redhat-lsb-core
- ubuntu 22:
set enable-bracketed-paste off
present in~/.inputrc
- maybe/maybe not, depends if you see failed tests:
export TERM=dumb
in~/.bashrc
:)
# downloads endly binary and runs endly tests
make
- place binary to the path of your liking
- place goslmailer.conf here:
/etc/slurm/goslmailer.conf
(default path)- OR: anywhere else, but then run the binary with
GOSLMAILER_CONF=/path/to/gosl.conf
in environment
- OR: anywhere else, but then run the binary with
- point slurm
MailProg
to the binary
- place binary to the path of your liking
- place gobler.conf to the path of your liking
- start the service (with -c switch pointing to config file)
- place binary to the path of your liking
- place tgslurmbot.conf to the path of your liking
- config file has the same format as goslmailer, so you can use the same one (other connectors configs are not needed)
- start the service (with -c switch pointing to config file)
- msteams webhook --mail-user=
msteams:
userid - telegram bot --mail-user=
telegram:
chatId - mailto --mail-user=
mailto:
email-addr
See each connector details below...
In high-throughput clusters or in situations where job/message spikes are common, it might not be advisable to try to send all of the incoming messages as they arrive. For these environments goslmailer can be configured to spool messages from certain connectors on disk, to be later processed by the gobler service.
gobler is a daemon program that can be configured to monitor specific spool directories for messages, process them and send out using the same connectors as goslmailer.
On startup, gobler reads its config file and spins-up a connector monitor
for each configured spool directory.
connector monitor
in turn spins up 3 goroutines: monitor
, picker
and numSenders
x sender
.
-
monitor :
- every
monitorT
seconds (or milliseconds) scans thespoolDir
for new messages and sends them to the picker
- every
-
picker :
- on receipt of new messages performs trimming of excessive messages, limiting the number of users messages in the system to
maxMsgPU
- every
pickerT
seconds (or milliseconds) picks the next message to be delivered and sends it to the sender (ordering by time of arrival)
- on receipt of new messages performs trimming of excessive messages, limiting the number of users messages in the system to
-
sender :
numSenders
goroutines are waiting for messages from the picker and try to deliver them. In case of failure, messages are returned to the picker for a later retry
Specifies which receiver scheme is the default one, in case when user didn't specify --mail-user
and slurm sent a bare username.
"defaultconnector": "msteams"
Mailto covers for original slurm e-mail sending functionality, plus a little bit more. With connector parameters, you can:
- specify your e-mail client (ex slurm:
MailProg
, e.g. /usr/bin/mutt) - template mail client command line (e.g. custom subject line)
- template message body
- allowList the recipients
To make sure that mutt properly renders the HTML email, add following lines to /etc/Muttrc.local
# Local configuration for Mutt.
set content_type="text/html"
See annotated configuration example
Sends 1on1 or group chat messages about jobs via telegram messenger app
Prerequisites for the telegram connector:
- a telegram bot must be created and
- the bot daemon service tgslumbot must be running.
Site admins can create a telegram bot by messaging botfather.
Once the bot is created, you will receive a bot token
. Place the bot token
in the goslmailer/gobler config file in the telegram
connector section (see example below).
Start the tgslurmbot binary that serves as the bot.
When the chat/group chat with the bot is initiated and/or the bot receives a /start
command, he will reply with a chat-specific --mail-user=telegram:nnn
message which the user can use in his slurm job scripts to get the job messages.
See annotated configuration example
Sends a message to a preconfigured ms teams channel webhook.
Since MS Teams does not provide with the option to send messages to users directly, only to channel webhooks, we have devised a way using MS Power Automate framework to pick up messages from this one configured sink channel and deliver them via private 1on1 chats to the recipient user.
Users listed in the --mail-user=msteams:userA,msteams:userB
will be sent as adaptive card mention entity.
A MS Power Automate workflow monitors the configured sink channel, parses the received adaptive card jsons, locates the mention
entity and delivers to it the copy of the message via private chat.
See annotated configuration example
- using adaptive card schema version 1.5 doesn't work with our adaptive card, check if some element changed in designer
- tested: 1.0, 1.2 - work