add queues to the adapter interface
johrstrom opened this issue · comments
Pretty straight forward: add queues to the adapter interface.
I think we can start off with a fairly simple interface for Queue to have a name and a list of accounts that can use the queue.
class Queue
attr_reader :name, :accounts
def initialize(name, accounts: [])
# ...
end
def to_s
name.to_s
end
endmodule OodCore
module Job
class Adapter
# return the list of Queues for this adapter. Default is an empty array.
def queues
[]
end
end┆Issue is synchronized with this Asana task by Unito
Just add that in Slurm you can associate records in Slurm database to be cluster + account + user + partition. I believe if sites use that feature you could say like sacctmgr list account withassoc where partition=<name> though I've never tested this as we don't use partition based associations. We mostly limit partition access using accounts via partition definition in slurm.conf which you can get with scontrol show partition and look at AllowAccounts but there is also DenyAccounts which could complicate things.
I don't think you'll want to select based on partition though? Rather, we want all of them and construct all the possibly queues (for the current user)
# sacctmgr -np list account format=account,partition,user withassoc where user=someuser
my_project|my_partition|someuser|
private_project|private_partition|someuser|
unrelated_project||| # <-- For some reason sacctmgr still insists on printing all the projects you aren't a member despite the filter...
and just create all the queues.
I was considering having a go at an implementation (if I can find the time); do we imagine this being added as a method onto the job Adapter (e.g. queue_info or some returning a list of queues)?
I'll just add that this way to limit access to partitions is one of many ways. At OSC we don't associate users or accounts to partitions via the database. We restrict access to partitions via AllowAccounts or DenyAccounts on the actual partition definition in slurm.conf. For CLI that is accessible via scontrol show partitions or with REST API:
export $(scontrol token)
curl -X GET -H "Content-Type: application/json" \
-H "X-SLURM-USER-NAME: $USER" \
-H "X-SLURM-USER-TOKEN: $SLURM_JWT" \
https://OMIT/slurm/v0.0.37/partitions
Would still need to get user's account associations from database which can also be done with REST API on newer versions of Slurm I believe.
I was considering having a go at an implementation (if I can find the time); do we imagine this being added as a method onto the job Adapter (e.g. queue_info or some returning a list of queues)?
I updated the comment. I think adapter#queues is intuitive enough. Of course then you'd have to add the Slurm's implementation for the same.
If the API is versioned we'd have to add the api version to the slurm config so folks can toggle back and forth, but that raises the question of what api versions would we support. We don't have a lot of support for multiple versions of schedulers (PBS I believe has a bug or 2 regarding the same. That is, the pbs adapter doesn't work for all versions of PBS).
Which is all to say - if we're going to use the API we should understand it's compatibility.
Only way to reliably check which versions of Slurm support which API version is checkout the code for Slurm from Github and look at OpenAPI spec. The official docs here: https://slurm.schedmd.com/rest_api.html , are only for the latest stable version which is 21.08 which we happen to run at OSC. When 22.05 comes out soon, the docs will be for that version and I believe the version of API will bump but Slurm tends to keep older API versions in newer Slurm versions so I'm pretty sure 22.05 will continue to have v0.0.37 even if they are adding functionality to v0.0.38 (or whatever next version is).
For example, our 21.08 if you query https://OMIT/openapi that will dump the spec and it appears like we have both v0.0.37 and v0.0.36 available to in 21.08.
Also I'll just add that getting the token requires scontrol token and you can specify a lifetime, I think max being 1 day. With Slurm exporters for Prometheus I worked on with UBCCR I added a sort of global variable in Go to keep track of last time token was refreshed to know when to generate new one so wasn't always executing scontrol token everytime exporter ran collection.
I've started this in auto-queues but I'm having difficulty with the association tuple because our systems don't really use that.
I can help provide example output if that helps?
sacctmgr -np show accounts withassoc format=account,partition,user
would produce output something like
project_123|||
project_123|partition_x|user_a|
project_123|partition_x|user_b|
project_123|partition_x|user_c|
project_123|partition_y|user_a|
project_123|partition_y|user_b|
project_123|partition_y|user_x|
project_124|||
project_124|partition_x|user_d|
project_124|partition_x|user_e|
project_124|partition_x|user_f|
So, it will be unique to each user, and would be some set of (project, partition) tuples basically.
It also includes partly empty lines (which have the fairshare association for the project itself), which I just ignore in my scripts that use this info.
My ruby is weak, but I was expecting something roughly equivalent to
for line in sacctmgr_output:
project, partition, user, _ = line.split('|')
if user == username:
project_partitions[project].append(partition)
# or the other way around, doesn't really matter
partition_projects[partition].append(project)
Right now we're querying like this. Do you also use partition in your tuple?
args = ['-nP', 'show', 'users', 'withassoc', 'format=account,cluster,qos', 'where', "user=#{user}"]
Sure, that works to if you ask for the partition
# sacctmgr -nP 'show' 'users' 'withassoc' 'format=account,partition,cluster,qos' 'where' "user=ohmanm"
c3se-staff-chair|chair|alvis|normal
c3se-staff|alvis|alvis|normal
OK - I'll add partition so it'll capture this 4 dimensional tuple account,partition,cluster,qos.
Is partition always singular here? qos column has multiple comma seperated values, I'm wondering if partition is the same. @treydock do you know?
It would be multiple lines as each (project+partition+cluster+user+qos) association is arbitrary.
I added myself to another partition on the same project here just to test;
# sacctmgr -nP 'show' 'users' 'withassoc' 'format=account,partition,cluster,qos' 'where' "user=ohmanm"
c3se-staff|chair|vera|normal
c3se-staff|vera|vera|normal
So, theoretically, it could be
project_a|partition_a|cluster_x|qos_a,qos_b
project_a|partition_b|cluster_x|qos_b
project_b|partition_a|cluster_x|qos_a
etc..
So.. if one wants to map this into a hierarchy, then to map it to
- cluster_projects - maps cluster to a list of projects
- project_partitions - maps a project to a list of partitions
- partition_qos -_ maps a partition to a list of QoS'es
if one desires to map this unto "data-option-for-xxx" type fields in the form.
Though, the direction of this mapping is arbitrary i think.
@johrstrom The QOS is not part of tuple in SLurm database, it's just an attribute on the cluster + account + user + (optional) partition association. The association can have multiple QOSes assigned to it but it's not what defines the association. An association can only have 1 of each of the tuple elements, cluster, account, user, optional partition.
Thanks! I've got the pull request now pulling all 4 dimensions, account,partition,cluster,qos and I guessed correctly that there's only 1 partition per record.
OK - I'll add partition so it'll capture this 4 dimensional tuple
account,partition,cluster,qos.Is partition always singular here?
qoscolumn has multiple comma seperated values, I'm wondering ifpartitionis the same. @treydock do you know?
you can ask for multiple partitions to a job if that's what you meant. the job will end up on whatever partition first have enough resources to fit it.
Thanks @akesandgren! It's really about those association records.
I was wondering if 1 assocication record could have multiple partitions like so (this is an edited example output from eariler):
# sacctmgr -nP 'show' 'users' 'withassoc' 'format=account,partition,cluster,qos' 'where' "user=ohmanm"
c3se-staff|chair,other_partition|vera|normal
c3se-staff|vera|vera|normal
or, if partition is 1:1 with these records and you'd get a new record for each partition.
# sacctmgr -nP 'show' 'users' 'withassoc' 'format=account,partition,cluster,qos' 'where' "user=ohmanm"
c3se-staff|chair|vera|normal
c3se-staff|other_partition|vera|normal
c3se-staff|vera|vera|normal
I've come to believe it's the latter.
Yep, sacctmgr produces one record per account/partition/cluster at least.
Thanks for the confirmation. I'm trying to get this into 2.1 now.