slurm-stats
extract slurm job stats
It uses slurm's sacct to extract job stats of finished jobs. It stores the last timestamp in a file called "lasttimestamp". This means the tool can fetch all jobs since the last time stamp. If there is no stored timestime it defaults to fetching all jobs since midnight.
Note: the default OpenHPC install doesn't enable the accounting service. It appears this must be enabled before --starttime and --endtime options work as expected.
Install
Requires python3:
python3 -m venv /path/to/venv
/path/to/venv/bin/pip install https://github.com/JohnGarbutt/slurm-stats
To run the script, try this cron ready example:
rm -f lasttimestamp # clear any old state, default to today's job
TZ=UTC /path/to/venv/bin/sacct.py >>finished_jobs.json
For example, you would expect output a bit like this:
tail -n2 finished_jobs.json
{"JobID": "20", "JobIDRaw": "20", "Cluster": "linux", "Partition": "normal", "Account": "", "Group": "centos", "GID": 1000, "User": "centos", "UID": 1000, "Submit": "2020-06-23T12:43:17", "Eligible": "2020-06-23T12:43:17", "Start": "2020-06-23T12:43:21", "End": "2020-06-23T12:43:23", "Elapsed": "00:00:02", "ExitCode": "1:0", "State": "FAILED", "NNodes": 1, "NCPUS": 1, "ReqCPUS": 1, "ReqMem": "500Mc", "ReqGRES": "", "ReqTRES": "bb/datawarp=2800G,billing=1,cpu=1,mem=500M,node=1", "Timelimit": "5-00:00:00", "NodeList": "c1", "JobName": "use-perjob.sh", "AllNodes": ["c1"]}
{"JobID": "21", "JobIDRaw": "21", "Cluster": "linux", "Partition": "normal", "Account": "", "Group": "centos", "GID": 1000, "User": "centos", "UID": 1000, "Submit": "2020-06-23T12:45:30", "Eligible": "2020-06-23T12:45:30", "Start": "2020-06-23T12:45:33", "End": "2020-06-23T12:45:35", "Elapsed": "00:00:02", "ExitCode": "1:0", "State": "FAILED", "NNodes": 1, "NCPUS": 1, "ReqCPUS": 1, "ReqMem": "500Mc", "ReqGRES": "", "ReqTRES": "bb/datawarp=2800G,billing=1,cpu=1,mem=500M,node=1", "Timelimit": "5-00:00:00", "NodeList": "c1", "JobName": "use-perjob.sh", "AllNodes": ["c1"]}
OpenDistro Setup
Best to see the data via Elasticsearch. We used OpenDistro.
Get started here: https://opendistro.github.io/for-elasticsearch-docs/docs/install/docker/
Install Filebeat
Be sure to use the oss filebeat, e.g. https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-oss-7.7.0-x86_64.rpm
Be sure to do add something like this to filebeat config:
setup.ilm.enabled: false
xpack.monitoring.enabled: false
output.elasticsearch:
hosts: ["localhost:9200"]
protocol: "https"
ssl.verification_mode: none
username: "admin"
password: "admin"
Then execute the setup:
sudo filebeat modules enable system
sudo filebeat setup -e --dashboards --pipelines --template
sudo systemctl start filebeat
To parse the files as json, add this:
- type: log
json.add_error_key: true
paths:
- '/mnt/ohpc/centos/*.json'
fields:
event.kind: event
fields_under_root: true
processors:
- timestamp:
field: json.End
layouts:
- '2006-01-02T15:04:05'
test:
- '2020-06-17T10:17:48'
- timestamp:
target_field: 'event.end'
field: json.End
layouts:
- '2006-01-02T15:04:05'
test:
- '2020-06-17T10:17:48'
- timestamp:
target_field: 'event.start'
field: json.Start
layouts:
- '2006-01-02T15:04:05'
test:
- '2020-06-17T10:17:48'
- convert:
fields:
- {from: "json.NNodes", type: "integer"}
- {from: "json.NCPUS", type: "integer"}
- {from: "json.ElapsedRaw", type: "integer"}