microsoft / pai

Resource scheduling and cluster management for AI

Home Page:https://openpai.readthedocs.io

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

how to set job-status-change-notification, can not receive job notification

HaoLiuHust opened this issue · comments

Organization Name:

Short summary about the issue/question: how to set job-status-change-notification

Brief what process you are following:
I am setting email notification for better experience, for now I can receive cluster warning about disk usage, but I can not receive job status change eventhough I have enable it.
image

How to reproduce it:

OpenPAI Environment:

  • OpenPAI version: v1.8.0
  • Cloud provider or hardware configuration:
  • OS (e.g. from /etc/os-release):
  • Kernel (e.g. uname -a):
  • Hardware (e.g. core number, memory size, storage size, GPU type etc.):
  • Others:

Anything else we need to know:

set flags in job config, I can receive running notification, is there a way without this set

 jobStatusChangeNotification:
    running: true
    succeeded: true
    stopped: true
    failed: true
    retried: false

@suiguoxin Any idea?

@HaoLiuHust Yes, this setting in job config is necessary: https://github.com/microsoft/pai/blob/master/docs/manual/cluster-user/how-to-use-advanced-job-settings.md#job-status-change-notification

OK, thanks for your reply, if it is by design, it make sense...Just from manual, maybe someone will have the same misunderstand