TFX Exit Handler for Slack Notifications (or Twilio et al.)
hanneshapke opened this issue · comments
Proposal
Since TFX 1.4, TFX supports exit handlers if the pipeline runs on Vertex (and KFP?). At @digits, we created an exit handler component to notify us about the exit state and any messages from the pipelines (failed or succeeded). If anyone finds this setup interesting, we can open source this component to TFX Addons.
This could also work as an example implementation for exit handlers and it could be used to trigger other pipelines.
Expected Component Behavior
The exit handler component would take four parameters (no input artifacts):
final_status: tfx.dsl.components.Parameter[str],
slack_token: tfx.dsl.components.Parameter[str],
slack_channel_id: tfx.dsl.components.Parameter[str],
on_failure_only: tfx.dsl.components.Parameter[int] = 0,
In addition to the final_status from the pipeline orchestrator, we are currently consuming the slack tokens and channel id. We added an additional flag to only alert on failures for frequently running pipelines.
The component could support other communication ways (e.g. Twilio SMS) too.
Example Output
Below you can find a screenshot of the current, internal implementation and its output to Slack.
Pipeline Success Message
Pipeline Failure Message
Visualization in Google Cloud Vertex Pipelines
Specifications
- TFX Version: >= 1.4.0
- Orchestrator Platforms: Vertex (aka KubeflowRunnerV2)
Out of Scope
- Initial support for Slack only (unless someone wants to implement other communication platforms)
- Decryption of credentials isn't supported through this proposal
Submitted to the Google team for review
Reviewed and approved by the Google team
Closing this issue since we just merged the initial component to main, cc: @rcrowe-google