dashbitco / broadway

Concurrent and multi-stage data ingestion and data processing with Elixir

Home Page:https://elixir-broadway.org

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

How to stop a Broadway Kafka pipeline?

aravindanck opened this issue · comments

Team, I tried Broadway.stop(MODULE) in a broadway_kafka pipeline. But the pipeline gets restarted after a minute by the brod supervisor :brod_sup. The restart option is fixed to permanent in brod library (see ref) - could it be the cause? Is there a way I can start my broadway kafka producer with 'restart' arg set to temporary/transient?

:supervisor: {:local, :brod_sup}
    :started: [
  pid: #PID<0.1456.0>,
  id: TestBroadway.Broadway.Producer_0.Client,
  mfargs: {:brod_client, :start_link,
   [[localhost: 9092], TestBroadway.Broadway.Producer_0.Client, []]},
  restart_type: {:permanent, 10},
  shutdown: 5000,
  child_type: :worker
]

Reference: https://github.com/kafka4beam/brod/blob/master/src/brod_sup.erl#L164

Tracking the issue in Broadway Kafka issues - dashbitco/broadway_kafka#86