Issues using Broadway with DynamicSupervisor

[Thomas Brewer]

I am running into a preplexing issue while using Broadway with a DynamicSupervisor and Registry. If I try to initialize more than one pipeline the second pipeline doesn't appear to initialize.

I added some logging to the start_supervisor function in the Broadway.Topology.

Logger.debug("-starting supervior-------------------------------")
IO.inspect(children: children)
result = Supervisor.start_link(children, supervisor_opts)

Logger.debug("-result-------------------> #{inspect(result)} <-----------")
result

When I add this I am seeing the first debug message get logged and the children are output but the second debug message is never logged.

I compared the list of children for the same pipeline when it is the second one started and when it is the only one and they are identical. When it is the only pipeline started it works fine.

Here is gist with the logs when I try to initialize 2 pipelines.

This is how I am starting the pipelines:

def start_destination_pipeline(%Destination{} = destination) do
  case ER.Destinations.Pipeline.factory(destination) do
    nil ->
      Logger.info(
        "#{__MODULE__}.start_destination_pipeline(#{inspect(destination)} not starting pipline."
      )

    pipeline ->
      Logger.debug(
        "#{__MODULE__}.start_destination_pipeline(#{inspect(destination)} starting pipeline. pipeline=#{inspect(pipeline)}"
      )

      result =
        DynamicSupervisor.start_child(
          ER.DynamicSupervisor,
          {pipeline, [destination: destination]}
        )

      Logger.debug("#{__MODULE__}.start_destination_pipeline with result=#{inspect(result)}")
  end
end

I have defined a process_name function for the pipeline:

@impl Broadway
def process_name({:via, module, {registry, name}}, base_name) do
  name = {:via, module, {registry, "#{base_name}.Broadway.#{name}"}}
  Logger.debug("#{__MODULE__}.process_name with name=#{inspect(name)}")
  name
end

I have also overriden the child_spec function:

def child_spec(arg) do
  destination = arg[:destination]

  child_spec = %{
    id: "#{__MODULE__}:base:#{name(destination.id)}",
    start: {__MODULE__, :start_link, [arg]},
    shutdown: :infinity
  }

  Logger.debug("#{__MODULE__}.child_spec with child_spec=#{inspect(child_spec)}")

  child_spec
end

I am starting to lean in the direction of this being a bug in Broadway but am open to this being something I am doing wrong. The frustrating thing is I can't get anything to return from Supervisor.start_link in Broadway.Topology.start_supervisor or at least I can't get it to log it out. So it makes tracing the issue down kind of difficult. I greatly appreciate any feedback you may have.