AMQP Sink does not recover after RabbitMQ server restart

[Vasily Kirichenko]

Run the following code

let system = ActorSystem.Create("test", ConfigurationFactory.Default())
    let mat = ActorMaterializer.Create(system, ActorMaterializerSettings.Create(system).WithSupervisionStrategy(Deciders.ResumingDecider))
    
    let connSettings = 
        AmqpConnectionDetails
            .Create("localhost", 5672)
            .WithAutomaticRecoveryEnabled(true)
            .WithNetworkRecoveryInterval(TimeSpan.FromSeconds 1.0)
            
    let queueName = "my-queue"
    let queueDeclaration = QueueDeclaration.Create(queueName).WithDurable(true).WithAutoDelete(false)
    let sink = AmqpSink.CreateSimple(AmqpSinkSettings.Create(connSettings).WithRoutingKey("my-queue").WithDeclarations(queueDeclaration))
    
    Source
        .From(seq { 1..1_000_000 })
        .Select(fun x -> x |> string |> ByteString.FromString)
        .To(sink)
        .Run(mat)
    |> ignore
    
    system.WhenTerminated.Wait()

Everything is OK, messages are being queued at ~20K/second rate.
Now restart RabbitMQ server. The following error is in the log

[ERROR][17-Jan-18 19:06:48][Thread 0012][[akka://test/user/StreamSupervisor-0/Flow-0-1-AmqpSink#1922506265]] Error in stage [AmqpSink]: Unable to write data to the transport connection: An existing connection was forcibly closed by the remote host.
Cause: System.IO.IOException: Unable to write data to the transport connection: An existing connection was forcibly closed by the remote host. ---> System.Net.Sockets.SocketException: An existing connection was forcibly closed by the remote host
   at System.Net.Sockets.NetworkStream.Write(Byte[] buffer, Int32 offset, Int32 size)
   --- End of inner exception stack trace ---
   at System.Net.Sockets.NetworkStream.Write(Byte[] buffer, Int32 offset, Int32 size)
   at System.IO.BufferedStream.Flush()
   at RabbitMQ.Client.Impl.SocketFrameHandler.WriteFrameSet(IList`1 frames)
   at RabbitMQ.Client.Impl.Command.TransmitAsFrameSet(Int32 channelNumber, Connection connection)
   at RabbitMQ.Client.Impl.SessionBase.Transmit(Command cmd)
   at RabbitMQ.Client.Impl.ModelBase.ModelSend(MethodBase method, ContentHeaderBase header, Byte[] body)
   at RabbitMQ.Client.Impl.ModelBase.BasicPublish(String exchange, String routingKey, Boolean mandatory, IBasicProperties basicProperties, Byte[] body)
   at Akka.Streams.Amqp.AmqpSinkStage.AmqpSinkStageLogic.<.ctor>b__3_0()
   at Akka.Streams.Implementation.Fusing.GraphInterpreter.ProcessEvent(Connection connection)
   at Akka.Streams.Implementation.Fusing.GraphInterpreter.Execute(Int32 eventLimit)

and the messages are not queued anymore.

I thought that WithAutomaticRecoveryEnabled(true) reconnects automatically, and WithSupervisionStrategy(Deciders.ResumingDecider) resumes the failing stream stage. As a result, I expected messages to continue being queued after server restart (and a single message to be lost).

[Boban Pavloski]

you need to use RestartSink ,if you want your sink to restart when the connection drops, so the sink will be recreated with new connection to rabbit. Also u can combine them with KillSwitch so you can also gracefully stop.

something like this:

var source = Source.Queue<string>(0, OverflowStrategy.Backpressure);
var restartSink = RestartSink.WithBackoff(() =>
                {
                    Console.WriteLine("Restarting AMQP Sink.");
                    var amqpSink = AmqpSink.CreateSimple(
                        AmqpSinkSettings.Create(connectionSettings)
                            .WithRoutingKey(queueName)
                            .WithDeclarations(queueDeclaration,bindingDeclaration,exchangeDeclaration));
                    return amqpSink;
                }, TimeSpan.FromSeconds(1), TimeSpan.FromSeconds(5), 0.2);

var tupple = source
                    .Select(ByteString.FromString)
                    .ViaMaterialized(KillSwitches.Single<ByteString>(),Keep.Both)
                    .ToMaterialized(restartSink,Keep.Left)
                    .Run(mat);

                var sinkKillSwitch = tupple.Item2;
                var queueSource = tupple.Item1;

                sinkKillSwitch.Shutdown();

[Vasily Kirichenko]

@bobanco Excellent, thanks! :) I completely forgot about RestartSink however read about it a week ago.

It works, but it uncover a bug in PostStop:

[ERROR][17-Jan-18 19:46:30][Thread 0003][[akka://test/user/StreamSupervisor-0/Flow-0-0-unknown-operation#1059276845]] Error during PostStop in [AmqpSink]
Cause: System.NullReferenceException: Object reference not set to an instance of an object.
   at Akka.Streams.Amqp.AmqpSinkStage.AmqpSinkStageLogic.PostStop()
   at Akka.Streams.Implementation.Fusing.GraphInterpreter.FinalizeStage(GraphStageLogic logic)

[Vasily Kirichenko]

The NRE is fixed in #23