Enable testing of backpressure from TCP connections

[Mostafa Elhemali]

There are certain error conditions we want to test out that only happen when the TCP connection stalls and doesn't write right away (returns Poll::Pending on write). Would be great if Turmoil allowed for that - so in the simple code below, the host b just waited forever instead of the simulation panicking with socket buffer full.

use std::{
    net::{IpAddr, Ipv4Addr},
    time::Duration,
};
use tokio::{io::AsyncWriteExt, time::sleep};
use turmoil::{
    net::{TcpListener, TcpStream},
    Result,
};

#[test]
fn want_backpressure() -> Result {
    let mut sim = turmoil::Builder::new().build();
    sim.host("b", || async {
        let listener = TcpListener::bind((IpAddr::from(Ipv4Addr::UNSPECIFIED), 9876))
            .await
            .expect("Bind to local host");
        let (mut conn, _addr) = listener.accept().await.expect("Accept conn");
        for _ in 0..10000 {
            conn.write_all(b"message").await.expect("Write");
            conn.flush().await.expect("flush");
        }
        Ok(())
    });
    sim.client("a", async move {
        let _conn = TcpStream::connect("b:9876").await.expect("Open to b");
        sleep(Duration::from_millis(100)).await;
        Ok(())
    });
    sim.run()
}

[James Mayclin]

I also just ran into this. I think this is related to turmoil skipping flow-control logic?

turmoil/src/envelope.rs

Lines 26 to 28 in 6728fbe

 /// We implement just enough to ensure fidelity and provide knobs for real world 
 /// scenarios, but we skip a ton of complexity (e.g. checksums, flow control, 
 /// etc) because said complexity isn't useful in tests. 

Is there any appetite to add flow-control emulation to turmoil?

In my use case I am trying to use turmoil for a test case that sends a large amount (~100 Gb) of data, but was running into this panic when trying to reproduce a different issue that I was running into 😄 .

[Brett McChesney]

I think there is an appetite for back pressure, but I'm not sure we need to implement a flow control message protocol. One idea for this is that we just poll on write when the egress tcp buffer fills (doesn't exist today) instead of putting packets directly on the network.