Giters

reactphp / socket

Async, streaming plaintext TCP/IP and secure TLS socket server and client connections for ReactPHP.

Home Page:https://reactphp.org/socket/

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Writes not being sent back to client

sburkett opened this issue · comments

commented

I am probably missing the obvious here - sorry, I'm still learning the framework :)

Our customer has a persistent (always-on) socket connection with us. The use-case is, we receive data from them, and we send back basically an acknowledgement (ACK) that we got the data. Receiving the data works great, but this doesn't seem to send a test message back to the client.

    $server = new \React\Socket\TcpServer($this->host . ':' . $this->port, $loop, array(
      'so_reuseport' => true
    ));
    ....
    $server->on('connection', function(ConnectionInterface $connection) use($loop, $log) {
        ...
        $connection->on('data', function ($receivedData) use ($log, $connection)
        {
           $message = "Testing\n";
           $connection->write($message);
        });
     });

Thanks for any guidance!

commented

Hi @sburkett, welcome to ReactPHP 🎉

The code snippet looks about right, are you sure the client does not receive anything? A trace (wireshark etc.) could help here. You can also take a look at one of our examples: https://github.com/reactphp/socket/blob/master/examples/01-echo-server.php

A common problem is if your code still includes blocking calls. The write() call only schedules the data to be written and has to wait for the EventLoop, so if you block your code before the EventLoop gets a chance to actually write, this data will not be sent until after your blocking code returns.

I hope this helps 👍

I believe this has been answered, so I'm closing this for now. Please come back with more details if this problem persists and we can reopen this 👍

commented

Thanks for the reply! The client isn't receiving anything, and nothing is blocking as far as I can tell. The write call is the last line in the data event handler.

commented

Also, I've dumped the data that is supposed to go out, and it looks solid. What is interesting is that I have a test line at the top of the data event handler that seems to work when testing with a local port. The only difference is that the client connection comes through a VPN.

commented

Can you test this with the example linked above? Might as well be a firewall / VPN setup issue, so if this example doesn't work either, I would suggest taking a look at a Wireshark dump.

commented

I'm wondering if, for some reason, whatever we are trying to send outbound to them is getting gobbled up by their firewall. Like, for some reason, the traffic appears to be coming from our public IP rather than our private/VPN IP. Not sure.

commented

I'm pretty sure this is on their end at this point, so I'll continue to investigate that. Thanks for the guidance!

commented

Ok, well that theory was short-lived. Sorry lol.

Question: is there a way to flush the buffer after the write? In other words, like the end call, which flushes the buffer then closes the socket (though I don't want the latter).

commented

Another followup question: Is there some sort of DNS resolve going on? Obviously, it won't be able to resolve a private IP. We have the private IP in our hosts file, but thought I would ask anyway.

commented

Some sort of timeout or something going on here. You can see below that we get the connection, receive the data, but when we send the ACK back, it basically sits there for ~30 seconds or so, then closes the connection.

Waiting for connections ...
Connection from: tcp://10.243.4.137:58426
Received data: 1030 bytes
Stored: 7f2641fe-ec19-48be-8596-c6463ab55108.hl7
Created ACK: 7f2641fe-ec19-48be-8596-c6463ab55108.hl7.ack
-------- CONNECTION CLOSED ----------
commented

Ok, here is a snippet of a tcpdump, with the IPs/hostnames cleansed.

13:41:43.063130 IP REMOTEHOST.33262 > OUR-PVT-IP.28137: Flags [P.], seq 0:1030, ack 1, win 115, options [nop,nop,TS val 1721548981 ecr 236858308], length 1030
13:41:43.119583 IP REMOTEHOST.33262 > OUR-PVT-IP.28137: Flags [.], ack 87, win 115, options [nop,nop,TS val 1721549038 ecr 236858366], length 0

You can see that received 1030 bytes from them, which is accurate, but I'm not quite sure how to interpret the second line, but that's when it hangs.

commented

Ok, so in the second line, the Flag dot . means that is a TCP ACK (not my application's ACK). And it has a length of zero. TCP ACKs should have a header, etc, correct?

commented

Sorry for all of the followups on this. Lots of pressure from our customer to get this working correctly. I do very much appreciate the guidance!!!

tcpdump is showing their data arriving, but not seeing any packets for our response trying to leave over the interface. Disabled the firewall on our end just to see if that was it, but no dice there.

I also tried calling end to see if it would flush the buffer and close the connection (again, don't want the latter, but figured it would be a good test), but I got the same result.

Arrrrrghhhh! :)

commented

Regarding my DNS comment above, do you think my issue may be related to this one?

#144

commented

This sounds frustrating :-) And it still sounds more like a network / VPN issue rather than a ReactPHP issue:

There's no way to explicitly flush the buffer because it would violate the non-blocking guarantees. The loop is really reliable in taking care of this and as you've found out with your end() tests, this should usually not affect program execution anyway, unless you're explicitly blocking the loop.

DNS resolution should be unrelated. It sounds like you're building a server component, i.e. a remote client connects to your server. DNS resolution will only be used if you're building a client that has to resolve a hostname before connecting to a remote server. For the reference: The related DNS issue has been fixed in the meantime, ReactPHP correctly respects the /etc/hosts and /etc/resolv.conf files.

The tcpdump LGTM, but it's hard to tell just from the snippet you've posted.

In order to further isolate the problem you're seeing, I would suggest using some other network programs to see this shouldn't be related to ReactPHP. For instance, you can run netcat as a simple server to just dump some predefined data and see if your client receives this. You may also try a simple PHP-based server:

$server = stream_socket_server('tcp://0.0.0.0:8000');
$client = stream_socket_accept($server);

// read 10 bytes and send back same data
$data = fread($client, 10);
fwrite($client, $data);

Keep us posted if you find anything 👍

commented

Hey @clue thanks for the response on all my ramblings above :) After some more debugging I'm sort of still in the same place. Let me share a bit more with you, as I may just simply be too close to the trees at this point, and not able to see the forest.

Here is my current server-side code harness, and the output I'm seeing when I fire it up and receive something from our customer. Maybe you can see something I haven't thus far.

public function foo()
{
    $this->host = $this->argument('host');
    $this->port = $this->argument('port');

    $loop = \React\EventLoop\Factory::create();

    $log = new \React\Stream\WritableResourceStream(STDOUT, $loop);

    $server = new \React\Socket\TcpServer($this->host . ':' . $this->port, $loop, array(
      'so_reuseport' => true,
    ));

    $log->write("Server started at: " . now() . PHP_EOL . PHP_EOL);
    $log->write("Waiting for connections ..." . PHP_EOL);

    $server->on('connection', function(ConnectionInterface $connection) use($loop, $log) {

        $log->write('Connection from: '.$connection->getRemoteAddress() . PHP_EOL);

        $connection->on('close', function () use ($log) {
            $log->write('-------- CONNECTION CLOSED ----------' . PHP_EOL);
        });

        $connection->on('error', function (Exception $e) {
            $log-write('error: ' . $e->getMessage() . PHP_EOL);
        });

        $connection->on('data', function ($chunk) use ($log, $connection)
        {
            $log->write('Received data: ' . strlen($chunk) . ' bytes' . PHP_EOL);
            $log->write('Sending ACK ...' . PHP_EOL);

            $connection->write("TESTING...!\n");

            $log->write('Write complete ...' . PHP_EOL);
       });
    });

    $loop->run();
}

Here is the output .. it hangs after the "Write complete", then I just CTRL-C'd it:

Waiting for connections ...
Connection from: tcp://10.243.4.137:52242
Received data: 1030 bytes
Sending ACK ...
Write complete ...
^C

It gets to the "write complete", which indicates that the write call scheduled the data to be transmitted. Then, as far as I can tell, the loop should continue, correct? At any rate, as you can see above, nothing else happens, so I just stopped the server. If I didn't, the remote end finally times out and disconnects the socket connection, then it received data again (since no acknowledgment was issued to the sender/client), and the cycle is repeated. According to tcpdump -X tcp port xxxx, we see the inbound data, but no outbound packets from us to the remote client.

I would expect to at least see the outbound packet of the "test" reply going out to the interface for routing, but I don't (even with the firewall temporarily disabled, to rule out a FW configuration issue). If I did see an outbound packet, but didn't get a TCP ack back from the remote host, I would be more inclined to lean towards it being an issue on their end.

The output of tcpdump is interesting. It receives the data from the remote host, but, check this out:

$ tcpdump -i any tcp port 28137

00:34:47.834815 IP REMOTEHOST.45902 > OURSERVER.28137: Flags [.], ack 1875970229, win 115, options [nop,nop,TS val 1847133752 ecr 362443082], length 0
00:34:47.836693 IP REMOTEHOST.45902 > OURSERVER.28137: Flags [P.], seq 0:1030, ack 1, win 115, options [nop,nop,TS val 1847133754 ecr 362443082], length 1030
00:34:47.893334 IP REMOTEHOST.45902 > OURSERVER.28137: Flags [.], ack 97, win 115, options [nop,nop,TS val 1847133811 ecr 362443140], length 0

HANGS until CTRL-C on the server ... then continues on...

00:34:58.131777 IP REMOTEHOST.45902 > OURSERVER.28137: Flags [F.], seq 1030, ack 98, win 115, options [nop,nop,TS val 1847144049 ecr 362453376], length 0
00:34:58.133848 IP REMOTEHOST.46036 > OURSERVER.28137: Flags [S], seq 2562621813, win 14600, options [mss 1200,sackOK,TS val 1847144051 ecr 0,nop,wscale 7], length 0
00:34:59.187426 IP REMOTEHOST.46052 > OURSERVER.28137: Flags [S], seq 3923279333, win 14600, options [mss 1200,sackOK,TS val 1847145105 ecr 0,nop,wscale 7], length 0

The tcpdump output pauses where you see the "HANGS" line above, until I CTRL-C the server, at which point, it continues on with normal traffic on the port. So something seems to be blocking there, just not sure what it could be.

If we aren't seeing outbound packets via tcpdump, something is getting in the way of that. Just don't really know what it could be.

If I change the host/port info and test it locally with a simple test client, it works great, but again, that isn't going across the VPN. The server code currently listens on the VPN private IP, (not our public IP) which the remote end connects to over the VPN. But obviously, since it works locally on the same host, the write is "working", but for some reason, it isn't getting routed out to the interface according to tcpdump.

I am truly perplexed on this one, Not really sure what could be causing this issue, but it seems to me to be on our end somewhere.

commented

This is very interesting. I added some debug lines to the simple server code you posted above and the fwrite returned the proper # of bytes that were "written" to the stream. Yet, I don't see anything in tcpdump that would indicate that an outbound packet was sent across the wire.

01:47:17.841364 IP REMOTEHOST.53230 > OURSERVER.28137: Flags [S], seq 3084156702, win 14600, options [mss 1200,sackOK,TS val 1851483759 ecr 0,nop,wscale 7], length 0
01:47:18.899436 IP REMOTEHOST.53242 > OURSERVER.28137: Flags [S], seq 54137162, win 14600, options [mss 1200,sackOK,TS val 1851484817 ecr 0,nop,wscale 7], length 0
...
01:47:18.952300 IP REMOTEHOST.53242 > OURSERVER.28137: Flags [.], ack 3953121818, win 115, options [nop,nop,TS val 1851484870 ecr 366794199], length 0
01:47:18.954718 IP REMOTEHOST.53242 > OURSERVER.28137: Flags [P.], seq 0:907, ack 1, win 115, options [nop,nop,TS val 1851484873 ecr 366794199], length 907
01:47:19.007542 IP REMOTEHOST.53242 > OURSERVER.28137: Flags [.], ack 79, win 115, options [nop,nop,TS val 1851484925 ecr 366794255], length 0
01:47:19.010914 IP REMOTEHOST.53242 > OURSERVER.28137: Flags [F.], seq 907, ack 80, win 115, options [nop,nop,TS val 1851484929 ecr 366794255], length 0
...
01:47:19.013571 IP REMOTEHOST.53246 > OURSERVER.28137: Flags [S], seq 379058127, win 14600, options [mss 1200,sackOK,TS val 1851484931 ecr 0,nop,wscale 7], length 0
01:47:20.067070 IP REMOTEHOST.53264 > OURSERVER.28137: Flags [S], seq 4148579442, win 14600, options [mss 1200,sackOK,TS val 1851485985 ecr 0,nop,wscale 7], length 0
commented

I don't see anything obviously wrong in your server code. Have you checked your connectivity without the ReactPHP server? For example, you can start a simple netcat server like this:

$ nc -l 0.0.0.0 28137

Then you can start your tcpdump and let the client connect to this server. Everything you type into the netcat window will be sent back to the client.

Your tcpdump doesn't appear to be complete and otherwise shows only incoming data and not even outgoing ACKS afaict? Here's an example output showing just the threeway handshake in my test and simple client ping message with a server pong message:

$ sudo tcpdump tcp port 28137 -i lo
12:02:59.149618 IP me-in.52500 > me-in.28137: Flags [S], seq 2086518227, win 65495, options [mss 65495,sackOK,TS val 3000968489 ecr 0,nop,wscale 7], length 0
12:02:59.149652 IP me-in.28137 > me-in.52500: Flags [S.], seq 856390461, ack 2086518228, win 65483, options [mss 65495,sackOK,TS val 3000968489 ecr 3000968489,nop,wscale 7], length 0
12:02:59.149682 IP me-in.52500 > me-in.28137: Flags [.], ack 1, win 512, options [nop,nop,TS val 3000968489 ecr 3000968489], length 0

12:06:45.623971 IP me-in.52500 > me-in.28137: Flags [P.], seq 1:7, ack 1, win 512, options [nop,nop,TS val 3001194963 ecr 3000968489], length 6                                                  
12:06:45.624004 IP me-in.28137 > me-in.52500: Flags [.], ack 7, win 512, options [nop,nop,TS val 3001194963 ecr 3001194963], length 0                                                            
12:06:48.164326 IP me-in.28137 > me-in.52500: Flags [P.], seq 1:6, ack 7, win 512, options [nop,nop,TS val 3001197503 ecr 3001194963], length 5                                                  
12:06:48.164365 IP me-in.52500 > me-in.28137: Flags [.], ack 6, win 512, options [nop,nop,TS val 3001197503 ecr 3001197503], length 0
commented

Well, we have connectivity, for sure. At least as far as receiving data from the remote host.

Very weird. I wonder what could be causing outbound traffic to just die in a fire? As I mentioned earlier, we're currently testing without the firewall enabled, just to rule that out.

commented

Turned out to be a stray iptables rule on our end. Whew! Sorry for all the back and forth, but hopefully this will all help someone else one day!

commented

@sburkett Happy to hear you've got this sorted out and it only seems to boil down to a network misconfiguration and not a bug in any software component! 🎉