CORK breakage

[Marek Knápek]

Hi, at out company we updated libmicrohttpd and suddenly, our application started sometimes (often) to work badly.

I performed binary search of libmicrohttpd releases to find which version broke our application and found out that v0.9.66 works fine, but v0.9.67 breaks our application. The newest build v0.9.71 also doesn't work for us.

Then I performed binary search among all commits between the two releases and found out that commit 6a9e9a5 from 2019-07-25 20:10:39+02:00 works fine, but commit 8567afc from 2019-07-29 15:39:22+00:00 breaks our application. Unfortunately, I'm not able to compile libmicrohttpd with changes between these two commit as the build is broken.

It seems the changes that causes us troubles are related to CORK and/or Nagle's algorithm.

It is possible the bug is in libmicrohttpd, could you please investigate the mentioned changes? It is also possible, that the bug is in our application and we are doing something wrong. Could you please help me how to disable CORK (or maybe how to enable TCP_NODELAY) in libmicrohttpd?

Best regards,
Marek Knápek

[Karlson2k]

Hi @MarekKnapek. Please, give more details. Which OS do you use? Which kernel version?

[Marek Knápek]

Hi @Karlson2k, I'm currently not at the computer. It is Linux 18.04 LTS, AMD64, the lib and our application is compiled with GCC (I guess version 7.5) as static library (with PIC enabled) compiled into dynamic .so loaded at runtime with dlopen. We use it as HTTPS server, client is Windows 10 C# application. It is long lasting connection, upon initial application level handshake we send approx 5 bytes chunks back to client once per second, client doesn't send us anything.

[Karlson2k]

What kind of response do you use? Fixed size or unknown size with callbacks?
Or, more precisely, do you use chunked encoding?

Did you try to do the same, but on HTTP (without TLS)?

[Alexander Petrov]

Hello. I have similar issue with https server application which use chunked encoding to transfer data from FreeBSD machine. After freebsd-ports libmicrohttpd updating from 0.9.58 to 0.9.70 wget(and curl) on client side (Ubuntu) getting stuck. As I saw in curl trace-ascii logs last CRLF in chunk ending is missing. After disabling CORK in MHD_connection_handle_write() on MHD_CONNECTION_FOOTERS_SENDING state problem was solved. Could it be a bug? Transfers with disabled chunked encoding works well.
Changes:

diff --git a/src/microhttpd/connection.c b/src/microhttpd/connection.c
index 3f47c924..cb338543 100644
--- a/src/microhttpd/connection.c
+++ b/src/microhttpd/connection.c
@@ -3151,7 +3151,7 @@ MHD_connection_handle_write (struct MHD_Connection *connection)
                                    [connection->write_buffer_send_offset],
                                    connection->write_buffer_append_offset
                                    - connection->write_buffer_send_offset,
-                                   MHD_SSO_HDR_CORK);
+                                   MHD_SSO_NO_CORK);
     if (ret < 0)
     {
       if (MHD_ERR_AGAIN_ == ret)

[Karlson2k]

@a1exanderpetrov, please open another issue as it most probably is unrelated.

[Karlson2k]

Hi @Karlson2k, I'm currently not at the computer. It is Linux 18.04 LTS, AMD64, the lib and our application is compiled with GCC (I guess version 7.5) as static library (with PIC enabled) compiled into dynamic .so loaded at runtime with dlopen. We use it as HTTPS server, client is Windows 10 C# application. It is long lasting connection, upon initial application level handshake we send approx 5 bytes chunks back to client once per second, client doesn't send us anything.

@MarekKnapek, what do you expect on application side? Receive portion of data every second? Is it chunked encoding or binary data with specified length?

[Marek Knápek]

Hi @Karlson2k, the kernel is Linux version 4.15.0-117-generic (buildd@lcy01-amd64-008) (gcc version 7.5.0 (Ubuntu 7.5.0-3ubuntu1~18.04)) #118-Ubuntu SMP Fri Sep 4 20:02:41 UTC 2020.

I made a mistake describing the issue, we do not use HTTPS in this particular case (we do use HTTPS at other places in the application unrelated to this issue).

We are using the unknown size with callback type of connection. We are using chunked type of connection.

I created a pcap capture libmicrohttpd-issue-6.zip of both good connection (libmicrohttpd 0.9.66) and bad connection (libmicrohttpd 0.9.67). Please see attachment.

In the good connection you can see we transmit single byte of data followed by bunch of newlines (probably added by libmicrohttpd, but I'm not sure about this), total of 6 Bytes, to the client once per second. We expect the client to receive the data and behave according to its contents. There should be no delay, for example such as client receiving one big data package each 10 seconds.

In the bad connection you can see that line 6 differs. The client closes the connection for no apparent reason. At least I cannot see any reason why it would do that.

I tested the change @a1exanderpetrov suggested and it didn't help.

// Edit: Typos and grammar.

Marek

[Karlson2k]

@MarekKnapek, as HTTP is designed to transfer data by request of the client and not designed to signal something to client by sending data by parts, MHD optimise network usage and ask OS to buffer the data until complete response is ready.
All data should be sent to the client as soon as response is formed. Could you check that client receive full response after data callback returns MHD_CONTENT_READER_END_OF_STREAM?

We can add flag to signal MHD that data must be sent to the client immediately when chunk is formed.

[Karlson2k]

@MarekKnapek New release version should fix your issue. Please try it.