Some FTP clients may disconnect TCP sessions just after connected

Question

Some FTP clients may disconnect TCP sessions just after connected

cat-in-136 opened this issue 2 years ago · comments

Some FTP clients such as gftp sometimes sends an RST packet and disconnect the TCP session before receiving all 220-- responses just after connecting.

The first 220- responses are consisting of the following 112 bytes (when welcomeMessage is default):

220---Welcome to Simply FTP server ---
220---   By Renzo Mischianti   ---
220 --    Version 2022-03-11    --

Subsequent messages are similar. This behavior depends on esp32 library's Nagle algorithm implementation.

When the messages are segmented excessively and there is a certain large latency between the server (esp32) and the client, some client seems to send an RST packet to disconnect.

cat_in_136 · Answer 1 · Sat Jun 18 2022 22:45:36 GMT+0800 (China Standard Time)

Sometimes the first six bytes of 220--- is sent on the only packet.

This boundary is come from the execution unit of client.print().

  client.print(F("220---")); client.print(welcomeMessage); client.println(F(" ---"));

So this issue can be resolved by storing the message in a buffer and sending it to the client when all messages stored. For example, the following workaround code:

template <size_t N = 256>
class StringPrintBuffer : public Print {
  public:
    virtual size_t write(const uint8_t *buffer, size_t size) override {
      if (_size + size < N) {
        memcpy(_buf + _size, buffer, size);
        _size += size;
        return size;
      } else {
        return 0;
      }
    }
    virtual size_t write(uint8_t c) override {
      if (_size < N) {
        _buf[_size++] = c;
        return 1;
      } else {
        return 0;
      }
    }

    const char* buffer() { return reinterpret_cast<const char*>(_buf); };
    const size_t size() { return _size; };
  private:
    size_t _size = 0;
    uint8_t _buf[N] = {0};
};


// ...snip...

  StringPrintBuffer<256> response;
  response.print(F("220---")); response.print(welcomeMessage); response.println(F(" ---"));
  response.println(F("220---   By Renzo Mischianti   ---"));
  response.print(F("220 --    Version ")); response.print(FTP_SERVER_VERSION); response.println(F("    --"));
  client.write(response.buffer(), response.size());

This can occur in any other response and it is necessary to apply the change to all responses.

Renzo Mischianti · Answer 2 · Sun Jun 19 2022 14:46:35 GMT+0800 (China Standard Time)

Thanks, @cat-in-136,
I must test the memory usage, when I return home probably I allow to select the buffer.
I use this library with Arduino Mega also with few memory.
Bye Renzo

Renzo Mischianti · Answer 3 · Mon Jun 27 2022 04:29:42 GMT+0800 (China Standard Time)

Hi @cat-in-136,
can you help me to replicate the problem?

And if you can, can you try to set this parameter (like explained here).
WiFi.setSleep(false);

Thanks Bye Renzo

Renzo Mischianti · Answer 4 · Wed Jun 29 2022 14:11:48 GMT+0800 (China Standard Time)

Hi @cat-in-136,
I try with gftp but no disconnection, It's working well without a problem.
Can you help me to replicate the issue?
Bye Renzo

Renzo Mischianti · Answer 5 · Mon Jul 04 2022 21:10:27 GMT+0800 (China Standard Time)

I try to do a double system to generate client write, but I need some time.
Bye Renzo

cat_in_136 · Answer 6 · Thu Jul 07 2022 23:32:45 GMT+0800 (China Standard Time)

@xreef

I understand my fix is ad-hoc and is not good from memory usage perspective.

Unfortunately, WiFi.setSleep(false); does not resolve the issue on my environment.

Environment details:

M5Stack Basic (black 2018.3)
WiFi mode: STA
Code https://gist.github.com/cat-in-136/fc2bcd59883088c43a927cccb7b5fd93

The packet capture is as follows:

where 192.168.0.4 is my linux PC (ftp client), 192.168.0.64 is M5Stack. RST were generated by gftp. Line 78-89 were generated by gftp's connection retry.

I originally saw this issue with my private app on M5Paper v1.1. After that, I reproduced this issue on other device M5Stack Basic. Does this issue only occurs on M5 product series 🤔!?

Renzo Mischianti · Answer 7 · Fri Jul 08 2022 14:04:53 GMT+0800 (China Standard Time)

Probably on M5 the latency is managed with different parameters, but I try to buffer the response (not now but in the future when I have more time).
I also need help with a lot of projects and tutorials if you are interested :P
I leave this issue open to remember, if you discover something else write here.
Bye Renzo

cat_in_136 · Answer 8 · Tue Oct 11 2022 00:09:04 GMT+0800 (China Standard Time)

I purchased and got ESP32-DevKitC V4 (board = esp32dev) which is not a M5Stack family product but a espressif official board. The esp32dev board does not have an SD card slot unlike M5Stack, I used a MicroSD card breakout board and connect it and esp32dev to handle SD card as same as M5Stack.

But, unfortunately, this issue was still reproduced with ESP32-DevKitC V4 on my environment. 😢 I believe I need to review my environment such as pio etc...

Environment details:

Espressif ESP32-DevKitC V4 with ESP32-WROOM-32D
WiFi mode: STA
Code: https://gist.github.com/cat-in-136/fc2bcd59883088c43a927cccb7b5fd93 (with #if defined(ARDUINO_ESP32_DEV))

LFrank2021 · Answer 9 · Sun Nov 27 2022 19:28:05 GMT+0800 (China Standard Time)

Hi. I just started with my first project and am running into the same problem.
Model ESP8266 WEMOS D1 mini
WiFi mode STA
Client Windows 10 command line ftp

I needed several attempt to get the password prompt at the correct time:

ftp esp8266.ftp.server
Verbindung mit esp8266.ftp.server wurde hergestellt.
220---Welcome to Simply FTP server ---
220---   By Renzo Mischianti   ---
220 --    Version 2.1.3 (2022-09-20)    --
530
Benutzer (esp8266.ftp.server:(none)): esp8266
221 Goodbye
220---Welcome to Simply FTP server ---
220---   By Renzo Mischianti   ---
220 --    Version 2.1.3 (2022-09-20)    --
ftp>
ftp> open esp8266.ftp.server
Verbindung zu esp8266.ftp.server besteht bereits. Beenden Sie die Verbindung zuerst.
ftp> bye
331 Ok. Password required

ftp esp8266.ftp.server
Verbindung mit esp8266.ftp.server wurde hergestellt.
220---Welcome to Simply FTP server ---
220---   By Renzo Mischianti   ---
220 --    Version 2.1.3 (2022-09-20)    --
530
Benutzer (esp8266.ftp.server:(none)): esp8266
221 Goodbye
220---Welcome to Simply FTP server ---
220---   By Renzo Mischianti   ---
220 --    Version 2.1.3 (2022-09-20)    --
ftp>
ftp> bye
331 Ok. Password required

ftp esp8266.ftp.server
Verbindung mit esp8266.ftp.server wurde hergestellt.
220---Welcome to Simply FTP server ---
220---   By Renzo Mischianti   ---
220 --    Version 2.1.3 (2022-09-20)    --
530
Benutzer (esp8266.ftp.server:(none)): esp8266
221 Goodbye
220---Welcome to Simply FTP server ---
220---   By Renzo Mischianti   ---
220 --    Version 2.1.3 (2022-09-20)    --
ftp> stat
Verbindung mit esp8266.ftp.server wurde hergestellt.
Typ: ascii; Verbose: EIN ; Bell: AUS ; Prompt: EIN ; Glob: EIN
Debug: AUS ; Hash: AUS
ftp> esp8266
Ungültiger Befehl
ftp>
ftp> quit
331 Ok. Password required

ftp esp8266.ftp.server
Verbindung mit esp8266.ftp.server wurde hergestellt.
220---Welcome to Simply FTP server ---
220---   By Renzo Mischianti   ---
220 --    Version 2.1.3 (2022-09-20)    --
530
221 Goodbye
Benutzer (esp8266.ftp.server:(none)):
220---Welcome to Simply FTP server ---
220---   By Renzo Mischianti   ---
220 --    Version 2.1.3 (2022-09-20)    --
ftp>
ftp>
ftp>
ftp> bye
530

ftp esp8266.ftp.server
Verbindung mit esp8266.ftp.server wurde hergestellt.
220---Welcome to Simply FTP server ---
220---   By Renzo Mischianti   ---
220 --    Version 2.1.3 (2022-09-20)    --
530
Benutzer (esp8266.ftp.server:(none)): esp8266
221 Goodbye
220---Welcome to Simply FTP server ---
220---   By Renzo Mischianti   ---
220 --    Version 2.1.3 (2022-09-20)    --
331 Ok. Password required
Kennwort:
230 Ok
ftp> ls
200 PORT command successful
150 Accepted data connection to port 50996
drwxrwsr-x      2       esp8266 4096    Jan 01  1970    logs
-rw-rw-r--      1       esp8266 7       Nov 26 16:06    version.txt
-rw-rw-r--      1       esp8266 7       Nov 26 14:44    version2.txt
-rw-rw-r--      1       esp8266 7       Nov 26 15:23    version4.txt
226 4 matches total
FTP: 197 Bytes empfangen in 0.09Sekunden 2.32KB/s
ftp> cd logs
250 Directory changed to /logs
ftp> ls
200 PORT command successful
150 Accepted data connection to port 50997
-rw-rw-r--      1       esp8266 17088   Nov 27 11:11    general.log
-rw-rw-r--      1       esp8266 11508   Nov 27 11:10    vitwifi.log
226 2 matches total
FTP: 109 Bytes empfangen in 0.09Sekunden 1.21KB/s
ftp>

Hope this helps a bit.

Cheers,
Frank

LFrank2021 · Answer 10 · Wed Nov 30 2022 01:44:34 GMT+0800 (China Standard Time)

I had a really bad day today and took a look at the source code.
Took me a while but after changing this line it is way more stable.

Granted, I do not exactly know when this 'else' is used:

FtpServer.cpp: Line 484 - 413 (

  else if( CommandIs( "AUTH" ))
    client.println(F("502 ") );
  //
  //  Unrecognized commands at stage of authentication
  //
  else if( cmdStage < FTP_Cmd )
  {
    client.println(F("530 ") );
---   cmdStage = FTP_Stop;
+++   cmdStage = FTP_User;
  }

Reasoning:
After the MS ftp.exe opens the connection a OPTS command is send. After that the USER command should be initiated. It seems that after the OPTS gets handled the program runs into this 'else' tree.

Renzo Mischianti · Answer 11 · Thu Dec 01 2022 22:57:18 GMT+0800 (China Standard Time)

Hi @LFrank2021,
I think the problem differs from the one proposed by @cat-in-136, but it can be a good point to do additional tests.

But what means "more stable", sometimes It's working.

Bye Renzo

LFrank2021 · Answer 12 · Thu Dec 01 2022 23:27:03 GMT+0800 (China Standard Time)

Hi @xreef
yes, sometimes the client connected fine.
How to explain it?
I am currently heavily experimenting with my first own project.
And am stressing the little ESP8266 (maybe) to its limits:
HTTP, FTP, LittleFS (for debugging), VitoWiFi
And it seems to me, the more (especially FS logging) it has to do, the more instable the FTP-server got.
Lately, prior to my manipulation, I was unable to get any connection.
Always a 530 message (without 'TIMEOUT') and no prompt for PASS.

Right now I am again strugeling with ftp.exe but FileZilla is able to connect reliable.

Kind regards
Frank

Renzo Mischianti · Answer 13 · Tue Oct 17 2023 17:42:47 GMT+0800 (China Standard Time)

After some time, this bug seems no longer appears.
Thanks to all