send_file drop connections (Error while writing data to the client)

[Muhammed Yaşar]

Description

Kemal is working on the backend service as file server behind of litespeed, but it drops client connections and sometimes not respond at all.

Server load is about 10K/s

Reproduces how often: 1% of all request.

Versions

$ crystal --version
Crystal 0.35.1 [5999ae29b] (2020-06-19)

LLVM: 8.0.0
Default target: x86_64-unknown-linux-gnu

$ ulimit -n
640000

$ hostnamectl 
  Operating System: CentOS Linux 7 (Core)
       CPE OS Name: cpe:/o:centos:centos:7
            Kernel: Linux 3.10.0-1127.18.2.el7.x86_64
      Architecture: x86-64

Example of usage:

get "/v/:sum" do |env|
  storage = FileServer.get_storage(env.params.url["sum"], "mp4")
  if storage.is_a? Arango::Cursor(Storage) && !storage.result.empty?
    doc = storage.result.first
    mp4_file = doc.files.select { |d| d["type"] == "mp4" }.first
    file_path = "#{CONFIG["storage"]["filepath"].as_s}#{mp4_file["url"]}"
    begin
      send_file env, file_path, filename: "#{doc.filename.chomp}.mp4", disposition: "attachment"
   rescue err
      puts "[#{Time.utc}] Error while file streaming - type: mp4, md5: #{env.params.url["sum"]} err: #{err.message}"
    end
    next
  end
  puts "[#{Time.utc}] 404 Not Found - type: mp4, md5: #{env.params.url["sum"]}"
  halt(env, status_code: 404, response: "not found.")
end

error message:

[2020-08-15 11:40:58 UTC] Error while file streaming - type: mp4, md5: 5fa2e6fd6ecf1fdd247288dff56a72da err: Error while writing data to the client
[2020-08-15 11:41:07 UTC] Error while file streaming - type: mp3, md5: caa6a9d408edac1ae18a864c8a0f383c err: Error while writing data to the client
[2020-08-15 11:41:26 UTC] Error while file streaming - type: mp3, md5: f03689916b37adda54d28a0e46ace7d7 err: Error while writing data to the client
[2020-08-15 11:41:27 UTC] Error while file streaming - type: mp3, md5: 0ad91dc2e28232f7559e1c88d70713f9 err: Error while writing data to the client
[2020-08-15 11:41:28 UTC] Error while file streaming - type: mp3, md5: 1bcbcdfecc8363cb0fa04b372d67fd95 err: Error while writing data to the client
[2020-08-15 11:41:31 UTC] Error while file streaming - type: mp3, md5: 59cd12085859f27e702dd1904113dc22 err: Error while writing data to the client
[2020-08-15 11:41:36 UTC] Error while file streaming - type: mp3, md5: c2976fde442520ac6e45823847393742 err: Error while writing data to the client
[2020-08-15 11:41:39 UTC] Error while file streaming - type: mp3, md5: c2976fde442520ac6e45823847393742 err: Error while writing data to the client
[2020-08-15 11:41:41 UTC] Error while file streaming - type: mp3, md5: a4f05f5c07cc9b5935308b414d52a8ee err: Error while writing data to the client
[2020-08-15 11:41:42 UTC] Error while file streaming - type: mp4, md5: 11766aac704afc2ac8d5657004724736 err: Error while writing data to the client
[2020-08-15 11:41:42 UTC] Error while file streaming - type: mp4, md5: 11766aac704afc2ac8d5657004724736 err: Error while writing data to the client
[2020-08-15 11:41:43 UTC] Error while file streaming - type: mp3, md5: 76067f58cd6805b353d66f9e39ce6866 err: Error while writing data to the client
[2020-08-15 11:41:49 UTC] Error while file streaming - type: mp4, md5: 72cdc1369b800bd85d074fd102d1e3a2 err: Error while writing data to the client
[2020-08-15 11:41:51 UTC] Error while file streaming - type: mp4, md5: 7d417d64994651970b2374cc7a61d0a5 err: Error while writing data to the client
[2020-08-15 11:42:03 UTC] Error while file streaming - type: mp3, md5: 3566393f60c7d3ed60f6ac2797fd862f err: Error while writing data to the client
[2020-08-15 11:42:05 UTC] Error while file streaming - type: mp3, md5: c246c6e3bdafc71bce52801416a35a6b err: Error while writing data to the client
[2020-08-15 11:42:05 UTC] Error while file streaming - type: mp3, md5: c2976fde442520ac6e45823847393742 err: Error while writing data to the client

And sometimes:

Unhandled exception in spawn: Error while flushing data to the client (HTTP::Server::ClientError)
  from ???
  from ???
  from ???
  from ???
  from ???
  from ???
  from ???
Caused by: Error writing to socket: Connection reset by peer (IO::Error)
  from ???
  from ???
  from ???
  from ???
  from ???
  from ???
  from ???
  from ???

** there is no unhandled io operation on program by the way. **

[Louis Brauer]

This is "normal" in a sense that clients (peers) sometimes abort downloads.
When you close i.e. the browser while it's downloading, this will lead to the error you show. I don't think it is a problem with Kemal.

[Muhammed Yaşar]

This is "normal" in a sense that clients (peers) sometimes abort downloads.
When you close i.e. the browser while it's downloading, this will lead to the error you show. I don't think it is a problem with Kemal.

I guess its not about aborting current download, because we tested that with both internet connection drop on client side and also abort download on client side, both situation were not throw any error. I am not sure where it causes that error but that service were rewritten in golang and it works perfectly now, i havent seen any "sending data" error since september yet and our users also stopped reporting this problem on client side.

Maybe its a bug in crystal http core maybe not, i dont know.

[Anton Maminov]

I believe this was fixed by #576. But I'm not sure 😃

[Serdar Dogruyol - Sedo セド]

@yorci any update on this? Can we close the issue?

[Muhammed Yaşar]

@sdogruyol i dont know, im not using currently crystal nor kemal on projects but there are some other developers having that issue. I think can close issue i cant give more feedback.