Buffer sizes, too much RAM is used
opened this issue · comments
After days of debugging I find that default chunkSize is using too much RAM, here is simple examples to check this.
Why you put by default 65536 ? Are there any reasons for this ?
After some tests I find that best settings are:
-
For
stream_get_contents->-1
1.1 If65536is used: for 1000 connections and1byte written you get ~73MB of RAM
1.2 If-1is used: for 1000 connections and1byte written you get ~6MB of RAM
1.3 If-1is used: for 1000 connections and65537byte written you get ~27MB of RAM
1.4 If65536is used: for 1000 connections and65537byte written you get ~73MB of RAM -
For
fwrite-> depends on data size
2.1 If your $chunkSize is small ex:1024and you send big data, you will spend more time to receive it all, but less RAM
2.2 vise versa2.1
PS: Sorry for my formatting, looks like this is not my best skill :)
Formatting is totally fine. 👍
I think using -1 makes sense, it's limited by the receive buffer anyway. Previously we didn't have any such possibility due to using fread(). @trowski thoughts?
@kelunik Using -1 makes sense with stream_get_contents, but we still need the setting for when fread is used. Should we add an option to SocketListenContext for the chunk size?
A major issue with using -1 with stream_get_contents() – when using a local stream where data is always immediately available, the function will keep consuming data until either the entire stream is consumed or memory is exhausted. I was able to crash a process using -1 and uploading a 50MB image because stream_get_contents() returned the entire image, then later code prepended some headers to the string and exceeded the memory limit.
In light of this, we need to supply a limit to stream_get_contents(). I recently changed the default to 8192, which should be a good balance between memory usage and performance.
Closing this, as the default size has been reduced and immediate reads make it still performant, which is basically equivalent to stream_get_contents with -1, but without the associated problems.