olivere / elastic

Deprecated: Use the official Elasticsearch client for Go at https://github.com/elastic/go-elasticsearch

Home Page:https://olivere.github.io/elastic/

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

io.ReadAll consumes considerable amount of memory when searching 10K hits

dilshandcs opened this issue · comments

Please use the following questions as a guideline to help me answer
your issue/question without further inquiry. Thank you.

Which version of Elastic are you using?

[X ] elastic.v7 (for Elasticsearch 7.x)
[ ] elastic.v6 (for Elasticsearch 6.x)
[ ] elastic.v5 (for Elasticsearch 5.x)
[ ] elastic.v3 (for Elasticsearch 2.x)
[ ] elastic.v2 (for Elasticsearch 1.x)
olivere/elastic/v7 --> v7.0.21

Please describe the expected behavior

I'm doing a performance test of an endpoint which is trying to get 10000 hits from ES. The profiling give me a high memory usage. But I am expecting a reasonable memory consumption

Please describe the actual behavior

pprof tool shows me that io.ReadAll is using considerable amount of memory when searching 10K hits from ES. This bound our application to a certain amount of concurrent requests due to the memory overhead. See the following part from the memory profile

image

I used the io.Copy instead of io.ReadAll and tried the same amount of concurrent requests. This improves the situation significantly. see the profile below

image

See the chages I made to response.go file below
image

According to above example, io.ReadAll has an allocated memory space of around 145MB while other has only around 57MB. I think this will significantly improve the memory usage of any application, and this is an issue we are facing as well. Shall I create a PR with this change?

Any steps to reproduce the behavior?

Difference of memory usage between io.Copy vs io.ReadAll can be compared when searching for 10K hits