Giters

ex-aws / ex_aws

A flexible, easy to use set of clients AWS APIs for Elixir

Home Page:https://hex.pm/packages/ex_aws

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

High memory usage

isavita opened this issue · comments

commented

My application is using ExAws for Lambda call with relatively large body in the response (400KB gzip) responses from aws lambda. I notice that in the request module ExAws.Request there is some copying of the response_body in for telemetry stats here.
Is there any way to avoid this (e.g. config)?

Environment

  • Elixir & Erlang versions (elixir --version): Elixir 1.14.3 (compiled with Erlang/OTP 23)
  • ExAws version: ex_aws 2.4.4
  • HTTP client version: finch 0.13.0

Current behavior

There is some copying the response body for :telemetry that increase the memory usage here.

Expected behavior

I would expect to be able to disable this copy of the response body to telemetry or in general to be able to disable telemetry.

commented

I'm not quite clear how you come to the conclusion that that line causes a copy of the body (not saying it's not, just that I don't understand). Telemetry operations all occur in-process, so unless the body is modified I don't see any reason to imaging it's being copied there.

commented

The flow looks to me as

  1. We use this telemetry wrapper that spawn a new :telemetry.span(telemetry_event, telemetry_metadata, fn -> <INNER_SCOPE> end)
  2. We do in <INNER_SCOPE> the http client call to lambda and get result = config[:http_client].request(...). This result has the body.
  3. After that we get stop_metadata = %{result: :ok, response_body: Map.get(resp, :body)}. Which is the place that we get the body again.
  4. Then we have this part telemetry_metadata = Map.merge(telemetry_metadata, stop_metadata), which I think makes copy of the body.
  5. We return {result, telemetry_metadata} in which we have the result with body and telemetry_metadata with the body again.

Am I wrong to think that we copy the body in telemetry_metadata when we merge the metadata? In my application I can see replacing the line stop_metadata = %{result: :ok, response_body: Map.get(resp, :body)} with stop_metadata = %{result: :ok, response_body: ""} decreases the memory usage.

commented

To the best of my understanding, step 4 should not create a copy of the body - it should just add a reference to the existing binary. How are you measuring memory use? It would be interesting to know exactly what aspect of memory use is increasing (using :erlang.memory() will show you the breakdown) - I think the body would be part of binary so you'd expect to see that jump up at that point if it's really being copied.

commented

@bernardd I've investigated more closely but cannot reproduce the issue. I am closing it.