rajanadar / VaultSharp

A comprehensive cross-platform .NET Library for HashiCorp's Vault, a secret management tool

Home Page:http://rajanadar.github.io/VaultSharp

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Large request cause OutOfMemoryException

GeirGrusom opened this issue · comments

In our project VaultSharp consistently causes an OutOfMemoryException when doing a batch encryption. It's probably caused by our large request sizes.

We're using 1.13.1

System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown. at 
   at System.Text.Json.JsonReaderHelper.TranscodeHelper(ReadOnlySpan`1 utf8Unescaped)
   at System.Text.Json.Utf8JsonReader.GetString()
   at System.Text.Json.Serialization.Metadata.JsonPropertyInfo`1.ReadJsonAndSetMember(Object obj, ReadStack& state, Utf8JsonReader& reader)
   at System.Text.Json.Serialization.Converters.ObjectDefaultConverter`1.OnTryRead(Utf8JsonReader& reader, Type typeToConvert, JsonSerializerOptions options, ReadStack& state, T& value)
   at System.Text.Json.Serialization.JsonConverter`1.TryRead(Utf8JsonReader& reader, Type typeToConvert, JsonSerializerOptions options, ReadStack& state, T& value)
   at System.Text.Json.Serialization.JsonCollectionConverter`2.OnTryRead(Utf8JsonReader& reader, Type typeToConvert, JsonSerializerOptions options, ReadStack& state, TCollection& value)
   at System.Text.Json.Serialization.JsonConverter`1.TryRead(Utf8JsonReader& reader, Type typeToConvert, JsonSerializerOptions options, ReadStack& state, T& value)
   at System.Text.Json.Serialization.Metadata.JsonPropertyInfo`1.ReadJsonAndSetMember(Object obj, ReadStack& state, Utf8JsonReader& reader)
   at System.Text.Json.Serialization.Converters.ObjectDefaultConverter`1.OnTryRead(Utf8JsonReader& reader, Type typeToConvert, JsonSerializerOptions options, ReadStack& state, T& value)
   at System.Text.Json.Serialization.JsonConverter`1.TryRead(Utf8JsonReader& reader, Type typeToConvert, JsonSerializerOptions options, ReadStack& state, T& value)
   at System.Text.Json.Serialization.Metadata.JsonPropertyInfo`1.ReadJsonAndSetMember(Object obj, ReadStack& state, Utf8JsonReader& reader)
   at System.Text.Json.Serialization.Converters.ObjectDefaultConverter`1.OnTryRead(Utf8JsonReader& reader, Type typeToConvert, JsonSerializerOptions options, ReadStack& state, T& value)
   at System.Text.Json.Serialization.JsonConverter`1.TryRead(Utf8JsonReader& reader, Type typeToConvert, JsonSerializerOptions options, ReadStack& state, T& value)
   at System.Text.Json.Serialization.JsonConverter`1.ReadCore(Utf8JsonReader& reader, JsonSerializerOptions options, ReadStack& state)
   at System.Text.Json.JsonSerializer.ReadFromSpan[TValue](ReadOnlySpan`1 utf8Json, JsonTypeInfo jsonTypeInfo, Nullable`1 actualByteCount)
   at System.Text.Json.JsonSerializer.ReadFromSpan[TValue](ReadOnlySpan`1 json, JsonTypeInfo jsonTypeInfo)
   at System.Text.Json.JsonSerializer.Deserialize[TValue](String json, JsonSerializerOptions options)
   at VaultSharp.Core.Polymath.MakeRequestAsync[TResponse](String resourcePath, HttpMethod httpMethod, Object requestData, IDictionary`2 headers, Boolean rawResponse, Action`1 postResponseAction)
   at VaultSharp.Core.Polymath.MakeVaultApiRequest[TResponse](String resourcePath, HttpMethod httpMethod, Object requestData, Boolean rawResponse, Action`1 postResponseAction, String wrapTimeToLive, Boolean unauthenticated)
   at VaultSharp.Core.Polymath.MakeVaultApiRequest[TResponse](String mountPoint, String path, HttpMethod httpMethod, Object requestData, Boolean rawResponse, Action`1 postResponseAction, String wrapTimeToLive, Boolean unauthenticated)
   at VaultSharp.V1.SecretsEngines.Transit.TransitSecretsEngineProvider.EncryptAsync(String keyName, EncryptRequestOptions encryptRequestOptions, String mountPoint, String wrapTimeToLive)

So I think the reason for this is fairly straightforward, and I think has a simple fix. In Polymath.cs on line 203 it uses StringContent. Switching this with JsonContent (which requires System.Net.Http.Json in Framework and .net standard) should enable streaming the object graph rather than duplicating the JSON content twice in memory (one for the immediate UTF-16 version, and then once more for the actual UTF-8 JSON)

This should likely improve performance, and reduce memory usage.