Giters

kriszyp / msgpackr

Ultra-fast MessagePack implementation with extension for record and structural cloning / msgpack.org[JavaScript/NodeJS]

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Memory leak in native extension

mrazorvin opened this issue · comments

commented

Hi, thanks for amazing lib,

We found that with enabled native extension rss size is continue growing

2023-07-12T10:34:29.078Z INFO Memory usage: 236.37 MB memoryUsage.heapUsed, 426.34 MB memoryUsage.rss, 352.94 MB memoryUsage.heapTotal, 32907.31 MB os.freemem, 63275.04 MB os.totalmem

2023-07-12T11:38:44.212Z INFO Memory usage: 233.28 MB memoryUsage.heapUsed, 509.14 MB memoryUsage.rss, 443.63 MB memoryUsage.heapTotal, 31857.98 MB os.freemem, 63275.04 MB os.totalmem

There no such problem when extension disabled

commented

I was unable to reproduce this issue; I ran 40,000,000 deserializations without any noticeable memory growth at all. Do you have any steps to reproduce this?

It would be somewhat surprising for this code to have a memory leak since it does virtually no memory allocations other than creating JS strings that are managed by JavaScript. However... the extract code definitely does increase the potential retained memory if a isolated strings are retained, since they are sliced from a larger extracted strings. That is, when all the extracted strings are in memory, the memory usage is efficient, and if all the strings are collectable, they can be GC'ed efficiently. However, if there individual string from an extraction that are retained, they will retain a reference to the larger source string. This is not a memory leak itself, but it could/would exacerbate the memory consumption if there is a memory leak in an application that isn't releasing certain strings that came from the extractions. In other words, it is quite possible for a memory leak to manifest as faster memory growth with the native extension than without.

commented

Thanks for your time, this explains slow memory grow.

We have a lot messages with size of 1-2mb. Every new message replace some part of previous message, but not completely, basically most of references to large strings will live foreve

commented

Fwiw, I believe there are some tricks to create a non- sliced copy of a string, by recreating with some appending, that may help to decouple the string from the source.