Heap memory flush needed
byongwu opened this issue · comments
Hi,
I am running pako and found out that following gzip() function call cause heap memory increase.
function garbageCollect() {
try {
if (global.gc) {
global.gc();
}
} catch (e) {
console.log("`node --expose-gc index.js`");
process.exit();
}
}
function memoryTest() {
const pako = require("pako");
garbageCollect();
const used_before = process.memoryUsage().heapUsed;
console.log("memory current: ", used_before);
pako.gzip("Test1234"); // or any other test string or method you choose.
garbageCollect();
const used_after = process.memoryUsage().heapUsed;
console.log(
"memory after: ",
used_after,
", increased: ",
used_after - used_before
);
}
memoryTest();
memoryTest();
memoryTest();
memoryTest();
memoryTest();
memoryTest();
This is the result.
% tsc main.ts
% node --expose-gc main.js
memory current: 2664072
memory after: 2845944 , increased: 181872
memory current: 2844520
memory after: 2855136 , increased: 10616
memory current: 2854608
memory after: 2855984 , increased: 1376
memory current: 2856416
memory after: 2858080 , increased: 1664
memory current: 2860112
memory after: 2861600 , increased: 1488
memory current: 2862696
memory after: 2863928 , increased: 1232
Eventually, the memory increase should diminish and there shouldn't be any more heap memory consumed.
Due to this issue, I had heap memory overflow.
Could you please reproduce the problem and enable the API to flush the pako's internal heap memory usage?
Thanks,
JIT gc behaviour depends on amount of available memory. Such tests are not relevant for checking package memory leaks.
Hi, @puzrin .
When I disable pako, I don't get memory failure like following.
<--- Last few GCs --->
[20203:0x1048d7000] 45544 ms: Scavenge 4067.7 (4095.4) -> 4067.4 (4096.4) MB, 1.8 / 0.1 ms (average mu = 0.920, current mu = 0.848) allocation failure
[20203:0x1048d7000] 45553 ms: Scavenge 4068.4 (4104.4) -> 4067.7 (4107.4) MB, 3.1 / 0.0 ms (average mu = 0.920, current mu = 0.848) allocation failure
[20203:0x1048d7000] 45635 ms: Mark-sweep 4074.7 (4107.4) -> 4072.7 (4115.4) MB, 36.5 / 0.1 ms (average mu = 0.861, current mu = 0.730) allocation failure scavenge might not succeed
<--- JS stacktrace --->
FATAL ERROR: MarkCompactCollector: young object promotion failed Allocation failed - JavaScript heap out of memory
1: 0x101306bb5 node::Abort() (.cold.1) [/Users/byochong/.nvm/versions/node/v14.17.5/bin/node]
2: 0x1000ad6c9 node::Abort() [/Users/byochong/.nvm/versions/node/v14.17.5/bin/node]
3: 0x1000ad82f node::OnFatalError(char const*, char const*) [/Users/byochong/.nvm/versions/node/v14.17.5/bin/node]
4: 0x1001f1b97 v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [/Users/byochong/.nvm/versions/node/v14.17.5/bin/node]
5: 0x1001f1b33 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [/Users/byochong/.nvm/versions/node/v14.17.5/bin/node]
6: 0x10039f695 v8::internal::Heap::FatalProcessOutOfMemory(char const*) [/Users/byochong/.nvm/versions/node/v14.17.5/bin/node]
7: 0x1003fb6f3 v8::internal::EvacuateNewSpaceVisitor::Visit(v8::internal::HeapObject, int) [/Users/byochong/.nvm/versions/node/v14.17.5/bin/node]
8: 0x1003e2f5b void v8::internal::LiveObjectVisitor::VisitBlackObjectsNoFail<v8::internal::EvacuateNewSpaceVisitor, v8::internal::MajorNonAtomicMarkingState>(v8::internal::MemoryChunk*, v8::internal::MajorNonAtomicMarkingState*, v8::internal::EvacuateNewSpaceVisitor*, v8::internal::LiveObjectVisitor::IterationMode) [/Users/byochong/.nvm/versions/node/v14.17.5/bin/node]
9: 0x1003e2aa5 v8::internal::FullEvacuator::RawEvacuatePage(v8::internal::MemoryChunk*, long*) [/Users/byochong/.nvm/versions/node/v14.17.5/bin/node]
10: 0x1003e27e6 v8::internal::Evacuator::EvacuatePage(v8::internal::MemoryChunk*) [/Users/byochong/.nvm/versions/node/v14.17.5/bin/node]
11: 0x10040011e v8::internal::PageEvacuationTask::RunInParallel(v8::internal::ItemParallelJob::Task::Runner) [/Users/byochong/.nvm/versions/node/v14.17.5/bin/node]
12: 0x1003ba0d2 v8::internal::ItemParallelJob::Task::RunInternal() [/Users/byochong/.nvm/versions/node/v14.17.5/bin/node]
13: 0x1003ba558 v8::internal::ItemParallelJob::Run() [/Users/byochong/.nvm/versions/node/v14.17.5/bin/node]
14: 0x1003e4855 void v8::internal::MarkCompactCollectorBase::CreateAndExecuteEvacuationTasks<v8::internal::FullEvacuator, v8::internal::MarkCompactCollector>(v8::internal::MarkCompactCollector*, v8::internal::ItemParallelJob*, v8::internal::MigrationObserver*, long) [/Users/byochong/.nvm/versions/node/v14.17.5/bin/node]
15: 0x1003e4456 v8::internal::MarkCompactCollector::EvacuatePagesInParallel() [/Users/byochong/.nvm/versions/node/v14.17.5/bin/node]
16: 0x1003cfbc7 v8::internal::MarkCompactCollector::Evacuate() [/Users/byochong/.nvm/versions/node/v14.17.5/bin/node]
17: 0x1003cd45b v8::internal::MarkCompactCollector::CollectGarbage() [/Users/byochong/.nvm/versions/node/v14.17.5/bin/node]
18: 0x10039fd5b v8::internal::Heap::MarkCompact() [/Users/byochong/.nvm/versions/node/v14.17.5/bin/node]
19: 0x10039c349 v8::internal::Heap::PerformGarbageCollection(v8::internal::GarbageCollector, v8::GCCallbackFlags) [/Users/byochong/.nvm/versions/node/v14.17.5/bin/node]
20: 0x10039a190 v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [/Users/byochong/.nvm/versions/node/v14.17.5/bin/node]
21: 0x1003a88ba v8::internal::Heap::AllocateRawWithLightRetrySlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [/Users/byochong/.nvm/versions/node/v14.17.5/bin/node]
22: 0x1003a8941 v8::internal::Heap::AllocateRawWithRetryOrFailSlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [/Users/byochong/.nvm/versions/node/v14.17.5/bin/node]
23: 0x100371abd v8::internal::FactoryBase<v8::internal::Factory>::NewRawOneByteString(int, v8::internal::AllocationType) [/Users/byochong/.nvm/versions/node/v14.17.5/bin/node]
24: 0x1003781c2 v8::internal::Factory::NewStringFromTwoByte(unsigned short const*, int, v8::internal::AllocationType) [/Users/byochong/.nvm/versions/node/v14.17.5/bin/node]
25: 0x100214320 v8::String::NewFromTwoByte(v8::Isolate*, unsigned short const*, v8::NewStringType, int) [/Users/byochong/.nvm/versions/node/v14.17.5/bin/node]
26: 0x10016ac7b node::(anonymous namespace)::ExternString<v8::String::ExternalStringResource, unsigned short>::NewFromCopy(v8::Isolate*, unsigned short const*, unsigned long, v8::Local<v8::Value>*) [/Users/byochong/.nvm/versions/node/v14.17.5/bin/node]
27: 0x10008ebfa void node::Buffer::(anonymous namespace)::StringSlice<(node::encoding)3>(v8::FunctionCallbackInfo<v8::Value> const&) [/Users/byochong/.nvm/versions/node/v14.17.5/bin/node]
28: 0x1002624a8 v8::internal::FunctionCallbackArguments::Call(v8::internal::CallHandlerInfo) [/Users/byochong/.nvm/versions/node/v14.17.5/bin/node]
29: 0x100261a3c v8::internal::MaybeHandle<v8::internal::Object> v8::internal::(anonymous namespace)::HandleApiCallHelper<false>(v8::internal::Isolate*, v8::internal::Handle<v8::internal::HeapObject>, v8::internal::Handle<v8::internal::HeapObject>, v8::internal::Handle<v8::internal::FunctionTemplateInfo>, v8::internal::Handle<v8::internal::Object>, v8::internal::BuiltinArguments) [/Users/byochong/.nvm/versions/node/v14.17.5/bin/node]
30: 0x1002611a2 v8::internal::Builtin_Impl_HandleApiCall(v8::internal::BuiltinArguments, v8::internal::Isolate*) [/Users/byochong/.nvm/versions/node/v14.17.5/bin/node]
31: 0x100a7c4d9 Builtins_CEntry_Return1_DontSaveFPRegs_ArgvOnStack_BuiltinExit [/Users/byochong/.nvm/versions/node/v14.17.5/bin/node]
zsh: abort ts-node main.ts
%
My nodejs application using pako increased heap memory usage until 10GB and it crashes like above.
I spent days chasing the heap memory issue and found out that pako kept on increasing the heap memory.
It would be nice if you have some evidence or stress test report that pako does not have heap memory issue.
It would be nice if you have some evidence or stress test report that pako does not have heap memory issue.
Many things would be nice, but i can't spend time "just for fun" for uncertain goal. AFAIK, scripts in lib/zlib folder do not allocate new memory after init. You could try to create your own wrapper, if you don't trust default one. Reporting global metric like "server memory leak" is not enough for me to investigate.
Also, note, pako was done for browsers, because those not have built-in zlib module like nodejs.
Hi, @puzrin . When I disable pako, I don't get memory failure like following.
<--- Last few GCs ---> [20203:0x1048d7000] 45544 ms: Scavenge 4067.7 (4095.4) -> 4067.4 (4096.4) MB, 1.8 / 0.1 ms (average mu = 0.920, current mu = 0.848) allocation failure [20203:0x1048d7000] 45553 ms: Scavenge 4068.4 (4104.4) -> 4067.7 (4107.4) MB, 3.1 / 0.0 ms (average mu = 0.920, current mu = 0.848) allocation failure [20203:0x1048d7000] 45635 ms: Mark-sweep 4074.7 (4107.4) -> 4072.7 (4115.4) MB, 36.5 / 0.1 ms (average mu = 0.861, current mu = 0.730) allocation failure scavenge might not succeed <--- JS stacktrace ---> FATAL ERROR: MarkCompactCollector: young object promotion failed Allocation failed - JavaScript heap out of memory 1: 0x101306bb5 node::Abort() (.cold.1) [/Users/byochong/.nvm/versions/node/v14.17.5/bin/node] 2: 0x1000ad6c9 node::Abort() [/Users/byochong/.nvm/versions/node/v14.17.5/bin/node] 3: 0x1000ad82f node::OnFatalError(char const*, char const*) [/Users/byochong/.nvm/versions/node/v14.17.5/bin/node] 4: 0x1001f1b97 v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [/Users/byochong/.nvm/versions/node/v14.17.5/bin/node] 5: 0x1001f1b33 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [/Users/byochong/.nvm/versions/node/v14.17.5/bin/node] 6: 0x10039f695 v8::internal::Heap::FatalProcessOutOfMemory(char const*) [/Users/byochong/.nvm/versions/node/v14.17.5/bin/node] 7: 0x1003fb6f3 v8::internal::EvacuateNewSpaceVisitor::Visit(v8::internal::HeapObject, int) [/Users/byochong/.nvm/versions/node/v14.17.5/bin/node] 8: 0x1003e2f5b void v8::internal::LiveObjectVisitor::VisitBlackObjectsNoFail<v8::internal::EvacuateNewSpaceVisitor, v8::internal::MajorNonAtomicMarkingState>(v8::internal::MemoryChunk*, v8::internal::MajorNonAtomicMarkingState*, v8::internal::EvacuateNewSpaceVisitor*, v8::internal::LiveObjectVisitor::IterationMode) [/Users/byochong/.nvm/versions/node/v14.17.5/bin/node] 9: 0x1003e2aa5 v8::internal::FullEvacuator::RawEvacuatePage(v8::internal::MemoryChunk*, long*) [/Users/byochong/.nvm/versions/node/v14.17.5/bin/node] 10: 0x1003e27e6 v8::internal::Evacuator::EvacuatePage(v8::internal::MemoryChunk*) [/Users/byochong/.nvm/versions/node/v14.17.5/bin/node] 11: 0x10040011e v8::internal::PageEvacuationTask::RunInParallel(v8::internal::ItemParallelJob::Task::Runner) [/Users/byochong/.nvm/versions/node/v14.17.5/bin/node] 12: 0x1003ba0d2 v8::internal::ItemParallelJob::Task::RunInternal() [/Users/byochong/.nvm/versions/node/v14.17.5/bin/node] 13: 0x1003ba558 v8::internal::ItemParallelJob::Run() [/Users/byochong/.nvm/versions/node/v14.17.5/bin/node] 14: 0x1003e4855 void v8::internal::MarkCompactCollectorBase::CreateAndExecuteEvacuationTasks<v8::internal::FullEvacuator, v8::internal::MarkCompactCollector>(v8::internal::MarkCompactCollector*, v8::internal::ItemParallelJob*, v8::internal::MigrationObserver*, long) [/Users/byochong/.nvm/versions/node/v14.17.5/bin/node] 15: 0x1003e4456 v8::internal::MarkCompactCollector::EvacuatePagesInParallel() [/Users/byochong/.nvm/versions/node/v14.17.5/bin/node] 16: 0x1003cfbc7 v8::internal::MarkCompactCollector::Evacuate() [/Users/byochong/.nvm/versions/node/v14.17.5/bin/node] 17: 0x1003cd45b v8::internal::MarkCompactCollector::CollectGarbage() [/Users/byochong/.nvm/versions/node/v14.17.5/bin/node] 18: 0x10039fd5b v8::internal::Heap::MarkCompact() [/Users/byochong/.nvm/versions/node/v14.17.5/bin/node] 19: 0x10039c349 v8::internal::Heap::PerformGarbageCollection(v8::internal::GarbageCollector, v8::GCCallbackFlags) [/Users/byochong/.nvm/versions/node/v14.17.5/bin/node] 20: 0x10039a190 v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [/Users/byochong/.nvm/versions/node/v14.17.5/bin/node] 21: 0x1003a88ba v8::internal::Heap::AllocateRawWithLightRetrySlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [/Users/byochong/.nvm/versions/node/v14.17.5/bin/node] 22: 0x1003a8941 v8::internal::Heap::AllocateRawWithRetryOrFailSlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [/Users/byochong/.nvm/versions/node/v14.17.5/bin/node] 23: 0x100371abd v8::internal::FactoryBase<v8::internal::Factory>::NewRawOneByteString(int, v8::internal::AllocationType) [/Users/byochong/.nvm/versions/node/v14.17.5/bin/node] 24: 0x1003781c2 v8::internal::Factory::NewStringFromTwoByte(unsigned short const*, int, v8::internal::AllocationType) [/Users/byochong/.nvm/versions/node/v14.17.5/bin/node] 25: 0x100214320 v8::String::NewFromTwoByte(v8::Isolate*, unsigned short const*, v8::NewStringType, int) [/Users/byochong/.nvm/versions/node/v14.17.5/bin/node] 26: 0x10016ac7b node::(anonymous namespace)::ExternString<v8::String::ExternalStringResource, unsigned short>::NewFromCopy(v8::Isolate*, unsigned short const*, unsigned long, v8::Local<v8::Value>*) [/Users/byochong/.nvm/versions/node/v14.17.5/bin/node] 27: 0x10008ebfa void node::Buffer::(anonymous namespace)::StringSlice<(node::encoding)3>(v8::FunctionCallbackInfo<v8::Value> const&) [/Users/byochong/.nvm/versions/node/v14.17.5/bin/node] 28: 0x1002624a8 v8::internal::FunctionCallbackArguments::Call(v8::internal::CallHandlerInfo) [/Users/byochong/.nvm/versions/node/v14.17.5/bin/node] 29: 0x100261a3c v8::internal::MaybeHandle<v8::internal::Object> v8::internal::(anonymous namespace)::HandleApiCallHelper<false>(v8::internal::Isolate*, v8::internal::Handle<v8::internal::HeapObject>, v8::internal::Handle<v8::internal::HeapObject>, v8::internal::Handle<v8::internal::FunctionTemplateInfo>, v8::internal::Handle<v8::internal::Object>, v8::internal::BuiltinArguments) [/Users/byochong/.nvm/versions/node/v14.17.5/bin/node] 30: 0x1002611a2 v8::internal::Builtin_Impl_HandleApiCall(v8::internal::BuiltinArguments, v8::internal::Isolate*) [/Users/byochong/.nvm/versions/node/v14.17.5/bin/node] 31: 0x100a7c4d9 Builtins_CEntry_Return1_DontSaveFPRegs_ArgvOnStack_BuiltinExit [/Users/byochong/.nvm/versions/node/v14.17.5/bin/node] zsh: abort ts-node main.ts %My nodejs application using pako increased heap memory usage until 10GB and it crashes like above. I spent days chasing the heap memory issue and found out that pako kept on increasing the heap memory. It would be nice if you have some evidence or stress test report that pako does not have heap memory issue.
is this true that pako deliberately kept increasing heap memory ?
i am getting the same issue @byongwu for small files it is working but files with size 5mb then while inflating it gives me the same error as you were getting
Did you found any solution to this ?? @byongwu