RAFT: snapshot should use ByteBuffer rather than ByteArray
belaban opened this issue · comments
If possible, would be better to use direct ByteBuffer to hold the snapshot: having big byte[] is dangerous both for GC (causing humoungous allocations) and for the I/O, because native I/O don't understand byte[] heap objects and is forced to copy everything into a direct ByteBuffer (or worst, by using malloc to allocate one and free when completed)
@pruivo : can you check if ByteBuffer is used correctly/efficiently in FileBasedLog?
@belaban @franz1981 sorry again, but I don't see any advantage of using ByteBuffer over ByteArray in the commit above :(
@pruivo I thought you wanted me to switch to ByteBuffer? Because it is part of the JDK, and ByteArray isn't?
@belaban I didn't ask for anything 👍 Francesco did
oops :-)
sorry again, but I don't see any advantage of using ByteBuffer over ByteArray in the commit above :(
It's not the ByteBuffer vs ByteArray the problem, but just this part:
os.write(snapshot.getArray(), snapshot.getOffset(), snapshot.getLength());OutputStream are backed by posix I/O that is not able to understand what byte[] means, hence a copy is performed for each write.
The copy is performed against a pooled direct ByteBuffer stored in a thread-local field; this can create severe stability issues and is not recommended (that's why i've implemented the file store reusing the same direct ByteBuffer over and over).
In short, allocating a direct ByteBuffer is preferred and it's highly suggested to pool (and reuse) it somewhere, because releasing direct ByteBuffer is not as easy that with heap based ones (ie they use Cleaner/PhantomReference and can cause OOM of native memory with unpredictable results).
A note on using big byte[] in general:
The allocation of big byte[] in itself is already a problem depending by the length of arrays, because GC won't allow to allocate a single contiguos byte[] with easy, but trigger a so-called "humoungous allocation" that's way less friendly with GC.
So, are you happy with the change from ByteArray to ByteBuffer, @franz1981 ? :-)
The good thing is these are high-hanging fruit, as a snapshot is performed infrequently...
The good thing is these are high-hanging fruit, as a snapshot is performed infrequently...
I'm warning about the stability problem in itself, because being bitten by it several times before in my life - you can still use byte[] and leave the issue as it is, when/if the problem arise 👍