A bunch of little problems with canary, ptrie, and failed allocations

[Matthew Gambrell]

Sorry for a bunch of problems all in one ticket. This is just a dump of everything I encountered. I hope it will be useful for someone doing similar work or doing a general round of work to make ltalloc more robust.

1. There is a problem when LTALLOC_OVERFLOW_DETECTION is used, along with a large block. ltmsize is returning the padded size in a normal small block case (because ltalloc is pretending like the user is requesting the extra canary bytes) but it is subtracting the canary size for the large block case. ltmsize is returning values useless to the user (for instance I allocated 19936 bytes, with the canary 19940, and ltmsize returns 20480) so there is no need for it to subtract the canary size to make it look perfect. (In fact, it should not even be a public API imo.) It should return the total bytes that ltalloc has reserved for the allocation's block internally, and the canary will always be occupying the final four bytes of that. The canary check assumes this, but ltmsize is messing that up. So ltmsize needs to lose its final size -= LTALLOC_CANARY_SIZE;. Now it will faithfully describe the actual bytes available for the block in every case.

This problem will manifest as spurious detections of overflows, due to the canary being mis-tested 4 bytes early, where the application may sensibly write into that position. For most people this probably would only ever happen in the case where the user's request was 4 bytes fewer than the size of a VM page. So it's not a common scenario.

2. There is a similar problem in ltrealloc. It checks ltmsize to see if it can just return the same block. The code is written in english roughly as "if needed bytes are less than or equal to ltmsize for the current block, return the current block". However, when a canary is in use, the current block would have to be that much larger. So we need to have this code instead:

	size_t osz = ltmsize( ptr );
#ifdef LTALLOC_OVERFLOW_DETECTION
	osz -= LTALLOC_CANARY_SIZE;
#endif
	if( sz <= osz ) {
		return ptr;
	}

Keep in mind, the user's request for memory is EXPANDED in ltmalloc to include the canary; conceptually, his request in ltrealloc should be too. However I don't think implementing it that way works as well.

This problem would also manifest as detections of overflows, but only in cases where there was a reallocation from, e.g. 64KB to 128KB. I am really unlucky!

Everything related to analysis of points 1 and 2 above could be a bit garbled. Hopefully it suffices to illustrate the problems, though.

3. In case allocation from the system fails, ltalloc can normally recover nicely and return a nullptr to the user. I would like to discover this, and execute an emergency squeeze (and then probably abort with memory diagnostics) at that time. However, when the overflow detection is enabled, ltmalloc will try to write the canary to a null pointer. So we need to check for null before writing the canary.
   3b. ltcalloc has a similar problem
   3c. ltrealloc attempts to memcpy on an allocation that may have failed; it should check first and return null of it fails

4. I know this sounds strange, but we need a static assert to verify that LTALLOC_VMALLOC_MIN_SIZE is >= (sizeof(PTrieNode) + SOME_MORE) due to the ptrie using it for allocating storage for 'newnode', and then some. (I was getting my memory from another heap instead of virtual memory, so it wasn't clear what the minimum vmalloc was)