recastnavigation / recastnavigation

What?

Addition of tile to navmesh sometimes works incorrect In Detour built in DT_POLYREF64 mode:

After call of addTile method in dtNavMesh, pointer to array of links in resulting dtMeshTile may has incorrect alignment. In cpp, if pointer has incorrect alignment, access to data is undefined behaviour.

To reproduce that issue you may:

Build RecastDemo with -DRECASTNAVIGATION_DT_POLYREF64=ON option
Add breakpoint or dbg check after following line:

recastnavigation/RecastDemo/Source/Sample_TileMesh.cpp

Line 761 in cd89890

dtStatus status = m_navMesh->addTile(data,dataSize,DT_TILE_FREE_DATA,0,0);

Launch RecastDemo, choose TileMesh sample with any navmesh and click Build
Check links pointer value (or throw some error if value incorrect).

My environment is x64-bit Ubuntu 18.04 LTS, but bug also checked at Windows 10.

Why?

The reason why this bug happens in following lines:

recastnavigation/Detour/Source/DetourNavMesh.cpp

Lines 978 to 996 in cd89890

    
           const int headerSize = dtAlign4(sizeof(dtMeshHeader)); 
        
           const int vertsSize = dtAlign4(sizeof(float)*3*header->vertCount); 
        
           const int polysSize = dtAlign4(sizeof(dtPoly)*header->polyCount); 
        
           const int linksSize = dtAlign4(sizeof(dtLink)*(header->maxLinkCount)); 
        
           const int detailMeshesSize = dtAlign4(sizeof(dtPolyDetail)*header->detailMeshCount); 
        
           const int detailVertsSize = dtAlign4(sizeof(float)*3*header->detailVertCount); 
        
           const int detailTrisSize = dtAlign4(sizeof(unsigned char)*4*header->detailTriCount); 
        
           const int bvtreeSize = dtAlign4(sizeof(dtBVNode)*header->bvNodeCount); 
        
           const int offMeshLinksSize = dtAlign4(sizeof(dtOffMeshConnection)*header->offMeshConCount); 
        
           unsigned char* d = data + headerSize; 
        
           tile->verts = dtGetThenAdvanceBufferPointer<float>(d, vertsSize); 
        
           tile->polys = dtGetThenAdvanceBufferPointer<dtPoly>(d, polysSize); 
        
           tile->links = dtGetThenAdvanceBufferPointer<dtLink>(d, linksSize); 
        
           tile->detailMeshes = dtGetThenAdvanceBufferPointer<dtPolyDetail>(d, detailMeshesSize); 
        
           tile->detailVerts = dtGetThenAdvanceBufferPointer<float>(d, detailVertsSize); 
        
           tile->detailTris = dtGetThenAdvanceBufferPointer<unsigned char>(d, detailTrisSize); 
        
           tile->bvTree = dtGetThenAdvanceBufferPointer<dtBVNode>(d, bvtreeSize); 
        
           tile->offMeshCons = dtGetThenAdvanceBufferPointer<dtOffMeshConnection>(d, offMeshLinksSize);

All sizes here alinged by 4 byte, but some tipes (i.e. dtLink) have alignof equal to 8 bytes if DT_POLYREF64 is ON.

So, dtGetThenAdvanceBufferPointer provides pointer with incorrect alignment to links pointer.

I did a pull request which should fix that behaviour and compute pointers values correctly.

PTAL and share your opinion: #670

Thank you for the detailed writeup! I unfortunately haven't been able to follow your repro steps as I'm not sure what you're looking for in step 4. I appreciate its alignment related, but it's unclear how this problem manifests.

Could you provide some more specifics in your repro steps, especially around why you're setting the breakpoint where you are, and what exactly you're looking at that's incorrect?

Hi @grahamboree, thank you to your reply.

I will try to be more clear this time in my repro steps and especially what's wrong in it.

Please take a look at my local commit m-bronnikov@ba5c072

I added assert which describes what I meant when opened this issue. This assert checks links array is aligned correctly (by alignment of dtLink). If this assert triggers, it's mean we have undefined behavior due to unaligned access.

To trigger this assert I did following steps locally:

mkdir build
cd build
cmake ../CMakeLists.txt -DRECASTNAVIGATION_DT_POLYREF64=ON
make
cd RecastDemo
./RecastDemo

Choose following Sample and Input Mesh

And press Build button below.

Result I got:

added links: 0x55d102799588
added links: 0x55d102787e1c
RecastDemo: /home/mbronnikov/Work/recastnavigation/RecastDemo/Source/Sample_TileMesh.cpp:770: void Sample_TileMesh::buildAllTiles(): Assertion `reinterpret_cast<uintptr_t>(addedTyle->links) % alignof(dtLink) == 0' failed.

The reason why we met this error is described above in issue description.

Please feel free to ask additional info if something still unclear.

	const int headerSize = dtAlign4(sizeof(dtMeshHeader));
	const int vertsSize = dtAlign4(sizeof(float)3header->vertCount);
	const int polysSize = dtAlign4(sizeof(dtPoly)*header->polyCount);
	const int linksSize = dtAlign4(sizeof(dtLink)*(header->maxLinkCount));
	const int detailMeshesSize = dtAlign4(sizeof(dtPolyDetail)*header->detailMeshCount);
	const int detailVertsSize = dtAlign4(sizeof(float)3header->detailVertCount);
	const int detailTrisSize = dtAlign4(sizeof(unsigned char)4header->detailTriCount);
	const int bvtreeSize = dtAlign4(sizeof(dtBVNode)*header->bvNodeCount);
	const int offMeshLinksSize = dtAlign4(sizeof(dtOffMeshConnection)*header->offMeshConCount);

	unsigned char* d = data + headerSize;
	tile->verts = dtGetThenAdvanceBufferPointer<float>(d, vertsSize);
	tile->polys = dtGetThenAdvanceBufferPointer<dtPoly>(d, polysSize);
	tile->links = dtGetThenAdvanceBufferPointer<dtLink>(d, linksSize);
	tile->detailMeshes = dtGetThenAdvanceBufferPointer<dtPolyDetail>(d, detailMeshesSize);
	tile->detailVerts = dtGetThenAdvanceBufferPointer<float>(d, detailVertsSize);
	tile->detailTris = dtGetThenAdvanceBufferPointer<unsigned char>(d, detailTrisSize);
	tile->bvTree = dtGetThenAdvanceBufferPointer<dtBVNode>(d, bvtreeSize);
	tile->offMeshCons = dtGetThenAdvanceBufferPointer<dtOffMeshConnection>(d, offMeshLinksSize);

[Detour] Incorrect layout of tile links in DT_POLYREF64 mode

What?

Why?