GVDB CUDA ERROR: an illegal memory access was encountered

[Tom Svilans]

Hi,

I've run into another error with the gDepthMap example. I previously used gImportVDB to convert a test VDB file to VBX (renaming it to bunny.vdb in the assets folder 😛 ) and then renamed it to explosion.vbx just so that I could run the example with my own data. I did this so that I could rule out any compilation issues on my side, before creating this issue.

Loading looks fine, and the Lucy model shows up in the viewport. However, as soon as I rotate the view to find the imported VBX file, it crashes with this output:

GVDB CUDA ERROR:
  Launch status: an illegal memory access was encountered
  Kernel status: no error
  Caller: VolumeGVDB::PrepareRender
  Call:   cuMemcpyHtoD
  Args:   cuScnInfo
Error. Application will exit.
GVDB CUDA ERROR:
  Launch status: an illegal memory access was encountered
  Kernel status: no error
  Caller: Allocator::AtlasReleaseAll
  Call:   cuSurfObjectDestroy
  Args:
Error. Application will exit.

The sample data I used can be found here. It is a VDB file with a single grid, named density. It has a simple scaling transformation of (1,1,10). It seems specific to either this dataset, or the conversion from VDB to VBX... Perhaps it is too large for the device memory? Is there a way to verify this?

Any tips would be great! Let me know if there is more info that I can provide.

[Nia Bickford]

Hmm, it seems to work OK on a GTX 1060 Max-Q laptop graphics card, so it's probably not a problem with VBX export. Here's what it looks like inside gDepthMap, after zooming out:

It's interesting that you mention it doesn't appear and then the app crashes when you rotate the view, since the volume appears to overlap with the model. Another thing that's interesting is that the call that's failing (line 4319 of gvdb_volume_gvdb.cpp) is a copy to a fixed-size buffer, which should always work.

Is it possible to post the full log, in case there are any other messages (like a device lost event)? Adding gvdb.SetDebug(true) after the gvdb.SetVerbose call in main_depthmap.cpp will also add synchronization after every CUDA call, in case a different kernel is performing an invalid memory access, and this is only getting picked up by cuMemcpyHtoD.

Calling gvdb.Measure(true), it looks like the volume takes up 247 MB in memory, which seems to be OK. Just to check if it's a memory size issue, what do you get if you run

size_t free, total;
cuMemGetInfo(&free, &total);
printf("free: %zu, total: %zu", free, total);

after the gvdb.UpdateApron(); call in main_depthmap.cpp?

[Tom Svilans]

Hmm very weird. Thanks for trying it out, and cool to see it work :)

I tried compiling with CUDA 10.1 but it hangs on something like cuda_gvdb_operators.cuh(208): error : no operator "*" matches these operands and then a few instances of operand types are: const uint3 * const dim3

Compiling with CUDA 11.0 seems to work, but then we're back to square one with the incompatibility with my old cards :)
In this case, I just copied over the gDepthMap.exe and gvdb.dll from my build to the prebuilt bin folder which has the cuda_gvdb_module.ptx which works on my machine. It boots fine, shows Lucy but not the VBX volume, and now instead of crashing, at a certain point during orbiting around the model, it spits out this, for every drawn frame:

GVDB CUDA ERROR:
  Launch status: an illegal memory access was encountered
  Kernel status: an illegal memory access was encountered
  Caller: VolumeGVDB::Render
  Call:   cuLaunch
  Args:   SHADE_VOLUME
  Generating assert so you can examine call stack.
Render volume. 11.170 ms

Trying with the original explosion.vbx file, it now results in the same error. Perhaps I need to fix this CUDA issue. I'll try CUDA 10.2.

[Tom Svilans]

That works 😃

Installed CUDA 10.2 and it compiles and runs. Thanks for all the help and sorry for the bother! I'm quite new at CUDA.

[Nia Bickford]

Awesome, no worries! These intersecting sets of compatibility between versions and devices can be confusing. Glad you fixed it!