Multiple Masks for Volumetric Data Rendering and Memory Optimization

[Shijou87]

Hello,

I have been exploring the capabilities of the OSPRay library for volumetric data rendering and I must say, it's been a great experience so far. However, I have come across a particular requirement that I couldn't find a solution for. I was wondering if there is any existing feature or planned development in OSPRay that allows for the provision of multiple masks for rendering volumetric data, while avoiding memory duplication.

To provide some context, I have a volumetric dataset where different regions of interest (segmentation of organs in medical context) need to be rendered with different masks. These masks help me isolate specific parts of the dataset for visualization or analysis purposes. Currently, I am achieving this by duplicating the entire volumetric dataset for each mask , which is not an efficient approach from a memory consumption perspective.

Therefore, I would like to know if there are any mechanisms in OSPRay that could help in achieving this without duplicating the entire dataset for each mask. It would be ideal if OSPRay could support rendering different regions of interest using multiple masks, while reusing the underlying volumetric data to optimize memory usage.

If such a feature does not currently exist in OSPRay, I would like to request it as a potential enhancement. It would greatly benefit users who deal with large volumetric datasets and need to visualize or analyze specific regions independently.

I would appreciate any insights, suggestions, or guidance from the OSPRay community regarding this matter. If there are any alternative approaches or workarounds that could help me achieve my goal, I would be eager to learn about them as well.

Thank you all in advance for your time and support!

Best regards,

[Johannes Günther]

How exactly do you apply the mask? A different transfer function?

You can create multiple VolumetricModels (with different OSPTransferFunction) but reusing the same OSPVolume, this would not duplicate memory usage.

You can also create multiple different structuredRegular OSPVolumes with the same OSPData object, which also would not duplicate memory usage.

You can even create multiple OSPData without duplicating memory if you use ospNewSharedData and the same memory pointer.

[Shijou87]

Thanks for the quick answer,

Masks for volumetric data are 3D arrays that have the same dimensions as the 3D volume they describe. These masks represent voxel visibility, indicating whether each voxel in the volume is visible or not. However, masks carry much less weight or importance compared to the actual volumetric data.

Unlike a transfer function, which maps a volume value to a specific opacity color, a mask operates differently. It maps a given XYZ coordinate to a corresponding boolean value, indicating whether the voxel at that coordinate is visible (true) or not (false). This mapping is not achieved through a transfer function, as the purpose and functionality of a mask differ from that of a transfer function.

Currently, the mask is being applied by assigning a null opacity value to the corresponding voxels in the volumetric data. However, this approach requires creating a new volumetric data set each time the mask is applied.

Indeed yes I saw that is possible to make buffers shared between OSPData and volumetricModels handling same volume.

[Johannes Günther]

Ok, OSPRay does not natively support masks for volumes yet (except the clipping feature, but that does not work well for this usecase).

One idea, although it also requires some preprocessing. Assuming you have a segmentation volume, i.e., some classification integer value per voxel (and the mask is created by mask = selection == id) – if not, multiple masks can be combined into a segmentation volume. Then you could further combine the scalar values of the actual volume with the classification, e.g., the fractional part of the float is the (rescaled) scalar, the integer part of the float the classification, resulting in a single volume. Then you can quickly change the selection by just manipulating the transfer function (which has opacity zero for all integers not equal the selection). Or have different colors like in the image above. All with a single volume.

[Shijou87]

Thank you so much for providing an answer!
I agree that combining volumetric data and classification information is an excellent idea, and I'm grateful for the tips you've shared.
In the field of medical imaging, the use of masks/classification is quite common.
It would be wonderful if Ospray could incorporate such a feature in the future :).
Thanks again for the clarification.

[FlauschBert]

@Shijou87 Many thanks for bringing up this topic. We do have the same use case (I guess).

One solution I could imagine would be to use the OpenVDB format to encode a single bit mask. This should be directly supported and thus could be used without preprocessing. See the examples in https://www.openvdb.org/documentation/doxygen/overview.html. Unfortunately, @johguenther stated already that the approach of using it in a GeometricModel as clipping object is not advisable.

In our case preprocessing is not applicable. We need a direct approach.
There are things possible with a mask as separate volume:

• the mask volume can be rotated (somewhat transformed) against the data volume interactively
• having several masks that have to be turned on and off interactively

Just like the volume describes a smooth surface with its gray value levels, a mask should be able to do the same: This enables the mask to intersect the volume without loss of visual quality. The quality of an ISO surface increases with the number of available bits. This enables the renderer to show all sides of the intersection of a volume with a mask in the same quality.
Hint: Think about a freeform surface clipping half of the volume which has to result in a smooth surface.

[Shijou87]

Hi,
I successfully modified ospray to handle 8-bit packed 3D masks and integrated this into the VolumetricModel Interface.

// Note: dimX volume dimension is not multiple of 8 it needs padding
unsigned int size = volDimX*volDimY*volDimZ/8;
unsigned char mask[size];
OSPData maskData = ospNewSharedData1D(mask,OSP_UCHAR,size);
ospCommit(maskData);
ospSetObject(volumetricModel, "mask", maskData);
ospSetVec3i(volumetricModel, "maskDims", volDimX/8, volDimY, volDimZ);
ospCommit(volumetricModel);

As I cannot create a PR please find below the diffs (available only for scivis renderer) which can be applied to current ospray code

---
 modules/cpu/render/scivis/volumes.ispc     | 28 ++++++++++++++++++----
 modules/cpu/volume/VolumetricModel.cpp     | 14 +++++++++++
 modules/cpu/volume/VolumetricModel.h       |  5 ++++
 modules/cpu/volume/VolumetricModel.ih      |  2 ++
 modules/cpu/volume/VolumetricModelShared.h |  4 ++++
 5 files changed, 49 insertions(+), 4 deletions(-)

diff --git a/modules/cpu/render/scivis/volumes.ispc b/modules/cpu/render/scivis/volumes.ispc
index 103d58c07..450cceacb 100644
--- a/modules/cpu/render/scivis/volumes.ispc
+++ b/modules/cpu/render/scivis/volumes.ispc
@@ -17,6 +17,24 @@ OSPRAY_BEGIN_ISPC_NAMESPACE
 
 #ifndef OSPRAY_TARGET_SYCL
 
+bool inMask(const VolumetricModel *uniform m, const varying vec3f& p)
+{
+  // mask is not defined, consider everything
+  if(!m->hasMask)
+  {
+    return true;
+  }
+  int x = floor(p.x);
+  int y = floor(p.y);
+  int z = floor(p.z);
+  int sliceSize = m->maskDims.x*m->maskDims.y;
+  // mask contains packed data
+  uint8 maskVal =  m->maskBuffer.addr[z*sliceSize + y*(m->maskDims.x) +x/8] ;
+  // upack value
+  uint8 bitMask = 1 << (x%8);
+  return maskVal & bitMask;
+}
+
 struct VolumeContext
 {
   uniform uint8 intervalIteratorBuffer[VKL_MAX_INTERVAL_ITERATOR_SIZE];
@@ -79,10 +97,12 @@ static void sampleVolume(SciVisRenderContext &rc,
 
     // Prepare sampling position
     p = vc.org + newDistance * vc.dir;
-
-    // Sample volume value in given point
-    sampleVal = vklComputeSampleV(
-        &m->volume->vklSampler, (const varying vkl_vec3f *uniform) & p);
+    if(inMask(m, p))
+    {
+      // Sample volume value in given point
+      sampleVal = vklComputeSampleV(
+          &m->volume->vklSampler, (const varying vkl_vec3f *uniform) & p);
+    }
     // Go to the next sub-interval
     vc.iuDistance += 1.f;
     dt = newDistance - vc.distance - emptySpace;
diff --git a/modules/cpu/volume/VolumetricModel.cpp b/modules/cpu/volume/VolumetricModel.cpp
index 2f72b14dd..7012db1ae 100644
--- a/modules/cpu/volume/VolumetricModel.cpp
+++ b/modules/cpu/volume/VolumetricModel.cpp
@@ -74,6 +74,20 @@ void VolumetricModel::commit()
   getSh()->densityScale = getParam<float>("densityScale", 1.f);
   getSh()->anisotropy = getParam<float>("anisotropy", 0.f);
   getSh()->gradientShadingScale = getParam<float>("gradientShadingScale", 0.f);
+
+  // Mask handling
+  if(hasParam("mask"))
+  {
+    maskBuffer = getParamDataT<uint8>("mask");
+    getSh()->maskBuffer = *ispc(maskBuffer);
+    getSh()->maskDims = getParam<vec3i>("maskDims");
+    getSh()->hasMask = true;
+  }
+  else
+  {
+    getSh()->hasMask = false;
+  }
+
   getSh()->userID = getParam<uint32>("id", RTC_INVALID_GEOMETRY_ID);
 
   featureFlagsOther = FFO_VOLUME_IN_SCENE;
diff --git a/modules/cpu/volume/VolumetricModel.h b/modules/cpu/volume/VolumetricModel.h
index 219178088..a84154269 100644
--- a/modules/cpu/volume/VolumetricModel.h
+++ b/modules/cpu/volume/VolumetricModel.h
@@ -11,6 +11,7 @@
 #include "openvkl/device/openvkl.h"
 #include "transferFunction/TransferFunction.h"
 // ispc shared
+#include "common/Data.h"
 #include "volume/VolumetricModelShared.h"
 
 namespace ospray {
@@ -36,6 +37,10 @@ struct OSPRAY_SDK_INTERFACE VolumetricModel
   box3f volumeBounds;
   Ref<Volume> volume;
   Ref<TransferFunction> transferFunction;
+  // contains 8bit packed mask data
+  Ref<const DataT<uint8>> maskBuffer;
+  // the mask dimension which is in general equal to [voldim.x/8,voldim.y,voldim.z]
+  vec3i maskDims;
   const Ref<Volume> volumeAPI;
   VKLIntervalIteratorContext vklIntervalContext = VKLIntervalIteratorContext();
 
diff --git a/modules/cpu/volume/VolumetricModel.ih b/modules/cpu/volume/VolumetricModel.ih
index 661e6858b..2c64067c7 100644
--- a/modules/cpu/volume/VolumetricModel.ih
+++ b/modules/cpu/volume/VolumetricModel.ih
@@ -9,8 +9,10 @@
 #include "Volume.ih"
 #include "transferFunction/TransferFunctionShared.h"
 // c++ shared
+#include "common/Data.ih"
 #include "VolumetricModelShared.h"
 
+
 OSPRAY_BEGIN_ISPC_NAMESPACE
 
 inline void VolumetricModel_postIntersect(const VolumetricModel *uniform self,
diff --git a/modules/cpu/volume/VolumetricModelShared.h b/modules/cpu/volume/VolumetricModelShared.h
index f7986fc22..e46c9fd69 100644
--- a/modules/cpu/volume/VolumetricModelShared.h
+++ b/modules/cpu/volume/VolumetricModelShared.h
@@ -22,6 +22,10 @@ struct VolumetricModel
   float densityScale;
   float anisotropy; // the anisotropy of the volume's phase function
                     // (Heyney-Greenstein)
+  Data1D maskBuffer;
+  vec3i maskDims;
+  bool hasMask;
+
   float gradientShadingScale;
   unsigned int userID;
 
--