Vertex color from OBJ
nezix opened this issue · comments
Hi,
I am looking for a way to load OBJ files containing color information for each vertex (even if it's not standard in the OBJ format) ?
I think tiny_obj_loader provides this feature.
Thanks
Hey @nezix ,
This isn't something I have plans to support right now in ChameleonRT but it'd be pretty easy to hack in a branch to support it by passing them through as "normals" since those aren't really used now. What kind of meshes are you looking to render? Also, if you're more looking for any interactive path tracer which supports per-vertex colors, OSPRay might work as well, and it does have support for per-vertex colors. OSPRay is an interactive CPU renderer using Embree/ISPC/TBB, a bit similar to the Embree backend here, though with far more features.
Thanks for the answer.
I am familiar with OSPray, I am interested by your multi-framework support :)
I will try to hack around this.
Thanks
Gotcha, so then for the hack what you can do is send the per-vertex colors through the "normals" buffer. Then for example in the DXR backend you'd make the following changes (each will be similar):
- Load the per-vertex colors and blend them instead of per-vertex normals in the closest hit program and pass it out instead of the uv coords. So the closest hit would look like this:
[shader("closesthit")]
void ClosestHit(inout HitInfo payload, Attributes attrib) {
uint3 idx = indices[NonUniformResourceIndex(PrimitiveIndex())];
float3 va = vertices[NonUniformResourceIndex(idx.x)];
float3 vb = vertices[NonUniformResourceIndex(idx.y)];
float3 vc = vertices[NonUniformResourceIndex(idx.z)];
float3 ng = normalize(cross(vb - va, vc - va));
float3 n = ng;
float3 vcolor = float3(0, 0, 0);
if (num_normals > 0) {
float3 na = normals[NonUniformResourceIndex(idx.x)];
float3 nb = normals[NonUniformResourceIndex(idx.y)];
float3 nc = normals[NonUniformResourceIndex(idx.z)];
vcolor = (1.f - attrib.bary.x - attrib.bary.y) * na
+ attrib.bary.x * nb + attrib.bary.y * nc;
}
float2 uv = float2(0, 0);
if (num_uvs > 0) {
float2 uva = uvs[NonUniformResourceIndex(idx.x)];
float2 uvb = uvs[NonUniformResourceIndex(idx.y)];
float2 uvc = uvs[NonUniformResourceIndex(idx.z)];
uv = (1.f - attrib.bary.x - attrib.bary.y) * uva
+ attrib.bary.x * uvb + attrib.bary.y * uvc;
}
payload.color_dist = float4(vcolor.x, vcolor.y, vcolor.z, RayTCurrent());
float3x3 inv_transp = float3x3(WorldToObject4x3()[0], WorldToObject4x3()[1], WorldToObject4x3()[2]);
payload.normal = float4(normalize(mul(inv_transp, n)), material_id);
}Then in the shading code when we unpack the material, use the first 3 elements of payload.color_dist to set the base color here. So unpack_material would be:
void unpack_material(inout DisneyMaterial mat, uint id, float3 vcolor) {
MaterialParams p = material_params[NonUniformResourceIndex(id)];
const uint32_t mask = asuint(p.base_color.x);
if (vcolor.z > 0.f) {
mat.base_color = vcolor;
} else {
mat.base_color = p.base_color;
}And you'll change how you call unpack_material to pass the full vcolor instead of just uv here:
unpack_material(mat, uint(payload.normal.w), payload.color_dist.rgb);Awesome ! Thanks again !
If you want to also still support texturing for objects w/o per-vertex colors you could still send the uv's through as payload.color_dist.rg and then if vcolor.b is 0 you can check if it's textured or not as before.
The loader part is here for OBJ, so you'd just swap reading from the tinyobj normals attrib to read from the colors using the color indices.
Well that was simple for the OSPray part :)
Nice, this looks great!
I am trying to modify the Vulkan part. I cannot test it because I don't have a compatible GPU.
After sending the data via the command buffer, I try to modify the hit shader but I am not sure what to do:
vec4 vcol = vec4(1);
if(color_buf != uint32_t(-1)){
const vec4 ca = unpack_float4(color_buffers[nonuniformEXT(color_buf)].v[idx.x]);
const vec4 cb = unpack_float4(color_buffers[nonuniformEXT(color_buf)].v[idx.y]);
const vec4 cc = unpack_float4(color_buffers[nonuniformEXT(color_buf)].v[idx.z]);
vcol = ;//not sure what goes there
}For the Vulkan one you'll also interpolate w/ the barycentric coords (stored in attrib.xy):
vec4 vcol = vec4(1);
if(color_buf != uint32_t(-1)){
const vec4 ca = unpack_float4(color_buffers[nonuniformEXT(color_buf)].v[idx.x]);
const vec4 cb = unpack_float4(color_buffers[nonuniformEXT(color_buf)].v[idx.y]);
const vec4 cc = unpack_float4(color_buffers[nonuniformEXT(color_buf)].v[idx.z]);
vcol = (1.f - attrib.x - attrib.y) * ca
+ attrib.x * cb + attrib.y * cc;
}Though if you're sending through the normal buffer you'd want to unpack float3's there, since i'm not uploading float4 data
I changed the render_vulkan.cpp/h to also send the color information as a vec4.
Let me fork the repo and do some commits to see what I have done. I am sure I forgot a lot of things but it's a start for me :)
Awesome, let me know if you have any other questions/issues and if you make any cool images with it! The lighting setup isn't as full featured as OSPRay but you can hard-code in some additional quad lights in the OBJ loader (at the end of the import function).
If you have time, could you test that for me ? nezix@d0c701d
I don't have a compatible GPU but I think I am on the right track.
This looks like it's on the right track, I'll have some time to do a test later tonight but two things stood out:
-
The descriptor pool size for the storage buffers should be
5 * total_geomhere since you're passing an additional set of color buffers -
There is no
pack_float4since it's not needed, the regular vec4 GLSL type will be aligned properly (https://github.com/nezix/ChameleonRT/blob/feature/vertexColor/vulkan/hit.rchit#L26). So you can just use vec4 directly. I only needed thepack_float3forstd430because the vec3 array is padded to use a vec4 stride. In thekhr_ray_tracingbranch I learned about thescalarlayout which makes that pack type also not needed, but I'll merge that later when I finish moving tokhr_ray_tracing
Oh also, here where you pick the vertex color index: https://github.com/nezix/ChameleonRT/blob/feature/vertexColor/util/scene.cpp#L129-L163 if it's always just the same as the vertex index, then you could skip setting this extra index and just use the vertex index. It looks like that's the case from tinyobjloader's example. It looks like you set 1 as the alpha channel though so you could just pass as vec3's. If you're planning to add transparency, there are some bugs in how I've got the Disney BSDF implemented so transparency may not look that great. In OSPRay it will look nice though
Hi,
I am done implementing vertex color for each framework but I cannot test it when a RTX GPU is needed (OptiX/DXR/Vulkan). If you have time one day, please tell me, I am curious to see if everything is ok :)
I fixed things you noted and for OSPRay and embree it works fine :)
Thanks again !
Sure! I noticed a few smaller things with the different backends that I forgot to mention since they're a bit less obvious (but are more obvious when running since they crash).
DXR Backend
This was just missing increasing the payload size set on the pipeline to match the addition of the vertex color to the payload.
diff --git a/dxr/render_dxr.cpp b/dxr/render_dxr.cpp
index 84c5f26..053a948 100644
--- a/dxr/render_dxr.cpp
+++ b/dxr/render_dxr.cpp
@@ -565,7 +565,7 @@ void RenderDXR::build_raytracing_pipeline()
.add_miss_shader(L"ShadowMiss")
.set_shader_root_sig({L"RayGen"}, raygen_root_sig)
.configure_shader_payload(
- shader_library.export_names(), 8 * sizeof(float), 2 * sizeof(float))
+ shader_library.export_names(), 12 * sizeof(float), 2 * sizeof(float))
.set_max_recursion(1);
// Setup hit groups and shader root signatures for our instances.OptiX Backend
This was super weird and still doesn't quite make sense, but when I swapped to use float4 and multiplied the colors I'd get black (but just picking one of cola/b/c gave what I expected). So I swapped to use float3 for the colors:
diff --git a/optix/render_optix.cu b/optix/render_optix.cu
index e5f78f5..2dd141c 100644
--- a/optix/render_optix.cu
+++ b/optix/render_optix.cu
@@ -15,8 +15,7 @@ struct RayPayload {
uint32_t material_id;
float3 normal;
- float pad;
- float4 vcol;
+ float3 vcol;
};
__device__ RayPayload make_ray_payload() {
@@ -25,7 +24,7 @@ __device__ RayPayload make_ray_payload() {
p.t_hit = -1.f;
p.material_id = 0;
p.normal = make_float3(0.f);
- p.vcol = make_float4(1.f);
+ p.vcol = make_float3(1.f);
return p;
}
@@ -257,11 +256,12 @@ extern "C" __global__ void __closesthit__closest_hit() {
+ bary.x * uvb + bary.y * uvc;
}
- float4 vcol = make_float4(1.f);
+ float3 vcol = make_float3(1.f);
if (params.color_buffer) {
- float2 cola = params.color_buffer[indices.x];
- float2 colb = params.color_buffer[indices.y];
- float2 colc = params.color_buffer[indices.z];
+ float3 cola = make_float3(params.color_buffer[indices.x]);
+ float3 colb = make_float3(params.color_buffer[indices.y]);
+ float3 colc = make_float3(params.color_buffer[indices.z]);
+ // Not sure why multiplying the float4's leads to all black
vcol = (1.f - bary.x - bary.y) * cola
+ bary.x * colb + bary.y * colc;
}Vulkan Backend
A few smaller things with how the descriptor sets and pipeline are set up. The color buffer is read in the closest hit stage so needed that bit flag added (maybe also needed for some of the other buffers, but I'm migrating off that style soon anyway). Then the color descriptor set was missing from the pipeline set list, and we needed to allow one more set to allocate from the pool.
diff --git a/vulkan/render_vulkan.cpp b/vulkan/render_vulkan.cpp
index 95b5eaf..3c6cc92 100644
--- a/vulkan/render_vulkan.cpp
+++ b/vulkan/render_vulkan.cpp
@@ -193,7 +193,6 @@ void RenderVulkan::set_scene(const Scene &scene)
upload_verts->unmap();
}
-
auto upload_indices = vkrt::Buffer::host(*device,
geom.indices.size() * sizeof(glm::uvec3),
VK_BUFFER_USAGE_TRANSFER_SRC_BIT);
@@ -220,10 +219,10 @@ void RenderVulkan::set_scene(const Scene &scene)
std::memcpy(map, geom.normals.data(), upload_normals->size());
upload_normals->unmap();
}
- if(!geom.colors.empty()) {
+ if (!geom.colors.empty()) {
upload_cols = vkrt::Buffer::host(*device,
- geom.colors.size() * sizeof(glm::vec4),
- VK_BUFFER_USAGE_TRANSFER_SRC_BIT);
+ geom.colors.size() * sizeof(glm::vec4),
+ VK_BUFFER_USAGE_TRANSFER_SRC_BIT);
color_buf = vkrt::Buffer::device(
*device,
upload_cols->size(),
@@ -232,7 +231,6 @@ void RenderVulkan::set_scene(const Scene &scene)
void *map = upload_cols->map();
std::memcpy(map, geom.colors.data(), upload_cols->size());
upload_cols->unmap();
-
}
std::shared_ptr<vkrt::Buffer> upload_uvs = nullptr;
@@ -767,11 +765,12 @@ void RenderVulkan::build_raytracing_pipeline()
// we use to send the mesh data
buffer_desc_layout =
vkrt::DescriptorSetLayoutBuilder()
- .add_binding(0,
- total_geom,
- VK_DESCRIPTOR_TYPE_STORAGE_BUFFER,
- VK_SHADER_STAGE_CLOSEST_HIT_BIT_NV,
- VK_DESCRIPTOR_BINDING_VARIABLE_DESCRIPTOR_COUNT_BIT_EXT)
+ .add_binding(
+ 0,
+ total_geom,
+ VK_DESCRIPTOR_TYPE_STORAGE_BUFFER,
+ VK_SHADER_STAGE_RAYGEN_BIT_NV | VK_SHADER_STAGE_CLOSEST_HIT_BIT_NV,
+ VK_DESCRIPTOR_BINDING_VARIABLE_DESCRIPTOR_COUNT_BIT_EXT)
.build(*device);
textures_desc_layout =
@@ -788,7 +787,8 @@ void RenderVulkan::build_raytracing_pipeline()
buffer_desc_layout,
buffer_desc_layout,
buffer_desc_layout,
- textures_desc_layout};
+ textures_desc_layout,
+ buffer_desc_layout};
VkPipelineLayoutCreateInfo pipeline_create_info = {};
pipeline_create_info.sType = VK_STRUCTURE_TYPE_PIPELINE_LAYOUT_CREATE_INFO;
@@ -832,7 +832,7 @@ void RenderVulkan::build_shader_descriptor_table()
VkDescriptorPoolCreateInfo pool_create_info = {};
pool_create_info.sType = VK_STRUCTURE_TYPE_DESCRIPTOR_POOL_CREATE_INFO;
- pool_create_info.maxSets = 6;
+ pool_create_info.maxSets = 7;
pool_create_info.poolSizeCount = pool_sizes.size();
pool_create_info.pPoolSizes = pool_sizes.data();
CHECK_VULKAN(vkCreateDescriptorPool(
@@ -881,11 +881,10 @@ void RenderVulkan::build_shader_descriptor_table()
} else {
idx.normal_buf = -1;
}
- if(geom.color_buf){
+ if (geom.color_buf) {
indices.col_buf++;
col_buffers.emplace_back(geom.color_buf);
- }
- else{
+ } else {
idx.col_buf = -1;
}In the shaders the color buffer is moved to set 6 b/c it was conflicting with the texture set (on set 5)
diff --git a/vulkan/hit.rchit b/vulkan/hit.rchit
index 1645ee4..df13545 100644
--- a/vulkan/hit.rchit
+++ b/vulkan/hit.rchit
@@ -22,7 +22,7 @@ layout(binding = 0, set = 4, std430) buffer UVBuffers {
vec2 uv[];
} uv_buffers[];
-layout(binding = 0, set = 5, std430) buffer ColorBuffers {
+layout(binding = 0, set = 6, std430) buffer ColorBuffers {
vec4 col[];
} color_buffers[];In raygen there's a small typo:
diff --git a/vulkan/raygen.rgen b/vulkan/raygen.rgen
index ceee48c..2c3b45f 100644
--- a/vulkan/raygen.rgen
+++ b/vulkan/raygen.rgen
@@ -194,7 +194,7 @@ void main() {
mat.base_color.x *= payload.vcolor.x;
mat.base_color.y *= payload.vcolor.y;
- mat.base_color.z *= payload.vcolor.y;
+ mat.base_color.z *= payload.vcolor.z;
vec3 v_x, v_y;
vec3 v_z = payload.normal;With those fixes I can render this vertex colored test triangle on each GPU backend:
Awesome ! Thank you !
Do you want me to do a pull request or is it dirty enough to keep it separated ?
Let's keep it separate for now, since it's increasing the ray payload size for all scenes even though the vertex color use case is probably less common. I think we could have something at runtime or some compile time switch for this in the future though to have it in master. I'm also going to change the vulkan backend some to fix a few things with the SBT setup and migrate to use the KHR_ray_tracing extension this weekend