Bug importing "pawn"-like shape

Question

Bug importing "pawn"-like shape

trevorld opened this issue 4 years ago · comments

I've found an obj file that readobj::read.obj doesn't seem to import correctly. However rayrender::obj_model and rgl::readOBJ both seem to import correctly. Since both rayrender and readobj embed tinyobjloader maybe rayrender is using a newer version of tinyobjloader that has fixed a bug?

Github won't let me upload my obj file so here is a link to a zip file containing "pawn.obj": https://trevorldavis.com/share/tmp/readobj/pawn.zip

library("rayrender")
library("readobj")
library("rgl")

png("rayrender.png")
obj <- obj_model(filename = "pawn.obj", material=diffuse(color="black"))
render_scene(obj, lookfrom=c(0, 0, 3))
dev.off()

open3d()
view3d(theta = 0, phi = 0)
shade3d(readOBJ("pawn.obj"), material=list(lit=FALSE))
rgl.snapshot("readOBJ.png")

rgl.clear()
shade3d(read.obj("pawn.obj", convert.rgl=TRUE))
rgl.snapshot("read.obj.png")

Incorrect render using readobj::read.obj:

Correct render using rgl::readOBJ and rayrender::obj_model:

Gregory Jefferis · Answer 1 · Fri Aug 07 2020 01:31:53 GMT+0800 (China Standard Time)

thanks a lot for the bug report @trevorld. Having looked into this briefly, I think the issue is with the triangulation code that tinyobjloader uses. See the results from readobj::read.obj (black) and rgl::readObj (red) below. I think your idea that this has been fixed in tinyobjloader is also correct, since it seems that rayrender accepts the default behaviour of tinyobjloader, which is to triangulate (link).
I'm not sure when the fix happened, but tinyobjloader/tinyobjloader#150 looks relevant, suggesting it may have been merged into mainline in Jan 2018.

The only problem is that the API has change quite a lot since I created the readobj R package and I will need to update my Rcpp wrapping code, something which is not too complicated, but not trivial either and I don't have the time to complete just now. I made a quick start here #8 if you want to take a look or contribute.

Trevor L. Davis · Answer 2 · Fri Aug 07 2020 02:54:50 GMT+0800 (China Standard Time)

Thanks for following up with this! I haven't done much C++ programming so I think I'll pass for now on updating the Rcpp wrapper code. However after that is done I can probably help in testing and (if necessary due to a breaking change) in updating mesh3d.R or the unit tests.

Gregory Jefferis · Answer 3 · Sun Aug 16 2020 19:34:47 GMT+0800 (China Standard Time)

@trevorld if you try installing #8

remotes::install_github("jefferis/readobj#8")

you can see that updating the tinyobjloader rgl reader indeed solves your problem above. Unfortunately it breaks texture rendering. I think the issue is with how duplicate vertices are represented. tinyobjloader now de-duplicates more effectively. For example your tile example in the package now has 8 nodes instead of 24 (which makes sense for a square playing tile if you stop to think about it). Similarly there are 12 texture coordinates, matching the contents of the obj file.

However, it looks like rgl::shade3d expects the texture coordinates to be indexable by the same indices used to define the faces in the mesh (i.e. 8 in this case), which you can see causes trouble if there are 12 texture coordinates. I have not had the time to investigate properly, but it feels like there is some inflexibility in how rgl handles texture mapping.

The only way that I can think to deal with this in the short term is to reduplicate the vertices, so that both vertices and texture coordinates can be indexed in the same way.

That said rgl author @dmurdoch has always been very responsive when I contact him by email and if you have the time to construct a full reprex for your tile case (as embedded in the new tests that you added to the package), he might well be able to help.

Best, Greg.

dmurdoch · Answer 4 · Sun Aug 16 2020 21:34:12 GMT+0800 (China Standard Time)

The indexing of textures in rgl is there because that's how OpenGL requires it. Fairly recently I've added the "meshColor" argument which allows colours to be interpreted as applying by vertex, edge, or face: it duplicates vertices as necessary to implement this when drawing. It would make sense to do the same for textures as well. I'll add it to the to-do list. Duncan Murdoch

…

On 16/08/2020 7:34 a.m., Gregory Jefferis wrote: @trevorld <https://github.com/trevorld> if you try installing #8 <#8> |remotes::install_github("jefferis/readobj#8") | you can see that updating the tinyobjloader rgl reader indeed solves your problem above. Unfortunately it breaks texture rendering. I think the issue is with how duplicate vertices are represented. tinyobjloader now de-duplicates more effectively. For example your tile example in the package now has 8 nodes instead of 24 (which makes sense for a square playing tile if you stop to think about it). Similarly there are 12 texture coordinates, matching the contents of the obj file <https://github.com/jefferis/readobj/blob/master/inst/obj/tile.wavefront>. However, it looks like rgl::shade3d <https://github.com/cran/rgl/blob/9aeda2e1bdc47258a727099e194e26bb8251d6de/R/mesh3d.R#L377-L378> expects the texture coordinates to be indexable by the same indices used to define the faces in the mesh (i.e. 8 in this case), which you can see causes trouble if there are 12 texture coordinates. I have not had the time to investigate properly, but it feels like there is some inflexibility in how rgl handles texture mapping. The only way that I can think to deal with this in the short term is to reduplicate the vertices, so that both vertices and texture coordinates can be indexed in the same way. That said rgl author @dmurdoch <https://github.com/dmurdoch> has always been very responsive when I contact him by email and if you have the time to construct a full reprex for your tile case (as embedded in the new tests that you added to the package, he might well be able to help). Best, Greg. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#6 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAOYSQF6PB5V6C5XZQJ2DHLSA676FANCNFSM4PU3Q4UA>.

Gregory Jefferis · Answer 5 · Sun Aug 16 2020 22:29:16 GMT+0800 (China Standard Time)

If you look at res (rgl::readOBJ) and res3 (read.obj from CRAN package) you will see that they both have 24 vertices and 24 pairs of texture coordinates, whereas the new version of read.obj has 8 vertices and 12 texture coordinates matching the contents of the obj file.

> res=rgl::readOBJ('https://raw.githubusercontent.com/jefferis/readobj/master/inst/obj/tile.wavefront')
Warning message:
In rgl::readOBJ("https://raw.githubusercontent.com/jefferis/readobj/master/inst/obj/tile.wavefront") :
  Instructions "mtllib", "usemtl" ignored

> str(res)
List of 7
 $ vb       : num [1:4, 1:24] -1 1 0.125 1 -1 -1 0.125 1 1 -1 ...
 $ it       : num[1:3, 0 ] 
 $ material : list()
 $ normals  : NULL
 $ texcoords: num [1:2, 1:24] 0.025 1 0.025 0 0.425 0 0.425 1 0.575 1 ...
 $ meshColor: chr "vertices"
 $ ib       : num [1:4, 1:6] 1 2 3 4 5 8 7 6 19 22 ...
 - attr(*, "class")= chr [1:2] "mesh3d" "shape3d"
> # new version of tinyobjloader
> res2=read.obj('inst/obj/tile.wavefront')
> str(res2)
List of 2
 $ shapes   :List of 1
  ..$ :List of 6
  .. ..$ positions   : num [1:3, 1:8] -1 1 0.125 -1 -1 0.125 1 -1 0.125 1 ...
  .. ..$ normals     : num[1:3, 0 ] 
  .. ..$ texcoords   : num [1:2, 1:12] 0.025 1 0.025 0 0.425 ...
  .. ..$ indices     : int [1:3, 1:12] 0 1 2 0 2 3 4 7 6 4 ...
  .. ..$ texindices  : int [1:3, 1:12] 0 1 2 0 2 3 4 7 6 4 ...
  .. ..$ material_ids: int [1:12] 0 0 0 0 0 0 0 0 0 0 ...
 $ materials:List of 1
  ..$ material_0:List of 13
  .. ..$ ambient         : num [1:3] 0 0 0
  .. ..$ diffuse         : num [1:3] 0 0 0
  .. ..$ specular        : num [1:3] 0 0 0
  .. ..$ transmittance   : num [1:3] 0 0 0
  .. ..$ emission        : num [1:3] 0 0 0
  .. ..$ shininess       : num 1
  .. ..$ ior             : num 1
  .. ..$ dissolve        : num 1
  .. ..$ illum           : int 0
  .. ..$ ambient_texname : chr ""
  .. ..$ diffuse_texname : chr "tile.png"
  .. ..$ specular_texname: chr ""
  .. ..$ normal_texname  : chr ""
> # previous version of tinyobjloader as present in CRAN version of package
> res3=read.obj('inst/obj/tile.wavefront')
> str(res3)
List of 2
 $ shapes   :List of 1
  ..$ :List of 5
  .. ..$ positions   : num [1:3, 1:24] -1 1 0.125 -1 -1 0.125 1 -1 0.125 1 ...
  .. ..$ normals     : num[1:3, 0 ] 
  .. ..$ texcoords   : num [1:48] 0.025 1 0.025 0 0.425 ...
  .. ..$ indices     : num [1:3, 1:12] 0 1 2 0 2 3 4 5 6 4 ...
  .. ..$ material_ids: int [1:12] 0 0 0 0 0 0 0 0 0 0 ...
 $ materials:List of 1
  ..$ material_0:List of 13
  .. ..$ ambient         : num [1:3] 0 0 0
  .. ..$ diffuse         : num [1:3] 0 0 0
  .. ..$ specular        : num [1:3] 0 0 0
  .. ..$ transmittance   : num [1:3] 0 0 0
  .. ..$ emission        : num [1:3] 0 0 0
  .. ..$ shininess       : num 1
  .. ..$ ior             : num 1
  .. ..$ dissolve        : num 1
  .. ..$ illum           : int 0
  .. ..$ ambient_texname : chr ""
  .. ..$ diffuse_texname : chr "tile.png"
  .. ..$ specular_texname: chr ""
  .. ..$ normal_texname  : chr ""

Gregory Jefferis · Answer 6 · Sun Aug 16 2020 22:31:15 GMT+0800 (China Standard Time)

The indexing of textures in rgl is there because that's how OpenGL
requires it. Fairly recently I've added the "meshColor" argument which
allows colours to be interpreted as applying by vertex, edge, or face:
it duplicates vertices as necessary to implement this when drawing. It
would make sense to do the same for textures as well. I'll add it to
the to-do list.

Many thanks for the very quick response and explanation @dmurdoch!

Trevor L. Davis · Answer 7 · Mon Aug 17 2020 02:01:48 GMT+0800 (China Standard Time)

@dmurdoch and @jefferis I've taken a look.
It doesn't seem to be an issue in shade3d per se but rather one with tmesh3d (interacting with the new behavior of tinyobjloader's LoadObj which doesn't duplicate vertices) since it seems tmesh3d doesn't seem to have any way of separately indicating "texture indices". It seems we'd either need

A new separate texindices matrix argument for rgl::tmesh3d that maps texture coordinates to face vertices (with an NA if that face doesn't use textures). Perhaps there is some internal code used by rgl::readOBJ that could be used by rgl::tmesh3d to convert vertices, indices, texcoords, texindices to the (duplicated) vertices, indices, texcoords expected by shade3d and OpenGL.

Or

We'd need to duplicate/match vertices and texCoords (and update the indices matrix accordingly) locally in readobj:::tinyshape2mesh3d before making the call to rgl::tmesh3d. This doesn't seem too hard to do.

I'm willing to re-implement #7 and either update the tmesh3d call to support a new texindices argument OR manually manipulate vertices, indices, texcoords, texindices locally in a way that supports the existing tmesh3d API's duplicated vertices, indices, texcoords setup.

dmurdoch · Answer 8 · Mon Aug 17 2020 08:47:04 GMT+0800 (China Standard Time)

I've just committed rgl version 0.102.19 on R-forge (and Github soon enough) that allows more flexibility in meshes and textures. In particular, it now allows the meshColor argument to various mesh functions to also apply to texture coordinates, and by default, texture coordinates are treated as if meshColor = "vertices", i.e. each vertex gets a single texture coordinate, and the duplication is handled when rendered.

This is probably going to cause trouble for other packages using textures, which will now need meshColor = "legacy" to get the old behaviour.

I thought about separate meshColor and meshTexture settings, but it seemed too complicated. I might change my mind on that before a CRAN update. I don't think it's likely I'll support having some faces with textures and some without: that would also be very complicated to set up, whereas drawing two objects, one with textures and one without, is already supported.

I still don't understand how the 12 texture coordinates in the new tinyobjloader code are supposed to be interpreted. What 12 things get those coordinates?

Trevor L. Davis · Answer 9 · Mon Aug 17 2020 09:53:19 GMT+0800 (China Standard Time)

I don't think it's likely I'll support having some faces with textures and some without: that would also be very complicated to set up,

In the version of rgl on my computer it seems to support having some faces with textures and some without i.e. code like the following with an "obj" file that has some faces with textures and some without seem to work as expected (i.e. non-textured faces are simply the material color). When we examine the mesh3d/shape3d object we see that the relevant texcoords for the non-textured faces imported by readOBJ are simply missing values:

> o = rgl::readOBJ("tile2.obj", material=list(color='blue', texture="tile.png"))
Warning in rgl::readOBJ("tile2.obj", material = list(color = "blue", texture = "tile.png")) :
  Instructions "mtllib", "usemtl" ignored
> shade3d(o)
> o$vb
       [,1]   [,2]   [,3]  [,4]   [,5]   [,6]   [,7]   [,8]   [,9]  [,10]
[1,] -1.000 -1.000  1.000 1.000 -1.000 -1.000  1.000  1.000 -1.000 -1.000
[2,]  1.000 -1.000 -1.000 1.000  1.000 -1.000 -1.000  1.000  1.000 -1.000
[3,]  0.125  0.125  0.125 0.125 -0.125 -0.125 -0.125 -0.125  0.125  0.125
[4,]  1.000  1.000  1.000 1.000  1.000  1.000  1.000  1.000  1.000  1.000
      [,11] [,12]  [,13]  [,14]  [,15]  [,16]
[1,]  1.000 1.000 -1.000 -1.000  1.000  1.000
[2,] -1.000 1.000  1.000 -1.000 -1.000  1.000
[3,]  0.125 0.125 -0.125 -0.125 -0.125 -0.125
[4,]  1.000 1.000  1.000  1.000  1.000  1.000
> o$texcoords
      [,1]  [,2]  [,3]  [,4]  [,5]  [,6]  [,7]  [,8] [,9] [,10] [,11] [,12]
[1,] 0.025 0.025 0.425 0.425 0.575 0.575 0.975 0.975   NA    NA    NA    NA
[2,] 1.000 0.000 0.000 1.000 1.000 0.000 0.000 1.000   NA    NA    NA    NA
     [,13] [,14] [,15] [,16]
[1,]    NA    NA    NA    NA
[2,]    NA    NA    NA    NA
> o$ib
     [,1] [,2] [,3] [,4] [,5] [,6]
[1,]    1    5   14   15   16   13
[2,]    2    8   15   16   13   14
[3,]    3    7   11   12    9   10
[4,]    4    6   10   11   12    9

I still don't understand how the 12 texture coordinates in the new tinyobjloader code are supposed to be interpreted. What 12 things get those coordinates?

The rectangular solid is broken up by tinobjloader into twelve triangles. In the original obj file there are eight vertices (which is what a rectangular solid has) and twelve texture coordinates (in the original obj file four used to illustrate the "top", four used to illustrate the "base", and the remaining four repeated four times to illustrate each of the four edges) which tinyobjloader preserves. indices is a 3x12 matrix describing is a mapping of the eight vertices to the vertices of the twelve triangles (re-using vertices as needed). texindices is a 3x12 matrix describing a mapping of the twelve texture coordinates to the vertices of the twelve triangles (re-using texture coordinates as needed).

dmurdoch · Answer 10 · Mon Aug 17 2020 19:41:32 GMT+0800 (China Standard Time)

Thanks @trevorld for the clarification. I didn't know that rgl would handle NA for a texture coordinate: it isn't doing anything special with it, that's all happening in OpenGL. I'm not certain that the behaviour is documented, so there might be problems in some drivers or in WebGL.

The issue with the texindices appears to be a different way of thinking about textures in rgl and in wavefront. In rgl, texture coordinates are associated with vertices. Sometimes the same vertex is shared between faces; it then shares the same texture coordinates (the texture wraps around), or you need to duplicate the vertex and associate different texture coordinates with each copy. In the res2 structure shown by @jefferis above, it appears that there's an extra level of indirection: texture coordinates are associated with corners of triangles, and corners of triangles are associated with vertices. Then there's no problem having different textures on a shared vertex, because they are in different triangles.

I don't think I'd want to add the extra level of indirection to rgl, because it would affect a lot of very low level things. So I think this needs to be handled in readobj.

Gregory Jefferis · Answer 11 · Mon Aug 17 2020 19:50:54 GMT+0800 (China Standard Time)

Thank you @dmurdoch for already implementing part of the solution for texture meshes here.
@trevorld as Duncan says, it's up to us to implement the rest as part of readobj:::tinyshape2mesh3d. If you would be willing to take a stab at that, I would appreciate.

Gregory Jefferis · Answer 12 · Mon Aug 17 2020 19:53:08 GMT+0800 (China Standard Time)

FWIW, I also added a triangulate argument now, so you can see what the original shape representation looked like

Original (no triangulation)

$positions
       [,1]   [,2]   [,3]  [,4]   [,5]   [,6]   [,7]   [,8]
[1,] -1.000 -1.000  1.000 1.000 -1.000 -1.000  1.000  1.000
[2,]  1.000 -1.000 -1.000 1.000  1.000 -1.000 -1.000  1.000
[3,]  0.125  0.125  0.125 0.125 -0.125 -0.125 -0.125 -0.125

$normals
    
[1,]
[2,]
[3,]

$texcoords
      [,1]  [,2]  [,3]  [,4]  [,5]  [,6]  [,7]  [,8] [,9] [,10] [,11] [,12]
[1,] 0.025 0.025 0.425 0.425 0.575 0.575 0.975 0.975 0.52  0.48  0.48  0.52
[2,] 1.000 0.000 0.000 1.000 1.000 0.000 0.000 1.000 0.00  0.00  1.00  1.00

$indices
     [,1] [,2] [,3] [,4] [,5] [,6]
[1,]    0    4    5    6    7    4
[2,]    1    7    6    7    4    5
[3,]    2    6    2    3    0    1
[4,]    3    5    1    2    3    0

$texindices
     [,1] [,2] [,3] [,4] [,5] [,6]
[1,]    0    4    8    8    8    8
[2,]    1    7    9    9    9    9
[3,]    2    6   10   10   10   10
[4,]    3    5   11   11   11   11

$material_ids
[1] 0 0 0 0 0 0

After triangulation

$positions
       [,1]   [,2]   [,3]  [,4]   [,5]   [,6]   [,7]   [,8]
[1,] -1.000 -1.000  1.000 1.000 -1.000 -1.000  1.000  1.000
[2,]  1.000 -1.000 -1.000 1.000  1.000 -1.000 -1.000  1.000
[3,]  0.125  0.125  0.125 0.125 -0.125 -0.125 -0.125 -0.125

$normals
    
[1,]
[2,]
[3,]

$texcoords
      [,1]  [,2]  [,3]  [,4]  [,5]  [,6]  [,7]  [,8] [,9] [,10] [,11] [,12]
[1,] 0.025 0.025 0.425 0.425 0.575 0.575 0.975 0.975 0.52  0.48  0.48  0.52
[2,] 1.000 0.000 0.000 1.000 1.000 0.000 0.000 1.000 0.00  0.00  1.00  1.00

$indices
     [,1] [,2] [,3] [,4] [,5] [,6] [,7] [,8] [,9] [,10] [,11] [,12]
[1,]    0    0    4    4    5    5    6    6    7     7     4     4
[2,]    1    2    7    6    6    2    7    3    4     0     5     1
[3,]    2    3    6    5    2    1    3    2    0     3     1     0

$texindices
     [,1] [,2] [,3] [,4] [,5] [,6] [,7] [,8] [,9] [,10] [,11] [,12]
[1,]    0    0    4    4    8    8    8    8    8     8     8     8
[2,]    1    2    7    6    9   10    9   10    9    10     9    10
[3,]    2    3    6    5   10   11   10   11   10    11    10    11

$material_ids
 [1] 0 0 0 0 0 0 0 0 0 0 0 0

Trevor L. Davis · Answer 13 · Tue Aug 18 2020 00:28:50 GMT+0800 (China Standard Time)

Okay, I'll take a stab at updating readobj:::tinyshape2mesh3d. It may take me a couple of weeks. Assuming texture coordinates are non-NULL (otherwise we can skip all of this) I think a working algorithm could be:

Uniquely label each triangle vertex
Build a data frame of the mapping of triangle vertices to shape vertices
Build a data frame of the mapping of triangle vertices to texture coordinates
Join the two data frames by triangle labels, drop the triangle labels columns, and compute unique values for the association of vertices with texture coordinates - this should give us the minimum amount of duplicated vertices / textures coordinates that rgl needs
Construct a new matching indices matrix for rgl from tinyobjloader's indices and texindices matrices