Streaming large 3D dataset, support for 3Dtiles
benoitdm-oslandia opened this issue · comments
QGIS Enhancement: Streaming large 3D dataset, support for 3Dtiles
Date 2021/05/03
Author Benoit De Mezzo (benoit.de.mezzo@oslandia.com)
Contact benoit dot de dot mezzo at oslandia dot com
maintainer @benoitdm-oslandia
Version QGIS 3.18
Summary
We want to provide a software architecture in QGIS to integrate the loading of large 3D datasets via the 3D Tiles format.
This architecture should:
- be extensible to other similar formats (eg. I3S)
- allow smooth viewing by managing the level of detail (aka LOD)
- provide early loading in the background
To fullfil this QEP we already have secured some funding to work on mainly with Métropole Européenne de Lille (@Jean-Roc).
Proposed Solution
As part of this mission we will develop a partial (but working) implementation of 3D Tiles which includes:
- the whole 3D Tiles tree parsing
- only
b3dmobjects handling - loading of
featureTablesandbatchTablesattribute tables - no 3D Tiles extension management
We also needs to:
- properly inherit from the
QgsChunkLoader,QgsChunkedEntityandQgsChunkLoaderFactoryclasses to support dynamic tile loading - inherit from the
QgsMapLayerclass to display tile datasets - inherit from the
QgsAbstract3DRendererclass to display the tiles in 3D - inherit from the
QgsDataProviderclass to create a mesh or vector provider to load a 3d Tiles file - integrate into
Qgs3DMapScene - support 3D identification map tool
We will try to provide a class architecture to be able to support other formats (I3S, etc.).
Example(s)
An example implementation without the management of featureTables and batchTables is available in this branch.
Implementations of the QgsChunkLoader, QgsChunkedEntity, QgsChunkLoaderFactory, QgsAbstract3DRenderer and QgsMapLayer classes have been created for the LOD in the directory src/3d/3dtiles.
As part of this POC, the qgis_3d_sandbox test has been modified in order to apply this workflow:
- load the tileset from a
tileset.jsonfile passed as a parameter (via the 3D Tiles classes we created) - create a
Qgs3dTilesLayerusing the previously loaded tileset - create a
Qgs3dTilesLayer3DRendererusing the previously created layer. This class will use theQgs3dTilesChunkLoaderFactoryand theQgs3dTilesChunkedEntity
For now, the loaded meshs are replaced by torus:
Affected Files
src/3d/qgs3dmapscene.cppsrc/3d/CMakeLists.txttests/src/3d/sandbox/qgis_3d_sandbox.cpp- new directory
src/3d/3dtiles
Issues
3D stack
The 3D stack to handle the chunk loading is missing documentation and samples to properly understood the usecases and the proper way to implement a new chunk loader.
As a result the proposed implementation may lack the proper software designs to match the requirement of the chunk loading API. For example, the background loading thread, the unloading process, loader factory usage, root node/children nodes creation, etc.
Gltf
The gltf support via Qt3D only allow to read specific format via url/uri.
It does NOT allow direct reading from the binary format included in the b3dm, it only supports the JSON text format and only via file name (in the GLTFImporter class, the setData function is not exported in the API).
To overcome this shortcoming, we use an extra lib (libtinygltf) to extract the gltf binary data and then dump the gltf components to files (JSON, bin, etc.). At final, we provide the url of the dumped JSON file to the Qt3DRender::QSceneLoader.
But this diverted process by using temporary files is not efficient but creates unusable geometries (no normal defined, could be fixed). To overcome it, we need to improve the gltf support.
We have 3 suggestions:
-
In Qt, without the help of the
libtinygltf, we could add the possibility to theQt3DRender::QSceneLoaderAPI to handle the data loading viaQByteArrayand add thegltfbinary format support toGLTFImporter. May be there are some leads into the existinggltfsupport in QML? -
In QGIS, without the help of the
libtinygltf, we would have to (re)create a basicgltfto support the binary format and extract meshes, material settings, light settings to rebuild the scene. -
In QGIS, WITH the help of the
libtinygltf, we could avoid the use of temporary files by inheriting fromQNetworkAccessManager(as suggested here) to handle a specific uri (ie.b3dm://...). This uri will be pass toQt3DRender::QSceneLoaderand via ourQNetworkAccessManagerwe could return a validgltf/JSONfile to the scene loader.
Float precision
For reasons of precision tests on matrix computation, the qgsmatrix4x4 and qgsvector4d classes have been added by using double instead of float.
Unloading
In our example, may be due to implementation, some objects persist even when unzooming but the Qt3DCore::QEntity are no more active. We will have to investigate further to resolve this point.
Further Considerations/Improvements
Further Considerations:
- inherit from the
QgsMapLayerRendererclass to display the tiles in 2D and support 2D identification map tool - implements other 3D Tiles subcontents:
i3dmandpnts - try to move to Qt6 to test the new RHI 3D support
Backwards Compatibility
The need of backwards compatibility may not be releavant if, in the short term, QGIS evolves to Qt6.
Votes
(required)
The gltf support via Qt3D only allow to read specific format via url/uri.
QgsModelPoint3DSymbolHandler::addSceneEntities already handles this situation by automatically storing the binary in a temporary file via QgsApplication::instance()->sourceCache()->localFilePath(...). Couldn't that approach be used here too?
Exciting addition, this sounds great! I'd love for the introduced code to be kept as generic as possible to also allow for potential future support for ArcGIS scene layers 👍 .
@nyalldawson I did not know this API part! Thanks for the tips I'll check this asap!
Exciting stuff - looking forward to that!
Various things that jump to my mind:
- what about 2D map rendering?
- what about APIs to access 3D Tiles data - will there be a way to access the hierarchy programmatically and to access the raw mesh data?
- how is it going to integrate with the concept of map layers - will there be a new layer type for this? or using mesh layer? or...?
- is it going to support local datasets or remote datasets (over HTTP(S)) or both?
- how is styling going to work? Are you planning to add new shaders?
- is identification of objects going to be supported?
3D tiles seem to have quite wide range of functionalities, so it would be good to further specify what will be supported and what not... Using the overview document - https://github.com/CesiumGS/3d-tiles/blob/master/3d-tiles-overview.pdf
- what types of bounding volumes?
- which refinement strategies?
- is the declarative styling going to be supported?
The 3D stack to handle the chunk loading is missing documentation and samples to properly understood the usecases and the proper way to implement a new chunk loader.
Please let me know if you have concrete questions - will try to answer them (and hopefully also add to the API documentation).
properly extend the QgsChunkLoader, QgsChunkedEntity and QgsChunkLoaderFactory classes to support dynamic tile loading
Dynamic tile loading is already used for remote EPT datasets for point clouds... what did you have in mind?
extend the QgsAbstract3DRenderer and QgsMapLayer classes to display the tiles
In what way do they need to be extended?
Thanks @wonder-sk!
what about 2D map rendering?
what about APIs to access 3D Tiles data - will there be a way to access the hierarchy programmatically and to access the raw mesh data?
For what I know, the 2D rendering is not planned! But we could try to provide mesh data picking/selection but the main difficulty may be to maintain the selection and the mesh changes due to LOD.
how is it going to integrate with the concept of map layers - will there be a new layer type for this? or using mesh layer? or...?
In the current implementation example, we inherit from the mesh map layer but may be we will need a new one.
is it going to support local datasets or remote datasets (over HTTP(S)) or both?
Both dataset will be available.
how is styling going to work? Are you planning to add new shaders?
In 3D Tiles the mesh styling is in the embedded gltf objects. The style customization is a good idea but must be discussed further.
is identification of objects going to be supported?
What do you mean? If you are talking about selection of object, I would like to do it.
3D tiles seem to have quite wide range of functionalities, so it would be good to further specify what will be supported and what not... Using the overview document - https://github.com/CesiumGS/3d-tiles/blob/master/3d-tiles-overview.pdf
what types of bounding volumes?
the 3 types are already extracted from the JSON but from QGIS point of view, they are all boxes.
which refinement strategies?
The refinement strategies are (or seems to be) already handled by QgsChunkedEntity::setUsingAdditiveStrategy
is the declarative styling going to be supported?
This would be really nice but I do not know if we will have enough time to do it.
About the "extension" I was talking, it is only from the class inheritance point of view (french mistranslated) :)
For what I know, the 2D rendering is not planned!
Hmm... IMHO that should be planned as well. All existing data types including point clouds are supported in 2D map rendering...
In the current implementation example, we inherit from the mesh map layer but may be we will need a new one.
This seems like an important implementation decision that should go to QEP - would it be a new map layer type (something like "scene layer" in ESRI (?)), that would be common for 3D Tiles, I3S and possibly other formats (implemented as data providers)? What would be the interface of data providers?
In 3D Tiles the mesh styling is in the embedded gltf objects.
Ah right - good point.
is identification of objects going to be supported?
What do you mean? If you are talking about selection of object, I would like to do it.
I mean support for identify map tool (in 2D and 3D view)...
Hmm... IMHO that should be planned as well. All existing data types including point clouds are supported in 2D map rendering...
good point. we'll check how to do that.
This seems like an important implementation decision that should go to QEP
Ok I will add it!
I mean support for identify map tool (in 2D and 3D view)...
Ok I will add it!
good point. we'll check how to do that.
Keeping the faces based on normal orientation ? It would seems less costly than involving gltf's uv map. However, while it is a good feature to have, could it be a task planned for a later stage ?
Hi,
so long without any updates! Let's try to fix that!
Introducing 3DTiles in QGIS was not as easy as I planned, mainly due to:
- got many issues with glTF reading and scene loading from Qt3D
- got geocentric projection issues (qgis/QGIS#44941)
- got object orientation issues
One can look at our presentation done for FOSS4G 2021/09. Here are the screencasts dragon, Japanese house and Nasa landscape.
glTF
As previously said, we have to use an external library to extract the data from the glb saved within the b3dm files to dump new files (json+bin) usable by Qt3D qsceneloader. Theses files load properly in blender.
Also If more than one mesh is defined, the Qt gltf importer failed at reading all meshes (only the first one). And as texture material cannot be retrieved/rendered properly with the generated qt scene we use our internal material.
To resolve those points, as we may modify the Qt and Assimp libraries hoping the changes will be backported in Qt 5.15 via compilation flag (like this one), thus available in future QGis builds.
Or we may have to retrieve from Qt some piece of code (eg. glTF/Assimp wrapper) and fix them in QGis (until we move to Qt6). Also it may be needed to add an updated Assimp library (via static linkage) in QGis. After that we should be able to backport the changes to Qt.
File cache
To load the 3Dtile files from remote location, I used, as suggested, the QgsSourceCache but sometimes the remote files cannot be loaded. Thus, I implemented one more file cache manager (as there are many ways in QGis to have a file cache). It should be improved or replaced by a better QGis solution.
Object orientation
By default 3dTiles objects are defined to be put on a globe thus the final object/mesh coordinates must be oriented accordingly. But we are not on a globe in QGis and the final object/mesh coordinates do not match our needs.
We need to re-orient the mesh to put it on a flat ground.
But the meshes may be pre-oriented ie. the mesh coordinates are already set to match the globe location without the need to apply the tile transformation matrix. When the mesh is pre-oriented it is quite difficult to compute the reverse transformation matrix.
Currently when the globe transformation is brought by the tile transformation matrix (ie. the mesh is on flat ground and 0 centered), we are able to compute the reverse transformation matrix.
LOD
I encountered some issues or questions when loading the 3DTiles data within the QGis chunked loader mechanism.
event loop
The Qt3D entity retrieval needs the QSceneLoader object to load a glTF file, but the setSource signal is never handled by the event loop thus the QSceneLoader object never load the file. I have to manually call the QCoreApplication::processEvents() in createEntity!
Qt tree
3Dtiles dataset can be huge and for now I use only one rootEntity for the whole dataset. The more I get deeper in the tileset, the more I enrich the rootEntiy with new Qt3DCore::QEntity (with mesh, material, etc.) and by the way the Qt object tree is getting bigger. This will soon induce a memory problem and I do not know how to change my loader or the QGis chunk loading mechanism to solve it.
Should I use multiple rootEntity, ie. by limiting the tree depth ? How to tell to QGis to switch from one rootEntity to another?
Or Should I chop off the Qt object tree of the furthest Qt3DCore::QEntity branches? How to do that?
Further works
We would like to thank here the Eurométropole de Lille which financed this work, thus opening the way to a direct visualization of the 3DTiles format in QGIS!
We have laid the groundwork for an implementation of 3DTiles and we are very happy about it :).
However, to make its use possible in QGIS, there is still work to be done to industrialize it and optimize its performance.
Here are the main points needing work:
- texture handling and correct object orientation
- better memory management (lod/Qt tree)
- object selection and attribute retrieval
- 3DTile file provider
- 2D reprojection
- add support for i3dm and pnts
Several actors have shown interest in contributing to these developments and we would also be happy to be able to work on it again!
@nyalldawson Great to see you managed a Cesium Grant for this work to be pursued. We would have appreciated to be contacted first to discuss the matter though, as @benoitdm-oslandia already put some significant efforts into this topic.
We are ready to collaborate to make 3D Tiles in QGIS a reality, but will need fair coordination and cooperation.
Hi, more details have been published on north road's blog but I do not see any mention of the present QEP or others about 3D. I would like to second Vincent's demand on this subject, on a personal point of view this does not encourage me to fund this kind of work again.
Not needed anymore!