Default Meshroom Pipeline
ErlerPhilipp opened this issue · comments
Implement the default Meshroom Pipeline with only minor changes. Minor changes mean the editor must still work and some parameters should be changed.
AliceVision binaries need to be called like from Meshroom. This will require adapting the lib path before the CLI calls:
cmdBase = '**export LD_LIBRARY_PATH**=$LD_LIBRARY_PATH:{}/aliceVision/lib'.format(absComputePath) + ' && {}/aliceVision/bin/aliceVision_convertSfMFormat'.format(absComputePath)
Adapting the default pipeline to the editor:
- Transforms can be applied to the data with a SfM Transform node.
- The handling of the cropping area in the editor needs to change. The meshing node has a parameter for a custom bounding box, which needs to come from the editor. If none is provided, the meshing will find its own.
Requires Meshroom Docker #162
Read this: https://sketchfab.com/blogs/community/tutorial-meshroom-for-beginners/
Here are some rough measurements and parameter suggestions:
| Step | Node Type | Duration | Critical for Duration | Critical for Quality | Parameters |
|---|---|---|---|---|---|
| 1 | Camera Init | 1s | |||
| 1 | Feature Extraction | 1m | + | ++ | describer quality high, force CPU? check issues |
| 1 | Image Matching | 1s | method?, advanced settings -> min for voc tree, sequential neighbors; max descriptors 0, Nb matches 0 | ||
| 1 | Feature Matching | 2m | ++ | + | cascade hashing l2; guided matching, cross matching, known poses geometric error max 0 |
| 1 | Structure from Motion | 4m | ++++ | +++ | Scaling on |
| 1 | Prepare Dense Scene | 20s | no save meta data? | ||
| 1 | Depth Map | 20s | + | downscale 1? (longer meshing) | |
| 1 | Depth Map Filter | 15s | + | number of nearest cameras 10 -> 15? | |
| Editor | +++ | +++ | Need to switch from cropping area to will-be-reconstructed area | ||
| 2 | SfM Transform | 10s? | Apply Editor transformations (check if non-uniform scaling is supported) | ||
| 2 | Meshing | 7m | +++++++ | ++ | use custom bounding box from editor; min step 1, margin init (final) coef 1; densify front/back 5 |
| 2 | Mesh Filtering | 30s | |||
| 2 | Texturing | 9m | ++++++++++ | ++ | padding, multiband high freq > low freq, subdiv target ration 1 |
Examples for calls:
Camera Init
aliceVision_cameraInit --sensorDatabase "C:\Users\pherl\Desktop\Meshroom-2023.2.0\aliceVision\share\aliceVision\cameraSensors.db" --lensCorrectionProfileInfo "${ALICEVISION_LENS_PROFILE_INFO}" --lensCorrectionProfileSearchIgnoreCameraModel True --defaultFieldOfView 45.0 --groupCameraFallback folder --allowedCameraModels pinhole,radial1,radial3,brown,fisheye4,fisheye1,3deanamorphic4,3deradial4,3declassicld --rawColorInterpretation LibRawWhiteBalancing --viewIdMethod metadata --verboseLevel info --output "E:/GoogleDrive/Pix2Model/elephants meshroom/MeshroomCache/CameraInit/b90727abc0f92c40c074360d60d8f9fe21728f5c/cameraInit.sfm" --allowSingleView 1 --input "E:/GoogleDrive/Pix2Model/elephants meshroom/MeshroomCache/CameraInit/b90727abc0f92c40c074360d60d8f9fe21728f5c/viewpoints.sfm"
Feature Extraction
aliceVision_featureExtraction --input "E:/GoogleDrive/Pix2Model/elephants meshroom/MeshroomCache/CameraInit/b90727abc0f92c40c074360d60d8f9fe21728f5c/cameraInit.sfm" --masksFolder "" --describerTypes dspsift --describerPreset normal --describerQuality normal --contrastFiltering GridSort --gridFiltering True --workingColorSpace sRGB --forceCpuExtraction True --maxThreads 0 --verboseLevel info --output "E:/GoogleDrive/Pix2Model/elephants meshroom/MeshroomCache/FeatureExtraction/7bb71d547c48f1543cdb1cc05fe6d5908fa1d5e2" --rangeStart 0 --rangeSize 40
Image Matching
aliceVision_imageMatching --input "E:/GoogleDrive/Pix2Model/elephants meshroom/MeshroomCache/CameraInit/b90727abc0f92c40c074360d60d8f9fe21728f5c/cameraInit.sfm" --featuresFolders "E:/GoogleDrive/Pix2Model/elephants meshroom/MeshroomCache/FeatureExtraction/7bb71d547c48f1543cdb1cc05fe6d5908fa1d5e2" --method SequentialAndVocabularyTree --tree "C:\Users\pherl\Desktop\Meshroom-2023.2.0\aliceVision\share\aliceVision\vlfeat_K80L3.SIFT.tree" --weights "" --minNbImages 0 --maxDescriptors 500 --nbMatches 40 --nbNeighbors 5 --verboseLevel info --output "E:/GoogleDrive/Pix2Model/elephants meshroom/MeshroomCache/ImageMatching/065c74f91d227855884634da377e3e3826ca767a/imageMatches.txt"
Feature Matching
aliceVision_featureMatching --input "E:/GoogleDrive/Pix2Model/elephants meshroom/MeshroomCache/CameraInit/b90727abc0f92c40c074360d60d8f9fe21728f5c/cameraInit.sfm" --featuresFolders "E:/GoogleDrive/Pix2Model/elephants meshroom/MeshroomCache/FeatureExtraction/7bb71d547c48f1543cdb1cc05fe6d5908fa1d5e2" --imagePairsList "E:/GoogleDrive/Pix2Model/elephants meshroom/MeshroomCache/ImageMatching/065c74f91d227855884634da377e3e3826ca767a/imageMatches.txt" --describerTypes dspsift --photometricMatchingMethod ANN_L2 --geometricEstimator acransac --geometricFilterType fundamental_matrix --distanceRatio 0.8 --maxIteration 2048 --geometricError 0.0 --knownPosesGeometricErrorMax 5.0 --minRequired2DMotion -1.0 --maxMatches 0 --savePutativeMatches False --crossMatching False --guidedMatching False --matchFromKnownCameraPoses False --exportDebugFiles False --verboseLevel info --output "E:/GoogleDrive/Pix2Model/elephants meshroom/MeshroomCache/FeatureMatching/a768effb45b2427e6223536e39a8fe5b34b77ec1" --rangeStart 0 --rangeSize 20
Structure from Motion
aliceVision_incrementalSfM --input "E:/GoogleDrive/Pix2Model/elephants meshroom/MeshroomCache/CameraInit/b90727abc0f92c40c074360d60d8f9fe21728f5c/cameraInit.sfm" --featuresFolders "E:/GoogleDrive/Pix2Model/elephants meshroom/MeshroomCache/FeatureExtraction/7bb71d547c48f1543cdb1cc05fe6d5908fa1d5e2" --matchesFolders "E:/GoogleDrive/Pix2Model/elephants meshroom/MeshroomCache/FeatureMatching/a768effb45b2427e6223536e39a8fe5b34b77ec1" --describerTypes dspsift --localizerEstimator acransac --observationConstraint Scale --localizerEstimatorMaxIterations 4096 --localizerEstimatorError 0.0 --lockScenePreviouslyReconstructed False --useLocalBA True --localBAGraphDistance 1 --nbFirstUnstableCameras 30 --maxImagesPerGroup 30 --bundleAdjustmentMaxOutliers 50 --maxNumberOfMatches 0 --minNumberOfMatches 0 --minInputTrackLength 2 --minNumberOfObservationsForTriangulation 2 --minAngleForTriangulation 3.0 --minAngleForLandmark 2.0 --maxReprojectionError 4.0 --minAngleInitialPair 5.0 --maxAngleInitialPair 40.0 --useOnlyMatchesFromInputFolder False --useRigConstraint True --rigMinNbCamerasForCalibration 20 --lockAllIntrinsics False --minNbCamerasToRefinePrincipalPoint 3 --filterTrackForks False --computeStructureColor True --initialPairA "" --initialPairB "" --interFileExtension .abc --verboseLevel info --output "E:/GoogleDrive/Pix2Model/elephants meshroom/MeshroomCache/StructureFromMotion/79d690639eb4ec5db3c82d212117b127b6432147/sfm.abc" --outputViewsAndPoses "E:/GoogleDrive/Pix2Model/elephants meshroom/MeshroomCache/StructureFromMotion/79d690639eb4ec5db3c82d212117b127b6432147/cameras.sfm" --extraInfoFolder "E:/GoogleDrive/Pix2Model/elephants meshroom/MeshroomCache/StructureFromMotion/79d690639eb4ec5db3c82d212117b127b6432147"
SfMTransform
aliceVision_sfmTransform --input "E:/GoogleDrive/Pix2Model/elephants meshroom/MeshroomCache/StructureFromMotion/79d690639eb4ec5db3c82d212117b127b6432147/sfm.abc" --method manual --manualTransform 0.0,0.0,0.0,0.0,0.0,0.0,1.0 --landmarksDescriberTypes sift,dspsift,akaze --scale 1.0 --verboseLevel info --output "E:/GoogleDrive/Pix2Model/elephants meshroom/MeshroomCache/SfMTransform/9879d91b1963063be49cef1e1cd8c7183890b06e/sfm.abc" --outputViewsAndPoses "E:/GoogleDrive/Pix2Model/elephants meshroom/MeshroomCache/SfMTransform/9879d91b1963063be49cef1e1cd8c7183890b06e/cameras.sfm"
Prepare Dense Scene
aliceVision_prepareDenseScene --input "E:/GoogleDrive/Pix2Model/elephants meshroom/MeshroomCache/SfMTransform/9879d91b1963063be49cef1e1cd8c7183890b06e/sfm.abc" --outputFileType exr --saveMetadata True --saveMatricesTxtFiles False --evCorrection False --verboseLevel info --output "E:/GoogleDrive/Pix2Model/elephants meshroom/MeshroomCache/PrepareDenseScene/61b60df8ca44abab29cdae4493b249bbbc36c54f" --rangeStart 0 --rangeSize 40
ExportColoredPointCloud
aliceVision_exportColoredPointCloud --input "E:/GoogleDrive/Pix2Model/elephants meshroom/MeshroomCache/SfMTransform/9879d91b1963063be49cef1e1cd8c7183890b06e/sfm.abc" --verboseLevel info --output "E:/GoogleDrive/Pix2Model/elephants meshroom/MeshroomCache/ExportColoredPointCloud/9196ffaa386983c0dff76b765e7461e7259d6f9f/pointCloud.abc"
ConvertSfMFormat
aliceVision_convertSfMFormat --input "E:/GoogleDrive/Pix2Model/elephants meshroom/MeshroomCache/SfMTransform/9879d91b1963063be49cef1e1cd8c7183890b06e/sfm.abc" --describerTypes dspsift --views True --intrinsics True --extrinsics True --structure True --observations True --verboseLevel info --output "E:/GoogleDrive/Pix2Model/elephants meshroom/MeshroomCache/ConvertSfMFormat/e09417617ad2cae5fb7217876efb4993908a465f/sfm.ply"
Depth Map
aliceVision_depthMapEstimation --input "E:/GoogleDrive/Pix2Model/elephants meshroom/MeshroomCache/SfMTransform/9879d91b1963063be49cef1e1cd8c7183890b06e/sfm.abc" --imagesFolder "E:/GoogleDrive/Pix2Model/elephants meshroom/MeshroomCache/PrepareDenseScene/61b60df8ca44abab29cdae4493b249bbbc36c54f" --downscale 4 --minViewAngle 2.0 --maxViewAngle 70.0 --tileBufferWidth 1024 --tileBufferHeight 1024 --tilePadding 64 --autoAdjustSmallImage True --chooseTCamsPerTile True --maxTCams 10 --sgmScale 2 --sgmStepXY 2 --sgmStepZ -1 --sgmMaxTCamsPerTile 4 --sgmWSH 4 --sgmUseSfmSeeds True --sgmSeedsRangeInflate 0.2 --sgmDepthThicknessInflate 0.0 --sgmMaxSimilarity 1.0 --sgmGammaC 5.5 --sgmGammaP 8.0 --sgmP1 10.0 --sgmP2Weighting 100.0 --sgmMaxDepths 1500 --sgmFilteringAxes "YX" --sgmDepthListPerTile True --sgmUseConsistentScale False --refineEnabled True --refineScale 1 --refineStepXY 1 --refineMaxTCamsPerTile 4 --refineSubsampling 10 --refineHalfNbDepths 15 --refineWSH 3 --refineSigma 15.0 --refineGammaC 15.5 --refineGammaP 8.0 --refineInterpolateMiddleDepth False --refineUseConsistentScale False --colorOptimizationEnabled True --colorOptimizationNbIterations 100 --sgmUseCustomPatchPattern False --refineUseCustomPatchPattern False --exportIntermediateDepthSimMaps False --exportIntermediateNormalMaps False --exportIntermediateVolumes False --exportIntermediateCrossVolumes False --exportIntermediateTopographicCutVolumes False --exportIntermediateVolume9pCsv False --exportTilePattern False --nbGPUs 0 --verboseLevel info --output "E:/GoogleDrive/Pix2Model/elephants meshroom/MeshroomCache/DepthMap/e9b865d6643dd0ef5972d8c3494b1874d0a91547" --rangeStart 0 --rangeSize 3
Depth Map Filter
aliceVision_depthMapFiltering --input "E:/GoogleDrive/Pix2Model/elephants meshroom/MeshroomCache/SfMTransform/9879d91b1963063be49cef1e1cd8c7183890b06e/sfm.abc" --depthMapsFolder "E:/GoogleDrive/Pix2Model/elephants meshroom/MeshroomCache/DepthMap/e9b865d6643dd0ef5972d8c3494b1874d0a91547" --minViewAngle 2.0 --maxViewAngle 70.0 --nNearestCams 10 --minNumOfConsistentCams 3 --minNumOfConsistentCamsWithLowSimilarity 4 --pixToleranceFactor 2.0 --pixSizeBall 0 --pixSizeBallWithLowSimilarity 0 --computeNormalMaps False --verboseLevel info --output "E:/GoogleDrive/Pix2Model/elephants meshroom/MeshroomCache/DepthMapFilter/fad1cb48e702e4fa2c92cf297b7618f58a23a356" --rangeStart 0 --rangeSize 10
Meshing
aliceVision_meshing --input "E:/GoogleDrive/Pix2Model/elephants meshroom/MeshroomCache/SfMTransform/9879d91b1963063be49cef1e1cd8c7183890b06e/sfm.abc" --depthMapsFolder "E:/GoogleDrive/Pix2Model/elephants meshroom/MeshroomCache/DepthMapFilter/fad1cb48e702e4fa2c92cf297b7618f58a23a356" --estimateSpaceFromSfM True --estimateSpaceMinObservations 3 --estimateSpaceMinObservationAngle 10.0 --maxInputPoints 50000000 --maxPoints 5000000 --maxPointsPerVoxel 1000000 --minStep 2 --partitioning singleBlock --repartition multiResolution --angleFactor 15.0 --simFactor 15.0 --pixSizeMarginInitCoef 2.0 --pixSizeMarginFinalCoef 4.0 --voteMarginFactor 4.0 --contributeMarginFactor 2.0 --simGaussianSizeInit 10.0 --simGaussianSize 10.0 --minAngleThreshold 1.0 --refineFuse True --helperPointsGridSize 10 --nPixelSizeBehind 4.0 --fullWeight 1.0 --voteFilteringForWeaklySupportedSurfaces True --addLandmarksToTheDensePointCloud False --invertTetrahedronBasedOnNeighborsNbIterations 10 --minSolidAngleRatio 0.2 --nbSolidAngleFilteringIterations 2 --colorizeOutput False --maxNbConnectedHelperPoints 50 --saveRawDensePointCloud False --exportDebugTetrahedralization False --seed 0 --verboseLevel info --outputMesh "E:/GoogleDrive/Pix2Model/elephants meshroom/MeshroomCache/Meshing/405e88218b3f4ac0b78047c8950c51f7d53755f9/mesh.obj" --output "E:/GoogleDrive/Pix2Model/elephants meshroom/MeshroomCache/Meshing/405e88218b3f4ac0b78047c8950c51f7d53755f9/densePointCloud.abc"
Mesh Filtering
aliceVision_meshFiltering --inputMesh "e:/GoogleDrive/Pix2Model/elephants meshroom/MeshroomCache/Meshing/405e88218b3f4ac0b78047c8950c51f7d53755f9/mesh.obj" --keepLargestMeshOnly False --smoothingSubset all --smoothingBoundariesNeighbours 0 --smoothingIterations 5 --smoothingLambda 1.0 --filteringSubset all --filteringIterations 1 --filterLargeTrianglesFactor 60.0 --filterTrianglesRatio 0.0 --verboseLevel info --outputMesh "e:/GoogleDrive/Pix2Model/elephants meshroom/MeshroomCache/MeshFiltering/c6b9c2e5ebed34188f4b60c4a3843fc7259fada8/mesh.obj"
MeshDecimate
aliceVision_meshDecimate --input "E:/GoogleDrive/Pix2Model/elephants meshroom/MeshroomCache/MeshFiltering/c6b9c2e5ebed34188f4b60c4a3843fc7259fada8/mesh.obj" --simplificationFactor 0.5 --nbVertices 0 --minVertices 0 --maxVertices 100000 --flipNormals False --verboseLevel info --output "E:/GoogleDrive/Pix2Model/elephants meshroom/MeshroomCache/MeshDecimate/5f47d58c696da73476793955bacf1dbf587660ba/mesh.obj"
Texturing
aliceVision_texturing --input "E:/GoogleDrive/Pix2Model/elephants meshroom/MeshroomCache/Meshing/405e88218b3f4ac0b78047c8950c51f7d53755f9/densePointCloud.abc" --imagesFolder "E:/GoogleDrive/Pix2Model/elephants meshroom/MeshroomCache/PrepareDenseScene/61b60df8ca44abab29cdae4493b249bbbc36c54f" --inputMesh "E:/GoogleDrive/Pix2Model/elephants meshroom/MeshroomCache/MeshDecimate/5f47d58c696da73476793955bacf1dbf587660ba/mesh.obj" --inputRefMesh "" --textureSide 2048 --downscale 8 --outputMeshFileType obj --colorMappingFileType jpg --unwrapMethod Basic --useUDIM True --fillHoles False --padding 5 --multiBandDownscale 4 --multiBandNbContrib 5 4 3 2 --useScore True --bestScoreThreshold 0.1 --angleHardThreshold 90.0 --workingColorSpace sRGB --outputColorSpace AUTO --correctEV False --forceVisibleByAllVertices False --flipNormals False --visibilityRemappingMethod PullPush --subdivisionTargetRatio 0.8 --verboseLevel info --output "E:/GoogleDrive/Pix2Model/elephants meshroom/MeshroomCache/Texturing/cece8ca9f920246842eb74fb65d1e09427964384"
See elephants test scene for details.
Instead of using Alice Vision library directly we can use the Meshroom command line feature instead:
https://meshroom-manual.readthedocs.io/en/latest/feature-documentation/cmd/photogrammetry.html
Configure a pipeline in Meshroom and save it as e.g. default_pipeline.mg.
Official site
If you use the application from the official site (https://alicevision.org/#meshroom) the executable is inside the root folder.
A simple shell script to start the pipeline looks like this:
#!/bin/sh
TIME=$(date +%s)
/home/joschi/Downloads/Meshroom-2023.2.0-av3.1.0-centos7-cuda11.3.1/meshroom_batch -i ../meshroom_test/test_images -p ./default_pipeline.mg --cache /home/joschi/Downloads/meshroom_test/$TIME
Use an absolute path for the cache folder, otherwise it messes things up. The pipeline should start and run trough without troubles. The output of each node is in the corresponding cache folder.
Inside Docker
If you want to start it from inside docker with this dockerfile 92e8449 you need to build the image with sudo docker build -t meshroom -f Dockerfile.meshroom . from inside the Pix2Model root folder.
And then run the container with docker run -it --runtime=nvidia meshroom. Test your setup by running the commands nvcc --version and nvidia-smi from inside the container. If the output looks good you did a great job!
The executable is in /opt/Meshroom_bundle.
Upload a folder with your images and pipeline file to the container, e.g. /opt/Test_data.
meshroom_batch searches for a config file (config.ocio) in the wrong place, rename the folder AliceVision_install to AliceVision_bundle to solve the issue.
A simple shell script to start the pipeline looks like this:
#!/bin/sh
TIME=$(date +%s)
./../Meshroom_bundle/meshroom_batch -i ./test_images -p ./default_pipeline.mg --cache /opt/Test_data/$TIME
Use an absolute path for the cache folder, otherwise it messes things up. The pipeline should start and run trough without troubles. The output of each node is in the corresponding cache folder.
If I understand this correctly, we need to adapt this config for the editor, right?
Meaning, that we need to store the bounding box for meshing, which we get from the editor, in the config.
Also, we need separate parts for before the editor and after the editor?
Right, currently I am working on the separation of the pipeline into two steps: the sfm part and the reconstruction part. I attach a little test environment in python if some one wants to test it.
test_meshroom.zip
Currently I am struggling with the reconstruction part, the provided input seems wrong, reconstruction fails. I also attach a error and log file.
When this is fixed I will look into providing a custom bounding box for the reconstruction.
are you sure that it runs through. if i see this right, there are only 34 images. there is a chance that the registration failed completely and SfM produced no results.
if the folder structure is a problem, you could try the publish node. you could probably copy the files from [guid]/sfm etc to a fixed folder.
The images were fine, it seems that meshroom_batch is not meant to be used on incomplete pipelines, meaning 2 separate pipelines that are split like this won't work.
But we can use a single complete pipeline and with the parameter --toNode we can specify which node should be executed. It will also execute all the node's dependencies and with the correct input parameters it will skip already executed nodes.
With this its very easy to separate the pipelines into 2 steps. Here is the working python script:
meshroom_test.zip
Another problem I encountered is with using the sfmTransform node. The transformation is applied to the point cloud, but then the reconstruction seems to use an inverse transformed version of the pointcloud. There is an issue that says this bug is fixed in the new version, but the bug seems to persist: alicevision/Meshroom#1994
This bug will make transforming the pointcloud a little bit trickier...
So the 2nd cameraInit does use the outputs from the 1st part? Or do you need to copy data manually?
For the bug, I see 2 options:
- Make a fork and pull request for this bug. Unless they provide docker images with nightly updates, we would need to pull from the fork and compile ourselves.
- Fix the sfmTransform outputs. This would require messing with the SfMData files, which could be done by a small e.g. Python script.
Am I missing anything?
The problem is that the second cameraInit won't take the input from the first part. The provided screen shots are from the setup where it doesn't work. Sorry for the confusion.
The solution is to re-use one complete pipeline and define which nodes should be executed, much like in meshroom where you right click a node and click "compute". Meshroom will execute the selected node and all its dependencies, also skipping nodes that were previously computed, shown here:
If we execute Texturing it will skip every green node and start with the PrepareDenseScene node. It also recognizes if one node has changed. That means if we change the bounding box of the Meshing node it will automatically recompute only the last 3 nodes, Meshing, MeshFiltering, Texturing. So a custom bounding box is also working now.
Regarding the bug:
A 3rd option would be skipping translation transforms completely, as rotations and scale transforms seem to work.
Another point would be thinking about dropping the feature, to transform the point cloud, completely, as this would also mean that we need to recompute the whole reconstruction pipeline. Doesn't make much sense to me, when we are also able to transform the reconstructed mesh in a fraction of the time.
I will give this another thought, and maybe we can discuss this further during our next meeting.
Actually, I think we can skip the transform node altogether for now. The important part of the editor is to specify a bounding box for the reconstruction. Everything else is not strictly necessary.
A transform for the final output would be nice, so that the users can e.g. scale the result to meters and center it on a specific part. But that's already optional.
too much work for now