RenderMatch

Implementation of our paper "Leveraging Photogrammetric Mesh Models for Aerial-Ground Feature Point Matching Toward Integrated 3D Reconstruction" submitted to ISPRS Journal of Photogrammetry and Remote Sensing

Dependencies

The code is extracted from our private code repository, currently many libraries are dependent for some legacy issues. To install the dependencies, we provide anaconda environment file. Using the environment file environment.Linux.yml and environment.msvc.yml for building with Linux gcc and VS 2017 (we only test it with MSVC 2017), respectively.

The environment is built using these recipies, and the binaries are deployed in anaconda.

Linux: create environment with conda env create -n h2o -f environment.Linux.yml to create an environment named h2o (other names are also OK).
MSVC 2017: create environment with conda env create -n h2o -f environment.msvc.yml.
For windows, we also provide a cmake tool chain for easy deployment, run

conda activate h2o
cd toolchains
robocopy ./ %CONDA_PREFIX% h2o.cmake
robocopy ./ %CONDA_PREFIX%/Library/plugins applocal.ps1
robocopy ./ %CONDA_PREFIX%/Library/plugins qtdeploy.ps1
robocopy ./ %CONDA_PREFIX% applocal.ps1

Building

activate the environment conda activate h2o
run the building script ./build.sh for Linux and ./build.bat for MSVC.

Running

We have made several routines and currently we only test it on MSVC, including

FeatureExtract: Extract SIFT keypoints and descriptors for the ground images, a AT file in the BlockExchange Format is needed.
RenderMesh: Render the mesh onto the ground views, color, depth, normal and XYZ maps can be generated.
RenderMeshMatch: Render the image, match with ground image, propagate to aerial views, refine the match and export the results.
TileImage: tile original image into blocked TIFF format. This is used for fast loading. We have optimized the file loading and encoding using multi-thread.
UndistortImage: This is currently not used.

A common routine to do the aerial-ground matching is like

TileImage -i Aerial.xml -o ./AerialTiled
FeatureExtract -i ./AerialTiled/Aerial.json
RenderMeshMatch -a ./AerialTiled/Aerial.json -g Ground.xml -m model.osgb -c config.json

DATASET

We release all the images, initial AT (Aerial Triangulation) Files and the Tiled 3D models.

Dortmund (3.1 GB): Google Drive, BaiduYun Code: 7n58
SWJTU-BLD (2.7 GB): Google Drive, BaiduYun Code: vgj5
SWJTU-LIB (2.0 GB): Google Drive, BaiduYun Code: aiwa
SWJTU-RES (2.2 GB): Google Drive, BaiduYun Code: bvje

TODO List

There are several differences for the code and the paper.

The normal is not implemented, because this requires a lot of tricks in the graphics pipeline and pre-processing of the models in tiled format.
The occlusion detection is currently not ported.

Licenses

Data: CC-BY-NC (Except for the images and derivatives of the Dortmund Datasets)
Code: MIT

bigkabi / RenderMatch