Instance Segmentation Results on Area5

[mohuamuyu]

Thank you for your great work! but I have some problem when trying to reproduce the results on Area5. Has anyone reproduced the results on Instance Segmentation task?
Can you provide some advice and guidance?

//=======================================
Area5 Results report in the paper:

Table 1: Instance segmentation results on S3DIS dataset.
Backbone Method mCov mWCov mPrec mRec
PN++ ASIS 44.6 47.8 55.3 42.4

Table 2: Semantic segmentation results on S3DIS dataset
Backbone Method mAcc mIoU oAcc
PN++ ASIS 60.9 53.4 86.9

//=======================================
Evaluate the trained model you provided on Area5 :
"(optional) Trained model can be downloaded from here."

I can reproduce the Semantic segmentation results with the trained model you provided,
but can not reproduce the Instance segmentation results. (Just testing with the trained model you provided, without training by myself)

Because of the evaluate method in test.py sampled 4096 points in each block to test on, each time the evaluation results are different. The mMUCov, mMWCov, mACC, mIoU oAcc will be a little different which is acceptable, but the difference is a bit big for mPrecision and mRecall.

I evaluate the provided model on Area5 using "test.py" and "eval_iou_accuracy.py" for three times, without changing any code. Results are as follows:
//--------------------------------------
test on Area5 for the 1st time:

Instance segmentation:
mMUCov: 0.42489331999047575 < 44.6
mMWCov: 0.4535493664127393 < 47.8
mPrecision: 0.5021224775661305 < 55.3
mRecall: 0.4004512705562707 < 42.4

Semantic segmentation:
mAcc: 0.6092345653223031 = 60.9
mIoU: 0.5331283341313834 = 53.4
oAcc: 0.8693586484215375 = 86.9
//--------------------------------------
test on Area5 for the 2nd time:

Instance segmentation:
mMUCov: 0.4291705590335591 < 44.6
mMWCov: 0.45548170503790186 < 47.8
mPrecision: 0.5474015599454721 = 55.3
mRecall: 0.4274631185866652 = 42.4

Semantic segmentation:
mAcc: 0.6084982131786074 = 60.9
mIoU: 0.5328714125575762 = 53.4
oAcc: 0.8693728293860369 = 86.9
//--------------------------------------
test on Area5 for the 3rd time:

Instance segmentation:
mMUCov: 0.4270858894049287 < 44.6
mMWCov: 0.45475902917080196 < 47.8
mPrecision: 0.5194688592978431 < 55.3
mRecall: 0.40944191619875714 < 42.4

Semantic segmentation:
mAcc: 0.6079589620032484 = 60.9
mIoU: 0.5315288320965799 = 53.4
oAcc: 0.8692114229308049 = 86.9
//--------------------------------------

//=======================================
So I trained the model by myself, and test it on Area5,
using your training and tesing code, with random initialization.

//--------------------------------------
test on Area5 for the 1st time:

mMUCov: 0.4192711216731248 < 44.6
mMWCov: 0.4444165547538725 < 47.8
mPrecision: 0.5096157303879506 < 55.3
mRecall: 0.40007411245273317 < 42.4

mAcc: 0.6049548823385776 = 60.9
mIoU: 0.528582705433821 = 53.4
oAcc: 0.8679205275945892 = 86.9

//--------------------------------------
test on Area5 for the 2nd time:

mMUCov: 0.4174190863147767 < 44.6
mMWCov: 0.44349561917787444 < 47.8
mPrecision: 0.5229537176620869 < 55.3
mRecall: 0.40793882667136827 < 42.4

mAcc: 0.6044562112313121 = 60.9
mIoU: 0.5287068091885877 = 53.4
oAcc: 0.8676705435570818 = 86.9

//=======================================
I also trained the model by initializing the network using the provided model weights.

//--------------------------------------
test on Area5 for the 1st time:

mMUCov: 0.4184383436403125 < 44.6
mMWCov: 0.4453902032528212 < 47.8
mPrecision: 0.5073059155435999 < 55.3
mRecall: 0.3895353315843023 < 42.4

mAcc: 0.6104495511204024 = 60.9
mIoU: 0.5355729541022203 = 53.4
oAcc: 0.868981285117484 = 86.9

Thank you again for your code.

[Xinlong Wang]

Hi @mohuamuyu , please check if the dataset including the annotations are collected and orgnized correctly. And could you provide your environment info including python version and tf vertion.

[mohuamuyu]

I run the experiments on Ubuntu16.06, Anaconda2, python3.5, tensorflow1.9.
Just as you mentioned, I have fixed the data bug in Stanford3dDataset_v1.2_Aligned_Version/Area_5/hallway_6/ceiling_1.txt manually. I can generate *.npy and *h5 files successfully。
Thank you for your advice, I plan to check the datasets more carefully, and
I will try to run the experiments in other environments.

[mohuamuyu]

when I use the python2.7 and tensorflow1.3, I get the results as your paper, using your provided trained models on Area5. thank you for your help.