cleardusk / MeGlass

An eyeglass face dataset collected and cleaned for face recognition evaluation, CCBR 2018.

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Evaluation protocol IV get the different result from paper

gangeqian opened this issue · comments

Hi
I evaluation the four testing protocols in paper. I don't confirm my method and result is correct.
there is 4 file list txt。[0] gallery_black_glass.txt [1]gallery_no_glass.txt [2]probe_black_glass.txt [3]probe_no_glass.txt
4 test protocols:
I) [1] vs [3]
II) [0] vs [2]
III) [1] vs [2]
IV) [0][1] vs [2][3] (Gallery images contain both eyeglass images and non eyeglass images, so as probe image.)
the result from paper show that III) has the better result than [IV)。
for ResNet-22-A RPR@FAR=10e-4 III): 88.13 IV):78.17 Rank1 III): 95.61 IV):92.31

in my experiment test, I use LightCNN29 model to evaluate, ang get the following result:
for LightCNN29 RPR@FAR=10e-4 III): 87.49 IV):87.03 Rank1 III): 93.39 IV):98.54
as showed, III) has litter better result than IV) and Rank1 smaller than IV).

I mixture [0][1] as the new Gallery, and mixture [2][3] as the new probe in protocol IV). which should has less degree of difficulty than protocol III), because all of pairs in III) contains non-eyeglass and eyeglass, otherwise,IV)contains some simple pairs of non-eyeglass.

So I don't think IV) could get better than III) from my consider and test

where is wrong for my test? could you tell me

commented

Would it depend on which combinations were taken?
Was there any given .txt file provided with the pairs list for any of the protocols?

Could you provide the pair lists you have used yourself and I could compare my results.