Bechmarking using caffe test tool

[Yeshwanth Venkatesha]

I modified the mobilenet_deploy.prototxt by adding data layers and accuracy layers.

layer {
    name: "data"
    type: "Data"
    top: "data"
    top: "label"
    include {
        phase: TRAIN
    }
    transform_param {
        mirror: true
        crop_size: 224
        mean_value: 104
        mean_value: 117
        mean_value: 123
    }
    data_param {
        source: "examples/imagenet/ilsvrc12_train_lmdb"
        batch_size: 10
        backend: LMDB
    }
}
layer {
    name: "data"
    type: "Data"
    top: "data"
    top: "label"
    include {
        phase: TEST
    }
    transform_param {
        mirror: false
        crop_size: 224
        mean_value: 104
        mean_value: 117
        mean_value: 123
    }
    data_param {
        source: "examples/imagenet/ilsvrc12_val_lmdb"
        batch_size: 10
        backend: LMDB
    }
}

...
A lot of layers here
...

layer {
    bottom: "fc7"
    bottom: "label"
    name: "loss"
    type: "SoftmaxWithLoss"
    top: "loss"
}

layer {
    bottom: "fc7"
    bottom: "label"
    top: "acc/top-1"
    name: "acc/top-1"
    type: "Accuracy"
    include {
        phase: TEST
    }
}

layer {
    bottom: "fc7"
    bottom: "label"
    top: "acc/top-5"
    name: "acc/top-5"
    type: "Accuracy"
    include {
        phase: TEST
    }
    accuracy_param {
        top_k: 5
    }
}

On trying to do a benchmark test using CAFFE_ROOT test --model mobilenet.prototxt --weights mobilenet.caffemodel, I am getting zero accuracy.
Any suggestions on this.

[ij10]

Hi,

I have the same problem. Although I have a very slightly different data layer in my altered mobilenet_deploy.prototxt file:

layer {
  name: "data"
  type: "Data"
  top: "data"
  top: "label"
  include {
    phase: TEST
  }
  transform_param {
    scale: 0.017
    mirror: false
    crop_size: 224
    mean_value: [103.94, 116.78, 123.68]
  }
  data_param {
    source: "/path/to/ilsvrc12_val_lmdb/"
    batch_size: 50
    backend: LMDB
  }
}

I use the following command to run inference on validation data (ImageNet 2012, 50K images):

./build/tools/caffe test -model /path/to/mobilenet_deploy.prototxt -weights /path/to/mobilenet.caffemodel -gpu 0 -iterations 1000

Any suggestions?

Thanks in advance!

[ij10]

Hi again,

I found the problem. It seems like I used a slightly modified caffe framework to run inference which got me zero in accuracy. When I then changed to the standard caffe framework I got slightly lower accuracies than what is presented.

[shicai]

@ij10 to reproduce the performance, you'd better use the original images. when generating lmdb datasets, images may be resized and cropped in a different way, so there may be some performance degradation.

[ij10]

@shicai I'll try that. Thanks!