DataAugmentation Process

[adhara123007]

Thanks Nikolaus fir being patient and answering so many of my questions, which brings me to my next one :)

I want to understand the data augmentation process. Lets look at FlowNet2-CSS. There are three layers. As I understand, Layer - 1 is DataAugmentation layer which applies the transformation to the "image-1" and generates "img0_aug" blob. It also generates "blob6" which has somehow stores the parameters. Both these are input to next layer (Layer-2) which only generates augmentation parameters (it doesn't apply them) by modifying/adding to the existing parameters and stores them in "blob7". Layer-3 takes "blob 7" and applies it to "image-2".
It looks like it the same transformations are not applied to both "image-1" and "image-2". Is this deliberate to put effect of lighting conditions?

Layer - 1

layer {
name: "img0s_aug"
type: "DataAugmentation"
bottom: "blob3"
top: "img0_aug"
top: "blob6"
augmentation_param {
max_multiplier: 1
augment_during_test: false
recompute_mean: 1000
mean_per_pixel: false
translate {
rand_type: "uniform_bernoulli"
exp: false
mean: 0
spread: 0.4
prob: 1.0
}
rotate {
rand_type: "uniform_bernoulli"
exp: false
mean: 0
spread: 0.4
prob: 1.0
}
zoom {
rand_type: "uniform_bernoulli"
exp: true
mean: 0.2
spread: 0.4
prob: 1.0
}
squeeze {
rand_type: "uniform_bernoulli"
exp: true
mean: 0
spread: 0.3
prob: 1.0
}
lmult_pow {
rand_type: "uniform_bernoulli"
exp: true
mean: -0.2
spread: 0.4
prob: 1.0
}
lmult_mult {
rand_type: "uniform_bernoulli"
exp: true
mean: 0.0
spread: 0.4
prob: 1.0
}
lmult_add {
rand_type: "uniform_bernoulli"
exp: false
mean: 0
spread: 0.03
prob: 1.0
}
sat_pow {
rand_type: "uniform_bernoulli"
exp: true
mean: 0
spread: 0.4
prob: 1.0
}
sat_mult {
rand_type: "uniform_bernoulli"
exp: true
mean: -0.3
spread: 0.5
prob: 1.0
}
sat_add {
rand_type: "uniform_bernoulli"
exp: false
mean: 0
spread: 0.03
prob: 1.0
}
col_pow {
rand_type: "gaussian_bernoulli"
exp: true
mean: 0
spread: 0.4
prob: 1.0
}
col_mult {
rand_type: "gaussian_bernoulli"
exp: true
mean: 0
spread: 0.2
prob: 1.0
}
col_add {
rand_type: "gaussian_bernoulli"
exp: false
mean: 0
spread: 0.02
prob: 1.0
}
ladd_pow {
rand_type: "gaussian_bernoulli"
exp: true
mean: 0
spread: 0.4
prob: 1.0
}
ladd_mult {
rand_type: "gaussian_bernoulli"
exp: true
mean: 0.0
spread: 0.4
prob: 1.0
}
ladd_add {
rand_type: "gaussian_bernoulli"
exp: false
mean: 0
spread: 0.04
prob: 1.0
}
col_rotate {
rand_type: "uniform_bernoulli"
exp: false
mean: 0
spread: 1
prob: 1.0
}
crop_width: 768
crop_height: 384
chromatic_eigvec: 0.51
chromatic_eigvec: 0.56
chromatic_eigvec: 0.65
chromatic_eigvec: 0.79
chromatic_eigvec: 0.01
chromatic_eigvec: -0.62
chromatic_eigvec: 0.35
chromatic_eigvec: -0.83
chromatic_eigvec: 0.44
noise {
rand_type: "uniform_bernoulli"
exp: false
mean: 0.03
spread: 0.03
prob: 1.0
}
}
}

Layer - 2

layer {
name: "aug_params1"
type: "GenerateAugmentationParameters"
bottom: "blob6"
bottom: "blob3"
bottom: "img0_aug"
top: "blob7"
augmentation_param {
augment_during_test: false
translate {
rand_type: "gaussian_bernoulli"
exp: false
mean: 0
spread: 0.03
prob: 1.0
}
rotate {
rand_type: "gaussian_bernoulli"
exp: false
mean: 0
spread: 0.03
prob: 1.0
}
zoom {
rand_type: "gaussian_bernoulli"
exp: true
mean: 0
spread: 0.03
prob: 1.0
}
gamma {
rand_type: "gaussian_bernoulli"
exp: true
mean: 0
spread: 0.02
prob: 1.0
}
brightness {
rand_type: "gaussian_bernoulli"
exp: false
mean: 0
spread: 0.02
prob: 1.0
}
contrast {
rand_type: "gaussian_bernoulli"
exp: true
mean: 0
spread: 0.02
prob: 1.0
}
color {
rand_type: "gaussian_bernoulli"
exp: true
mean: 0
spread: 0.02
prob: 1.0
}
}
coeff_schedule_param {
half_life: 50000
initial_coeff: 0.5
final_coeff: 1
}
}

Layer - 3

layer {
name: "img1s_aug"
type: "DataAugmentation"
bottom: "blob4"
bottom: "blob7"
top: "img1_aug"
augmentation_param {
max_multiplier: 1
augment_during_test: false
recompute_mean: 1000
mean_per_pixel: false
crop_width: 768
crop_height: 384
chromatic_eigvec: 0.51
chromatic_eigvec: 0.56
chromatic_eigvec: 0.65
chromatic_eigvec: 0.79
chromatic_eigvec: 0.01
chromatic_eigvec: -0.62
chromatic_eigvec: 0.35
chromatic_eigvec: -0.83
chromatic_eigvec: 0.44
}
}

[Nikolaus Mayer]

Yes, an additional set of augmentations is applied to the second image. This simulates effects like lighting condition changes, but also introduces geometric variation (we took a look at these elements in an IJCV paper).
Note that blob6 and blob7 are used by the layer which processes the groundtruth flow so it matches the images' geometric augmentations.

[adhara123007]

I tried taking the output from blob7 (using the following prototxt file) and have a look at it.
blob = np.squeeze(net.blobs['blob7'].data). I get the following error:

[libprotobuf ERROR google/protobuf/text_format.cc:274] Error parsing text-format caffe.NetParameter: 67:9: Message type "caffe.LayerParameter" has no field named "layer".
WARNING: Logging before InitGoogleLogging() is written to STDERR
F0325 10:10:06.553112 752 upgrade_proto.cpp:79] Check failed: ReadProtoFromTextFile(param_file, param) Failed to parse NetParameter file: /tmp/tmpc4WuJY
*** Check failure stack trace: ***
Aborted (core dumped)

Also, what is the parameter "coeff_schedule_param" in "GenerateAugmentationParameters" layer?

coeff_schedule_param {
half_life: 50000
initial_coeff: 0.5
final_coeff: 1
}

"Prototxt file"

input: "img0"
input_shape {
dim: 1
dim: 3
dim: $TARGET_HEIGHT$
dim: $TARGET_WIDTH$
}
layer {
name: "Eltwise1"
type: "Eltwise"
bottom: "img0"
top: "blob3"
eltwise_param {
operation: SUM
coeff: 0.00392156862745098
}
}
layer {
name: "img0s_aug"
type: "DataAugmentation"
bottom: "blob3"
top: "img0_aug"
top: "blob6"
augmentation_param {
max_multiplier: 1
augment_during_test: true
recompute_mean: 1000
mean_per_pixel: false
translate {
rand_type: "uniform_bernoulli"
exp: false
mean: 0
spread: 0.4
prob: 1.0
}
rotate {
rand_type: "uniform_bernoulli"
exp: false
mean: 0
spread: 0.4
prob: 1.0
}
zoom {
rand_type: "uniform_bernoulli"
exp: true
mean: 0.2
spread: 0.4
prob: 1.0
}
squeeze {
rand_type: "uniform_bernoulli"
exp: true
mean: 0
spread: 0.3
prob: 1.0
}
crop_width: 584
crop_height: 1001
noise {
rand_type: "uniform_bernoulli"
exp: false
mean: 0.0009733
spread: 0.03
prob: 1.0
}
}
layer {
name: "aug_params1"
type: "GenerateAugmentationParameters"
bottom: "blob6"
bottom: "blob3"
bottom: "img0_aug"
top: "blob7"
augmentation_param {
augment_during_test: false
translate {
rand_type: "gaussian_bernoulli"
exp: false
mean: 0
spread: 0.03
prob: 1.0
}
rotate {
rand_type: "gaussian_bernoulli"
exp: false
mean: 0
spread: 0.03
prob: 1.0
}
zoom {
rand_type: "gaussian_bernoulli"
exp: true
mean: 0
spread: 0.03
prob: 1.0
}
gamma {
rand_type: "gaussian_bernoulli"
exp: true
mean: 0
spread: 0.02
prob: 1.0
}
brightness {
rand_type: "gaussian_bernoulli"
exp: false
mean: 0
spread: 0.02
prob: 1.0
}
contrast {
rand_type: "gaussian_bernoulli"
exp: true
mean: 0
spread: 0.02
prob: 1.0
}
color {
rand_type: "gaussian_bernoulli"
exp: true
mean: 0
spread: 0.02
prob: 1.0
}
}
coeff_schedule_param {
half_life: 50000
initial_coeff: 0.5
final_coeff: 1
}
}

}

[Nikolaus Mayer]

The error message is due to your file missing a } after the img0s_aug layer, I think.
The coeff_schedule_param looks like it ramps up the magnitude of all augmentations over time.