Training issue - multiple image pairs

Question

Training issue - multiple image pairs

quizz0n opened this issue 3 years ago · comments

After a successful run a while ago (only one pair of images: Sentinel-2/Spot-6 - 1276 patches), I wanted to train a new model with 4 additional pairs of images above different cities. So in total 5 image pairs - 2208 total number of patches (1276+345+158+264+165 patches).

I fallowed the same steps when preparing the patches images, however the training with 5 image pairs is not successful (incomplete parts in the final image).

I also did a run with only 2 image pairs, first image pair used in the first run a while ago + a random image pair to check if maybe there's an issue with one of the pairs. It looks to be the same just not as spread, probably because of the smaller quantity of additional patches.

Another test that I did was to apply the initial successful model to all the new Sentinel-2 images used for the new patches of the 5 image pairs. I thought I can check by this if there's something wrong with the dynamic of the new images or anything else. All of the resulting images were good.

TensorBoard:

successful run with only one pair of images (a while ago):

unsuccessful run with 5 pairs of images / unsuccessful run with 2 pairs of images:

python sr4rs/code/train.py \
--lr_patches 4/patch_extraction/lr_patches.tif 4/patch_extraction/lr_b_patches.tif 4/patch_extraction/lr_d_patches.tif 4/patch_extraction/lr_s2_patches.tif 4/patch_extraction/lr_s_patches.tif \
--hr_patches 4/patch_extraction/hr_patches.tif 4/patch_extraction/hr_b_patches.tif 4/patch_extraction/hr_d_patches.tif 4/patch_extraction/hr_s2_patches.tif 4/patch_extraction/hr_s_patches.tif \
--preview 4/images/lr_subset.tif \
--vggfile sr4rs/code/vgg19.npy \
--save_ckpt sr4rs/code/ckpts/ \
--logdir sr4rs/code/results/logs/ \
--lr_scale 0.0001 \
--hr_scale 0.0001 \
--epochs 150 \
--vggweight 0.000025 \
--l1weight 180 \
--l2weight 0.0 \
--depth 32 \
--vggfeatures 1234 \
--batchsize 4 \
--adam_lr 0.002 \
--savedmodel sr4rs/code/savedmodel_t4

python sr4rs/code/train.py \
--lr_patches 4/patch_extraction/lr_patches.tif 4/patch_extraction/lr_s2_patches.tif \
--hr_patches 4/patch_extraction/hr_patches.tif 4/patch_extraction/hr_s2_patches.tif \
--preview 4/images/lr_subset_old.tif \
--vggfile sr4rs/code/vgg19.npy \
--save_ckpt sr4rs/code/ckpts/ \
--logdir sr4rs/code/results/logs/ \
--lr_scale 0.0001 \
--hr_scale 0.0001 \
--epochs 150 \
--vggweight 0.000025 \
--l1weight 180 \
--l2weight 0.0 \
--depth 32 \
--vggfeatures 1234 \
--batchsize 4 \
--adam_lr 0.002 \
--savedmodel sr4rs/code/savedmodel_t5

Applying the model to a Sentinel-2 image:

successful run with only one pair of images (a while ago):

unsuccessful run with 5 pairs of images / unsuccessful run with 2 pairs of images:

I'm thinking maybe it's the way the files are read in the train command?
Thanks!

Rémi Cresson · Answer 1 · Wed Nov 17 2021 02:05:24 GMT+0800 (China Standard Time)

Hi @quizz0n

Nice images!

I suspect that this comes from the dynamic of the input HR images used to train the model.

Indeed, when you train a model from several (LR, HR) pairs, the most important thing - and difficult to achieve, is to preserve the LR --> HR radiometric transformation: it must be the same for all images. Else, the network might perform some "random" radiometric transform, because it has learnt multiple "plausible" mappings, but that do not transform the radiometry the same way. The dark spots impacting your images still looks "plausible" locally, because the model might have learnt multiple
radiometric transformations from (LR, HR) couples having very different radiometry transformations.

Maybe you want that your model preserves the input image radiometry. In this case, one simple thing do to prior to training, is to align radiometrically the HR patches over the LR patches. To do that, you can for instance normalize each HR patches image over the LR patches image (e.g. compute the y=ax+b from the averaged HR patches and the LR patches, or from their 1st and 2nd order statistics).
You can then transform easily your HR patches with BandMathX once you have computed your a and b:

# Generate quickly a new patches image
otbcli_BandMathX 
-il 4/patch_extraction/hr_patches.tif 
-out 4/patch_extraction/hr_patches_calibrated.tif 
-exp "{1.2*im1b1+0.2; 0.8*im1b2-0.3; ...; 1.8*im1b4+0.02}"  # here is y=ax+b applied

quizz0n · Answer 2 · Tue Nov 23 2021 01:07:20 GMT+0800 (China Standard Time)

Hi @remicres, thank you for the fast reply!
That makes sense.

In order to normalize each HR patches image over the LR patches image, I'm not sure I understand what would the a and b represent in this case and how to compute them.

Rémi Cresson · Answer 3 · Tue Nov 23 2021 07:37:22 GMT+0800 (China Standard Time)

a and b are some vector (N components, like your images) to linear transform the HR images.

For instance you can compute them with least square regression for an entire patches-image, considering that one point is the average pixel value for a single patch.

quizz0n · Answer 4 · Tue Nov 23 2021 09:18:06 GMT+0800 (China Standard Time)

I've seen that in GRASS there's the r.regression.line algorithm:

r.regression.line calculates a linear regression from two raster maps, according to the formula
y = a + b*x
where
x
y
represent the input raster maps.
Optionally, it saves regression coefficients.

This should compute the a and b, but I'm not sure which patches image (LR/HR) should be the x / y here.

Rémi Cresson · Answer 5 · Tue Nov 23 2021 17:04:37 GMT+0800 (China Standard Time)

Ah nice if there is such thing in another OSGeo software ! 👍

You could compute the averaged HR image over the same pixel spacing as LR. Maybe using gdalwarp -r average -tr 10 10 input_hr.tif avg_input_hr.tif as x (and y is your LR patches)

quizz0n · Answer 6 · Fri Nov 26 2021 11:41:12 GMT+0800 (China Standard Time)

Thanks @remicres!
Unfortunately r.regression.line doesn't work properly. Do you have any suggestion for computing the a and b ?

Rémi Cresson · Answer 7 · Wed Dec 01 2021 17:55:44 GMT+0800 (China Standard Time)

Yes, you could compute them with least square regression for an entire patches-image, considering that one point is the average pixel value for a single patch.
Use gdal or otbApplication to read your LR and HR and patches images, then use numpy to average per patches and compute the polynomial fitting

quizz0n · Answer 8 · Tue Feb 15 2022 00:27:51 GMT+0800 (China Standard Time)

Hi @remicres,
I was able to fix the issue then with your suggestion (many thanks for that!) but didn't manage to give you an update so here it is.

From left to right: pre-trained model, original Spot-6 image, own model (about 2k patches).

Rémi Cresson · Answer 9 · Tue Feb 15 2022 04:08:05 GMT+0800 (China Standard Time)

Hi @quizz0n ,

Thanks for the feedback.
This is an interesting comparison, and your images look great 👍

Out of curiosity, which part of the world is that? (our pre-trained model has been only trained from patches over France mainland)

quizz0n · Answer 10 · Tue Feb 15 2022 15:34:12 GMT+0800 (China Standard Time)

Thanks!
This is above one city in Romania, Alba-Iulia.
Looking at the results over the Montpellier area I found the overall characteristics to be pretty similar between the two.
Own model seems to be better however this still has to go through a more in-depth validation.