Giters

CAMMA-public / rendezvous

A transformer-inspired neural network for surgical action triplet recognition from laparoscopic videos.

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Unable to reproduce results

carlosh93 opened this issue · comments

commented

Hi,

Thanks for sharing the code. I am trying different configurations, but I cannot reproduce the results shown in the results table. I have read the paper and used the same configurations. Could you please point me in the right direction to reproduce the results? This is my last result:

******************************************************************************************************************************************************************************************************************************************************
****************************************************** Run: run.py | Framework: PyTorch | Method: rendezvous_l8_cholectcholect50_k0_batchnorm_lowres | Version: 2 | Data: CholecT50 | Batch: 8 ******************************************************
*** Time: Wed Aug 23 14:55:39 2023 | Start: 0-epoch  0-steps | Init CKPT: ./__checkpoint__/run_2/rendezvous_l8_cholectcholect50_k0_batchnorm_lowres.pth | Save CKPT: ./__checkpoint__/run_2/rendezvous_l8_cholectcholect50_k0_batchnorm_lowres.pth ***
******************************************* LR Config: Init: [0.01, 0.01, 0.01] | Peak: [0.1, 0.1, 0.1] | Warmup Epoch: [9, 18, 58] | Rise: 0.1 | Decay 0.99 | train params 17120331 | all params 17120331 *******************************************
******************************************************************************************************************************************************************************************************************************************************
--------------------------------------------------
Test Results
Per-category AP:
I   : [0.92181609 0.85288714 0.97363326 0.61907781 0.83257944 0.67367684]
V   : [0.48829482 0.82626818 0.88521027 0.67931162 0.64592751 0.49003775
 0.43218691 0.14370269 0.1306369  0.19295252]
T   : [0.871987   0.12368118 0.30478707 0.14585582 0.06987836 0.18197214
 0.40765081 0.18287872 0.54285064        nan 0.35480296 0.0240869
 0.13138804 0.72797697 0.18824905]
IV  : [0.54444766 0.79120853 0.01177371 0.23637784 0.20182244 0.03897717
 0.03874252 0.05718203 0.73780621 0.07912493 0.04822602 0.86529259
 0.02323853        nan 0.12416068        nan        nan 0.39624429
 0.03983428 0.61592876 0.11862642 0.09059719 0.14822621 0.42056827
 0.11183068 0.13470465]
IT  : [0.77511714 0.07456015 0.06778698 0.01289309 0.03660033 0.51446138
 0.29508407 0.04227959 0.06049198 0.76342534 0.20182244 0.21152347
 0.10469678 0.0543054  0.16423916 0.03426366 0.44334165 0.31948076
 0.60744472        nan 0.1551666  0.15533951 0.14031986 0.07912493
 0.62475157 0.25613741 0.25274804 0.12195124 0.00366298 0.00277182
 0.01952348 0.44832755 0.01094626 0.12416068        nan 0.05288743
 0.22047608 0.29889868 0.027742   0.02237051        nan        nan
        nan 0.03983428 0.00538955 0.43032926 0.27359654        nan
 0.032429   0.11862642 0.01638695        nan        nan 0.03939073
 0.42056827 0.12017234 0.10679277 0.11742487 0.13470465]
IVT : [0.01374257 0.02038286 0.00372976 0.01289309 0.04931454 0.0392882
 0.02308338 0.1663519         nan 0.01079275 0.0341114  0.05172696
 0.76342534 0.23637784        nan        nan 0.02650409 0.68919188
 0.06081275 0.51397189 0.26189577 0.01631181 0.31948076 0.44334165
 0.03697838 0.00847221 0.05472511 0.10591901 0.1479067  0.69648711
 0.1035896  0.15533951        nan 0.00535301 0.04599283        nan
 0.07951776        nan        nan 0.00635413 0.14031986        nan
        nan        nan 0.03499391 0.00242605 0.00313091        nan
 0.00106157 0.00366298        nan 0.00821645 0.01648485 0.07803644
        nan        nan        nan 0.12243284 0.25178041 0.26738621
 0.62216952 0.44471886 0.00776213 0.03067817 0.01541671        nan
        nan 0.027742   0.29889868 0.22047608 0.08860436 0.02237051
        nan        nan        nan        nan        nan 0.032429
 0.27359654 0.43032926        nan 0.00538955 0.42056827        nan
 0.07258926        nan 0.00832395 0.07588882 0.12017234 0.00823753
 0.02369177 0.00835352 0.09756697 0.04323386 0.20182244 0.07912493
 0.12416068 0.03983428 0.11862642 0.13470465]
--------------------------------------------------
Mean AP:  I  |  V  |  T  |  IV  |  IT  |  IVT
:::::: : 0.8123 | 0.4915 | 0.3041 | 0.2554 | 0.1893 | 0.1364
==================================================

I trained with:

python run.py -t  --data_dir=datasets/CholecT50/ -l 1e-3 1e-4 1e-5 --dataset_variant=cholect50 --version=2 -b 8 --epochs 200

and tested with:

python run.py -e  --data_dir=datasets/CholecT50/ --dataset_variant=cholect50 --version=2 -b 8 --test_ckpt=__checkpoint__/run_2/rendezvous_l8_cholectcholect50_k0_batchnorm_lowres.pth

Also, the links to download the pretrained checkpoints don't work.

See, for example https://s3.unistra.fr/camma_public/github/rendezvous/rendezvous_l8_cholect50_batchnorm_lowres.pth

I really appreciate your help in reproducing your results to make a fair comparison with our method.

commented

Hi @carlosh93 , your lower result is most likely due to training from scratch. The result you are trying to reproduce is pretrained on Cholec80 for tool recognition. You can either use a pretrained model or train your model for longer epoch. On our side, we will fix the issues with downloading the pretrained checkpoints.