khanrc / swad

Official Implementation of SWAD (NeurIPS 2021)

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Can't reproduce the result of ERM.

justopit opened this issue · comments

train PACS and Office Home one time, and the result is

Office Home: Last: 62.4, SWAD: 70.8 , Oracle: 66.3
PACS: Last: 82.2, SWAD: 88.3, Oracle: 88.1

The Last result if ERM? It is not consistent with the result of your paper. So the ERM result is Oracle?

So, where I can see the result of ERM w/o SWAD

Our ERM results were from HP search. Searched ERM HPs are different from the SWAD reproduced commands we provided. This repository does not aim to reproduce ERM results, but here are some tips for this: 1) Set HP by referring to #3 (comment), 2) Turn off SWAD, and 3) Check the iid row results instead of last or oracle row.

OK, thank you.