Training MixNet models for Cifar 10

[Mete Ozay]

Hello,

Have you trained MixNet models for Cifar-10 from scratch (not transfer learning from ImageNet models)?

With default configuration, I obtain about 70% accuracy for MixNet small, and the accuracy decreases as the model size increases. Do I need to change some particular parameters in the configuration while training Cifar-10?

Thank you.

[Myeongjun Kim]

Hi, I'm Myeongjun Kim. How many epochs are you training? I think It was not completely training. Thank you. Best regards, Myeongjun Kim

…

On Thu, Nov 7, 2019 at 8:15 AM Mete Ozay ***@***.***> wrote: Hello, Have you trained MixNet models for Cifar-10 from scratch (not transfer learning from ImageNet models)? With default configuration, I obtain about 70% accuracy for MixNet small, and the accuracy decreases as the model size increases. Do I need to change some particular parameters in the configuration while training Cifar-10? Thank you. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#4?email_source=notifications&email_token=AFIOHZR2A3QZBQ7DRDNBIVTQSNFZZA5CNFSM4JJ7C2FKYY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4HXONZAA>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AFIOHZWTLQTSZAPXERN7GR3QSNFZZANCNFSM4JJ7C2FA> .

[Mete Ozay]

Hi Myeongjun, I train usually for 1000 epochs, however, accuracy does not change much after 300 epochs (converges to some points at about 300 epochs). What is the best accuracy you achieved for Cifar-10 or 100? Thank you, Best, Mete. On Thu, Nov 7, 2019 at 10:58 AM Myeongjun Kim <[e-mail]> wrote:

…

Hi, I'm Myeongjun Kim. How many epochs are you training? I think It was not completely training. Thank you. Best regards, Myeongjun Kim On Thu, Nov 7, 2019 at 8:15 AM Mete Ozay ***@***.***> wrote: > Hello, > > Have you trained MixNet models for Cifar-10 from scratch (not transfer > learning from ImageNet models)? > > With default configuration, I obtain about 70% accuracy for MixNet small, > and the accuracy decreases as the model size increases. Do I need to change > some particular parameters in the configuration while training Cifar-10? > > Thank you. > > — > You are receiving this because you are subscribed to this thread. > Reply to this email directly, view it on GitHub > < #4?email_source=notifications&email_token=AFIOHZR2A3QZBQ7DRDNBIVTQSNFZZA5CNFSM4JJ7C2FKYY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4HXONZAA >, > or unsubscribe > < https://github.com/notifications/unsubscribe-auth/AFIOHZWTLQTSZAPXERN7GR3QSNFZZANCNFSM4JJ7C2FA > > . > — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#4?email_source=notifications&email_token=ABCQZY4RT4Q23ME5Y4XVKNTQSNY4NA5CNFSM4JJ7C2FKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEDIUDVQ#issuecomment-550584790>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABCQZY5EKODEOUG3OBG33OTQSNY4NANCNFSM4JJ7C2FA> .

[Myeongjun Kim]

What is written in the readme is the accuracy report. Or maybe the problem is caused by a different pytorch version. I used the 1.1.0 version. Thank you.

…

On Thu, Nov 7, 2019 at 11:29 AM Mete Ozay ***@***.***> wrote: Hi Myeongjun, I train usually for 1000 epochs, however, accuracy does not change much after 300 epochs (converges to some points at about 300 epochs). What is the best accuracy you achieved for Cifar-10 or 100? Thank you, Best, Mete. On Thu, Nov 7, 2019 at 10:58 AM Myeongjun Kim ***@***.***> wrote: > Hi, I'm Myeongjun Kim. > > How many epochs are you training? I think It was not completely training. > > Thank you. > > Best regards, > > Myeongjun Kim > > On Thu, Nov 7, 2019 at 8:15 AM Mete Ozay ***@***.***> wrote: > > > Hello, > > > > Have you trained MixNet models for Cifar-10 from scratch (not transfer > > learning from ImageNet models)? > > > > With default configuration, I obtain about 70% accuracy for MixNet small, > > and the accuracy decreases as the model size increases. Do I need to > change > > some particular parameters in the configuration while training Cifar-10? > > > > Thank you. > > > > — > > You are receiving this because you are subscribed to this thread. > > Reply to this email directly, view it on GitHub > > < > #4?email_source=notifications&email_token=AFIOHZR2A3QZBQ7DRDNBIVTQSNFZZA5CNFSM4JJ7C2FKYY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4HXONZAA > >, > > or unsubscribe > > < > https://github.com/notifications/unsubscribe-auth/AFIOHZWTLQTSZAPXERN7GR3QSNFZZANCNFSM4JJ7C2FA > > > > . > > > > — > You are receiving this because you authored the thread. > Reply to this email directly, view it on GitHub > < #4?email_source=notifications&email_token=ABCQZY4RT4Q23ME5Y4XVKNTQSNY4NA5CNFSM4JJ7C2FKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEDIUDVQ#issuecomment-550584790 >, > or unsubscribe > < https://github.com/notifications/unsubscribe-auth/ABCQZY5EKODEOUG3OBG33OTQSNY4NANCNFSM4JJ7C2FA > > . > — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#4?email_source=notifications&email_token=AFIOHZXMLRRDJ55AY55RC6DQSN4P7A5CNFSM4JJ7C2FKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEDIV4GA#issuecomment-550592024>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AFIOHZQCXQVBPZQZ7RK77OTQSN4P7ANCNFSM4JJ7C2FA> .

[Mete Ozay]

Thank you. - Did you train these models from scratch, or - did you apply transfer learning on pre-trained ImageNet models to train Cifar-10 models (as suggested in the paper MixConv: Mixed Depthwise Convolutional Kernels)? Update: I implemented various other code of MixNets provided in different projects to train MixNet models using Cifar-10. The best achieved accuracy is about 80%. Imo, the problem is algorithmic , i.e. training MixNet from scratch does not perform well, for Cifar 10. Alternatively, there are some "implementation tricks" missed in these code. Please let me know the details, if you can succeed to achieve >= 92% accuracy by training MixNets using Cifar-10 from scratch. Best, Mete. On Thu, Nov 7, 2019 at 1:14 PM Myeongjun Kim <[e-mail]> wrote:

…

What is written in the readme is the accuracy report. Or maybe the problem is caused by a different pytorch version. I used the 1.1.0 version. Thank you. On Thu, Nov 7, 2019 at 11:29 AM Mete Ozay ***@***.***> wrote: > Hi Myeongjun, > > I train usually for 1000 epochs, however, accuracy does not change much > after 300 epochs (converges to some points at about 300 epochs). > > What is the best accuracy you achieved for Cifar-10 or 100? > > Thank you, > > Best, > > Mete. > > > On Thu, Nov 7, 2019 at 10:58 AM Myeongjun Kim ***@***.***> > wrote: > > > Hi, I'm Myeongjun Kim. > > > > How many epochs are you training? I think It was not completely training. > > > > Thank you. > > > > Best regards, > > > > Myeongjun Kim > > > > On Thu, Nov 7, 2019 at 8:15 AM Mete Ozay ***@***.***> > wrote: > > > > > Hello, > > > > > > Have you trained MixNet models for Cifar-10 from scratch (not transfer > > > learning from ImageNet models)? > > > > > > With default configuration, I obtain about 70% accuracy for MixNet > small, > > > and the accuracy decreases as the model size increases. Do I need to > > change > > > some particular parameters in the configuration while training > Cifar-10? > > > > > > Thank you. > > > > > > — > > > You are receiving this because you are subscribed to this thread. > > > Reply to this email directly, view it on GitHub > > > < > > > #4?email_source=notifications&email_token=AFIOHZR2A3QZBQ7DRDNBIVTQSNFZZA5CNFSM4JJ7C2FKYY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4HXONZAA > > >, > > > or unsubscribe > > > < > > > https://github.com/notifications/unsubscribe-auth/AFIOHZWTLQTSZAPXERN7GR3QSNFZZANCNFSM4JJ7C2FA > > > > > > . > > > > > > > — > > You are receiving this because you authored the thread. > > Reply to this email directly, view it on GitHub > > < > #4?email_source=notifications&email_token=ABCQZY4RT4Q23ME5Y4XVKNTQSNY4NA5CNFSM4JJ7C2FKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEDIUDVQ#issuecomment-550584790 > >, > > or unsubscribe > > < > https://github.com/notifications/unsubscribe-auth/ABCQZY5EKODEOUG3OBG33OTQSNY4NANCNFSM4JJ7C2FA > > > > . > > > > — > You are receiving this because you commented. > Reply to this email directly, view it on GitHub > < #4?email_source=notifications&email_token=AFIOHZXMLRRDJ55AY55RC6DQSN4P7A5CNFSM4JJ7C2FKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEDIV4GA#issuecomment-550592024 >, > or unsubscribe > < https://github.com/notifications/unsubscribe-auth/AFIOHZQCXQVBPZQZ7RK77OTQSN4P7ANCNFSM4JJ7C2FA > > . > — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#4?email_source=notifications&email_token=ABCQZY3WXFTHG2QUTWMQQPTQSOIZPA5CNFSM4JJ7C2FKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEDJP3ZA#issuecomment-550698468>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABCQZY5KBA4QLMT4A423RFLQSOIZPANCNFSM4JJ7C2FA> .

[sporthand]

Hi, I also have the same problem which my Mixnet was originally installed from pip install Keras_Mixnet. I wonder if your code is similar to Keras_Mixnet.

So how many epoch did you set to get the over 90% of validation accuracy? I initially set the number of epoch as 10,000 and the number of patient as 50. However, it stopped between from 110th to 120 epoch. So, the result is 73% of validation accuracy. But, I am planning to change the number of patient from 50 to 100 in order to get the same validation accuracy. I am looking forward to have your instruction.
Thank you!

[Myeongjun Kim]

The code currently uploaded to github is model architecture in the ImageNet dataset. ImageNet and CIFAR have different input image sizes, so you need to train by modifying the stem layer and model architecture.
Thank you.