microsoft / mup

maximal update parametrization (µP)

https://arxiv.org/abs/2203.03466

microsoft/mup Issues

Should `base=None` be used in `set_base_shapes` for model used for tuning?
Closed 2 years ago2
coord_check for model that returns loss function directly
Updated 4 months ago
Interpreting jitter in coordcheck
Closed 4 months ago2
Reproducing the validation accuracy vs learning rates curve on ResNet
Updated 5 months ago1
Reproducing the training loss vs learning rates curve on MLP
Closed 10 months ago5
Questions for training gpt-2 using mup
Closed 5 months ago5
About Learning rate decay
Updated 6 months ago1
FSDP support?
Updated 7 months ago1
Usage with torch.compile in Pytorch 2?
Updated 7 months ago1
Unclear `assert_hidden_size_inf` triggers
Closed 7 months ago1
dim_feedforward
Closed 7 months ago
Coord check looks good, but μTransfer is not working as expected
Closed a year ago6
Some questions about the implementation of muP.
Updated 8 months ago
µTransfer across batch size && weight decay setting
Updated 9 months ago
Is it possible to also scale the depth of the model?
Updated 9 months ago3
_rescale_parameters() inconsistent with the paper for the tied embedding scenario?
Updated 9 months ago2
Are Sequentials with list comprehension handled incorrectly?
Updated 10 months ago2
Once the best HPs have been found, does the final model have to be trained with `mup` or can one just use the found HPs and train the model in a standard way?
Closed a year ago
Is this compatible with DeepSpeed / ZeRO?
Closed a year ago6
Warmup schedule when changing the number of tokens/steps (GPT-3 experiment detail)
Updated a year ago
Positional Embeddings should be MuReadout parameters ?
Updated a year ago2
Does mup support fine tuning pretrained models
Closed a year ago2
Embedding Multiplier for Transformer - Clarification
Closed a year ago2
interpreting coord checks
Closed a year ago2
in mlp example: 2 problems
Updated a year ago1
Questions on learning schedule and binary classification
Closed a year ago12
Can base model be larger than target model?
Closed a year ago3
Should query layers in self-attention be initialized to 0 in practice?
Closed a year ago2
Does mup support Swin Transformer v2 model?
Updated a year ago2
Batch size, Seq len, Step Transfering
Closed a year ago2
Has MuP been tested on segmentation models?
Updated a year ago4
Finetuning a Pretrained Model Using MuP
Closed a year ago3
Issue in reproducing the training loss vs learning rates curve
Closed a year ago5
LayerNorm Gain and Bias Multipliers
Closed a year ago2
Are parameters with no "infinite" dimensions allowed?
Closed a year ago5
MuP Coord Check not Working with Electra Style Model
Closed 2 years ago8
Conv1D Coord check looks good (I think), but μTransfer does not seem to work?
Closed 2 years ago20
missing os import in mup/examples/MLP/main.py ?
Closed 2 years ago1
muP for contrastive losses
Closed 2 years ago2
Coord-check for conv1d
Closed 2 years ago17
mu parametrization for channel attention
Closed 2 years ago5
mu parametrization for multi-head attention / grouped convolution
Closed 2 years ago3
Optimizers for coord check
Closed 2 years ago2
ResNet readout_zero_init=True?
Closed 2 years ago2
Hyperparameter search on base models
Closed 2 years ago2
integration with Flax?
Updated 2 years ago4
Does MuReadout apply to all outputs on which loss is computed?
Closed 2 years ago2
How to use 'attn_mult' config
Closed 2 years ago2
Examples with ConvNets
Closed 2 years ago2
MuAdam not adjusting lr for output weights
Closed 2 years ago4