microsoft / mup

maximal update parametrization (µP)

https://arxiv.org/abs/2203.03466

microsoft/mup Issues

More options of input/output types in coord_check
Updated 2 months ago
How to use with SSL methods like DINOv2?
Updated 5 months ago
MuP for RNNs
Updated 5 months ago
Not getting perf improvements from muP at ~1.5B scale
Updated 5 months ago
Increasing coord check for the network output
Updated 6 months ago2
MuP for Mamba
Updated 7 months ago
FSDP support?
Updated 8 months ago3
About Learning rate decay
Updated 8 months ago2
Questions for training gpt-2 using mup
Closed a year ago6
mu parametrization for gated-mlp and group-query attention
Updated 9 months ago
Is it possible to also scale the depth of the model?
Updated 10 months ago5
Usage with torch.compile in Pytorch 2?
Updated 10 months ago2
Reproducing Figure 1 using 'examples/Transformer/main.py'
Updated a year ago
Should `base=None` be used in `set_base_shapes` for model used for tuning?
Closed 2 years ago2
coord_check for model that returns loss function directly
Updated a year ago
Interpreting jitter in coordcheck
Closed a year ago2
Reproducing the validation accuracy vs learning rates curve on ResNet
Updated a year ago1
Reproducing the training loss vs learning rates curve on MLP
Closed a year ago5
Unclear `assert_hidden_size_inf` triggers
Closed a year ago1
dim_feedforward
Closed a year ago
Coord check looks good, but μTransfer is not working as expected
Closed 2 years ago6
Some questions about the implementation of muP.
Updated a year ago
µTransfer across batch size && weight decay setting
Updated a year ago
_rescale_parameters() inconsistent with the paper for the tied embedding scenario?
Updated a year ago2
Are Sequentials with list comprehension handled incorrectly?
Updated a year ago2
Once the best HPs have been found, does the final model have to be trained with `mup` or can one just use the found HPs and train the model in a standard way?
Closed 2 years ago
Warmup schedule when changing the number of tokens/steps (GPT-3 experiment detail)
Updated 2 years ago
Positional Embeddings should be MuReadout parameters ?
Updated 2 years ago2
Does mup support fine tuning pretrained models
Closed 2 years ago2
Embedding Multiplier for Transformer - Clarification
Closed 2 years ago2
interpreting coord checks
Closed 2 years ago2
in mlp example: 2 problems
Updated 2 years ago1
Questions on learning schedule and binary classification
Closed 2 years ago12
Can base model be larger than target model?
Closed 2 years ago3
Should query layers in self-attention be initialized to 0 in practice?
Closed 2 years ago2
Does mup support Swin Transformer v2 model?
Updated 2 years ago2
Batch size, Seq len, Step Transfering
Closed 2 years ago2
Has MuP been tested on segmentation models?
Updated 2 years ago4
Finetuning a Pretrained Model Using MuP
Closed 2 years ago3
Issue in reproducing the training loss vs learning rates curve
Closed 2 years ago5
LayerNorm Gain and Bias Multipliers
Closed 2 years ago2
Are parameters with no "infinite" dimensions allowed?
Closed 2 years ago5
MuP Coord Check not Working with Electra Style Model
Closed 2 years ago8
Conv1D Coord check looks good (I think), but μTransfer does not seem to work?
Closed 2 years ago20
missing os import in mup/examples/MLP/main.py ?
Closed 2 years ago1
muP for contrastive losses
Closed 2 years ago2
Coord-check for conv1d
Closed 3 years ago17
mu parametrization for channel attention
Closed 3 years ago5
mu parametrization for multi-head attention / grouped convolution
Closed 3 years ago3
Optimizers for coord check
Closed 3 years ago2