microsoft / Megatron-DeepSpeed

Ongoing research training transformer language models at scale, including: BERT & GPT-2

microsoft/Megatron-DeepSpeed Issues

屎山代码DeepSpeed
Updated 2 days ago2
about the optimizer param group
Updated 2 days ago
Error message in checkpoint_{loader, saver}_megatron.py is usually a red herring
Updated 5 days ago1
Sequence Parallel is incompatible with Rotary Positional Embedding
Updated 9 days ago1
AttributeError: 'Namespace' object has no attribute 'deepspeed_config_dict'. Did you mean: 'deepspeed_config'? && batch = next(self.data_iterator)
Updated 10 days ago1
ImportError: /mnt/afs/hzy/Megatron-DeepSpeed/megatron/fused_kernels/build/scaled_upper_triang_masked_softmax_cuda.so
Updated 15 days ago4
[QUESTION] How to convert pretrained MoE model to FP32?
Updated 18 days ago6
Spurious all gather performance drop.
Updated 19 days ago
Call for Conversion from Huggingface to Megads with MoE
Updated 25 days ago
Expert deepcopy raises PickleError
Updated a month ago
[BUG] grad_scale_func is not working when using Float16OptimizerWithFloat16Params, causing slow loss drops in fp16
Updated a month ago2
Assertion failure when there are more than 255 tokenized data files (assert num_datasets < 255 in blendable_dataset.py)
Updated a month ago
FileNotFoundError: [Errno 2] No such file or directory: 'dataset/index-cache/xxx_doc_idx.npy'
Updated a month ago3
Fine-tune llama2 with sequence parallelism
Updated 2 months ago3
Pipeline parallelism + CPU offload?
Updated 2 months ago
[REQUEST] Could you add a new release version tag to Megatron-Deepspeed？Thanks
Closed 2 months ago2
[BUG] Problems with Mixture-of-Experts (MoE)
Updated 2 months ago
Bugs in GPT2 Inference Example
Updated 2 months ago2
loss of model decreased well but has terrible prediction result
Updated 2 months ago2
Problem in hf2megads_weight_converter.py
Updated 2 months ago
Loss is increasing when fine-tuning from a Megatron-Deepspeed pretrained checkpoint.
Updated 2 months ago
Some issues in the Twin-Flow feature provided by Zero-offload ++
Closed 3 months ago1
Unreasonably low throughput on HGX-H100s
Updated 3 months ago
The link (https://the-eye.eu/public/AI/pile_neox/data/BookCorpusDataset_text_document.bin ) has expired.
Closed 3 months ago1
The pile and its BookCorpus subset are not available
Closed 3 months ago2
how to convert deepspeed model to megatron, when pp=2, tp=2, nnode=2
Updated 3 months ago1
2nodes, 4 gpu, tp=2,pp=2, timeout
Updated 4 months ago
Error set num-experts>1 when running the generate_test.sh
Updated 4 months ago
Support TransformerEngine
Closed 4 months ago1
[QUESTION] Does the dev team have a plan to merge Mega-LM 0.4?
Updated 4 months ago
it seems there is a version problem please help me
Updated 4 months ago
[QUESTION]how to use' nsight compute' to profile 'pretrain_llama2_distributed.sh' in the 'examples_deepspeed' folder ?
Updated 4 months ago
Doubts about GPU memory
Updated 4 months ago
bos_token_id is assigned to be eos_token_id in _HFTokenizer
Updated 4 months ago1
error in generate_text.sh
Updated 4 months ago1
It should be no data at this website....
Closed 5 months ago8
Will offline++be forcibly uninstalled after specifying a ratio?
Updated 5 months ago1
[BUG] suspicious bf16 config
Updated 5 months ago2
Zero2/3 segmentation fault
Closed 5 months ago1
Question about order of transpose operation.
Closed 5 months ago
Support for distributed finetuning
Updated 5 months ago
load_args_from_checkpoint fails if arguments are missing
Updated 6 months ago
[BUG] An error occurs due to mismatched shapes during the process of splitting mixed_x_layer
Updated 6 months ago4
[QUESTION] How can I finetune a model without zero_pp_rank_*_mp_rank_00_optim_states.pt
Updated 6 months ago
[BUG] FP8+PP+Recompute+GA>1, loss = nan
Closed 6 months ago1
Deepspeed Zero Stage 3 save a empty model state_dict
Updated 6 months ago1
In megatron/model/transformer.py, repeat_kv when using GQA can be applied after rotary embeddings; can give a performance gain for RoPE
Updated 6 months ago2
Loss curve changes after checkpoint load
Closed 6 months ago17
Details on ZeRO++ tutorials
Updated 6 months ago2
Can't load GPT Model: ImportError: 'TP_REPLICATED_PARAMETER_PATTERNS' from 'deepspeed.checkpoint'
Updated 7 months ago7