microsoft / DeepSpeed

DeepSpeed is a deep learning optimization library that makes distributed training and inference easy, efficient, and effective.

https://www.deepspeed.ai/

microsoft/DeepSpeed Issues

[BUG] Tensors are on different devices when model.step()
Updated in 13 hours6
When using pure DeepSpeed ulysses and zero stage 3 to continue pre-training, the loss gap between each GPU is too large.[BUG]
Updated in 13 hours2
[BUG] No `universal_checkpoint_info` in the Accelerate+Deepspeed Checkpoint
Updated 11 hours ago4
[BUG: Whisper model pipeline parallel training] logits and ground truth size mismatch during loss calculation
Updated 11 hours ago
[BUG] Deepspeed hangs when setting ds_accelerator to cuda via VS Code Debugger
Updated 11 hours ago5
[BUG]pipeline/zero1 don't support layers less than 3 in first stage
Updated 11 hours ago1
[BUG] (flops_profiler) Duplicate registration check for start_time_hook is not working
Updated 11 hours ago
nv-nightly CI test failure
Updated 11 hours ago
nv-ds-chat CI test failure
Updated 11 hours ago
[BUG] AttributeError deepspeed.comm has no attribute Processgroup
Updated a day ago1
[BUG] Trying to finetune mistral using deepspeed but running into an error: Error building extension 'cpu_adam'
Updated a day ago
Is there any solusion to overcome underflow issues?
Updated a day ago
[BUG] CompiledModuleWrapper causing issues with checkpoints and pipeline module
Updated a day ago1
How to use DeepSpeed-Ulysses with the Pure Deepspeed Zero (i.e., deepspeed.initialize)
Closed 8 days ago4
[BUG]Zero inference return bad result and low speed inference
Closed 2 days ago1
[BUG] Fail to Resume From Checkpoint with Different GPU Number(Huggingface Trainer + Deepspeed)
Updated 2 days ago16
[BUG] Gradient Accumulation Steps Initialization Bug in Pipeline Parallel Mode
Updated 3 days ago
[QUESTION/HELP] about ignore_unused_parameters hang
Updated 3 days ago1
[BUG] Training Mamba with ZeRO stage 3 and long context lengths - CUDA OOM error
Updated 5 days ago3
[BUG] Mis-typed free_blocks
Updated 5 days ago
Pip failure subprocess-exited-with-error
Closed 5 days ago5
[REQUEST] Deepspeed support finetune extra model with lora ?
Updated 6 days ago
[BUG] fused_adam cannot be installed
Updated 6 days ago1
pydantic.errors.PydanticUserError: A non-annotated attribute was detected: `enabled = True`. All model fields require a type annotation; if `enabled` is not meant to be a field, you may be able to resolve this error by annotating it as a `ClassVar` or updating `model_config['ignored_types']`.
Updated 6 days ago2
[BUG] inference generate raises ValueError
Updated 6 days ago2
[Question] Deepspeed inference stage 3 + quantization
Updated 7 days ago
UserWarning: CUDA initialization: Unexpected error from cudaGetDeviceCount(). Did you run some cuda functions before calling NumCudaDevices() that might have already set an error? Error 803: system has unsupported display driver / cuda driver combination
Updated 7 days ago1
[BUG] DeepSpeed hangs during evaluation under multi-GPU
Updated 7 days ago
[BUG]binascii.Error: Incorrect padding
Updated 8 days ago
Baichuan fine-tune.py bug
Updated 8 days ago
[BUG] in `runtime/zero/partitioned_param_coordinator.py` -- Parameter not in `self.__inflight_param_registry` but has ds_status = `ZeroParamStatus.INFLIGHT`
Updated 9 days ago
ubuntu 20.04 DS_BUILD_FUSED_ADAM=1 pip install deepspeed failed
Updated 9 days ago
[REQUEST]Can we cache the results of pipeline stages distribution to the disk? It costs much time when I restart the training.
Updated 10 days ago
Problem when training with FULL FP32 model
Updated 10 days ago
[BUG] Multi-Node training "os.PathLike object" and "ds_opt_adam" error.
Updated 10 days ago
[BUG]Pipeline Parallelism Training: AttributeError: 'list' object has no attribute 'dtype'
Updated 11 days ago3
[REQUEST] deepspeed-inference support flash-decoding or the similar idea
Updated 11 days ago
[BUG] Dynamically switching freeze layers during the training process
Updated 12 days ago2
Change in checkpoint sizes?
Updated 13 days ago
[BUG] cannot import name 'log' from elastic agent
Closed 13 days ago3
[BUG] how to parse the parameters of the respective section using hydra and argparse?
Updated 15 days ago
[BUG] Grad_norm is nan and Loss is 0
Updated 16 days ago
pip install deepspeed error ：Getting requirements to build wheel did not run successfully.
Closed 16 days ago4
[BUG]When I run the DeepSpeed program, I will automatically enter the Python interface
Updated 16 days ago1
DeepSpeed Inference with Model not From HuggingFace
Closed 17 days ago
[BUG] ZeRO-3 gradient partitioning issue
Updated 20 days ago
Backward time grows linearly to the number of to zero3_consolidated_16bit_state_dict called
Updated 20 days ago1
On the training problem of mixed-precision fp16
Updated 20 days ago
[QUESTION/HELP] ZERO3 weight modification after load
Closed 21 days ago2
Request for Mixtral 8X7B inference with DP+EP+TP
Updated 21 days ago