Giters

deepseek-ai / DeepSeek-Coder-V2

DeepSeek-Coder-V2: Breaking the Barrier of Closed-Source Models in Code Intelligence

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

DeepSeek-Coder-V2-Lite model GPU/RAM requirement

HashedViking opened this issue · comments

commented

Hi, thank you for the amazing work! In the readme you say "DeepSeek-Coder-V2 in BF16 format for inference, 80GB*8 GPUs are required".
How much GPU/RAM is needed for inference for DeepSeek-Coder-V2-Lite model?

commented

40GB *1 GPUs in BF16 format

commented

40GB *1 GPUs in BF16 format

3*24G 4090 GPU doesn’t work?

commented

40GB *1 GPUs in BF16 format

3*24G 4090 GPU doesn’t work?

If you have 3*24G 4090 GPUs, you need to enable TP 2 or PP 2. Alternatively, you can use the quantized version of the model on Ollama.