microsoft / DeepSpeed-MII

MII makes low-latency and high-throughput inference possible, powered by DeepSpeed.

microsoft/DeepSpeed-MII Issues

Can't use Llama 3.1 with MII, ImportError: cannot import name 'Conversation' from 'transformers'
Closed 2 days ago1
non-persistent example doesn't work on Mixtral-8*7B-v0.1
Updated 5 days ago
Import Error, not compatible with transformer package
Closed 5 days ago4
non-persistent simple example does not work
Updated 11 days ago5
deepseed-mii支持多节点推理么
Updated 12 days ago2
[BUG] MII Backend Hangs After 9999 Exceptions in `MIIAsyncPipeline.put_request`
Closed 12 days ago2
Dummy data loading?
Updated 17 days ago
Client cannot find deployment error
Updated 19 days ago
Performance with vllm
Updated 21 days ago1
ValueError: Unsupported model type phi3
Updated 21 days ago1
CUDA device rank in mii.pipeline
Updated 21 days ago
Configure server log level
Updated a month ago2
support Qwen
Closed a month ago
[QUERY] Expert Parallelism Supported?
Updated a month ago
Attempting to flush sequence N which does not exist
Updated a month ago
Compute perplexity
Updated a month ago
[BUG] Issue serving Mixtral 8x7B on H100
Updated a month ago9
Cannot run Yi-34B-Chat => ValueError: Unsupported q_ratio: 7
Updated 2 months ago2
few questions regarding the implementation of streaming and batching
Updated 2 months ago
support stream
Updated 2 months ago
support Qwen1.5
Updated 2 months ago
Support LLava next stronger
Updated 2 months ago
How can I use the same prompt to produce the same text output as vllm
Updated 2 months ago
Tf32 support
Updated 2 months ago
DeepSpeed-MII 能加载量化的int4或者int8的模型吗？
Updated 2 months ago
Does deepspeed-mii support prefix_allowed_tokens_fn?
Updated 3 months ago
qwen1.5 model Support?
Updated 3 months ago3
Is openai compatible server still working?
Updated 3 months ago1
[REQUEST] LLAMA-3 support
Updated 3 months ago
[REQUEST] Mixtral-8x22B support
Updated 3 months ago
How can i use this library with langchain or llama_index?
Updated 3 months ago2
BUG in run_batch_processing
Updated 3 months ago
[FEATURE] Access to logits and final hidden layer
Updated 3 months ago1
How do I launch the api on a graphics card other than cuda: 0
Updated 3 months ago1
Block when Call client inference in multiprocessing.Process
Updated 3 months ago3
error when using Qwen1.5-32B
Updated 3 months ago
[Problem]errno: 98 - Address already in use
Closed 3 months ago
Only running one replica even though setting many replicas
Updated 3 months ago
RuntimeError: The server socket has failed to listen on any local network address
Updated 3 months ago1
How is the prompt segmentation specifically implemented for Dynamic SplitFuse? Is there any code implement or code snippet ？
Updated 3 months ago
[FEATURE REQUEST] Add Support for Qwen1.5-MoE Architecture in DeepSpeed-MII
Updated 4 months ago1
inference_core_ops.so: undefined symbol: _Z19cuda_wf6af16_linearRN2at6TensorES1_S1_S1_S1_S1_iiii
Updated 4 months ago6
how can I use deepspeed to split the model to submit GPU?
Updated 4 months ago
Cohere's Command-R model support
Updated 4 months ago1
Add support for DBRX
Updated 4 months ago
Any plans for produnction-ready services?
Updated 4 months ago
Limit VRAM usage in serving the model
Updated 4 months ago2
I can't tell from documentation if we're meant to use a chat template or if it's automatically implemented?
Updated 4 months ago
On M3 Pro Macbook having issues with installation
Closed 4 months ago2
server crashed for some reason, unable to proceed
Updated 4 months ago1