Giters

huggingface / text-generation-inference

Large Language Model Text Generation Inference

Home Page:http://hf.co/docs/text-generation-inference

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

What is TGI's graceful shutdown behavior?

seongminp opened this issue · comments

commented

When SIGKILL arrives,

  • does TGI process all pending inputs?
  • does TGI blocks incoming inputs?

I saw a PR that adds graceful shutdown but it did not specify the exact program behavior.

commented
  • does TGI process all pending inputs?

Yes all inputs in the queue until a timeout which is currently set at 90 seconds.

  • does TGI blocks incoming inputs?

Yes