NOTE: You are currently on the master branch which tracks under-development progress towards the next release. The latest release of the TensorRT Inference Server is 0.8.0 beta and is available on branch r18.11.
The NVIDIA TensorRT Inference Server provides a cloud inferencing solution optimized for NVIDIA GPUs. The server provides an inference service via an HTTP or gRPC endpoint, allowing remote clients to request inferencing for any model being managed by the server. The NVIDIA Inference Server provides the following features:
- Multiple model support. The server can manage any number and mix of models (limited by system disk and memory resources). Supports TensorRT, TensorFlow GraphDef, TensorFlow SavedModel and Caffe2 NetDef model formats. Also supports TensorFlow-TensorRT integrated models.
- Multi-GPU support. The server can distribute inferencing across all system GPUs.
- Multi-tenancy support. Multiple models (or multiple instances of the same model) can run simultaneously on the same GPU.
- Batching support. For models that support batching, the server can accept requests for a batch of inputs and respond with the corresponding batch of outputs. The server also supports dynamic batching where individual inference requests are dynamically combined together to improve inference throughput. Dynamic batching is transparent to the client requesting inference.
- Model repositories may reside on a locally accessible file system or in Google Cloud Storage.
- Readiness and liveness health endpoints suitable for Kubernetes-style orchestration.
- Metrics indicating GPU utiliization, server throughput, and server latency.
The current release of the TensorRT Inference Server is 0.8.0 beta and corresponds to the 18.11 release of the tensorrtserver container on NVIDIA GPU Cloud (NGC). The branch for this release is r18.11. The User Guide, Developer Guide, and API Reference documentation provide guidance on installing, building and running the inference server.
You can also view the documentation for the master branch and for earlier releases.
Contributions to TensorRT Inference Server are more than welcome. To contribute make a pull request and follow the guidelines outlined in the Contributing document.
We appreciate any feedback, questions or bug reporting regarding this project. When help with code is needed, follow the process outlined in the Stack Overflow (https://stackoverflow.com/help/mcve) document. Ensure posted examples are:
- minimal – use as little code as possible that still produces the same problem
- complete – provide all parts needed to reproduce the problem. Check if you can strip external dependency and still show the problem. The less time we spend on reproducing problems the more time we have to fix it
- verifiable – test the code you're about to provide to make sure it reproduces the problem. Remove all other problems that are not related to your request/question.