ETA for TensorFlow Runtime Integration?
jeongukjae opened this issue · comments
Feature Request
May I know the expected release date of TensorFlow Serving with TensorFlow Runtime?
For the background, recently, I found that tensorflow runtime is integrated into tensorflow/serving codes. (2.15.0-rc0 release)
Line 42 in 0bfaf6a
I'm really interested in this feature, so I tried to patch platform_config_utils.cc & server_init.cc, I could successfully build tensorflow serving with TFRT and inference arbitrary saved models (usually in CPU, transformer-based models).
Following the blog posts and GCP Vertex AI's guide, I expected that TFRT could show better performance.
- https://blog.tensorflow.org/2020/04/tfrt-new-tensorflow-runtime.html
- https://blog.tensorflow.org/2022/02/tfrt-progress-update.html
- https://cloud.google.com/vertex-ai/docs/predictions/optimized-tensorflow-runtime
But I think maybe due to plenty of tuning points, I couldn't get the great performance til now.
So it would be great if I could know when TF Serving with TFRT is released.