Code Examples for the TensorFlow World talk on "Advanced model deployments with TensorFlow Serving"
This repository contains all code examples for my TensorFlow World talk about "Advanced model deployments with TensorFlow Serving"
If you have questions, please reach out via twitter @hanneshapke. You can also find further information on buildingmlpipelines.com.
Model Data Set
The example model was trained with the dataset for "IMDB movie reviews for Sentiment Analysis" which can be found here. In order to retrain the model, log in to Kaggle and obtain the dataset. It is assumed that the uncompressed file is saved in the same folder as train.py
file.
For copyright reasons, the data set isn't included in this repository.
Getting started
$ virtualenv -p python3 tf-world-tf-serving
$ source tf-world-tf-serving/bin/activate
$ pip3 install -r requirements.txt
Starting up TensorFlow Serving
90 sec Model Deployment
With the example below you can start your TensorFlow Serving instance on your host machine.
$ docker run -p 8500:8500 \
-p 8501:8501 \
--mount type=bind, source=saved_models/, target=/models/my_model \
-e MODEL_NAME=my_model
-t tensorflow/serving
Loading two Versions of the same Model
The example below lets you server multiple models. You'll need to update your configuration file before starting the server.
docker run -p 8501:8501 \
--mount type=bind,source=`pwd`,target=/models/my_model \
--mount type=bind,source=`pwd`/../../example_tf_serving_configurations,target=/models/model_config \
-t tensorflow/serving \
--model_config_file=/models/model_config/model_config_list.txt
Loading two Versions of the same Model with Version Labels
The example below lets you server multiple model version. You'll need to update your configuration file before starting the server.
docker run -p 8501:8501 \
--mount type=bind,source=`pwd`,target=/models/my_model \
--mount type=bind,source=`pwd`/../../example_tf_serving_configurations,target=/models/model_config \
-t tensorflow/serving \
--model_config_file=/models/model_config/model_config_list_with_labels.txt \
--allow_version_labels_for_unavailable_models=true
Run Prometheus Container
With the Prometheus config file below, you can run a basic instance of Prometheus.
global:
scrape_interval: 15s
evaluation_interval: 15s
external_labels:
monitor: 'tf-serving-monitor'
scrape_configs:
- job_name: 'prometheus'
scrape_interval: 5s
metrics_path: /monitoring/prometheus/metrics
static_configs:
- targets: ['host.docker.internal:8501']
docker run -p 9090:9090 -v /tmp/prometheus.yml:/etc/prometheus/prometheus.yml \
prom/prometheus
Once Prometheus is running, you can config TFServing to provide an monitoring endpoint for Prometheus.
docker run -p 8501:8501 \
--mount type=bind,source=`pwd`,target=/models/my_model \
--mount type=bind,source=`pwd`/../../example_tf_serving_configurations,target=/models/model_config \
-t tensorflow/serving \
--model_config_file=/models/model_config/model_config_list_with_labels.txt \
--monitoring_config_file=/models/model_config/monitoring_config_file.txt
Loading TF Lite Models
If you want to host TFLite models, use the command below to run your TFServing instance.
docker run -p 8501:8501 \
--mount type=bind,source=`pwd`/tflite,target=/models/my_model \
-e MODEL_BASE_PATH=/models \
-e MODEL_NAME=my_model \
-t tensorflow/serving:latest \
--use_tflite_model=true