Application performs stream processing through video clips to identify Pikachu within them.
Building model
- Ensure TensorFlow is installed locally, follow reference 1 to install.
- Collect images of Pikachu. E.g. use images from Pikachu datasets project[2]
- Use tool like RectLabel [3] to do object detection of Pikachu among the collected images.
- Separate images to training and testing, and put into separate folder. E.g. dataset module resources/train_pikachu for training, resources/test_pikachu for testing.
- Then execute
python3 xml_to_csv.py
[4] to create CSV base on annotated XMLs. It's placed in dataset module resources folder, need to adjustimage_path
to correct folder. - Next, create TensorFlow TFRecord using
generate_tfrecord.py
[5] in dataset module resources folder. Need to provide correct arguments to specify CSV label file (csv_input), source folder (specified in code to raw/pikachu folder) and output file(output_path).
- Clone TensorFlow Models project.
- Install [TensorFlow Object Detection] (https://tensorflow-object-detection-api-tutorial.readthedocs.io/en/latest/install.html)
- Verify by:
# From within TensorFlow/models/research/
python object_detection/builders/model_builder_tf2_test.py
# You may need to upgrade opencv
pip install opencv-python --upgrade
python3 object_detection/model_main.py --logtostderr --model_dir=/Users/kai/Documents/Projects/MachineLearning/PikachuDetector/dataset/src/main/ssd_mobilenet_v2_training --pipeline_config_path=/Users/kai/Documents/Projects/MachineLearning/PikachuDetector/dataset/src/main/ssd_mobilenet_v2_pikachu.config --num_train_steps=2000 --sample_1_of_n_eval_examples=1
python3 object_detection/model_main.py --logtostderr --model_dir=/Users/kai/Documents/Projects/MachineLearning/PikachuDetector/dataset/src/main/ssd_inception_v3_training --pipeline_config_path=/Users/kai/Documents/Projects/MachineLearning/PikachuDetector/dataset/src/main/ssd_inception_v3_pikachu.config --num_train_steps=2000 --sample_1_of_n_eval_examples=1
Execute Tensorboard to see the progress
tensorboard --logdir <path to training folder>
Example result with SSD inception v3 after 2000 rounds Start 29.12 23 Finish 30.12 15 2000 steps, loss for final step is 5.444458.
Show saved model:
saved_model_cli show --dir <training folder path> --tag_set serve --signature_def serving_default`
https://github.com/tensorflow/models/blob/master/research/object_detection/export_inference_graph.py
tf_example
: Accepts a 1-D string tensor of shape [None] containing
serialized TFExample protos. Image resolutions are expected to be the same
if more than 1 image is provided.
The given SavedModel SignatureDef contains the following input(s):
inputs['serialized_example'] tensor_info:
dtype: DT_STRING
shape: ()
name: tf_example:0
The given SavedModel SignatureDef contains the following output(s):
outputs['detection_boxes'] tensor_info:
dtype: DT_FLOAT
shape: (1, 100, 4)
name: detection_boxes:0
outputs['detection_classes'] tensor_info:
dtype: DT_FLOAT
shape: (1, 100)
name: detection_classes:0
outputs['detection_multiclass_scores'] tensor_info:
dtype: DT_FLOAT
shape: (1, 100, 38)
name: detection_multiclass_scores:0
outputs['detection_scores'] tensor_info:
dtype: DT_FLOAT
shape: (1, 100)
name: detection_scores:0
outputs['num_detections'] tensor_info:
dtype: DT_FLOAT
shape: (1)
name: num_detections:0
outputs['raw_detection_boxes'] tensor_info:
dtype: DT_FLOAT
shape: (1, 5919, 4)
name: raw_detection_boxes:0
outputs['raw_detection_scores'] tensor_info:
dtype: DT_FLOAT
shape: (1, 5919, 38)
name: raw_detection_scores:0
Method name is: tensorflow/serving/predict
python3 object_detection/export_inference_graph.py --input_type image_tensor --pipeline_config_path =/Users/kai/Documents/Projects/MachineLearning/PikachuDetector/dataset/src/main/ssd_inception_v3_training --pipeline_config_path=/Users/kai/Documents/Projects/MachineLearning/PikachuDetector/dataset/src/main/ssd_inception_v3_pikachu.config --trained_checkpoint_prefix /Users/kai/Documents/Projects/MachineLearning/PikachuDetector/dataset/src/main/ssd_inception_v3_training/model.ckpt-2000 --output_directory /Users/kai/Documents/Projects/MachineLearning/PikachuDetector/dataset/src/main/ssd_inception_v3_image_tensor
image_tensor
: Accepts a uint8 4-D tensor of shape [None, None, None, 3]
Show saved model:
saved_model_cli show --dir <training folder path> --tag_set serve --signature_def serving_default`
The given SavedModel SignatureDef contains the following input(s):
inputs['inputs'] tensor_info:
dtype: DT_UINT8
shape: (-1, -1, -1, 3)
name: image_tensor:0
The given SavedModel SignatureDef contains the following output(s):
outputs['detection_boxes'] tensor_info:
dtype: DT_FLOAT
shape: (-1, 100, 4)
name: detection_boxes:0
outputs['detection_classes'] tensor_info:
dtype: DT_FLOAT
shape: (-1, 100)
name: detection_classes:0
outputs['detection_multiclass_scores'] tensor_info:
dtype: DT_FLOAT
shape: (-1, 100, 38)
name: detection_multiclass_scores:0
outputs['detection_scores'] tensor_info:
dtype: DT_FLOAT
shape: (-1, 100)
name: detection_scores:0
outputs['num_detections'] tensor_info:
dtype: DT_FLOAT
shape: (-1)
name: num_detections:0
outputs['raw_detection_boxes'] tensor_info:
dtype: DT_FLOAT
shape: (-1, -1, 4)
name: raw_detection_boxes:0
outputs['raw_detection_scores'] tensor_info:
dtype: DT_FLOAT
shape: (-1, -1, 38)
name: raw_detection_scores:0
Method name is: tensorflow/serving/predict
python3 object_detection/export_inference_graph.py --input_type encoded_image_string_tensor --pipeline_config_path=/Users/kai/Documents/Projects/MachineLearning/PikachuDetector/dataset/src/main/ssd_inception_v3_pikachu.config --trained_checkpoint_prefix /Users/kai/Documents/Projects/MachineLearning/PikachuDetector/dataset/src/main/ssd_inception_v3_training/model.ckpt-2000 --output_directory /Users/kai/Documents/Projects/MachineLearning/PikachuDetector/dataset/src/main/ssd_inception_v3_encoded_image_input_2
encoded_image_string_tensor
: Accepts a 1-D string tensor of shape [None]
containing encoded PNG or JPEG images. Image resolutions are expected to be
the same if more than 1 image is provided.
Show saved model:
saved_model_cli show --dir . --tag_set serve --signature_def serving_default
The given SavedModel SignatureDef contains the following input(s):
inputs['inputs'] tensor_info:
dtype: DT_STRING
shape: (-1)
name: encoded_image_string_tensor:0
The given SavedModel SignatureDef contains the following output(s):
outputs['detection_boxes'] tensor_info:
dtype: DT_FLOAT
shape: (-1, 100, 4)
name: detection_boxes:0
outputs['detection_classes'] tensor_info:
dtype: DT_FLOAT
shape: (-1, 100)
name: detection_classes:0
outputs['detection_multiclass_scores'] tensor_info:
dtype: DT_FLOAT
shape: (-1, 100, 38)
name: detection_multiclass_scores:0
outputs['detection_scores'] tensor_info:
dtype: DT_FLOAT
shape: (-1, 100)
name: detection_scores:0
outputs['num_detections'] tensor_info:
dtype: DT_FLOAT
shape: (-1)
name: num_detections:0
outputs['raw_detection_boxes'] tensor_info:
dtype: DT_FLOAT
shape: (-1, -1, 4)
name: raw_detection_boxes:0
outputs['raw_detection_scores'] tensor_info:
dtype: DT_FLOAT
shape: (-1, -1, 38)
name: raw_detection_scores:0
Method name is: tensorflow/serving/predict
- Path to the input video file.
E.g.
/Users/kai/Documents/Projects/MachineLearning/PikachuDetector/flink-processing/src/main/resources/pokemon_detective_pikachu.mp4
- Path to the output video file. Folders involved must be existing.
E.g.
/Users/kai/Documents/Projects/MachineLearning/PikachuDetector/flink-processing/src/main/resources/output.mp4
- Path to TensorFlow model file.
E.g.
/Users/kai/Documents/Projects/MachineLearning/PikachuDetector/flink-processing/src/main/resources/ssd_mobilenet_v2_encoded_image_input/saved_model
The example video of pokemon_detective_pikachu.mp4
has:
- time length: 79s
- Frame rate: 24 fps
- width: 640
- height: 360
- pixel format: PIX_FMT_YUV420P
- codec: H264 Decoded to 1897 images. Encode need to wait roughly 42ms in between images.
Execute PikachuDetectJobExecutor
main with parameter like
<path to input movie folder>/pokemon_detective_pikachu.mp4 <path to movie output folder>/output.mp4 <path to saved model>/ssd_mobilenet_v2_encoded_image_input/saved_model
In case you wanna use Anaconda, at the time when tried by author (12.2021), tensorflow-io is not in its repo, thus end up with error when use TensorFlow from there. To deactivate conda environment when open new terminal use:
deactivate
conda config --set auto_activate_base false
The movie pokemon_detective_pikachu.mp4
mentioned in the document is not uploaded in the repo.
It's the preview video of Detective Pikachu.
[1] Install tensorflow locally, follow https://www.tensorflow.org/install/.
Docker install
docker pull tensorflow/tensorflow:1.15.0
docker run -it -p 8888:8888 tensorflow/tensorflow:latest-jupyter # Start Jupyter server
[2] Pikachu datasets
[3] RectLabel
[4] xml_to_csv.py
[6] Train a Hand Detector using Tensorflow 2 Object Detection API in 2021
[7] LabelImg