The training project which includes MO and IE for OpenVINO unsupported layer - cosh
.
-
Create
tf_cosh.py
to generate the model and save ascosh.pb
import tensorflow as tf import tensorflow.contrib.layers as layers import numpy as np weights = { 'wc1': tf.Variable(tf.truncated_normal([1, 1, 1, 32])) } biases = { 'bc1': tf.Variable(tf.zeros([32])) } def model (inputs) : conv1 = tf.nn.conv2d(inputs, weights['wc1'], strides=[1, 1, 1, 1], padding='SAME') + biases['bc1'] out = tf.math.cosh(conv1) return out x = tf.placeholder(tf.float32, (None, 32, 32, 1)) o = model(x) sess = tf.Session() sess.run(tf.initialize_all_variables()) print (sess.run(o, {x : np.ones((1, 32, 32, 1))})) from tensorflow.python.framework import graph_io frozen = tf.graph_util.convert_variables_to_constants(sess, sess.graph_def, ['Cosh']) graph_io.write_graph(frozen, '.', 'cosh.pb', as_text=False)
-
Run
tf_cosh.py
$ python tf_cosh.py
-
Convert
cosh.pb
to IR-
Run:
$ python $MO_ROOT/mo.py \ --input_model=$ROOT_OF_TF_MODEL/cosh.pb \ --input_shape=[1,32,32,1] \ --disable_nhwc_to_nchw \ -o $OUTPUT_DIR
-
Output:
[ ERROR ] List of operations that cannot be converted to Inference Engine IR: [ ERROR ] Cosh (1) [ ERROR ] Cosh [ ERROR ] Part of the nodes was not converted to IR. Stopped. For more information please refer to Model Optimizer FAQ (https://docs.openvinotoolkit.org/ latest/_docs_MO_DG_prepare_model_Model_Optimizer_FAQ.html), question #24.
Got the errors since the
Cosh
layer was not supported by model optimizer.
-
-
Use existing operation to construct a new operation - Add
Cosh
operator intoactivation_ops.py
.-
Add
Cosh
operator into listactivation_ops
-
From:
activation_ops = ['Sigmoid', 'Tanh', 'ReLU6', 'Exp', 'Elu', 'Not', 'Floor']
-
To:
activation_ops = ['Sigmoid', 'Tanh', 'ReLU6', 'Exp', 'Elu', 'Not', 'Floor', 'Cosh']
-
-
Create a class for
Cosh
operator which inherit from classActivation
class Cosh(Activation): op = 'Cosh' operation = staticmethod(lambda x: np.cosh(x))
-
Create the file
cosh_ext.py
in the directory$OPENVINO_ROOT/deployment_tools/model_optimizer/extensions/front/tf to add the layer extractor for unsupported layer
-Cosh
from extensions.ops.activation_ops import Cosh from mo.front.extractor import FrontExtractorOp class LeakyReLUFrontExtractor(FrontExtractorOp): op = 'Cosh' enabled = True @staticmethod def extract(node): Cosh.update_node_stat(node) return __class__.enabled
-
Convert the model again
-
Run:
$ python $MO_ROOT/mo.py \ --input_model=$ROOT_OF_TF_MODEL/cosh.pb \ --input_shape=[1,32,32,1] \ --disable_nhwc_to_nchw \ -o $OUTPUT_DIR
-
Output:
[ SUCCESS ] Generated IR model. [ SUCCESS ] XML file: $OUTPUT_DIR/cosh.xml [ SUCCESS ] BIN file: $OUTPUT_DIR/cosh.bin [ SUCCESS ] Total execution time: 1.32 seconds.
-
-
-
Construct a new operation
-
Create a templates for
Cosh
custom layer-
Run
extgen.py
with parameter--mo-tf-ext
,--mo-op
,--ie-cpu-ext
, and--ie-gpu-ext
.python $OPENVINO_ROOT/deployment_tools/tools/extension_generator/extgen.py new \ --mo-tf-ext \ --mo-op \ --ie-cpu-ext \ --ie-gpu-ext \ --output_dir=${OUTPUT_DIR}
-
Information based on execution commands
Generating: Model Optimizer: Extractor for Caffe Custom Layer: No Extractor for MxNet Custom Layer: No Extractor for TensorFlow Custom Layer: Yes Framework-agnostic operation extension: Yes Inference Engine: CPU extension: Yes GPU extension: Yes
-
Answer the questions
Enter layer name: Cosh Do you want to automatically parse all parameters from the model file? (y/n) Yes means layer parameters will be automatically parsed during Model Optimizer work as is. No means you will be prompted for layer parameters in the following section n Enter all parameters in the following format: <param1> <new name1> <type1> <param2> <new name2> <type2> ... Where type is one of the following types: b - Bool, padding - Padding type, list.b - List of bools, f - Float, batch - Get batch from dataFormat, list.f - List of floats, i - Int, channel - Get channel from dataFormat, list.i - List of ints, s - String, spatial - Get spatial from dataFormat, list.s - List of strings, shape - TensorShapeProto, list.shape - List of TensorShapeProto, type - DataType, list.type - List of DataType, Example: length attr_length i If your attribute type is not shown in the list above, or you want to implement your own attribute parsing, omit the <type> parameter. Enter 'q' when finished: q ********************************************************************************************** Check your answers for TensorFlow* extractor generation: 1. Layer name: Cosh 2. Automatically parse all parameters from model file: No 3. Parameters entered: <param1> <new name1> <type1> [] ********************************************************************************************** Do you want to change any answer (y/n) ? Default 'no' n Do you want to use the layer name as the operation name? (y/n) y Does your operation change shape? (y/n) n ********************************************************************************************** Check your answers for the Model Optimizer operation generation: 4. Use layer name as operation name? (y/n) Yes 5. Operation changes shape? (y/n) No ********************************************************************************************** Do you want to change any answer (y/n) ? Default 'no' n
-
Generate files
Stub file for TensorFlow Model Optimizer extractor is in ${OUTPUT_DIR}/user_mo_extensions/front/tf folder Stub file for the Model Optimizer operation is in ${OUTPUT_DIR}/user_mo_extensions/ops folder Stub files for the Inference Engine CPU extension are in ${OUTPUT_DIR}/user_ie_extensions/cpu folder Stub files for the Inference Engine GPU extension are in ${OUTPUT_DIR}/user_ie_extensions/gpu folder
-
Tree structure of generate files
├── user_ie_extensions │ ├── cpu │ │ ├── CMakeLists.txt │ │ ├── ext_base.cpp │ │ ├── ext_base.hpp │ │ ├── ext_cosh.cpp │ │ ├── ext_list.cpp │ │ └── ext_list.hpp │ └── gpu │ ├── cosh_kernel.cl │ └── cosh_kernel.xml └── user_mo_extensions ├── front │ ├── caffe │ │ └── __init__.py │ ├── __init__.py │ ├── mxnet │ │ └── __init__.py │ └── tf │ ├── cosh_ext.py │ └── __init__.py ├── __init__.py └── ops ├── cosh.py └── __init__.py
-
-
Construct a new operation -
Cosh
-
Patch the file
cosh.py
and move to directory$OPENVINO_ROOT/deployment_tools/model_optimizer/extensions/ops
from mo.ops.op import Op from mo.front.common.partial_infer.elemental import copy_shape_infer from mo.graph.graph import Node class CoshOp(Op): op = 'Cosh' def __init__(self, graph, attrs): mandatory_props = dict( type=__class__.op, op=__class__.op, infer=CoshOp.infer ) super().__init__(graph, mandatory_props, attrs) @staticmethod def infer(node: Node): return copy_shape_infer(node)
-
Cosh Extractor (Choose one method)
-
Move the file
cosh_ext.py
to directory$OPENVINO_ROOT/deployment_tools/model_optimizer/extensions/front/tf
import numpy as np from mo.front.extractor import FrontExtractorOp from mo.ops.op import Op from mo.front.tf.extractors.utils import * from mo.front.common.partial_infer.utils import convert_tf_padding_to_str class CoshFrontExtractor(FrontExtractorOp): op = 'Cosh' enabled = True @staticmethod def extract(node): proto_layer = node.pb param = proto_layer.attr # extracting parameters from TensorFlow layer and prepare them for IR attrs = { 'op': __class__.op } # update the attributes of the node Op.get_op_class_by_name(__class__.op).update_node_stat(node, attrs) return __class__.enabled
-
Patch the file
cosh_ext.py
and move to directory$OPENVINO_ROOT/deployment_tools/model_optimizer/extensions/front/tf
from mo.front.extractor import FrontExtractorOp from extensions.ops.cosh_tf import CoshOp class CoshFrontExtractor(FrontExtractorOp): op = 'Cosh' enabled = True @staticmethod def extract(node): # update the attributes of the node CoshOp.update_node_stat(node) return __class__.enabled
-
-
Convert the model again
-
Run:
$ python $MO_ROOT/mo.py \ --input_model=$ROOT_OF_TF_MODEL/cosh.pb \ --input_shape=[1,32,32,1] \ --disable_nhwc_to_nchw \ -o $OUTPUT_DIR
-
Output:
[ SUCCESS ] Generated IR model. [ SUCCESS ] XML file: $OUTPUT_DIR/cosh.xml [ SUCCESS ] BIN file: $OUTPUT_DIR/cosh.bin [ SUCCESS ] Total execution time: 1.32 seconds.
-
-
-
-
Create
infer.py
for inferenceimport sys, os, time import numpy as np import argparse import time from openvino.inference_engine import IENetwork, IECore # @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ def parsing(): parser = argparse.ArgumentParser(add_help = False) parser.add_argument('-m', '--model', default='', type=str) parser.add_argument('-d', '--device', default='CPU', type=str) parser.add_argument('-c', '--gpu_extension', default='', type=str) parser.add_argument('-l', '--cpu_extension', default='', type=str) return parser def main() : args = parsing().parse_args() model_graph = args.model model_weight = args.model[:-3] + 'bin' net = IENetwork(model = model_graph, weights = model_weight) iter_inputs = iter(net.inputs) iter_outputs = iter(net.outputs) inputs_num = len(net.inputs) print (inputs_num) input_blob = [] for _inputs in iter_inputs: input_blob.append(_inputs) output_blob = [] for _outputs in iter_outputs: output_blob.append(_outputs) input_l = [] for i in input_blob: input_l.append(np.ones(shape=net.inputs[i].shape, dtype=np.float32)) inputs = dict() for i in range (inputs_num): inputs[input_blob[i]] = input_l[i] plugin = IECore() if args.cpu_extension and 'CPU' in args.device: plugin.add_extension(args.cpu_extension, "CPU") if args.gpu_extension and 'GPU' in args.device: plugin.set_config({"CONFIG_FILE": args.gpu_extension}, "GPU") exec_net = plugin.load_network(network = net, device_name = args.device) s_time = time.time() out = exec_net.infer(inputs) e_time = time.time() print (out) print ('execution time: ', e_time - s_time) if "__main__" : main()
-
Run inference on CPU
-
Execute
infer.py
with parameter–m
,–d
and–l
.-
Run:
-
Use existing layer to construct an operation
$ python infer.py \ -m $USING_EXISTING_LAYER/cosh.xml \ -l $SAMPLE_BUILD_DIR/intel64/Release/lib/libcpu_extension.so \ -d CPU
-
Construct a new operation
$ python infer.py \ -m $AS_NEW/cosh.xml \ -l $SAMPLE_BUILD_DIR/intel64/Release/lib/libcpu_extension.so \ -d CPU
-
-
Output:
-
Use existing layer to construct an operation
RuntimeError: Unsupported primitive of type: cosh name: Cosh
Since the math extension is supported
Cosh
layer notcosh
layer. Solution: Add CPU extension for IE -
Construct a new operation
Run the model well
Since the math extension is supported ‘Cosh’ layer
-
-
-
Add CPU extension for IE
-
Create the file
ext_cosh.cpp
in the directory$OPENVINO_ROOT/deployment_tools/inference_engine/src/extensio or patch the file
ext_cosh.cppwhich under
$GEN_OUTPUT/user_ie_extensions/cpu/`#include "ext_list.hpp" #include "ext_base.hpp" #include <algorithm> #include <string> #include <vector> #include <cmath> #include <utility> #include <functional> namespace InferenceEngine { namespace Extensions { namespace Cpu { class coshImpl: public ExtLayerBase { public: explicit coshImpl(const CNNLayer* layer) { try { if (layer->insData.size() != 1 || layer->outData.empty()) THROW_IE_EXCEPTION << "Incorrect number of input/output edges!"; addConfig(layer, {DataConfigurator(ConfLayout::PLN)}, {DataConfigurator(ConfLayout::PLN)}); } catch (InferenceEngine::details::InferenceEngineException &ex) { errorMsg = ex.what(); } } StatusCode execute(std::vector<Blob::Ptr>& inputs, std::vector<Blob::Ptr>& outputs, ResponseDesc *resp) noexcept override { SizeVector in_dims = inputs[0]->getTensorDesc().getDims(); SizeVector out_dims = outputs[0]->getTensorDesc().getDims(); float* src_data = inputs[0]->buffer(); float* dst_data = outputs[0]->buffer(); int _dsize = 1; for (size_t i = 0; i < in_dims.size(); i++) _dsize *= in_dims[i]; for (int i = 0; i < _dsize; i++) { double __exp = exp(src_data[i]); dst_data[i] = (__exp + (1 / __exp)) / 2; } return OK; } private: }; REG_FACTORY_FOR(ImplFactory<coshImpl>, cosh); } // namespace Cpu } // namespace Extensions } // namespace InferenceEngine
-
-
Build extension file
-
Create file by yourself
$ ./$OPENVINO_ROOT/deployment_tools/inference_engine/demos/build_demos.sh
-
Generate by
extgen.py
$ cd ${OUTPUT_DIR}/user_ie_extensions/cpu $ mkdir build && cd build $ cmake .. && make -j8
-
-
Execute
infer.py
with parameter–m
,–d
and–l
again.-
Run:
-
Construct a new operation
$ python infer.py \ -m $AS_NEW/cosh.xml \ -l $SAMPLE_BUILD_DIR/intel64/Release/lib/libcpu_extension.so \ -d CPU
-
Generate by
extgen.py
$ python infer.py \ -m $AS_NEW/cosh.xml \ -l ${GEN_OUTPUT}/user_ie_extensions/cpu/build/intel64/Release/lib/libcpu_extension.so \ -d CPU
-
-
-
-
Run inference on GPU
-
Execute
infer.py
with parameter–m
and–d
.-
Run:
-
Use existing layer to construct an operation
$ python infer.py \ -m $USING_EXISTING_LAYER/cosh.xml \ -d GPU
-
Construct a new operation
$ python infer.py \ -m $AS_NEW/cosh.xml \ -d GPU
-
-
Output:
-
Use existing layer to construct an operation
File "ie_api.pyx", line 85, in openvino.inference_engine.ie_api.IECore.load_network File "ie_api.pyx", line 92, in openvino.inference_engine.ie_api.IECore.load_network RuntimeError: Unknown Layer Type: cosh
-
Construct a new operation
File "ie_api.pyx", line 85, in openvino.inference_engine.ie_api.IECore.load_network File "ie_api.pyx", line 92, in openvino.inference_engine.ie_api.IECore.load_network RuntimeError: Unknown Layer Type: Cosh
Since the GPU didn't support
Cosh
andcosh
layer. Solution: Add GPU extension for IE
-
-
-
Add GPU extension for IE
Here provide 2 method to enable GPU extension for IE.
-
Use existing layer to construct an operation
-
Patch
cosh_kernel.cl
which under$GEN_OUTPUT/user_ie_extensions/gpu/
#pragma OPENCL EXTENSION cl_khr_fp16 : enable __kernel void cosh_kernel( // Insert pointers to inputs, outputs as arguments here // If your layer has one input and one output, arguments will be: const __global INPUT0_TYPE* input0, __global OUTPUT0_TYPE* output ) { // Add the kernel implementation here: // Get data dimensions const uint T_ = INPUT0_DIMS[0]; const uint N_ = INPUT0_DIMS[1]; const uint X_ = INPUT0_DIMS[2]; const uint Y_ = INPUT0_DIMS[3]; // Perform the hyperbolic cosine given by: // cosh(x) = (e^x + e^-x)/2 for (int ii = 0; ii < T_*N_*X_*Y_; ii++) { output[ii] = (exp(input0[ii]) + exp(-input0[ii]))/2; } }
-
Patch
cosh_kernel.xml
which under$GEN_OUTPUT/user_ie_extensions/gpu/
<CustomLayer name="cosh" type="SimpleGPU" version="1"> <Kernel entry="cosh_kernel"> <Source filename="cosh_kernel.cl"/> <!-- Parameters description /--> </Kernel> <!-- Buffer descriptions /--> <Buffers> <Tensor arg-index="0" type="input" port-index="0" format="BFYX"/> <Tensor arg-index="1" type="output" port-index="1" format="BFYX"/> </Buffers> <CompilerOptions options="-cl-mad-enable"/> <!-- define the global worksize. The formulas can use the values of the B,F,Y,X dimensions and contain the operators: +,-,/,*,% (all evaluated in integer arithmetics). Default value: global="B*F*Y*X,1,1"/--> <WorkSizes global="B,F"/> </CustomLayer>
-
-
Construct a new operation
-
Create
Cosh_kernel.cl
which under$GEN_OUTPUT/user_ie_extensions/gpu/
#pragma OPENCL EXTENSION cl_khr_fp16 : enable __kernel void Cosh(const __global INPUT0_TYPE* input0, __global OUTPUT0_TYPE* output) { // global index definition set in the XML configuration file const uint idx = get_global_id(0); const uint idy = get_global_id(1); const uint idbf = get_global_id(2); const uint feature = idbf%OUTPUT0_DIMS[1]; const uint batch = idbf/OUTPUT0_DIMS[1]; const uint in_id = batch*INPUT0_PITCHES[0] + feature*INPUT0_PITCHES[1] + idy*INPUT0_PITCHES[2] + idx*INPUT0_PITCHES[3] + INPUT0_OFFSET; const uint out_id = batch*OUTPUT0_PITCHES[0] + feature*OUTPUT0_PITCHES[1] + idy*OUTPUT0_PITCHES[2] + idx*OUTPUT0_PITCHES[3] + OUTPUT0_OFFSET; INPUT0_TYPE value = input0[in_id]; output[out_id] = (exp(value) + exp(-value))/2; }
-
Create
Cosh_kernel.xml
which under$GEN_OUTPUT/user_ie_extensions/gpu/
<CustomLayer name="Cosh" type="SimpleGPU" version="1"> <Kernel entry="Cosh"> <Source filename="Cosh_kernel.cl"/> </Kernel> <Buffers> <Tensor arg-index="0" type="input" port-index="0" format="BFYX"/> <Tensor arg-index="1" type="output" port-index="0" format="BFYX"/> </Buffers> <CompilerOptions options="-cl-mad-enable"/> <WorkSizes global="X,Y,B*F"/> </CustomLayer>
-
-
-
Execute
infer.py
with parameter–m
,–d
and–c
.-
Run:
-
Construct a new operation
python infer.py \ -m $USING_EXISTING_LAYER/cosh.xml \ -c $GEN_OUTPUT/user_ie_extensions/gpu//cosh_kernel.xml \ -d GPU
-
Generate by
extgen.py
python infer.py \ -m $AS_NEW/cosh.xml \ -c $GEN_OUTPUT/user_ie_extensions/gpu//Cosh_kernel.xml \ -d GPU
-
-
-