[tflite] Support Tensorflow Lite Selected Ops in nntrainer

Question

[tflite] Support Tensorflow Lite Selected Ops in nntrainer

DonghakPark opened this issue a year ago · comments

Donghak PARK commented a year ago

we will update tensorflow lite package for support tensorflow lite selected ops

below is temp solution for using them

Build Tensorflow lite .so file with selected ops

Requirements
gcc, g++, glibc

Download Tensorflow, Flatbuffer

$ wget https://github.com/tensorflow/tensorflow/archive/refs/tags/v2.11.0.tar.gz 
$ wget https://github.com/google/flatbuffers/archive/refs/tags/v22.11.23.tar.gz

unzip tar.gz files

$ tar -zxvf {filename}

Add this dependency to /tensorflow/lite/BUILD → tflite_cc_shared_object

→(remove this line) "//tensorflow/lite/kernels:builtin_ops_all_linked",
"//tensorflow/lite/kernels:builtin_ops",
"//tensorflow/lite/delegates/flex:delegate",
"//tensorflow/lite/delegates/flex:exported_symbols.lds",
"//tensorflow/lite/delegates/flex:version_script.lds",

Download dependency

$ sudo apt install python3-dev python3-pip
$ pip install pip numpy wheel packaging requests opt_einsum
$ pip install keras_preprocessing --no-deps
$ sudo apt-get install gcc-aarch64-linux-gnu

install bazelisk

$ sudo apt install npm
$ sudo npm install -g @bazel/bazelisk 
$ bazelisk

for x86_64

$ bazelisk build --config=monolithic --config=noaws --config=nogcp --config=nohdfs --config=nonccl  --fat_apk_cpu=x86,x86_64  --experimental_ui_max_stdouterr_bytes=1073741819 -c opt //tensorflow/lite:libtensorflowlite.so

for aarch64

You need to hide the system's openssl library. For this go to usr/include folder and make sudo mv openssl openssl.original.

$ bazelisk build --config=monolithic --config=noaws --config=nogcp --config=nohdfs --config=nonccl --config=elinux_aarch64 --experimental_ui_max_stdouterr_bytes=1073741819 -c opt //tensorflow/lite:libtensorflowlite.so

Apply to local environments

update "/usr/lib/pkgconfig/tensorflow2-lite.pc" file -- update like below

    Name: tensorflow lite
    Description: tensorflow lite static library
    Version: 2.11.0
    Libs: -L/usr/lib -ltensorflowlite
    Cflags: -I/usr/include/tensorflowlite

move libtensorflowlite.so file to /usr/lib
make dir in /usr/include as name "tensorflowlite"
in /usr/include/tensorflowlite put flatbuffer and tensorflow (it come from github)
in nntrainer meson build -> ninja install again
in your app (you can run)

summary : just change pkgconfig file and make them to use new .so file we can run Simpleshot nntrainer Application

I test this method on my local PC (Ubuntu 22.04, x86_64)

taos-ci · Answer 1 · Fri Apr 21 2023 17:01:52 GMT+0800 (China Standard Time)

cibot: Thank you for posting issue #2193. The person in charge will reply soon.

KirillP2323 · Answer 2 · Fri Apr 21 2023 20:02:47 GMT+0800 (China Standard Time)

@DonghakPark Thanks for the instructions! I'm following the instructions on Ubuntu 20.04 machine. I rebuild the libtensorflowlite.so with bazelisk. However, I encounter an error while doing ninja install:

$ ninja -C build install
ninja: Entering directory `build'
[254/443] Linking target nntrainer/libnntrainer.so.
FAILED: nntrainer/libnntrainer.so 
c++  -o nntrainer/libnntrainer.so 'nntrainer/d48ed23@@nntrainer@sha/ini_interpreter.cpp.o' 'nntrainer/d48ed23@@nntrainer@sha/activation_realizer.cpp.o' 'nntrainer/d48ed23@@nntrainer@sha/flatten_realizer.cpp.o' 'nntrainer/d48ed23@@nntrainer@sha/recurrent_realizer.cpp.o' 'nntrainer/d48ed23@@nntrainer@sha/remap_realizer.cpp.o' 'nntrainer/d48ed23@@nntrainer@sha/slice_realizer.cpp.o' 'nntrainer/d48ed23@@nntrainer@sha/input_realizer.cpp.o' 'nntrainer/d48ed23@@nntrainer@sha/previous_input_realizer.cpp.o' 'nntrainer/d48ed23@@nntrainer@sha/multiout_realizer.cpp.o' 'nntrainer/d48ed23@@nntrainer@sha/bn_realizer.cpp.o' 'nntrainer/d48ed23@@nntrainer@sha/loss_realizer.cpp.o' 'nntrainer/d48ed23@@nntrainer@sha/tflite_interpreter.cpp.o' 'nntrainer/d48ed23@@nntrainer@sha/tflite_opnode.cpp.o' 'nntrainer/d48ed23@@nntrainer@sha/flatbuffer_interpreter.cpp.o' 'nntrainer/d48ed23@@nntrainer@sha/iteration_queue.cpp.o' 'nntrainer/d48ed23@@nntrainer@sha/databuffer.cpp.o' 'nntrainer/d48ed23@@nntrainer@sha/data_iteration.cpp.o' 'nntrainer/d48ed23@@nntrainer@sha/databuffer_factory.cpp.o' 'nntrainer/d48ed23@@nntrainer@sha/random_data_producers.cpp.o' 'nntrainer/d48ed23@@nntrainer@sha/func_data_producer.cpp.o' 'nntrainer/d48ed23@@nntrainer@sha/raw_file_data_producer.cpp.o' 'nntrainer/d48ed23@@nntrainer@sha/dir_data_producers.cpp.o' 'nntrainer/d48ed23@@nntrainer@sha/loss_layer.cpp.o' 'nntrainer/d48ed23@@nntrainer@sha/kld_loss_layer.cpp.o' 'nntrainer/d48ed23@@nntrainer@sha/mse_loss_layer.cpp.o' 'nntrainer/d48ed23@@nntrainer@sha/cross_entropy_sigmoid_loss_layer.cpp.o' 'nntrainer/d48ed23@@nntrainer@sha/cross_entropy_softmax_loss_layer.cpp.o' 'nntrainer/d48ed23@@nntrainer@sha/constant_derivative_loss_layer.cpp.o' 'nntrainer/d48ed23@@nntrainer@sha/activation_layer.cpp.o' 'nntrainer/d48ed23@@nntrainer@sha/addition_layer.cpp.o' 'nntrainer/d48ed23@@nntrainer@sha/attention_layer.cpp.o' 'nntrainer/d48ed23@@nntrainer@sha/mol_attention_layer.cpp.o' 'nntrainer/d48ed23@@nntrainer@sha/multi_head_attention_layer.cpp.o' 'nntrainer/d48ed23@@nntrainer@sha/concat_layer.cpp.o' 'nntrainer/d48ed23@@nntrainer@sha/bn_layer.cpp.o' 'nntrainer/d48ed23@@nntrainer@sha/layer_normalization_layer.cpp.o' 'nntrainer/d48ed23@@nntrainer@sha/conv2d_layer.cpp.o' 'nntrainer/d48ed23@@nntrainer@sha/conv1d_layer.cpp.o' 'nntrainer/d48ed23@@nntrainer@sha/fc_layer.cpp.o' 'nntrainer/d48ed23@@nntrainer@sha/flatten_layer.cpp.o' 'nntrainer/d48ed23@@nntrainer@sha/input_layer.cpp.o' 'nntrainer/d48ed23@@nntrainer@sha/multiout_layer.cpp.o' 'nntrainer/d48ed23@@nntrainer@sha/layer_node.cpp.o' 'nntrainer/d48ed23@@nntrainer@sha/pooling2d_layer.cpp.o' 'nntrainer/d48ed23@@nntrainer@sha/preprocess_flip_layer.cpp.o' 'nntrainer/d48ed23@@nntrainer@sha/preprocess_translate_layer.cpp.o' 'nntrainer/d48ed23@@nntrainer@sha/preprocess_l2norm_layer.cpp.o' 'nntrainer/d48ed23@@nntrainer@sha/embedding.cpp.o' 'nntrainer/d48ed23@@nntrainer@sha/rnn.cpp.o' 'nntrainer/d48ed23@@nntrainer@sha/rnncell.cpp.o' 'nntrainer/d48ed23@@nntrainer@sha/acti_func.cpp.o' 'nntrainer/d48ed23@@nntrainer@sha/lstm.cpp.o' 'nntrainer/d48ed23@@nntrainer@sha/lstmcell.cpp.o' 'nntrainer/d48ed23@@nntrainer@sha/lstmcell_core.cpp.o' 'nntrainer/d48ed23@@nntrainer@sha/zoneout_lstmcell.cpp.o' 'nntrainer/d48ed23@@nntrainer@sha/time_dist.cpp.o' 'nntrainer/d48ed23@@nntrainer@sha/common_properties.cpp.o' 'nntrainer/d48ed23@@nntrainer@sha/split_layer.cpp.o' 'nntrainer/d48ed23@@nntrainer@sha/permute_layer.cpp.o' 'nntrainer/d48ed23@@nntrainer@sha/layer_impl.cpp.o' 'nntrainer/d48ed23@@nntrainer@sha/gru.cpp.o' 'nntrainer/d48ed23@@nntrainer@sha/grucell.cpp.o' 'nntrainer/d48ed23@@nntrainer@sha/dropout.cpp.o' 'nntrainer/d48ed23@@nntrainer@sha/centroid_knn.cpp.o' 'nntrainer/d48ed23@@nntrainer@sha/layer_context.cpp.o' 'nntrainer/d48ed23@@nntrainer@sha/reshape_layer.cpp.o' 'nntrainer/d48ed23@@nntrainer@sha/reduce_mean_layer.cpp.o' 'nntrainer/d48ed23@@nntrainer@sha/positional_encoding_layer.cpp.o' 'nntrainer/d48ed23@@nntrainer@sha/identity_layer.cpp.o' 'nntrainer/d48ed23@@nntrainer@sha/tflite_layer.cpp.o' 'nntrainer/d48ed23@@nntrainer@sha/model_loader.cpp.o' 'nntrainer/d48ed23@@nntrainer@sha/neuralnet.cpp.o' 'nntrainer/d48ed23@@nntrainer@sha/model_common_properties.cpp.o' 'nntrainer/d48ed23@@nntrainer@sha/dynamic_training_optimization.cpp.o' 'nntrainer/d48ed23@@nntrainer@sha/adam.cpp.o' 'nntrainer/d48ed23@@nntrainer@sha/optimizer_devel.cpp.o' 'nntrainer/d48ed23@@nntrainer@sha/sgd.cpp.o' 'nntrainer/d48ed23@@nntrainer@sha/optimizer_context.cpp.o' 'nntrainer/d48ed23@@nntrainer@sha/lr_scheduler_constant.cpp.o' 'nntrainer/d48ed23@@nntrainer@sha/lr_scheduler_exponential.cpp.o' 'nntrainer/d48ed23@@nntrainer@sha/lr_scheduler_step.cpp.o' 'nntrainer/d48ed23@@nntrainer@sha/optimizer_wrapped.cpp.o' 'nntrainer/d48ed23@@nntrainer@sha/blas_interface.cpp.o' 'nntrainer/d48ed23@@nntrainer@sha/cache_elem.cpp.o' 'nntrainer/d48ed23@@nntrainer@sha/cache_loader.cpp.o' 'nntrainer/d48ed23@@nntrainer@sha/cache_pool.cpp.o' 'nntrainer/d48ed23@@nntrainer@sha/lazy_tensor.cpp.o' 'nntrainer/d48ed23@@nntrainer@sha/manager.cpp.o' 'nntrainer/d48ed23@@nntrainer@sha/tensor.cpp.o' 'nntrainer/d48ed23@@nntrainer@sha/tensor_dim.cpp.o' 'nntrainer/d48ed23@@nntrainer@sha/var_grad.cpp.o' 'nntrainer/d48ed23@@nntrainer@sha/weight.cpp.o' 'nntrainer/d48ed23@@nntrainer@sha/basic_planner.cpp.o' 'nntrainer/d48ed23@@nntrainer@sha/memory_pool.cpp.o' 'nntrainer/d48ed23@@nntrainer@sha/swap_device.cpp.o' 'nntrainer/d48ed23@@nntrainer@sha/tensor_pool.cpp.o' 'nntrainer/d48ed23@@nntrainer@sha/optimized_v1_planner.cpp.o' 'nntrainer/d48ed23@@nntrainer@sha/optimized_v2_planner.cpp.o' 'nntrainer/d48ed23@@nntrainer@sha/optimized_v3_planner.cpp.o' 'nntrainer/d48ed23@@nntrainer@sha/task_executor.cpp.o' 'nntrainer/d48ed23@@nntrainer@sha/util_func.cpp.o' 'nntrainer/d48ed23@@nntrainer@sha/profiler.cpp.o' 'nntrainer/d48ed23@@nntrainer@sha/ini_wrapper.cpp.o' 'nntrainer/d48ed23@@nntrainer@sha/node_exporter.cpp.o' 'nntrainer/d48ed23@@nntrainer@sha/base_properties.cpp.o' 'nntrainer/d48ed23@@nntrainer@sha/nntr_threads.cpp.o' 'nntrainer/d48ed23@@nntrainer@sha/network_graph.cpp.o' 'nntrainer/d48ed23@@nntrainer@sha/graph_core.cpp.o' 'nntrainer/d48ed23@@nntrainer@sha/connection.cpp.o' 'nntrainer/d48ed23@@nntrainer@sha/nntrainer_logger.cpp.o' 'nntrainer/d48ed23@@nntrainer@sha/app_context.cpp.o' -Wl,--as-needed -Wl,--no-undefined -Wl,-O1 -shared -fPIC -Wl,--start-group -Wl,-soname,libnntrainer.so /usr/lib/x86_64-linux-gnu/openblas-pthread/libopenblas.so -liniparser /usr/lib/x86_64-linux-gnu/libcapi-ml-common.so -lm -ldl -pthread -fopenmp /usr/lib/libtensorflowlite.so -Wl,--end-group -Wl,-rpath,/usr/lib/x86_64-linux-gnu/openblas-pthread -Wl,-rpath-link,/usr/lib/x86_64-linux-gnu/openblas-pthread
/usr/bin/ld: nntrainer/d48ed23@@nntrainer@sha/tflite_layer.cpp.o: in function `nntrainer::TfLiteLayer::~TfLiteLayer()':
tflite_layer.cpp:(.text+0xe6): undefined reference to `tflite::impl::FlatBufferModel::~FlatBufferModel()'
/usr/bin/ld: tflite_layer.cpp:(.text+0x104): undefined reference to `tflite::impl::Interpreter::~Interpreter()'
/usr/bin/ld: nntrainer/d48ed23@@nntrainer@sha/tflite_layer.cpp.o: in function `nntrainer::TfLiteLayer::forwarding(nntrainer::RunLayerContext&, bool)':
tflite_layer.cpp:(.text+0x41a): undefined reference to `tflite::impl::Interpreter::Invoke()'
/usr/bin/ld: tflite_layer.cpp:(.text+0x4c4): undefined reference to `tflite::impl::Interpreter::Invoke()'
/usr/bin/ld: nntrainer/d48ed23@@nntrainer@sha/tflite_layer.cpp.o: in function `nntrainer::TfLiteLayer::finalize(nntrainer::InitLayerContext&)':
tflite_layer.cpp:(.text+0xb45): undefined reference to `tflite::impl::FlatBufferModel::BuildFromFile(char const*, tflite::ErrorReporter*)'
/usr/bin/ld: tflite_layer.cpp:(.text+0xb6a): undefined reference to `tflite::impl::FlatBufferModel::~FlatBufferModel()'
/usr/bin/ld: tflite_layer.cpp:(.text+0xb89): undefined reference to `tflite::impl::FlatBufferModel::~FlatBufferModel()'
/usr/bin/ld: tflite_layer.cpp:(.text+0xbc2): undefined reference to `tflite::impl::InterpreterBuilder::InterpreterBuilder(tflite::impl::FlatBufferModel const&, tflite::OpResolver const&, tflite::InterpreterOptions const*)'
/usr/bin/ld: tflite_layer.cpp:(.text+0xbcf): undefined reference to `tflite::impl::InterpreterBuilder::operator()(std::unique_ptr<tflite::impl::Interpreter, std::default_delete<tflite::impl::Interpreter> >*)'
/usr/bin/ld: tflite_layer.cpp:(.text+0xbd7): undefined reference to `tflite::impl::InterpreterBuilder::~InterpreterBuilder()'
/usr/bin/ld: tflite_layer.cpp:(.text+0xbea): undefined reference to `tflite::impl::Interpreter::AllocateTensors()'
/usr/bin/ld: nntrainer/d48ed23@@nntrainer@sha/tflite_layer.cpp.o: in function `nntrainer::TfLiteLayer::finalize(nntrainer::InitLayerContext&) [clone .cold]':
tflite_layer.cpp:(.text.unlikely+0x2a6): undefined reference to `tflite::impl::InterpreterBuilder::~InterpreterBuilder()'
collect2: error: ld returned 1 exit status
[319/443] Compiling C++ object 'test/unittest/f284a9c@@unittest_nntrainer_tensor@exe/unittest_nntrainer_tensor.cpp.o'.
ninja: build stopped: subcommand failed.

I guess this is the issue with linking NNtrainer's libnntrainer.so with the newly-generated libtensorflowlite.so?

Donghak PARK · Answer 3 · Fri Apr 21 2023 20:11:42 GMT+0800 (China Standard Time)

@KirillP2323
If you rebuild .so file you should put that tensorflow file in include dir

KirillP2323 · Answer 4 · Fri Apr 21 2023 20:45:12 GMT+0800 (China Standard Time)

Great, it run successfully, thanks!

Donghak PARK · Answer 5 · Fri May 12 2023 16:32:50 GMT+0800 (China Standard Time)

@KirillP2323

As mentioned you can build aarch64 with this command
if failed to build, i recommand to build in docker container (with provided image from tensorflow)

for aarch64

You need to hide the system's openssl library. For this go to usr/include  folder and make sudo mv openssl openssl.original. 
bazelisk build --config=monolithic --config=noaws --config=nogcp --config=nohdfs --config=nonccl --config=elinux_aarch64 --experimental_ui_max_stdouterr_bytes=1073741819 -c opt //tensorflow/lite:libtensorflowlite.so

after build this .so file you should replace tensorflow lib in android build in
~~~/nntrainer/builddir/tensorflow-2.11.0/tensorflow-lite/lib/arm64/libtensorflow-lite.a

you should run package_android.sh file in tools dir And then you do several jobs like x86_64

Related Issue
tensorflow/tensorflow#54517
tensorflow/tensorflow#48401

ref - Install Tensorflow Lite 2.x on Raspverry Pi 4 (https://qengineering.eu/install-tensorflow-2-lite-on-raspberry-pi-4.html)
guid : https://www.tensorflow.org/lite/guide/build_arm → just guide For build follow this docs

If it doesn't work for Android selected ops please let me know

Donghak PARK · Answer 6 · Tue Oct 24 2023 17:51:37 GMT+0800 (China Standard Time)

I will close this issue