Build error for tf-trusted custom op

Question

Build error for tf-trusted custom op

mirsci opened this issue 5 years ago · comments

Hi,

I am trying to build the tf-trusted for custom op to run models inside SGX enclave. I am using an Ubuntu 14.04 machine.
The first script to build the custom operation worked, but the build fails when executing:
bazel build model_enclave_op.so

with this error:

INFO: From SkylarkAction external/com_github_grpc_grpc/src/proto/grpc/reflection/v1alpha/reflection.grpc.pb.h:
bazel-out/k8-fastbuild/genfiles/external/com_github_grpc_grpc/external/com_github_grpc_grpc: warning: directory does not exist.
ERROR: /opt/my-project/tf_trusted_custom_op/BUILD:17:1: C++ compilation of rule '//:model_enclave_op.so' failed (Exit 1): gcc failed: error executing command 
  (cd /root/.cache/bazel/_bazel_root/6a072cedc59c5d9384722d447b964014/execroot/tf_trusted_custom_op && \
  exec env - \
    PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin \
    PWD=/proc/self/cwd \
    TF_HEADER_DIR=/usr/local/lib/python2.7/dist-packages/tensorflow/include \
    TF_SHARED_LIBRARY_DIR=/usr/local/lib/python2.7/dist-packages/tensorflow \

Could you please advise? Thanks in advance!

Justin Patriquin · Answer 1 · Thu Mar 21 2019 19:27:34 GMT+0800 (China Standard Time)

Hi @mirsci ,

Thanks for the issue. I think there might be more to the error than you've reported. If you could send the whole thing I'll be able to help you better!

Thanks

mirsci · Answer 2 · Thu Mar 21 2019 19:37:44 GMT+0800 (China Standard Time)

Hi @justin1121 ,

thanks too for responding so fast :).
I just re-executed the script above and here there are all the messages:

root@c36fdbefd6b6:/opt/my-project/tf_trusted_custom_op# bazel build model_enclave_op.so
Extracting Bazel installation...
WARNING: --batch mode is deprecated. Please instead explicitly shut down your Bazel server using the command "bazel shutdown".
WARNING: /root/.cache/bazel/_bazel_root/6a072cedc59c5d9384722d447b964014/external/local_config_tf/BUILD:3588:1: target 'libtensorflow_framework.so' is both a rule and a file; please choose another name for the rule
INFO: SHA256 (https://github.com/nanopb/nanopb/archive/f8ac463766281625ad710900479130c7fcb4d63b.tar.gz) = 8bbbb1e78d4ddb0a1919276924ab10d11b631df48b657d960e0c795a25515735
DEBUG: /root/.cache/bazel/_bazel_root/6a072cedc59c5d9384722d447b964014/external/bazel_tools/tools/build_defs/repo/http.bzl:43:9: ctx.attr.build_file @com_github_grpc_grpc//third_party:nanopb.BUILD, path /root/.cache/bazel/_bazel_root/6a072cedc59c5d9384722d447b964014/external/com_github_grpc_grpc/third_party/nanopb.BUILD
INFO: SHA256 (https://github.com/madler/zlib/archive/cacf7f1d4e3d44d871b605da3b647f07d718623f.tar.gz) = 6d4d6640ca3121620995ee255945161821218752b551a1a180f4215f7d124d45
INFO: SHA256 (https://github.com/c-ares/c-ares/archive/3be1924221e1326df520f8498d704a5c4c8d0cce.tar.gz) = e69e33fd40a254fcf00d76efa76776d45f960e34307bd9cea9df93ef79a933f1
DEBUG: /root/.cache/bazel/_bazel_root/6a072cedc59c5d9384722d447b964014/external/bazel_tools/tools/build_defs/repo/http.bzl:43:9: ctx.attr.build_file @com_github_grpc_grpc//third_party:zlib.BUILD, path /root/.cache/bazel/_bazel_root/6a072cedc59c5d9384722d447b964014/external/com_github_grpc_grpc/third_party/zlib.BUILD
DEBUG: /root/.cache/bazel/_bazel_root/6a072cedc59c5d9384722d447b964014/external/bazel_tools/tools/build_defs/repo/http.bzl:43:9: ctx.attr.build_file @com_github_grpc_grpc//third_party:cares/cares.BUILD, path /root/.cache/bazel/_bazel_root/6a072cedc59c5d9384722d447b964014/external/com_github_grpc_grpc/third_party/cares/cares.BUILD
INFO: SHA256 (https://boringssl.googlesource.com/boringssl/+archive/afc30d43eef92979b05776ec0963c9cede5fb80f.tar.gz) = 36bed1fcd4f290bff4aaa8decf67192e6ba727d76c1a2442aeb69c89b1e47379
WARNING: /root/.cache/bazel/_bazel_root/6a072cedc59c5d9384722d447b964014/external/local_config_tf/BUILD:5:12: in hdrs attribute of cc_library rule @local_config_tf//:tf_header_lib: file '_api_implementation.so' from target '@local_config_tf//:tf_header_include' is not allowed in hdrs
WARNING: /root/.cache/bazel/_bazel_root/6a072cedc59c5d9384722d447b964014/external/local_config_tf/BUILD:5:12: in hdrs attribute of cc_library rule @local_config_tf//:tf_header_lib: file '_message.so' from target '@local_config_tf//:tf_header_include' is not allowed in hdrs
INFO: Analysed target //:model_enclave_op.so (21 packages loaded).
INFO: Found 1 target...
INFO: From SkylarkAction external/com_github_grpc_grpc/src/proto/grpc/reflection/v1alpha/reflection.pb.h:
bazel-out/k8-fastbuild/genfiles/external/com_github_grpc_grpc/external/com_github_grpc_grpc: warning: directory does not exist.
INFO: From SkylarkAction external/com_github_grpc_grpc/src/proto/grpc/reflection/v1alpha/reflection.grpc.pb.h:
bazel-out/k8-fastbuild/genfiles/external/com_github_grpc_grpc/external/com_github_grpc_grpc: warning: directory does not exist.
ERROR: /opt/my-project/tf_trusted_custom_op/BUILD:17:1: C++ compilation of rule '//:model_enclave_op.so' failed (Exit 1): gcc failed: error executing command 
  (cd /root/.cache/bazel/_bazel_root/6a072cedc59c5d9384722d447b964014/execroot/tf_trusted_custom_op && \
  exec env - \
    PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin \
    PWD=/proc/self/cwd \
    TF_HEADER_DIR=/usr/local/lib/python2.7/dist-packages/tensorflow/include \
    TF_SHARED_LIBRARY_DIR=/usr/local/lib/python2.7/dist-packages/tensorflow \
  /usr/bin/gcc -U_FORTIFY_SOURCE -fstack-protector -Wall -B/usr/bin -B/usr/bin -Wunused-but-set-parameter -Wno-free-nonheap-object -fno-omit-frame-pointer '-std=c++0x' -MD -MF bazel-out/k8-fastbuild/bin/_objs/model_enclave_op.so/model_enclave_op.pic.d '-frandom-seed=bazel-out/k8-fastbuild/bin/_objs/model_enclave_op.so/model_enclave_op.pic.o' -fPIC '-DPB_FIELD_32BIT=1' -DCARES_STATICLIB -iquote . -iquote bazel-out/k8-fastbuild/genfiles -iquote external/local_config_tf -iquote bazel-out/k8-fastbuild/genfiles/external/local_config_tf -iquote external/bazel_tools -iquote bazel-out/k8-fastbuild/genfiles/external/bazel_tools -iquote external/com_github_grpc_grpc -iquote bazel-out/k8-fastbuild/genfiles/external/com_github_grpc_grpc -iquote external/com_github_madler_zlib -iquote bazel-out/k8-fastbuild/genfiles/external/com_github_madler_zlib -iquote external/com_github_nanopb_nanopb -iquote bazel-out/k8-fastbuild/genfiles/external/com_github_nanopb_nanopb -iquote external/com_github_cares_cares -iquote bazel-out/k8-fastbuild/genfiles/external/com_github_cares_cares -iquote external/boringssl -iquote bazel-out/k8-fastbuild/genfiles/external/boringssl -iquote external/com_google_protobuf -iquote bazel-out/k8-fastbuild/genfiles/external/com_google_protobuf -isystem external/local_config_tf/include -isystem bazel-out/k8-fastbuild/genfiles/external/local_config_tf/include -isystem bazel-out/k8-fastbuild/bin/external/local_config_tf/include -isystem external/com_github_grpc_grpc/include -isystem bazel-out/k8-fastbuild/genfiles/external/com_github_grpc_grpc/include -isystem bazel-out/k8-fastbuild/bin/external/com_github_grpc_grpc/include -isystem external/com_github_madler_zlib -isystem bazel-out/k8-fastbuild/genfiles/external/com_github_madler_zlib -isystem bazel-out/k8-fastbuild/bin/external/com_github_madler_zlib -isystem external/com_github_cares_cares -isystem bazel-out/k8-fastbuild/genfiles/external/com_github_cares_cares -isystem bazel-out/k8-fastbuild/bin/external/com_github_cares_cares -isystem external/com_github_grpc_grpc/third_party/address_sorting/include -isystem bazel-out/k8-fastbuild/genfiles/external/com_github_grpc_grpc/third_party/address_sorting/include -isystem bazel-out/k8-fastbuild/bin/external/com_github_grpc_grpc/third_party/address_sorting/include -isystem external/boringssl/src/include -isystem bazel-out/k8-fastbuild/genfiles/external/boringssl/src/include -isystem bazel-out/k8-fastbuild/bin/external/boringssl/src/include -isystem external/com_google_protobuf/src -isystem bazel-out/k8-fastbuild/genfiles/external/com_google_protobuf/src -isystem bazel-out/k8-fastbuild/bin/external/com_google_protobuf/src -isystem cpp_model_proto_pb -isystem bazel-out/k8-fastbuild/genfiles/cpp_model_proto_pb -isystem bazel-out/k8-fastbuild/bin/cpp_model_proto_pb -pthread '-std=c++11' '-D_GLIBCXX_USE_CXX11_ABI=0' -fno-canonical-system-headers -Wno-builtin-macro-redefined '-D__DATE__="redacted"' '-D__TIMESTAMP__="redacted"' '-D__TIME__="redacted"' -c model_enclave_op.cc -o bazel-out/k8-fastbuild/bin/_objs/model_enclave_op.so/model_enclave_op.pic.o)
In file included from bazel-out/k8-fastbuild/genfiles/cpp_model_proto_pb/proto/model_server.grpc.pb.h:7:0,
                 from model_enclave_grpc.h:14,
                 from model_enclave_op.cc:2:
bazel-out/k8-fastbuild/genfiles/cpp_model_proto_pb/proto/model_server.pb.h:17:2: error: #error This file was generated by an older version of protoc which is
 #error This file was generated by an older version of protoc which is
  ^
bazel-out/k8-fastbuild/genfiles/cpp_model_proto_pb/proto/model_server.pb.h:18:2: error: #error incompatible with your Protocol Buffer headers. Please
 #error incompatible with your Protocol Buffer headers.  Please
  ^
bazel-out/k8-fastbuild/genfiles/cpp_model_proto_pb/proto/model_server.pb.h:19:2: error: #error regenerate this file with a newer version of protoc.
 #error regenerate this file with a newer version of protoc.
  ^
Target //:model_enclave_op.so failed to build
INFO: Elapsed time: 460.935s, Critical Path: 37.07s
INFO: 1092 processes: 1092 local.
FAILED: Build did NOT complete successfully
root@c36fdbefd6b6:/opt/my-project/tf_trusted_custom_op# exit

Justin Patriquin · Answer 3 · Thu Mar 21 2019 20:05:00 GMT+0800 (China Standard Time)

Hi,

I think I see the problem. Inside the configure.sh file it installs tensorflow but the version isn't pinned to the 1.12.0 version. So its installing 1.13.1 which we don't yet support. I can submit a PR soon fixing this or in the mean time you can just change the pip install tensorflow line to pip install tensorflow==1.12.0. Feel free to submit your own PR fixing this as well.

mirsci · Answer 4 · Thu Mar 21 2019 22:14:26 GMT+0800 (China Standard Time)

Many thanks, @justin1121, it worked perfectly this time!

Justin Patriquin · Answer 5 · Fri Mar 22 2019 23:04:59 GMT+0800 (China Standard Time)

Going to keep this open for now to track that something needs to be fixed inside the configure.sh file!

mirsci · Answer 6 · Sun Mar 24 2019 00:59:20 GMT+0800 (China Standard Time)

Hi @justin1121,

just one question on tf-trusted framework: would you consider extending it with model training at some point in the future? It will be great way to support this level of abstraction for SGX enclaves operations...

Thanks in advance!

Justin Patriquin · Answer 7 · Sun Mar 24 2019 19:22:51 GMT+0800 (China Standard Time)

Training is definitely something we're interested it. Tflite doesn't currently support training but there's a good chance it might in the future. Tensorflow has recently released an RFC proposing the addition of control flow ops to tflite. This should help enable training in tflite in the future. For tf-trusted to enable training right now we'd have to build the tensorflow runtime into the SGX device.

mirsci · Answer 8 · Mon Mar 25 2019 21:34:04 GMT+0800 (China Standard Time)

Thanks @justin1121. It will be great to have training within SGX enclave, for increased privacy along the ML workflow. For example, Chiron paper describes an enclave training approach, using Theano, C compiler in a Ryoan sandbox: https://arxiv.org/pdf/1803.05961.pdf. Unfortunately, the Chiron approach is not open sourced yet....

Justin Patriquin · Answer 9 · Wed Mar 27 2019 23:15:22 GMT+0800 (China Standard Time)

Hey @mirsci,

If you haven't heard of it already I'd recommend checking out slalom https://github.com/ftramer/slalom its open source and takes an interesting approach to training models with SGX!

mirsci · Answer 10 · Wed Mar 27 2019 23:35:31 GMT+0800 (China Standard Time)

Thanks so much, @justin1121!
Indeed, Slalom has a very interesting approach on SGX, plus it includes blinding and unblinding operations to boost the overall computational privacy. Will be trying it out as well :)