Generating compile_commands.json for TensorFlow

[Jiawei Liu]

Hi, I am using bazel-compile-commands-extractor to generate compile_commands.json for tensorflow. By adding:

load("@bazel_tools//tools/build_defs/repo:http.bzl", "http_archive")


# Hedron's Compile Commands Extractor for Bazel
# https://github.com/hedronvision/bazel-compile-commands-extractor
http_archive(
    name = "hedron_compile_commands",

    # Replace the commit hash in both places (below) with the latest, rather than using the stale one here.
    # Even better, set up Renovate and let it do the work for you (see "Suggestion: Updates" in the README).
    url = "https://github.com/hedronvision/bazel-compile-commands-extractor/archive/5ceebcf367f51c6a18fffe726755951a43b50e45.tar.gz",
    strip_prefix = "bazel-compile-commands-extractor-5ceebcf367f51c6a18fffe726755951a43b50e45",
    # When you first run this tool, it'll recommend a sha256 hash to put here with a message like: "DEBUG: Rule 'hedron_compile_commands' indicated that a canonical reproducible form can be obtained by modifying arguments sha256 = ..."
)
load("@hedron_compile_commands//:workspace_setup.bzl", "hedron_compile_commands_setup")
hedron_compile_commands_setup()

to the end of its WORKSPACE file.

Using bazel run @hedron_compile_commands//:refresh_all it gets failed at:

>>> Analyzing commands used in @//...
WARNING: /home/jiawei/dev/tensorflow/tensorflow/python/ops/distributions/BUILD:6:11: target '//tensorflow/python/ops/distributions:distributions' is deprecated: TensorFlow Distributions has migrated to TensorFlow Probability (https://github.com/tensorflow/probability). Deprecated copies remaining in tf.distributions will not receive new features, and will be removed by early 2019. You should update all usage of `tf.distributions` to `tfp.distributions`.
WARNING: /home/jiawei/dev/tensorflow/tensorflow/core/kernels/BUILD:1091:18: target '//tensorflow/core/kernels:bitcast_op' is deprecated: use //third_party/tensorflow/c/kernels:bitcast_op instead
ERROR: /home/jiawei/dev/tensorflow/tensorflow/lite/experimental/acceleration/configuration/BUILD:74:24: every rule of type java_lite_proto_library implicitly depends upon the target '@com_google_protobuf//:javalite_toolchain', but this target could not be found because of: no such target '@com_google_protobuf//:javalite_toolchain': target 'javalite_toolchain' not declared in package '' (did you mean 'java_toolchain'?) defined by /home/jiawei/.cache/bazel/_bazel_jiawei/b619a772dd31287f179cb3c11ac7f523/external/com_google_protobuf/BUILD
ERROR: Analysis of target '//tensorflow/lite/experimental/acceleration/configuration:configuration_java_proto_lite' failed; build aborted: Analysis failed
Bazel aquery failed. Command: ['bazel', 'aquery', "mnemonic('(Objc|Cpp)Compile',deps(@//...))", '--output=jsonproto', '--include_artifacts=false', '--ui_event_filters=-info', '--noshow_progress', '--features=-compiler_param_file']
>>> Failed extracting commands for @//...
    Continuing gracefully...

So I am wondering how I can work it around. Thanks!

[Chris Sauer]

Hey, @ganler (and @dpar39)! thanks for writing in and giving this tool a whirl. Sorry I haven't gotten back to you with my usual speed. I've been traveling.

Tensorflow is a big project with platform-specific configuration, so I suspect you'll want to specify which output targets you care about via a refresh_compile_commands rule. (Please see README) You'll need to do that if you wouldn't expect bazel build @//... to work on its own. Let me know how that works for you!

[Chris Sauer]

(I'm pretty sure that's what this issue needs, so I'm going to optimistically close it, but if that doesn't work, just holler and I'll open things right back up.)

[Fabian Schuetze]

I can confirm that this worked in my case - thanks @cpsauer .