Build Runs Every Time Despite Using Remote Cache

[aryeh]

We are doing the following on CI:

    bazel build //... \
      --remote_cache=https://storage.googleapis.com/<path-to-bucket> \
      --google_default_credentials

We have a build target that uses rules docker that bazel runs every time, despite using the remote cache. It does not depend on other targets in our application, except for ones that are rarely modified. Is this expected behavior? It's not completely clear to me whether this is a bug. Any guidance on a workaround or otherwise would be appreciated. Unfortunately, the docker build target is expensive, as it involves compilation of third-party sources. We are looking at sharding the build on CI to multiple machines due to this issue.

[Sasha Avalon]

We're having the same issue. Bazel doesn't even seem to detect that it's a cacheable target - it reports a 100% cache hit rate.

[Mika Sun]

i found ImageLayer is not "remote_cacheable"，from the execution log(with option --execution_log_binary_file)

remote-cached:

listed_outputs: "bazel-out/k8-fastbuild/bin/apps/fc/ifbook/srcs/pubsub/[e-mail]256"
remotable: true
cacheable: true
progress_message: "SHA256 apps/fc/ifbook/srcs/pubsub/[e-mail]256"
mnemonic: "SHA256"
actual_outputs {
  path: "bazel-out/k8-fastbuild/bin/apps/fc/ifbook/srcs/pubsub/[e-mail]256"
  digest {
    hash: "55f9eac53ee92f92f3afefa32e1a9cecff3a98397df2ce81826b495a1e66209a"
    size_bytes: 64
    hash_function_name: "SHA-256"
  }
}
runner: "remote cache hit"
remote_cache_hit: true
remote_cacheable: true   # <--- here
walltime {
}

not remote-cached(remote_cacheable disappeared):

listed_outputs: "bazel-out/k8-fastbuild/bin/apps/fc/ifbook/srcs/pubsub/[e-mail]"
remotable: true
cacheable: true
progress_message: "ImageLayer apps/fc/ifbook/srcs/pubsub/[e-mail]"
mnemonic: "ImageLayer"
actual_outputs {
  path: "bazel-out/k8-fastbuild/bin/apps/fc/ifbook/srcs/pubsub/[e-mail]"
  digest {
    hash: "664346b8273e5c46f88c70cf3717bf01a01a3f32d65472b6448058962bdd6109"
    size_bytes: 27709440
    hash_function_name: "SHA-256"
  }
}
runner: "processwrapper-sandbox"
walltime {
  nanos: 138000000
}

ImageLayer code: https://github.com/bazelbuild/rules_docker/blob/master/container/layer.bzl#L170