Actions using ctx.actions.declare_directory() are never cached

[srdjanstipic]

Description of the bug:

Buildbudy version: BuildBuddy v2.12.43

Actions using ctx.actions.declare_directory() are never cached.

This bug is also reported as a bazel bug: bazelbuild/bazel#20899
but it looks like possible BuildBuddy bug.

Which category does this issue belong to?

Core

What's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

I am using bazel with remote caching enabled.
First I will show an example that uses declare_file() and is remote cacheable.
Second, I will modify the example to use declare_directory()
and I will show that bazel never caches modified example (even thought it should).
Before each bazel build I will clean all bazel data.

Remote cacheable example:

> ls non_cacheable_target
BUILD  defs.bzl

> cat non_cacheable_target/BUILD
load(":defs.bzl", "generate_output_files_rule")

generate_output_files_rule(name = "output_files")

> cat non_cacheable_target/defs.bzl
def _generate_output_files_rule_impl(ctx):
    generated_output = ctx.actions.declare_file(ctx.attr.name + "/file.txt")
    file_path = generated_output.path

    ctx.actions.run_shell(
        outputs = [generated_output],
        command = "echo Hola1 > " + file_path,
    )

    return [DefaultInfo(files = depset(direct = [generated_output]))]

generate_output_files_rule = rule(
    implementation = _generate_output_files_rule_impl,
)

I build the target like this (first invocation is not cached 1 processwrapper-sandbox):

> bazel clean --expunge; bazel shutdown
> sudo rm -rf external /home/dockeruser/.cache/bazel /home/dockeruser/.cache/bazel_disk_cache/
> bazel build //non_cacheable_target/...
INFO: Starting clean (this may take a while). Consider using --async if the clean takes more than several minutes.
Extracting Bazel installation...
Starting local Bazel server and connecting to it...
INFO: Analyzed target //non_cacheable_target:output_files (8 packages loaded, 14 targets configured).
INFO: Found 1 target...
Target //non_cacheable_target:output_files up-to-date:
  bazel-bin/non_cacheable_target/output_files/file.txt
INFO: Elapsed time: 7.775s, Critical Path: 0.18s
INFO: 2 processes: 1 internal, 1 processwrapper-sandbox.

I build the same target again (second invocation is cached 1 remote cache hit):

> bazel clean --expunge; bazel shutdown
> sudo rm -rf external /home/dockeruser/.cache/bazel /home/dockeruser/.cache/bazel_disk_cache/
> bazel build //non_cacheable_target/...
INFO: Starting clean (this may take a while). Consider using --async if the clean takes more than several minutes.
Extracting Bazel installation...
Starting local Bazel server and connecting to it...
INFO: Analyzed target //non_cacheable_target:output_files (8 packages loaded, 14 targets configured).
INFO: Found 1 target...
Target //non_cacheable_target:output_files up-to-date:
  bazel-bin/non_cacheable_target/output_files/file.txt
INFO: Elapsed time: 8.140s, Critical Path: 0.07s
INFO: 2 processes: 1 remote cache hit, 1 internal.

Now I modify the example to use declare_directory():

diff --git a/non_cacheable_target/defs.bzl b/non_cacheable_target/defs.bzl
--- a/non_cacheable_target/defs.bzl
+++ b/non_cacheable_target/defs.bzl
@@ -1,6 +1,6 @@
 def _generate_output_files_rule_impl(ctx):
-    generated_output = ctx.actions.declare_file(ctx.attr.name + "/file.txt")
-    file_path = generated_output.path
+    generated_output = ctx.actions.declare_directory(ctx.attr.name)
+    file_path = generated_output.path + "/file.txt"

     ctx.actions.run_shell(
         outputs = [generated_output],

I build the modified example (first invocation is not cached 1 processwrapper-sandbox):

> bazel clean --expunge; bazel shutdown
> sudo rm -rf external /home/dockeruser/.cache/bazel /home/dockeruser/.cache/bazel_disk_cache/
> bazel build //non_cacheable_target/...
INFO: Starting clean (this may take a while). Consider using --async if the clean takes more than several minutes.
Extracting Bazel installation...
Starting local Bazel server and connecting to it...
INFO: Analyzed target //non_cacheable_target:output_files (8 packages loaded, 14 targets configured).
INFO: Found 1 target...
Target //non_cacheable_target:output_files up-to-date:
  bazel-bin/non_cacheable_target/output_files
INFO: Elapsed time: 7.891s, Critical Path: 0.18s
INFO: 2 processes: 1 internal, 1 processwrapper-sandbox.

I build the modified example again (and it is not cached even thought it should be 1 processwrapper-sandbox):

> bazel clean --expunge; bazel shutdown
> sudo rm -rf external /home/dockeruser/.cache/bazel /home/dockeruser/.cache/bazel_disk_cache/
> bazel build //non_cacheable_target/...
INFO: Starting clean (this may take a while). Consider using --async if the clean takes more than several minutes.
Extracting Bazel installation...
Starting local Bazel server and connecting to it...
INFO: Analyzed target //non_cacheable_target:output_files (8 packages loaded, 14 targets configured).
INFO: Found 1 target...
Target //non_cacheable_target:output_files up-to-date:
  bazel-bin/non_cacheable_target/output_files
INFO: Elapsed time: 7.388s, Critical Path: 0.15s
INFO: 2 processes: 1 internal, 1 processwrapper-sandbox.

This is the execution_json_log_file of the last bazel build:

{
  "commandArgs": ["/bin/bash", "-c", "echo Hola1 \u003e bazel-out/k8-dbg/bin/non_cacheable_target/output_files/file.txt"],
  "environmentVariables": [],
  "platform": {
    "properties": []
  },
  "inputs": [],
  "listedOutputs": ["bazel-out/k8-dbg/bin/non_cacheable_target/output_files"],
  "remotable": true,
  "cacheable": true,
  "timeoutMillis": "0",
  "progressMessage": "Action non_cacheable_target/output_files",
  "mnemonic": "Action",
  "actualOutputs": [{
    "path": "bazel-out/k8-dbg/bin/non_cacheable_target/output_files/file.txt",
    "digest": {
      "hash": "9bea9f5b70eb77cb7dc82b9515c68aaa91b7c0cb50122389713fe0c155fd4626",
      "sizeBytes": "6",
      "hashFunctionName": "SHA-256"
    },
    "isTool": false
  }],
  "runner": "processwrapper-sandbox",
  "remoteCacheHit": false,
  "status": "",
  "exitCode": 0,
  "remoteCacheable": true,
  "walltime": "0.005973s",
  "targetLabel": "//non_cacheable_target:output_files"
}

Bazel says that the target is remoteCacheable but it is never cached
and this is the bug.

Which operating system are you running Bazel on?

Linux 4e32033cbff8 5.4.0-1109-azure #115~18.04.1-Ubuntu SMP Mon May 22 20:06:37 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

What is the output of bazel info release?

INFO: Invocation ID: 84276203-854f-4133-80c3-44552407c4a0 release 6.4.0

If bazel info release returns development version or (@non-git), tell us how you built Bazel.

No response

What's the output of git remote get-url origin; git rev-parse master; git rev-parse HEAD ?

No response

Is this a regression? If yes, please try to identify the Bazel commit where the bug was introduced.

No response

Have you found anything relevant by searching the web?

This bug might be related: bazelbuild/bazel#17162

Any other information, logs, or outputs that you want to share?

No response

[srdjanstipic]

I think that the issue is not in buildbuddy but in my BuildBarn settings.
I think that I didn't enable upload of REv2 Directory objects.

Once I am able to find the correct setting, I will update this ticket.

P.S.
BuildBarn version:

    bbSchedulerImage: 'ghcr.io/buildbarn/bb-scheduler:20231212t221000z-fb20d59',
    bbFrontendImage: 'ghcr.io/buildbarn/bb-storage:20231205t110558z-8de3263',
    bbFrontendWithAuthImage: 'ghcr.io/buildbarn/bb-storage:20231205t110558z-8de3263',
    bbStorageImage: 'ghcr.io/buildbarn/bb-storage:20231205t110558z-8de3263',
    bbBrowserImage: 'ghcr.io/buildbarn/bb-browser:20231103t134227z-794e380',
    bbRunnerInstallerImage: 'ghcr.io/buildbarn/bb-runner-installer:20231212t221000z-fb20d59',
    bbWorkerImage: 'ghcr.io/buildbarn/bb-worker:20231212t221000z-fb20d59',

[Brandon Duffany]

Hey @srdjanstipic, since it's a BuildBarn configuration issue, I'll mark this issue closed - but please feel free to share any updates in case it helps someone else :)