bake: hcl `index` function doesn't work as expected

Question

bake: hcl `index` function doesn't work as expected

Usual-Coder opened this issue 3 months ago · comments

Usual Coder commented 3 months ago

Contributing guidelines

I've read the contributing guidelines and wholeheartedly agree

I've found a bug and checked that ...

... the documentation does not mention anything about my problem
... there are no open or closed issues that are related to my problem

Description

When given the expected parameters, a collection as first parameter and a value as second, index function should return the corresponding index (number), if the value is present in the collection?! Instead it raises an error (...)Call to function "index" failed: key for tuple must be number.(...) (presumably because of collection.go of go-cty)

index is in fact behaving exactly as the element function. What's the purpose of an index function expecting a number (aka the searched index itself) = why search for something you already know?!!!

Expected behaviour

Knowing the differences between element and index functions (per hcl docs and/or based on comments in collection.go and the corresponding functions mapped from go-cty):

element(list, index)

element retrieves a single element from a list (prefer the built-in index syntax list[index]).
The index is zero-based. This function produces an error if used with an empty list.

index(list, value)

index should find the element index for a given value in a list.
The returned index is zero-based.
This function should produce an error if the given value is not present in the list.

With bug.hcl (aka dummy config ... just for demonstration / nothing specifically related to annotations = same error no matter where you use index function):

target "bug" {
  annotations = [
    "foo=${element(["a", "b", "c"], 1)}",  // Work as expected/documented = returns "b"
    "bar=${index(["a", "b", "c"], "b")}",  // Raise an error !!! Should return 1
    // "baz=${index(["a", "b", "c"], 1)}", // NOK = same behavior as "element" function = if you already know the index ... what's the purpose of this function ?!!!
  ]
}

Expected:

$ docker buildx bake --file path/to/bug.hcl --print bug
[+] Building 0.0s (1/1) FINISHED
 => [internal] load local bake definitions                                   0.0s
 => => reading path/to/bug.hcl 479B / 479B                                   0.0s
{
  "group": {
    "default": {
      "targets": [
        "bug"
      ]
    }
  },
  "target": {
    "bug": {
      "annotations": [
        "foo=b",
        "bar=1"
      ],
      "context": ".",
      "dockerfile": "Dockerfile"
    }
  }
}

Actual behaviour

Raise an error with the same/previous bug.hcl:

$ docker buildx bake --file path/to/bug.hcl --print bug
[+] Building 0.0s (1/1) FINISHED
 => [internal] load local bake definitions                                   0.0s
 => => reading path/to/bug.hcl 479B / 479B                                   0.0s
path/to/bug.hcl:4
--------------------
   2 |       annotations = [
   3 |         "foo=${element(["a", "b", "c"], 1)}",  // Work as expected/documented = returns "b"
   4 | >>>     "bar=${index(["a", "b", "c"], "b")}",  // Raise an error !!! Should return 1
   5 |         // "baz=${index(["a", "b", "c"], 1)}", // NOK = same behavior as "element" function = if you already know the index ... what's the purpose of this function ?!!!
   6 |       ]
--------------------
ERROR: path/to/bug.hcl:4,12-18: Error in function call; Call to function "index" failed: key for tuple must be number., and 1 other diagnostic(s)

Buildx version

github.com/docker/buildx v0.13.1 7884339

Docker info

Client: Docker Engine - Community
 Version:    26.0.0
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.13.1
    Path:     /usr/libexec/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.4.1
    Path:     /usr/local/lib/docker/cli-plugins/docker-compose
  scan: Docker Scan (Docker Inc.)
    Version:  v0.23.0
    Path:     /usr/libexec/docker/cli-plugins/docker-scan

Server:
 Containers: 1
  Running: 0
  Paused: 0
  Stopped: 1
 Images: 2
 Server Version: 26.0.0
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: ae07eda36dd25f8a1b98dfbf587313b99c0190bb
 runc version: v1.1.12-0-g51d5e94
 init version: de40ad0
 Security Options:
  apparmor
  seccomp
   Profile: builtin
  cgroupns
 Kernel Version: 6.5.0-26-generic
 Operating System: Ubuntu 22.04.4 LTS
 OSType: linux
 Architecture: x86_64
 CPUs: ...
 Total Memory: ...
 Name: ...
 ID: ...
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

Builders list

NAME/NODE     DRIVER/ENDPOINT   STATUS    BUILDKIT   PLATFORMS
default*      docker                                 
 \_ default    \_ default       running   v0.13.1    linux/amd64, linux/amd64/v2, linux/amd64/v3, linux/386

Configuration

target "bug" {
  annotations = [
    "foo=${element(["a", "b", "c"], 1)}",  // Work as expected/documented = returns "b"
    "bar=${index(["a", "b", "c"], "b")}",  // Raise an error !!! Should return 1
    // "baz=${index(["a", "b", "c"], 1)}", // NOK = same behavior as "element" function = if you already know the index ... what's the purpose of this function ?!!!
  ]
}

Build logs

No response

Additional info

First things first: bake feature is great 😍 Thanks for that 🙇

Since index is such a fundamental function, not having found an issue about it (and seen others not bothering/using it as/instead of element), I doubted myself for a while before filling this bug (not being fluent in Go, I stopped when digging deeper in go-cty sources became needed).

I don't know if it's related to how you're mapping go-cty in your hcl parser or the library itself ... but we should expect index or at least a function (no matter how it's named ... it should have a different behavior than the one expected from element ... the features subset being limited, may I say duplicates should not be expected?!) returning an index (as a number) ... not already known (the expected result shouldn't be provided as one of its parameters 🤐🤷)

Hope it can be fixed easily.

Thanks for everything.

Usual Coder · Answer 1 · Sun Apr 07 2024 04:20:57 GMT+0800 (China Standard Time)

With great humility (my deepest Go experience was limited to playing with Hugo modules -handling Bootstrap vendor folder problem- until recently ... I really started discovering Go syntax yesterday 🤐 ... please, be merciful 😅), I propose an opiniated™ solution ... since it's introducing a (much needed?!) breaking change 😱

You can easily test it, by spinning a Codespace on my fork (feature-hcl-index branch) as expected to prepare a PR (per CONTRIBUTING.md doc).

I'm providing a feature-hcl-index/test.hcl available in the demo container environment (using make shell, as explained in the very same CONTRIBUTING.md)

variable "APP_VERSIONS" {
  default = [
    "1.42.4",
    "1.42.3",
    "1.42.2",
  ]
}

group "default" {
    targets = [ "feature-hcl-index" ]
}

target "feature-hcl-index" {
    args = {
        APP_VERSION = app_version
    }
    matrix = {
        app_version = APP_VERSIONS
    }
    name="app-${replace(app_version, ".", "-")}"
    tags = [
        "app:${app_version}",
        index(APP_VERSIONS, app_version) == 0 ? "app:latest" : "",
    ]
}

btw, I'm not providing tests since none is present in that part of the project (probably because it's mainly relying on imported code, tested elsewhere + learning how to write them would have consumed way too much time before ensuring anyone was actually in the loop of this issue 😏) = the previous test.hcl is my gift to the (still) too rare TDD approach 😭😅

At least, the tests suite is still running after my modifications (easy since none exist for the hclparser part 😏)!

If you use it without the patched version, you'll get, as (un)expected (the reason I opened this issue 😅):

$ docker buildx bake --file path/to/feature-hcl-index/test.hcl --print
[+] Building 0.0s (1/1) FINISHED
 => [internal] load local bake definitions                            0.0s
 => => reading path/to/feature-hcl-index/test.hcl 453B / 453B         0.0s
path/to/feature-hcl-index/test.hcl:23
--------------------
  21 |         tags = [
  22 |             "app:${app_version}",
  23 | >>>         index(APP_VERSIONS, app_version) == 0 ? "app:latest" : "",
  24 |         ]
  25 |     }
--------------------
ERROR: path/to/feature-hcl-index/test.hcl:23,9-15: Error in function call; Call to function "index" failed: key for tuple must be number., and 1 other diagnostic(s)

But within the demo container (aka using the proposed solution):

/work # docker buildx bake --file feature-hcl-index/test.hcl --print
[+] Building 0.0s (1/1) FINISHED
 => [internal] load local bake definitions                    0.0s
 => => reading feature-hcl-index/test.hcl 453B / 453B         0.0s
{
  "group": {
    "default": {
      "targets": [
        "feature-hcl-index"
      ]
    },
    "feature-hcl-index": {
      "targets": [
        "app-1-42-4",
        "app-1-42-3",
        "app-1-42-2"
      ]
    }
  },
  "target": {
    "app-1-42-2": {
      "context": ".",
      "dockerfile": "Dockerfile",
      "args": {
        "APP_VERSION": "1.42.2"
      },
      "tags": [
        "app:1.42.2"
      ]
    },
    "app-1-42-3": {
      "context": ".",
      "dockerfile": "Dockerfile",
      "args": {
        "APP_VERSION": "1.42.3"
      },
      "tags": [
        "app:1.42.3"
      ]
    },
    "app-1-42-4": {
      "context": ".",
      "dockerfile": "Dockerfile",
      "args": {
        "APP_VERSION": "1.42.4"
      },
      "tags": [
        "app:1.42.4",
        "app:latest"
      ]
    }
  }
}

I can't take much credit for that solution, I'm just following the good people of Hashicorp stating stdlib.IndexFunc is not compatible (stdlib.IndexFunc being imported from the same source -obviously, why reinvent the wheel?! 😏- as it's done here: github.com/zclconf/go-cty/cty/function/stdlib)

To do so, I'm applying the same logic used 2 years ago by jedevc with timestampFunc = importing the solution from terraform itself! Same minor change(s) has(have) been applied to the imported parts.

By doing so, while still being a noob in Go, I hope to fall not so far from what's expected for a PR to be accepted?! 🤷

Anyway, I think I can see the philosophy behind the portions included in the bake feature (I am also a true believer in the motto: "The best code is the one you don't maintain" 😏) and how those choices can mislead people (index from go-cty not behaving as the one being part of the hcl syntax could lead to great frustration 🤐😅)

I understand the purpose of the bake subcommand is not to reproduce the full featureset of hcl from Hashicorp, but even on a limited subset we should expect everything required to manipulate collections without the trickery I've seen around ... just my 2 cents on the philosophical topic 🤷 (or why if it was up to me, I would probably add few other things 😏 ... making it easier to write -and maintain- things related to tagging for example: handling latest, major/minor/specific tags, etc. 🤷)

And, as conclusion, looking at the history of bake related sources ... that killing feature™ is not receiving the love it deserves, if I may 😅 With mine ... hoping all of that will help improve the common experience with Docker ❤️

Usual Coder · Answer 2 · Sun Apr 07 2024 18:58:13 GMT+0800 (China Standard Time)

For the crazy people living (thriving?! 😏) on the edge™, you can test this fix (aka use it locally until a maintainer says what about its future 😅) by following the (quite simple) Building instructions from the README:

# Buildx 0.6+
# $ docker buildx bake "https://github.com/docker/buildx.git"
$ docker buildx bake "https://github.com/Usual-Coder/buildx.git#feature-hcl-index"
$ mkdir -p ~/.docker/cli-plugins
$ mv ./bin/build/buildx ~/.docker/cli-plugins/docker-buildx

Rolling back to the default behavior is as easy as deleting the newly built plugin (or completely removing the cli-plugins folder if you don't have anything else inside)

Before installation (aka buildx distributed with docker itself):

$ docker buildx version
github.com/docker/buildx v0.13.1 7884339

After (using the one built previously):

$ docker buildx version
github.com/docker/buildx a88a281 a88a2814a43f633d75c9671911598c8c3e5222f7

Shown outputs (before/after versions) depend on your own configuration (don't expect a perfect match)

index function will now behave as its hcl definition 🎉

Enjoy while you can 🤐😅❤️

Usual Coder · Answer 3 · Wed Apr 10 2024 17:57:30 GMT+0800 (China Standard Time)

Per the maintainer's request, this proposed function is now named indexof (mimicking the hcl behavior of index function) while the old index builtin (wrapped from go-cty, unpatched like done by the people of Hashicorp and the very source of this proposal) remains in place.

Even without the proper/expected fix, I hope it will help you enjoy the bake feature. Have fun 🤘