Handling escaped characters when converting json to hcl

[Loren Gordon]

When using functions where arguments are wrapped in double quotes, so there are both outer double quotes around the variable syntax and inner double quotes around the values, json2hcl handles the hcl to json part fine (other than the extra arrays already reported in a different issue), but when converting that json back to hcl the escape characters are not removed.

example.tf:

data "template_file" "iam_instance_policies" {
  count    = "${length(split(",", var.child["instance_policy_paths"]))}"
  template = "${file(join("/", list(path.module, replace(element(split(",", var.child["instance_policy_paths"]), count.index), "\n", ""))))}"

  vars {
    partition = "${data.aws_partition.current.partition}"
    account   = "${lookup(var.account_ids, terraform.workspace)}"
  }
}

cat example.tf | json2hcl -reverse:

{
  "data": [
    {
      "template_file": [
        {
          "iam_instance_policies": [
            {
              "count": "${length(split(\",\", var.child[\"instance_policy_paths\"]))}",
              "template": "${file(join(\"/\", list(path.module, replace(element(split(\",\", var.child[\"instance_policy_paths\"]), count.index), \"\\n\", \"\"))))}",
              "vars": [
                {
                  "account": "${lookup(var.account_ids, terraform.workspace)}",
                  "partition": "${data.aws_partition.current.partition}"
                }
              ]
            }
          ]
        }
      ]
    }
  ]
}

cat example.tf | json2hcl -reverse | json2hcl:

"data" = {
  "template_file" = {
    "iam_instance_policies" = {
      "count" = "${length(split(\",\", var.child[\"instance_policy_paths\"]))}"

      "template" = "${file(join(\"/\", list(path.module, replace(element(split(\",\", var.child[\"instance_policy_paths\"]), count.index), \"\\n\", \"\"))))}"

      "vars" = {
        "account" = "${lookup(var.account_ids, terraform.workspace)}"

        "partition" = "${data.aws_partition.current.partition}"
      }
    }
  }
}

[Marius]

Are these escape characters in the resulting HCL harmful and cause it to be invalid or is it just an aesthetic concern?

[Loren Gordon]

The resulting HCL is invalid.

[Marius]

I have debugged this issue a bit and, if it's a bug, it originates in Hashicorp's HCL library (https://github.com/hashicorp/hcl). Since I am not well versed in HCL on my own, could you explain to me when a double quote needs to be escaped in HCL? Only when it's not inside a ${ ... } or never?

[Loren Gordon]

I'm on holiday this week and can't really test or research further as i don't have a computer. However, I cannot recall needing to escape double quotes at all in any of the HCL templates I've written.

[beatcracker]

I've similar issue with my own automated tool that generates JSON configuration.

Here is the generated configuration:

{
  "resource":[
    {
      "bitbucket_hook":{
        "bamboo":{
          "events":[
            "repo:push",
            "pullrequest:fulfilled"
          ],
          "url":"${format(var.bamboo_url, \"PRJ-KEY\", var.skip_branches ? \"true\" : \"false\")}",
          "description":"Trigger Bamboo build: PRJ-KEY",
          "owner":"${var.owner}",
          "repository":"${bitbucket_repository.PRJ-KEY}"
        }
      }
    }
  ]
}

This configuration is converted to:

"resource" "bitbucket_hook" "bamboo" {
  "events" = ["repo:push", "pullrequest:fulfilled"]
  "url" = "${format(var.bamboo_url, \"PRJ-KEY\", var.skip_branches ? \"true\" : \"false\")}"
  "description" = "Trigger Bamboo build: PRJ-KEY"
  "owner" = "${var.owner}"
  "repository" = "${bitbucket_repository.PRJ-KEY}"
}

As you can see, the arguments for the format function are not de-escaped. It should be this way:

  "url" = "${format(var.bamboo_url, "PRJ-KEY", var.skip_branches ? "true" : "false")}"

[Marius]

To be honest, I don't know HCL that well. Is "${format(var.bamboo_url, \"PRJ-KEY\", var.skip_branches ? \"true\" : \"false\")}" invalid?

[beatcracker]

Sorry, should've made that clear: yes, it's not a valid HCL.

[Marius]

This seems to be a problem in the upstream HCL package we use. So I reported the error there: hashicorp/hcl#268. However, this has already been reported but there was no further activity: hashicorp/hcl#233