kitchen converge fails in GitHub actions environment

Question

kitchen converge fails in GitHub actions environment

fdansey-ostmodern opened this issue 4 years ago · comments

Howdy. Running the kitchen converge command locally (using the example terraform code) works, but when I come to automate this in GitHub Actions, it fails after having created the resources in terraform with error,

$$$$$$ Parsing the Terraform output variables as JSON...
>>>>>> Parsing the Terraform output variables as JSON failed.
>>>>>> ------Exception-------
>>>>>> Class: Kitchen::ActionFailed
>>>>>> Message: 1 actions failed.
>>>>>>     Converge failed on instance <extensive-suite-aws>.  Please see .kitchen/logs/extensive-suite-aws.log for more details```

I have tried formatting the terraform output to be JSON and also removing the output completely with no success.

I have checked that the program versions between environments are the same, they are:
Terraform v0.11.14
Test Kitchen version 2.5.3
gem 2.7.6
ruby 2.5.1p57
Bundler version 2.1.4

Please advise

Edward Bartholomew · Answer 1 · Fri Jul 24 2020 19:36:20 GMT+0800 (China Standard Time)

Thank you for your interest in kitchen-terraform @fdansey-ostmodern Do you have that example with Actions in a public repo, that we can look at to help troubleshoot? I'm busy for the next few days but am happy to help debug early next week.

fdansey-ostmodern · Answer 2 · Fri Jul 24 2020 23:22:01 GMT+0800 (China Standard Time)

Howdy, I'm afraid the repo in which I am working is private. However, I can tell you I am running these four setup commands on the ubuntu-18.04 virtual env:

gem install bundler
bundle install
kitchen create
kitchen converge

I tried running kitchen via bundle exec and also completely changed the terraform I was testing without success.

Aaron Lane · Answer 3 · Sat Jul 25 2020 00:26:38 GMT+0800 (China Standard Time)

Hi @fdansey-ostmodern! You should run with bundle exec to ensure that all dependencies are correctly resolved. Can you try running kitchen converge -l debug and share the output?

fdansey-ostmodern · Answer 4 · Mon Jul 27 2020 20:17:58 GMT+0800 (China Standard Time)

Howdy, this is partial output when I added the -l debug argument:

...
Apply complete! Resources: 2 added, 0 changed, 0 destroyed.
       
       Outputs:
       
       example = example
       example2 = blarn
D      Finished running command `terraform apply -auto-approve -lock=true -lock-timeout=0s -input=false -no-color -parallelism=10 -refresh=true  ` in 0.4400000000000002 seconds.
$$$$$$ Finished building the infrastructure based on the Terraform configuration.
$$$$$$ Reading the output variables from the Terraform state...
D      Running command `terraform output -json` in directory /home/runner/work/<GITHUB_REPO>/<GITHUB_REPO>/example with a timeout of 600 seconds...
ommand]/home/runner/work/_temp/7427c1c0-d26c-43f4-bbcd-09e5eea1a04a/terraform-bin output -json

D      {
    "example": {
        "sensitive": false,
        "type": "string",
        "value": "example"
    },
    "example2": {
        "sensitive": false,
        "type": "string",
        "value": "blarn"
    }
}

D      ::debug::Terraform exited with code 0.

D      ::debug::exitcode: 0

D      ::set-output name=stderr::

D      Finished running command `terraform output -json` in 0.05 seconds.
$$$$$$ Finished reading the output variables from the Terraform state.
$$$$$$ Parsing the Terraform output variables as JSON...
>>>>>> Parsing the Terraform output variables as JSON failed.
>>>>>> ------Exception-------
>>>>>> Class: Kitchen::ActionFailed
>>>>>> Message: 1 actions failed.
>>>>>>     Converge failed on instance <extensive-suite-aws>.  Please see .kitchen/logs/extensive-suite-aws.log for more details
>>>>>> ----------------------
>>>>>> Please see .kitchen/logs/kitchen.log for more details
>>>>>> Also try running `kitchen diagnose --all` for configuration
D      ------Exception-------
D      Class: Kitchen::ActionFailed
D      Message: 1 actions failed.
>>>>>>     Converge failed on instance <extensive-suite-aws>.  Please see .kitchen/logs/extensive-suite-aws.log for more details
...

Aaron Lane · Answer 5 · Mon Jul 27 2020 23:31:36 GMT+0800 (China Standard Time)

@fdansey-ostmodern thank you! Can you please confirm which version of Kitchen-Terraform is present in the Gemfile.lock file?

fdansey-ostmodern · Answer 6 · Mon Jul 27 2020 23:56:28 GMT+0800 (China Standard Time)

The specs section of the Gemfile.lock includes: kitchen-terraform (5.4.0)

fdansey-ostmodern · Answer 7 · Fri Jul 31 2020 00:24:46 GMT+0800 (China Standard Time)

I also tried using a self-hosted runner. Running the commands directly on the host yielded the same error, however when I used a docker container action, everything worked as expected including the InSpec tests.

Aaron Lane · Answer 8 · Thu Aug 13 2020 07:32:13 GMT+0800 (China Standard Time)

@fdansey-ostmodern sorry for the delay in responding! Are you able to share reproductions that we may poke at?

amitkumardube · Answer 9 · Sat Apr 24 2021 17:32:01 GMT+0800 (China Standard Time)

@aaron-lane , @edwardbartholomew - I have the similar issue. This is under a public repo. Below is the URL. Do you think you can help on this please?

D ::debug::Terraform exited with code 0.

D ::debug::stdout: ***%0A

D ::set-output name=stdout::***%0A

D Finished running command terraform output -json in 0.3300000000000001 seconds.
$$$$$$ Finished reading the output variables from the Terraform state.
$$$$$$ Parsing the Terraform output variables as JSON...

Parsing the Terraform output variables as JSON failed.

https://github.com/amitkumardube/terraform-misc/runs/2426133878?check_suite_focus=true

Aaron Lane · Answer 10 · Sat Apr 24 2021 20:59:03 GMT+0800 (China Standard Time)

Hi @amitkumardube! Could you please manually provision the secret fixture and share the output of terraform output -json using the same version of Terraform as is used in your pipeline?

amitkumardube · Answer 11 · Sat Apr 24 2021 22:12:09 GMT+0800 (China Standard Time)

@aaron-lane - thanks for looking into this.
I tried with Terraform version 0.14.4 at both local and pipeline. Below is the output for local.

C:\Users\User\Downloads\terraform_0.14.4_windows_amd64\terraform-misc\test\fixtures\secret>terraform output -json
{
"project_id": {
"sensitive": false,
"type": "string",
"value": "my-first-project-298218"
},
"secret_name": {
"sensitive": false,
"type": "string",
"value": "secret_test"
}
}

The output of pipeline is still same -
https://github.com/amitkumardube/terraform-misc/runs/2427103700?check_suite_focus=true

D ::set-output name=stderr::

D Finished running command terraform output -json in 0.3000000000000001 seconds.
$$$$$$ Finished reading the output variables from the Terraform state.
$$$$$$ Parsing the Terraform output variables as JSON...

Parsing the Terraform output variables as JSON failed.

amitkumardube · Answer 12 · Wed Apr 28 2021 04:27:09 GMT+0800 (China Standard Time)

@aaron-lane - Sorry for follow-up. Wondering if you got some clue around why this is happening?

Aaron Lane · Answer 13 · Wed Apr 28 2021 05:03:35 GMT+0800 (China Standard Time)

I don't have a firm explanation yet, but this log message from your latest build looks suspicious:

***
"project_id": ***
"sensitive": false,
"type": "string",
"value": "my-first-project-298218"
***,
"secret_name": ***
"sensitive": false,
"type": "string",
"value": "secret_test"
***
***
https://github.com/amitkumardube/terraform-misc/runs/2427103700?check_suite_focus=true#step:8:97

It may just be the way that GitHub Actions is rendering the message, but it appears like it is modifying the output of terraform output -json by replacing brace characters { and } with ***.

amitkumardube · Answer 14 · Wed Apr 28 2021 16:32:25 GMT+0800 (China Standard Time)

@aaron-lane - No worries . thanks for your time.

I got a workaround in the meantime by adding a container and running all the steps inside a container and there it seems to be working fine. Below is the change I made if you like to have a look.

https://github.com/amitkumardube/terraform-misc/blob/master/.github/workflows/gcloud.yml#L22

Below is the successful run
https://github.com/amitkumardube/terraform-misc/runs/2455737193?check_suite_focus=true

jnahelou · Answer 15 · Thu Oct 28 2021 22:47:52 GMT+0800 (China Standard Time)

Hi @amitkumardube, @aaron-lane

I had the same issue using the hashicorp/setup-terraform action to install terraform.

Following hashicorp/setup-terraform/issues/20, you can see that hashicorp wrap terraform binary, that's why there is a different behavior.
I imagine that using your image, you install terraform using curl instead of the GH action.

You can disable the wrapper :

    - name: Setup Terraform
      uses: hashicorp/setup-terraform@v1
      with:
        terraform_wrapper: false

After that, kitchen works as expected.

Hope it will help other kitchen-terraform users :)

Aaron Lane · Answer 16 · Thu Oct 28 2021 23:15:39 GMT+0800 (China Standard Time)

@jnahelou thank you for confirming the fix!