CLI should return non-zero when stack is invalid

[Iain Samuel McLean Elder]

I'm setting up a quick demo infrastructure that includes an RDS instance. I hardcode the master password for my database because it's the easiest thing to do.

  RDSInstance:
    Type: AWS::RDS::DBInstance
    Properties:
      DBInstanceIdentifier: !Ref DBInstanceIdentifier
      DBInstanceClass: db.t3.micro
      Engine: mysql
      EngineVersion: 8.0.21
      DBSubnetGroupName: !Ref DBSubnetGroup
      PubliclyAccessible: false
      StorageEncrypted: true
      AllocatedStorage: "20"
      MasterUsername: root
      MasterUserPassword: password

CFRipper correctly calls me out on that and declares the stack invalid from a security perspective. Great!

$ cfripper stack.yml
Analysing stack.yml...
Valid: False
Issues found:
	- HardcodedRDSPasswordRule: RDS Instance password parameter missing NoEcho for RDSInstance.
$ echo $?
0

However, as you can see, the exit code is still 0.

That means this doesn't automatically trigger a failure in my CI system.

I was expecting to see a 1 or something else falsy.

I'm using Bitbucket Pipelines which passes each step if the exit code is 0 and fails otherwise. I believe most other CI systems work the same way.

Because this looks like a pass to Bitbucket Pipelines, I have to remember to inspect the output every time I want to know if there's a potential security issue.

So could we have CFRipper return a non-zero exit code when the stack is found to be invalid?

It could just be as simple as 0 for valid and 1 for invalid without any more detail.

The other tools in my pipeline work in a similar way, although they are more complex or nuanced that what is porbably needed in CFRipper.

• cfn_nag

  • A failing violation will return a non-zero exit code.
  • A warning will return a zero/success exit code.

• cfn-lint

  • 0 is no issue was found
  • 2 is an error
  • 4 is a warning
  • 6 is an error and a warning
  • 8 is an informational
  • 10 is an error and informational
  • 12 is an warning and informational
  • 14 is an error and a warning and an informational

[Jordi Soucheiron]

That's awesome feedback Iain, we'll fix it.

[Iain Samuel McLean Elder]

@jsoucheiron , @ocrawford555 , I see you addressed the issue in the #143. That's great.

However, the latest version of cfripper on PyPI (version 0.23.0) still returns 0 for invalid stacks.

Which version do I need to get the non-zero exit codes?

In the code comments I see documentation for the exit codes:

    Exit codes:
      - 0 = all templates valid
      - 1 = at least one template is not valid
      - 2 = error in scanning at least one template

Can we get this into the README?

[Oliver Crawford]

Hey, yeah I will update the README. It will be 0.23.1 which will be released once this is merged in (I imagine that will be tomorrow): #145

[Oliver Crawford]

@iainelder 0.23.1 now released and README updated

[Iain Samuel McLean Elder]

Yes, thanks, confirmed working with 0.23.1!

For anyone else reading this thread, the meaning of the error codes changed in #145.

From my informal local testing:

• a deployable CloudFormation template that passes all security checks returns 0
• a file that isn't a CloudFormation template returns 1
• a deployable CloudFormation template that fails a security check returns 2
• I was unable to provoke a return value of 3