steven-hadfield / tool-compare

tool-compare

In the world of infrastructure-as-code security there are several tools for users to choose from. The goal of this repository is to help compare the different options so that users can choose the tool that best fits their own needs.

What tools are there?

	Checkov	Indeni Cloudrail	Kics	Snyk	Terrascan	Tfsec
License	OSS	Freemium	OSS	Freemium	OSS	OSS

(there are others, anyone can add to this list, sorted A-Z)

How does this repo work?

This repository has a set of test-cases and a main script, called run_all_tools.sh which runs the above-listed tools against each of the test-cases. This allows any potential user to see what the tool can do, and how it compares, before even installing it.

Test case catch rate

The tables below list test cases included in this repository. For each case, it shows which tools are able to catch it specifically, and which don't.

Summary

Last update: 2021-05-14

	Checkov	Indeni Cloudrail	Kics	Snyk	Terrascan	Tfsec
Tested Version	2.0.135	1.2.130	1.3.0	1.563.0	1.4.0	0.39.34
Total Catch Rate	66%	69%	31%	43%	16%	53%

test-cases/terraform/aws/best-practices

Test Case	Checkov	Indeni Cloudrail	Kics	Snyk	Terrascan	Tfsec
alb_drop_http_headers	✅	❌	❌	❌	❌	✅
cloudfront_not_using_waf	✅	❌	✅	✅	❌	✅
cloudtrail_enabled_on_multi_region	✅	❌	✅	✅	❌	✅
config_aggregator_all_regions	✅	❌	❌	❌	❌	✅
deploy_ec2_to_default_vpc	✅	✅	✅	❌	✅	✅
deploy_redshift_in_ec2_classic_mode	✅	✅	❌	❌	❌	✅
dynamodb_without_recovery_enabled	✅	✅	❌	✅	❌	✅
ec2_ebs_not_optimized	✅	❌	❌	❌	❌	❌
ecr_make_tags_immutable	✅	❌	✅	✅	❌	✅
ecr_use_image_scanning	✅	❌	✅	✅	✅	✅
ecs_cluster_container_insights	✅	❌	❌	✅	❌	✅
elasticache_automatic_backup	✅	❌	❌	❌	❌	✅
kms_uses_rotation	✅	✅	❌	✅	❌	✅
rds_retention_period_set	✅	❌	❌	❌	❌	✅
security_group_no_description_for_rules	❌	✅	❌	❌	❌	✅
security_group_no_description_for_securi..	❌	✅	❌	✅	❌	✅
tag_all_items	❌	✅	❌	❌	❌	❌
using_public_amis	❌	✅	❌	❌	❌	❌
Category Catch Rate	78%	44%	28%	44%	11%	83%

test-cases/terraform/aws/encryption/at-rest

Test Case	Checkov	Indeni Cloudrail	Kics	Snyk	Terrascan	Tfsec
athena_not_encrypted	✅	❌	❌	✅	❌	✅
cloudtrail_not_encrypted	✅	✅	✅	✅	❌	✅
cloudwatch_groups_not_encrypted	✅	✅	❌	❌	✅	✅
codbuild_using_aws_key	❌	✅	❌	❌	❌	❌
dax_cluster_not_encrypted	✅	✅	❌	✅	❌	✅
docdb_cluster_encrypted_at_rest_using_cm..	❌	✅	❌	❌	❌	✅
docdb_cluster_encrypted_without_kms_key	❌	✅	❌	❌	❌	❌
docdb_clusters_non_encrypted	✅	✅	❌	✅	❌	❌
dynamodb_not_encrypted	✅	❌	❌	❌	❌	✅
ecr_repo_not_encrypted	✅	❌	❌	❌	❌	✅
elasticache_replication_group_not_encryp..	✅	✅	❌	✅	❌	✅
elasticsearch_not_encrypted	✅	✅	✅	✅	❌	✅
kinesis_stream_not_encrypted	✅	✅	✅	✅	❌	✅
neptune_cluster_no_encryption	✅	✅	✅	✅	❌	❌
rds_cluster_encrypt_at_rest_disabled	✅	✅	❌	❌	❌	✅
redshift_not_encrypted	✅	❌	✅	✅	✅	✅
rest_api_cache_non_encrypted	❌	✅	❌	❌	❌	❌
s3_bucket_non_encrypted	✅	✅	✅	✅	❌	✅
s3_bucket_object_non_encrypted	❌	✅	❌	❌	❌	❌
sagemaker_not_encrypted	✅	✅	❌	❌	❌	❌
secretsmanager_secrets_encrypted_at_rest..	✅	✅	❌	❌	❌	✅
secretsmanager_secrets_encrypted_at_rest..	❌	✅	❌	❌	❌	✅
sns_topic_encrypted_at_rest_with_aws_man..	❌	✅	❌	❌	❌	✅
sqs_queue_not_encrypted	✅	✅	✅	✅	❌	✅
workgroups_non_encrypted	✅	✅	❌	❌	❌	✅
workspace_root_volume_not_encrypted_at_r..	✅	✅	❌	❌	❌	✅
workspace_user_volume_not_encrypted_at_r..	✅	✅	❌	❌	❌	✅
Category Catch Rate	74%	85%	26%	41%	7%	74%

test-cases/terraform/aws/encryption/in-transit

Test Case	Checkov	Indeni Cloudrail	Kics	Snyk	Terrascan	Tfsec
alb_use_http	✅	✅	❌	❌	❌	✅
cloudfront_distribution_not_encrypted	✅	✅	✅	✅	✅	✅
cloudfront_protocol_version_is_low	✅	✅	✅	✅	✅	✅
ecs_task_definition_not_encrypted_in_tra..	✅	✅	❌	❌	❌	✅
elasticache_replication_group_not_encryp..	✅	✅	❌	✅	❌	✅
elasticsearch_encrypt_node_to_node_disab..	❌	✅	❌	✅	❌	✅
load_balancer_listener_http	✅	✅	✅	✅	❌	✅
vpc_has_only_dynamodb_vpce_gw_connection	❌	✅	❌	❌	❌	❌
Category Catch Rate	75%	100%	38%	62%	25%	88%

test-cases/terraform/aws/iam/iam-entities

Test Case	Checkov	Indeni Cloudrail	Kics	Snyk	Terrascan	Tfsec
human_users_defined	✅	✅	❌	❌	❌	❌
iam_user_inline_policy_attach	✅	✅	❌	✅	❌	❌
iam_user_managed_policy_direct_attachmen..	✅	✅	❌	✅	❌	❌
passrole_and_lambda_permissions_cause_pr..	❌	✅	❌	❌	❌	❌
policy-too-broad	❌	✅	❌	❌	❌	❌
policy_missing_principal	❌	✅	❌	❌	❌	❌
public_and_private_ec2_same_role	❌	✅	❌	❌	❌	❌
Category Catch Rate	43%	100%	0%	29%	0%	0%

test-cases/terraform/aws/iam/resource-authentication

Test Case	Checkov	Indeni Cloudrail	Kics	Snyk	Terrascan	Tfsec
rds_without_authentication	❌	❌	✅	❌	✅	❌
rest_api_without_authorization	✅	❌	✅	✅	❌	❌
Category Catch Rate	50%	0%	100%	50%	50%	0%

test-cases/terraform/aws/iam/resource-policies

Test Case	Checkov	Indeni Cloudrail	Kics	Snyk	Terrascan	Tfsec
cloudwatch_log_destination_insecure_poli..	❌	✅	❌	❌	❌	❌
ecr_not_secure_policy	❌	✅	❌	❌	❌	❌
efs_not_secure_policy	❌	✅	❌	❌	❌	❌
elasticsearch_domain_not_secure_policy	❌	✅	❌	❌	❌	❌
glue_data_catalog_not_secure_policy	❌	✅	❌	❌	❌	❌
kms_key_not_secure_policy	❌	✅	❌	❌	❌	❌
lambda_not_secure_policy	❌	✅	❌	❌	❌	❌
rest_api_not_secure_policy	❌	✅	❌	❌	❌	❌
s3_bucket_acl_public_all_authenticated_u..	✅	✅	✅	✅	✅	✅
s3_bucket_acl_public_all_users_canned	✅	✅	✅	✅	✅	✅
s3_bucket_acl_public_all_users_canned_wi..	❌	✅	✅	❌	❌	❌
s3_bucket_policy_public_to_all_authentic..	❌	✅	✅	❌	✅	❌
secrets_manager_not_secure_policy	❌	✅	❌	❌	❌	❌
Category Catch Rate	15%	100%	31%	15%	23%	15%

test-cases/terraform/aws/logging

Test Case	Checkov	Indeni Cloudrail	Kics	Snyk	Terrascan	Tfsec
api_gateway_no_xray	✅	❌	✅	✅	✅	❌
cloudfront_distribution_without_logging	✅	❌	✅	✅	✅	✅
cloudtrail_file_log_validation_disabled	✅	✅	✅	✅	❌	✅
cloudwatch_log_groups_no_retention	✅	✅	✅	✅	❌	❌
docdb_audit_logs_missing	✅	❌	❌	❌	❌	❌
ec2_without_monitoring	✅	❌	❌	❌	❌	❌
eks_logging_disabled	✅	✅	❌	✅	❌	✅
elasticsearch_domain_logging_disabled	✅	❌	❌	✅	✅	✅
elb_without_access_logs	✅	❌	❌	❌	❌	❌
globalaccelerator_accelerator_no_flow_lo..	✅	❌	❌	✅	❌	❌
lambda_without_explicit_log_group	❌	✅	❌	❌	❌	❌
lambda_without_xray	✅	❌	✅	✅	✅	❌
neptune_cluster_no_logging	✅	❌	❌	❌	❌	❌
rds_without_logging	✅	❌	❌	❌	❌	❌
redshift_without_logging	✅	❌	✅	✅	❌	❌
rest_api_no_access_logging	✅	❌	✅	✅	✅	✅
s3_access_logging_disabled	✅	❌	✅	✅	❌	✅
Category Catch Rate	94%	24%	47%	65%	29%	35%

test-cases/terraform/aws/networking/vpc-endpoints

Test Case	Checkov	Indeni Cloudrail	Kics	Snyk	Terrascan	Tfsec
dynamodb-vpce-exist-without-routeassocia..	❌	✅	❌	❌	❌	❌
sqs-vpc-endpoint-without-dns-resolution	❌	✅	❌	❌	❌	❌
Category Catch Rate	0%	100%	0%	0%	0%	0%

Contributing

Anyone can contribute to this repository. The main areas of contribution are:

• Adding an additional tool - simply add the tool to this readme and the run_all_tools.sh script. Then, execute that script and add all of its results as part of your PR. That's it, you're good to go.

• Adding test-cases - you can add the test case in the correct spot in the tree under test-cases and run the run_all_tools.sh script against it. Make sure to include all of the tools' results as part of your PR.

NOTE: This repository has been initiated by @yi2020, CEO & Founder of Indeni, the company behind Indeni Cloudrail. While this was initiated by an employee of a vendor in the community, the intention is for this repository to be neutral and truly serve as a non-biased comparison tool of products offered. Contributions that help users make that choice, and are unbiased in nature, are very welcome. The aspiration is that over time all vendors will become equal contributors in this repository.