Giters

cloud-custodian / cloud-custodian

Rules engine for cloud security, cost optimization, and governance, DSL in yaml for policies to query, filter, and take actions on resources

Home Page:https://cloudcustodian.io

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Slack API and Token

davidclin opened this issue · comments

commented

Per Custodian Mailer there is a section that covers Slack support.

By way of context, we have an existing Slack app that makes calls to a Slack Incoming Webhook to post to channels.

Based on Slack's announcement of deprecating vintage tokens (of which 'xoxo-' is noted among them), several questions arise:

(1) Vintage Token Support
Is this the only supported implementation today?

queue_url: https://sqs.us-east-1.amazonaws.com/1234567890/c7n-mailer-test
role: arn:aws:iam::123456790:role/c7n-mailer-test
slack_token: xoxo-token123

(2) Incoming Webhook Support
Does Cloud Custodian also support Incoming Webhook and, if so, is there any example of how to use it?

(3) Slack Permissions/Scope
Has anyone used either the vintage or Incoming Webhook (assuming support exists) approaches with success and can share what level of permissions and scope were needed to get things working?

(4) Alternatives
Are there alternative/short-term suggestions and/or examples on getting Cloud Custodian to integrate with Slack using an Incoming Webhook (assuming unsupported)?

Thanks!

commented

I believe the intent was not 1) vintage tokens but 4) using a oauth scope bot token (see https://api.slack.com/docs/oauth#bots ). The current slack transport, delivers individual user private messages potentially looking up via ldap to get email from a user id tag on a resource (or pass through if email is present), and using email to get a slack user id.

The web hook support for more static room style delivery is a desired feature enhancement

commented

also wrt to delivery or integration to custom channels that aren't currently implemented, the invoke-lambda action provides a means of integrating custom code, of course pull requests for the mailer around better slack integration are also welcome.

commented

I've opened a PR to add basic webhook support for the Slack integration. #2350 @davidclin can you review and confirm whether this satisfies your use case? Thanks!

commented

Thank you, Lykins! I'm tied up at the moment but will get back to you very soon. I also have a colleague who has been doing both bot user and webhook Slack integrations recently. I plan on leveraging some of her experience and have her review alongside me to make sure it covers what we think we'll need. Appreciate the quick response! Amazing.

commented

Aside, I attempted to run the c7n-mailer with a slack bot and got the following msg:

jsonschema.exceptions.ValidationError: Additional properties are not allowed ('slack_token' was unexpected)

It's likely I'm missing something, but I don't know where.

See below for details (click triangles to expand):

c7n-mailer --config mailer.yml --update-lambda && custodian run -c slack-notify.yml 
(c7n_mailer) $ c7n-mailer --config mailer.yml --update-lambda && custodian run -c slack-notify.yml
Traceback (most recent call last):
  File "/home/ec2-user/c7n_mailer/bin/c7n-mailer", line 11, in 
    load_entry_point('c7n-mailer', 'console_scripts', 'c7n-mailer')()
  File "/home/ec2-user/cloud-custodian/tools/c7n_mailer/c7n_mailer/cli.py", line 131, in main
    mailer_config = get_and_validate_mailer_config(args)
  File "/home/ec2-user/cloud-custodian/tools/c7n_mailer/c7n_mailer/cli.py", line 95, in get_and_validate_mailer_config
    jsonschema.validate(config, CONFIG_SCHEMA)
  File "/home/ec2-user/c7n_mailer/local/lib/python2.7/site-packages/jsonschema/validators.py", line 541, in validate
    cls(schema, *args, **kwargs).validate(instance)
  File "/home/ec2-user/c7n_mailer/local/lib/python2.7/site-packages/jsonschema/validators.py", line 130, in validate
    raise error
jsonschema.exceptions.ValidationError: Additional properties are not allowed ('slack_token' was unexpected)

Failed validating u'additionalProperties' in schema:

{u'additionalProperties': False,

u'properties': {u'account_emails': {u'type': u'object'},

u'cache_engine': {u'type': u'string'},

u'contact_tags': {u'items': {u'type': u'string'},

u'type': u'array'},

u'cross_accounts': {u'type': u'object'},

u'datadog_api_key': {u'type': u'string'},

u'datadog_application_key': {u'type': u'string'},

u'dead_letter_config': {u'type': u'object'},

u'debug': {u'type': u'boolean'},

u'from_address': {u'type': u'string'},

u'http_proxy': {u'type': u'string'},

u'https_proxy': {u'type': u'string'},

u'lambda_description': {u'type': u'string'},

u'lambda_name': {u'type': u'string'},

u'lambda_schedule': {u'type': u'string'},

u'lambda_tags': {u'type': u'object'},

u'ldap_bind_dn': {u'type': u'string'},

u'ldap_bind_password': {u'type': u'string'},

u'ldap_bind_password_in_kms': {u'type': u'boolean'},

u'ldap_bind_user': {u'type': u'string'},

u'ldap_email_attribute': {u'type': u'string'},

u'ldap_email_key': {u'type': u'string'},

u'ldap_manager_attribute': {u'type': u'string'},

u'ldap_uid_attribute': {u'type': u'string'},

u'ldap_uid_regex': {u'type': u'string'},

u'ldap_uid_tags': {u'items': {u'type': u'string'},

u'type': u'array'},

u'ldap_uri': {u'type': u'string'},

u'memory': {u'type': u'integer'},

u'profile': {u'type': u'string'},

u'queue_url': {u'type': u'string'},

u'redis_host': {u'type': u'string'},

u'redis_port': {u'type': u'integer'},

u'region': {u'type': u'string'},

u'role': {u'type': u'string'},

u'runtime': {u'type': u'string'},

u'security_groups': {u'items': {u'type': u'string'},

u'type': u'array'},

u'ses_region': {u'type': u'string'},

u'smtp_password': {u'type': u'string'},

u'smtp_port': {u'type': u'integer'},

u'smtp_server': {u'type': u'string'},

u'smtp_ssl': {u'type': u'boolean'},

u'smtp_username': {u'type': u'string'},

u'subnets': {u'items': {u'type': u'string'},

u'type': u'array'},

u'timeout': {u'type': u'integer'}},

u'required': [u'queue_url', u'role'],

u'type': u'object'}


On instance:

{'contact_tags': ['OwnerContact', 'OwnerEmail', 'SNSTopicARN'],

'from_address': 'david.lin.ctr@tri.global',

'queue_url': 'https://sqs.us-east-1.amazonaws.com/xxxxxxxxxxxx/sandbox',

'region': 'us-east-1',

'role': 'arn:aws:iam::xxxxxxxxxxxx:role/CloudCustodianRole',

'slack_token': 'xoxb-slack_token'}

mailer.yml 
queue_url: https://sqs.us-east-1.amazonaws.com/xxxxxxxxxxxx/sandbox
from_address: email@address.com
contact_tags:
  - OwnerContact
  - OwnerEmail
  - SNSTopicARN
region: us-east-1
role: arn:aws:iam::xxxxxxxxxxxx:role/CloudCustodianRole
slack_token: xoxb-slack_token
slack-notify.yml 
policies:
  - name: security-groups-unused
    resource: security-group
    description: |
      Retrieve unused security groups using regex.
      Notify using Slack.
    filters:
      - unused
      - type: value
        key: GroupName
        op: regex
        value: .*
    actions:
      - type: notify
        template_slack: slack_default
        template_format: 'html'
        priority_header: '5'
        subject: 'CloudCustodian: Unused Security Groups'
        to:
          - slack://#my-slack-channel
        owner_absent_contact:
          - email@address.com
        transport:
          type: sqs
          queue: https://sqs.us-east-1.amazonaws.com/xxxxxxxxxxxx/sandbox
commented

Are you running from the PR branch? It looks like your code isn't picking up the schema change for slack_token.

commented

I'm using the branch that comes with the instructions provided in the Developer Install (OS X El Capitan).

I assumed that would be enough based on the documentation.
If not, what additional steps do I need to take in order to pick up the schema change?

commented

The change was merged into master a few commits ago, but the webhook integration is still outstanding as a PR, on one of my branches. You can do a git clone against my development branch to test it: https://github.com/LykinsN/cloud-custodian/tree/slack-webhook

commented

Also, you're welcome to reach out to me on Gitter and we can collaborate directly.

commented

I cloned https://github.com/LykinsN/cloud-custodian.git and installed Cloud Custodian and c7n_mailer from a clean EC2 instance.

The slack_token schema error is gone now.

I'm now seeing the following in the head and tail of the CloudWatch logs for the cloud-custodian-mailer lambda:

Error: [Errno 2] No such file or directory (appears in head of log)

SSLError: [Errno 2] No such file or directory (appears in tail of log)

This is probably unrelated, but I forgot to mention that I've been getting UserWarning msgs every time I invoke the c7n-mailer as well:

UserWarning: Duplicate name: 'ruamel/yaml/some_name.py

I've been ignoring these user warnings from the day I first started, and it's never stopped me from moving forward. Not sure if it's related but worth mentioning since I'm in the thick of things now.

Latest artifacts
CloudWatch Log 
[ERROR]	2018-05-09T07:28:08.363Z	7f2b82d9-535a-11e8-af84-0f073e98d6ca	Error starting mailer MailerSqsQueueProcessor(). 
Error: [Errno 2] No such file or directory 

Traceback (most recent call last):

File "/var/task/c7n_mailer/handle.py", line 46, in start_c7n_mailer

mailer_sqs_queue_processor.run(parallel)

File "/var/task/c7n_mailer/sqs_queue_processor.py", line 113, in run

for sqs_message in sqs_messages:

File "/var/task/c7n_mailer/sqs_queue_processor.py", line 56, in next

MessageAttributeNames=self.msg_attributes)

File "/var/runtime/botocore/client.py", line 314, in _api_call

return self._make_api_call(operation_name, kwargs)

File "/var/runtime/botocore/client.py", line 599, in _make_api_call

operation_model, request_dict)

File "/var/runtime/botocore/endpoint.py", line 148, in make_request

return self._send_request(request_dict, operation_model)

File "/var/runtime/botocore/endpoint.py", line 177, in _send_request

success_response, exception):

File "/var/runtime/botocore/endpoint.py", line 273, in _needs_retry

caught_exception=caught_exception, request_dict=request_dict)

File "/var/runtime/botocore/hooks.py", line 227, in emit

return self._emit(event_name, kwargs)

File "/var/runtime/botocore/hooks.py", line 210, in _emit

response = handler(**kwargs)

File "/var/runtime/botocore/retryhandler.py", line 183, in call

if self._checker(attempts, response, caught_exception):

File "/var/runtime/botocore/retryhandler.py", line 251, in call

caught_exception)

File "/var/runtime/botocore/retryhandler.py", line 277, in _should_retry

return self._checker(attempt_number, response, caught_exception)

File "/var/runtime/botocore/retryhandler.py", line 317, in call

caught_exception)

File "/var/runtime/botocore/retryhandler.py", line 223, in call

attempt_number, caught_exception)

File "/var/runtime/botocore/retryhandler.py", line 359, in _check_caught_exception

raise caught_exception

SSLError: [Errno 2] No such file or directory

UserWarning: Duplicate name: 'ruamel/yaml/*.py 
9: UserWarning: Duplicate name: 'ruamel/yaml/reader.py'
  self._zip_file.writestr(dest, contents)
/home/ec2-user/cloud-custodian/c7n_mailer/local/lib/python2.7/site-packages/c7n/mu.py:17                                   
9: UserWarning: Duplicate name: 'ruamel/yaml/cyaml.py'
  self._zip_file.writestr(dest, contents)
/home/ec2-user/cloud-custodian/c7n_mailer/local/lib/python2.7/site-packages/c7n/mu.py:17                                   
9: UserWarning: Duplicate name: 'ruamel/yaml/resolver.py'
  self._zip_file.writestr(dest, contents)
/home/ec2-user/cloud-custodian/c7n_mailer/local/lib/python2.7/site-packages/c7n/mu.py:17                                   
9: UserWarning: Duplicate name: 'ruamel/yaml/scalarint.py'
  self._zip_file.writestr(dest, contents)
/home/ec2-user/cloud-custodian/c7n_mailer/local/lib/python2.7/site-packages/c7n/mu.py:17                                   
9: UserWarning: Duplicate name: 'ruamel/yaml/composer.py'
  self._zip_file.writestr(dest, contents)
/home/ec2-user/cloud-custodian/c7n_mailer/local/lib/python2.7/site-packages/c7n/mu.py:17                                   
9: UserWarning: Duplicate name: 'ruamel/yaml/scalarstring.py'
  self._zip_file.writestr(dest, contents)
/home/ec2-user/cloud-custodian/c7n_mailer/local/lib/python2.7/site-packages/c7n/mu.py:17                                   
9: UserWarning: Duplicate name: 'ruamel/yaml/timestamp.py'
  self._zip_file.writestr(dest, contents)
/home/ec2-user/cloud-custodian/c7n_mailer/local/lib/python2.7/site-packages/c7n/mu.py:17                                   
9: UserWarning: Duplicate name: 'ruamel/yaml/__init__.py'
  self._zip_file.writestr(dest, contents)
/home/ec2-user/cloud-custodian/c7n_mailer/local/lib/python2.7/site-packages/c7n/mu.py:17                                   
9: UserWarning: Duplicate name: 'ruamel/yaml/constructor.py'
  self._zip_file.writestr(dest, contents)
/home/ec2-user/cloud-custodian/c7n_mailer/local/lib/python2.7/site-packages/c7n/mu.py:17                                   
9: UserWarning: Duplicate name: 'ruamel/yaml/parser.py'
  self._zip_file.writestr(dest, contents)
/home/ec2-user/cloud-custodian/c7n_mailer/local/lib/python2.7/site-packages/c7n/mu.py:17                                   
9: UserWarning: Duplicate name: 'ruamel/yaml/comments.py'
  self._zip_file.writestr(dest, contents)
/home/ec2-user/cloud-custodian/c7n_mailer/local/lib/python2.7/site-packages/c7n/mu.py:17                                   
9: UserWarning: Duplicate name: 'ruamel/yaml/representer.py'
  self._zip_file.writestr(dest, contents)
/home/ec2-user/cloud-custodian/c7n_mailer/local/lib/python2.7/site-packages/c7n/mu.py:17                                   
9: UserWarning: Duplicate name: 'ruamel/yaml/compat.py'
  self._zip_file.writestr(dest, contents)
/home/ec2-user/cloud-custodian/c7n_mailer/local/lib/python2.7/site-packages/c7n/mu.py:17                                   
9: UserWarning: Duplicate name: 'ruamel/yaml/main.py'
  self._zip_file.writestr(dest, contents)
/home/ec2-user/cloud-custodian/c7n_mailer/local/lib/python2.7/site-packages/c7n/mu.py:17                                   
9: UserWarning: Duplicate name: 'ruamel/yaml/emitter.py'
  self._zip_file.writestr(dest, contents)
/home/ec2-user/cloud-custodian/c7n_mailer/local/lib/python2.7/site-packages/c7n/mu.py:17                                   
9: UserWarning: Duplicate name: 'ruamel/yaml/scanner.py'
  self._zip_file.writestr(dest, contents)
/home/ec2-user/cloud-custodian/c7n_mailer/local/lib/python2.7/site-packages/c7n/mu.py:17                                   
9: UserWarning: Duplicate name: 'ruamel/yaml/tokens.py'
  self._zip_file.writestr(dest, contents)
/home/ec2-user/cloud-custodian/c7n_mailer/local/lib/python2.7/site-packages/c7n/mu.py:17                                   
9: UserWarning: Duplicate name: 'ruamel/yaml/configobjwalker.py'
  self._zip_file.writestr(dest, contents)
/home/ec2-user/cloud-custodian/c7n_mailer/local/lib/python2.7/site-packages/c7n/mu.py:17                                   
9: UserWarning: Duplicate name: 'ruamel/yaml/loader.py'
  self._zip_file.writestr(dest, contents)
/home/ec2-user/cloud-custodian/c7n_mailer/local/lib/python2.7/site-packages/c7n/mu.py:17                                   
9: UserWarning: Duplicate name: 'ruamel/yaml/dumper.py'
  self._zip_file.writestr(dest, contents)
/home/ec2-user/cloud-custodian/c7n_mailer/local/lib/python2.7/site-packages/c7n/mu.py:17                                   
9: UserWarning: Duplicate name: 'ruamel/yaml/events.py'
  self._zip_file.writestr(dest, contents)
/home/ec2-user/cloud-custodian/c7n_mailer/local/lib/python2.7/site-packages/c7n/mu.py:17                                   
9: UserWarning: Duplicate name: 'ruamel/yaml/nodes.py'
  self._zip_file.writestr(dest, contents)
/home/ec2-user/cloud-custodian/c7n_mailer/local/lib/python2.7/site-packages/c7n/mu.py:17                                   
9: UserWarning: Duplicate name: 'ruamel/yaml/util.py'
  self._zip_file.writestr(dest, contents)
/home/ec2-user/cloud-custodian/c7n_mailer/local/lib/python2.7/site-packages/c7n/mu.py:17                                   
9: UserWarning: Duplicate name: 'ruamel/yaml/error.py'
  self._zip_file.writestr(dest, contents)
/home/ec2-user/cloud-custodian/c7n_mailer/local/lib/python2.7/site-packages/c7n/mu.py:17                                   
9: UserWarning: Duplicate name: 'ruamel/yaml/serializer.py'
  self._zip_file.writestr(dest, contents)
/home/ec2-user/cloud-custodian/c7n_mailer/local/lib/python2.7/site-packages/c7n/mu.py:17                                   
9: UserWarning: Duplicate name: 'ruamel/ordereddict/__init__.py'
  self._zip_file.writestr(dest, contents)
c7n-mailer --config mailer_sandbox.yml --update-lambda && custodian run -c slack-notify.yml -s . 
(c7n_mailer) $ c7n-mailer --config mailer_sandbox.yml --update-lambda && custodian run -c slack-notify.yml -s .
2018-05-09 07:45:07,264: custodian.policy:INFO policy: slack-test resource:security-group region:us-east-1 count:10 time:1.56
2018-05-09 07:45:07,524: custodian.actions:INFO sent message:e906df63-936c-45a3-b225-72b71c9f0985 policy:slack-test template:default_html count:10
2018-05-09 07:45:07,525: custodian.policy:INFO policy: slack-test action: notify resources: 10 execution_time: 0.26
mailer_sandbox.yml 
queue_url: https://sqs.us-east-1.amazonaws.com/1234567890/sandbox
from_address: email@address.com
contact_tags:
  - OwnerContact
  - OwnerEmail
  - SNSTopicARN
region: us-east-1
role: arn:aws:iam::1234567890:role/CloudCustodianRole
slack_token: xoxb-slack_token
slack-notify.yml 
policies:
  - name: slack-test
    resource: security-group
    description: |
      Retrieve unused security groups using regex.
      Notify using Slack.
    filters:
      - unused
      - type: value
        key: GroupName
        op: regex
        value: .*
    actions:
      - type: notify
        template: default_html
        template_format: 'html'
        priority_header: '5'
        subject: 'CloudCustodian: Unused Security Groups'
        to:
          - email@address.com
          - slack://#slack-channel
        owner_absent_contact:
          - email@address.com
        transport:
          type: sqs
          queue: https://sqs.us-east-1.amazonaws.com/1234567890/sandbox
commented

Thanks for the feedback. I suspect there might be an issue with some of the updated lambda dependencies. I've opened a new branch https://github.com/LykinsN/cloud-custodian/commits/mailer-test and reverted that set of changes. Can you try against that one?

commented

The errors are gone.

Now I'm getting:

OSError: [Errno 13] Permission denied: './slack-notify' (where slack-notify is the name of my policy)

when executing the c7n-mailer command.

c7n-mailer --config mailer_iesandbox.yml --update-lambda && custodian run -c slack-notify.yml -s . 
(c7n_mailer) $ c7n-mailer --config mailer_iesandbox.yml --update-lambda && custodian run -c slack-notify.yml -s .
/home/ec2-user/cloud-custodian/c7n_mailer/local/lib/python2.7/site-packages/c7n/mu.py:179: 
UserWarning: Duplicate name: 'ruamel/yaml/reader.py'
  self._zip_file.writestr(dest, contents)
/home/ec2-user/cloud-custodian/c7n_mailer/local/lib/python2.7/site-packages/c7n/mu.py:179: 
UserWarning: Duplicate name: 'ruamel/yaml/cyaml.py'
  self._zip_file.writestr(dest, contents)
/home/ec2-user/cloud-custodian/c7n_mailer/local/lib/python2.7/site-packages/c7n/mu.py:179: 
UserWarning: Duplicate name: 'ruamel/yaml/resolver.py'
  self._zip_file.writestr(dest, contents)
/home/ec2-user/cloud-custodian/c7n_mailer/local/lib/python2.7/site-packages/c7n/mu.py:179: 
UserWarning: Duplicate name: 'ruamel/yaml/scalarint.py'
  self._zip_file.writestr(dest, contents)
/home/ec2-user/cloud-custodian/c7n_mailer/local/lib/python2.7/site-packages/c7n/mu.py:179: UserWarning: Duplicate name: 'ruamel/yaml/composer.py'
  self._zip_file.writestr(dest, contents)
/home/ec2-user/cloud-custodian/c7n_mailer/local/lib/python2.7/site-packages/c7n/mu.py:179: 
UserWarning: Duplicate name: 'ruamel/yaml/scalarstring.py'
  self._zip_file.writestr(dest, contents)
/home/ec2-user/cloud-custodian/c7n_mailer/local/lib/python2.7/site-packages/c7n/mu.py:179: 
UserWarning: Duplicate name: 'ruamel/yaml/timestamp.py'
  self._zip_file.writestr(dest, contents)
/home/ec2-user/cloud-custodian/c7n_mailer/local/lib/python2.7/site-packages/c7n/mu.py:179: 
UserWarning: Duplicate name: 'ruamel/yaml/__init__.py'
  self._zip_file.writestr(dest, contents)
/home/ec2-user/cloud-custodian/c7n_mailer/local/lib/python2.7/site-packages/c7n/mu.py:179: 
UserWarning: Duplicate name: 'ruamel/yaml/constructor.py'
  self._zip_file.writestr(dest, contents)
/home/ec2-user/cloud-custodian/c7n_mailer/local/lib/python2.7/site-packages/c7n/mu.py:179: 
UserWarning: Duplicate name: 'ruamel/yaml/parser.py'
  self._zip_file.writestr(dest, contents)
/home/ec2-user/cloud-custodian/c7n_mailer/local/lib/python2.7/site-packages/c7n/mu.py:179: 
UserWarning: Duplicate name: 'ruamel/yaml/comments.py'
  self._zip_file.writestr(dest, contents)
/home/ec2-user/cloud-custodian/c7n_mailer/local/lib/python2.7/site-packages/c7n/mu.py:179: 
UserWarning: Duplicate name: 'ruamel/yaml/representer.py'
  self._zip_file.writestr(dest, contents)
/home/ec2-user/cloud-custodian/c7n_mailer/local/lib/python2.7/site-packages/c7n/mu.py:179: 
UserWarning: Duplicate name: 'ruamel/yaml/compat.py'
  self._zip_file.writestr(dest, contents)
/home/ec2-user/cloud-custodian/c7n_mailer/local/lib/python2.7/site-packages/c7n/mu.py:179: 
UserWarning: Duplicate name: 'ruamel/yaml/main.py'
  self._zip_file.writestr(dest, contents)
/home/ec2-user/cloud-custodian/c7n_mailer/local/lib/python2.7/site-packages/c7n/mu.py:179: 
UserWarning: Duplicate name: 'ruamel/yaml/emitter.py'
  self._zip_file.writestr(dest, contents)
/home/ec2-user/cloud-custodian/c7n_mailer/local/lib/python2.7/site-packages/c7n/mu.py:179: 
UserWarning: Duplicate name: 'ruamel/yaml/scanner.py'
  self._zip_file.writestr(dest, contents)
/home/ec2-user/cloud-custodian/c7n_mailer/local/lib/python2.7/site-packages/c7n/mu.py:179: 
UserWarning: Duplicate name: 'ruamel/yaml/tokens.py'
  self._zip_file.writestr(dest, contents)
/home/ec2-user/cloud-custodian/c7n_mailer/local/lib/python2.7/site-packages/c7n/mu.py:179: 
UserWarning: Duplicate name: 'ruamel/yaml/configobjwalker.py'
  self._zip_file.writestr(dest, contents)
/home/ec2-user/cloud-custodian/c7n_mailer/local/lib/python2.7/site-packages/c7n/mu.py:179: 
UserWarning: Duplicate name: 'ruamel/yaml/loader.py'
  self._zip_file.writestr(dest, contents)
/home/ec2-user/cloud-custodian/c7n_mailer/local/lib/python2.7/site-packages/c7n/mu.py:179: 
UserWarning: Duplicate name: 'ruamel/yaml/dumper.py'
  self._zip_file.writestr(dest, contents)
/home/ec2-user/cloud-custodian/c7n_mailer/local/lib/python2.7/site-packages/c7n/mu.py:179: 
UserWarning: Duplicate name: 'ruamel/yaml/events.py'
  self._zip_file.writestr(dest, contents)
/home/ec2-user/cloud-custodian/c7n_mailer/local/lib/python2.7/site-packages/c7n/mu.py:179: 
UserWarning: Duplicate name: 'ruamel/yaml/nodes.py'
  self._zip_file.writestr(dest, contents)
/home/ec2-user/cloud-custodian/c7n_mailer/local/lib/python2.7/site-packages/c7n/mu.py:179: 
UserWarning: Duplicate name: 'ruamel/yaml/util.py'
  self._zip_file.writestr(dest, contents)
/home/ec2-user/cloud-custodian/c7n_mailer/local/lib/python2.7/site-packages/c7n/mu.py:179: 
UserWarning: Duplicate name: 'ruamel/yaml/error.py'
  self._zip_file.writestr(dest, contents)
/home/ec2-user/cloud-custodian/c7n_mailer/local/lib/python2.7/site-packages/c7n/mu.py:179: 
UserWarning: Duplicate name: 'ruamel/yaml/serializer.py'
  self._zip_file.writestr(dest, contents)
/home/ec2-user/cloud-custodian/c7n_mailer/local/lib/python2.7/site-packages/c7n/mu.py:179: 
UserWarning: Duplicate name: 'ruamel/ordereddict/__init__.py'
  self._zip_file.writestr(dest, contents)
Traceback (most recent call last):
  File "/home/ec2-user/cloud-custodian/c7n_mailer/bin/custodian", line 11, in 
    sys.exit(main())
  File "/home/ec2-user/cloud-custodian/c7n_mailer/local/lib/python2.7/site-packages/c7n/cli.py", line 362, in main
    command(config)
  File "/home/ec2-user/cloud-custodian/c7n_mailer/local/lib/python2.7/site-packages/c7n/commands.py", line 66, in _load_policies
    collection = policy_load(options, fp, validate=validate, vars=vars)
  File "/home/ec2-user/cloud-custodian/c7n_mailer/local/lib/python2.7/site-packages/c7n/policy.py", line 74, in load
    collection = PolicyCollection.from_data(data, options)
  File "/home/ec2-user/cloud-custodian/c7n_mailer/local/lib/python2.7/site-packages/c7n/policy.py", line 93, in from_data
    for p in data.get('policies', ())]
  File "/home/ec2-user/cloud-custodian/c7n_mailer/local/lib/python2.7/site-packages/c7n/policy.py", line 614, in __init__
    self.ctx = ExecutionContext(self.session_factory, self, self.options)
  File "/home/ec2-user/cloud-custodian/c7n_mailer/local/lib/python2.7/site-packages/c7n/ctx.py", line 40, in __init__
    self.output = factory(self)
  File "/home/ec2-user/cloud-custodian/c7n_mailer/local/lib/python2.7/site-packages/c7n/output.py", line 207, in __init__
    os.makedirs(self.ctx.output_path)
  File "/home/ec2-user/cloud-custodian/c7n_mailer/lib64/python2.7/os.py", line 157, in makedirs
    mkdir(name, mode)
OSError: [Errno 13] Permission denied: './slack-notify'
commented

Can you verify the read/write permissions on your policy file are correct?

chmod 644 ./slack-notify

Also, can you confirm permissions are correct on the folder you're running in? You may not have write access there. Try a known folder that you can access. /tmp, etc.

commented

Good catch. I need to get more sleep. 💤

My policy was created as a root user and I was trying to invoke the c7n-mailer as an ec2-user.

The file permissions have been fixed and all aforementioned errors are gone now.

The CloudWatch logs are now showing:

ImportError: No module named slackclient

CloudWatch logs [ERROR] 2018-05-09T20:41:47.652Z 63f36ed0-53c9-11e8-a2eb-3d0a51e4f417 Error starting mailer MailerSqsQueueProcessor(). Error: No module named slackclient

Traceback (most recent call last):
File "/var/task/c7n_mailer/handle.py", line 46, in start_c7n_mailer
mailer_sqs_queue_processor.run(parallel)
File "/var/task/c7n_mailer/sqs_queue_processor.py", line 126, in run
self.process_sqs_message(sqs_message)
File "/var/task/c7n_mailer/sqs_queue_processor.py", line 170, in process_sqs_message
from .slack_delivery import SlackDelivery
File "/var/task/c7n_mailer/slack_delivery.py", line 22, in
from slackclient import SlackClient
ImportError: No module named slackclient

commented

I had to resync the branch with the other changes I was doing, but it's updated now. Can you try again?

commented

Looking much better!

The CloudWatch log is now showing:

[WARNING] 2018-05-09T22:13:37.606Z 3848ec92-53d6-11e8-adaa-091678e4c799 Error: An error occurred (InvalidCiphertextException) when calling the Decrypt operation: Unable to decrypt slack_token with kms, will assume plaintext.

I'm using:

slack-notify.yml 
policies:
  - name: slack-notification
    resource: security-group
    description: |
      Retrieve unused security groups using regex.
      Notify using Slack.
    filters:
      - unused
      - type: value
        key: GroupName
        op: regex
        value: .*launch-wizard.*
    actions:
      - type: notify
        slack_template: slack_default
        template_format: 'html'
        priority_header: '5'
        subject: 'CloudCustodian: Unused Security Groups'
        to:
          - slack://#my-channel
        owner_absent_contact:
          - email@address.com
        transport:
          type: sqs
          queue: https://sqs.us-east-1.amazonaws.com/1234567890/sandbox
mailer_sandbox.yml 
queue_url: https://sqs.us-east-1.amazonaws.com/1234567890/sandbox
from_address: email@address.com
contact_tags:
  - OwnerContact
  - OwnerEmail
  - SNSTopicARN
region: us-east-1
role: arn:aws:iam::1234567890:role/CloudCustodianRole
slack_token: xoxb-123456789012-1234567890123456789012345
commented

Good deal! That particular message is normal if your config.yml has an unencrypted slack_token field. The mailer tried to run it through KMS decryption but it was already unencrypted.

commented

For some reason, I'm not seeing anything in my Slack channel when I run my policy.

Other than the warning message noted above, CloudWatch isn't reporting any errors in its logs.

It's clean as a whistle.

Now, I notice when I run my policy from c7n-mailer (or the custodian CLI command), there's no reference to the slack_default template anywhere. I do see template:default returned.

Is that expected? Or an indication something failed using the token?

I'm using a bot token (eg: xoxb-). Is that okay?

How can I see what's going on with the bot user?

c7n-mailer --config mailer_sandbox.yml --update-lambda && custodian run -c slack-notify.yml 
$ c7n-mailer --config mailer_sandbox.yml --update-lambda && custodian run -c slack-notify.yml

2018-05-09 22:44:11,560: custodian.policy:INFO policy: slack-notification resource:security-group region:us-east-1 count:6 time:1.05

2018-05-09 22:44:11,820: custodian.actions:INFO sent message:246d8cd6-f71d-4087-8434-ef4f6d5b6921 policy:slack-notification template:default count:6

2018-05-09 22:44:11,820: custodian.policy:INFO policy: slack-notification action: notify resources: 6 execution_time: 0.26
commented

At this point, the mailer has been provisioned and the Custodian payload has been sent to your SQS queue. You'll need to reference the CloudWatch logs to see the output from the mailer. Any logging details regarding Slack will show up there, including references to the slack_default template.

commented

Sounds good.

I just ran the mailer again and am able to see the Custodian payload enter in the SQS queue which is a good sign.

Unfortunately, the CloudWatch logs don't have much in them after that.

I need to go check the Slack end of things to make sure the bot user has permissions to post to the channel.

I'll report back if I'm successful getting this to work.

Thanks again for your assistance and time. Really appreciate it!

commented

Not a problem. Please let me know if I can be of any further help.

For reference, this is the list of OAuth permissions attached to my test integration and it includes all the ones needed for Slack to work:

incoming-webhook
channels:history
channels:read
chat:write:bot
chat:write:user
groups:history
groups:read
im:write
users:read
users:read.email
commented

Okay, I finally got the Slack incoming webhook to work with my bot user!
I was even able to get a custom slack_template to work.

I think we can finally close this one out! Whew.

Appreciate the team's patience and assistance with this!

Pretty jazzed to finally get this working. 🚀