nice-registry / heroku-log-s3

log drain from heroku to s3. grep by line prefix and buffer upload to s3 as timestamped files

Heroku Log S3

Configure

Setup the following ENV (aka heroku config:set)

• FILTER_PREFIX this is the prefix string to look out for. every other log lines are ignored
• S3_KEY, S3_SECRET, S3_BUCKET necessary ACL to your s3 bucket
• DURATION (default 60) seconds to buffer until we close the IO to AWS::S3::S3Object#write
• STRFTIME (default %Y%m/%d/%H/%M%S.:thread_id.log) format of your s3 object_id
  • :thread_id will be replaced by a unique number to prevent overwriting of the same file between reboots, in case the timestamp overlaps
• HTTP_USER, HTTP_PASSWORD (default no password protection) credentials for HTTP Basic Authentication
• WRITER_LIB (default ./writer/s3.rb) defines the ruby script to load Writer class

Using

In your heroku app, add this drain (changing HTTP_USER, HTTP_PASSWORD and DRAIN_APP_NAME to appropriate values)

heroku drains:add https://HTTP_USER:HTTP_PASSWORD@DRAIN_APP_NAME.herokuapp.com/

or if you have no password protection

heroku drains:add https://DRAIN_APP_NAME.herokuapp.com/

Credits

• https://github.com/rwdaigle/heroku-log-parser
• https://github.com/rwdaigle/heroku-log-store