Giters

kelseyhightower / vault-on-google-kubernetes-engine

How to guide on running HashiCorp's Vault on Google Kubernetes Engine

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

[Question] - How to enable audit devices to stdout and save in Stackdriver Logs?

samuelbaruffi opened this issue · comments

commented

Hello,
First of all, thanks for the awesome tutorial. It is very handy.

We have implemented this in our production cluster and were having issues getting the audit device logs to Stackdriver logs.

I have enabled the audit device to stdout by doing the following:

vault audit enable file file_path=stdout

Which I can confirm that is outputting to stdout on the vault container, if I check the logs with:

kubectl logs vault-0 -f vault

But unfortunately those logs are not being saved in Stackdriver for some reason, and I was not able to find more info on how to enable or troubleshoot it. See picture below for my stackdriver log on the vault container:

screen shot 2018-10-11 at 2 17 54 pm

Thanks in advance for the help.

Sam.

commented

Hi @samuelbaruffi

Can you share more of that screenshot? The logs should be there, including the Vault startup logs. Can you make sure you're looking at the correct container? What does kubectl get logs show for that container?

commented

Thanks for the quick reply @sethvargo .

See the full screenshot below (hiding few fields for security reasons):

screen shot 2018-10-11 at 2 29 24 pm

The logs for the vault-init containers are being saved to Stackdriver, but for the vault container it does not seem they are. See screenshot below fir the vault-init container in the pod that is working on Stackdriver:

screen shot 2018-10-11 at 2 35 16 pm

If I run kubectl logs vault-0 -f vault I'm able to see all the audit logs.

Thanks for the help!

commented

Hmm - that's really weird, since they are deployed the same. Are you able to reproduce it on a new cluster?

commented

I'd have to try creating a new cluster and building Vault again.

I'll post the results once I am able to replicate the environment in a new cluster.

Let me know if you find anything meanwhile.

Thank you.

commented

I'm not able to reproduce it on my end. If you're familiar with Terraform, github.com/sethvargo/vault-on-gke is a one-command version of this same thing.

commented

Thank you @sethvargo,

I'll try to use the Terrraform script for my testing.

For now I'll go ahead and close this ticket.

Sam.

commented

I'm seeing issues with logging with the Terraform script. Absolutely 0 Kubernetes logs in Stackdriver. But viewing kubectl logs for the Vault container shows the audit logs properly.

For a cluster created "manually" through the GCloud UI, you can see the K8S logs:
image

For the Vault cluster created through the Terraform, nothing.
image