This contains a simple go program that can read Stackdriver K8s Audit Logs and forward them to a configurable webhook. This is a useful way to route K8s Audit Events to a program like falco or the Sysdig Agent.
These instructions assume you already have created a cluster and have configured the gcloud
and kubectl
command line programs to interact with the cluster.
-
Create a google cloud (not k8s) service account and key that has the ability to read logs:
$ gcloud iam service-accounts create swb-logs-reader --description "Service account used by stackdriver-webhook-bridge" --display-name "stackdriver-webhook-bridge logs reader" $ gcloud projects add-iam-policy-binding <your gce project id> --member serviceAccount:swb-logs-reader@<your gce project id>.iam.gserviceaccount.com --role 'roles/logging.viewer' $ gcloud iam service-accounts keys create $HOME/swb-logs-reader-key.json --iam-account swb-logs-reader@<your gce project id>.iam.gserviceaccount.com
-
Create a k8s secret containing the service account keys:
kubectl create secret generic stackdriver-webhook-bridge --from-file=key.json=$HOME/swb-logs-reader-key.json
-
Deploy the bridge program to your cluster using the provided stackdriver-webhook-bridge.yaml file:
kubectl apply -f stackdriver-webhook-bridge.yaml -n sysdig-agent
The bridge program routes audit events to the domain name sysdig-agent.sysdig-agent.svc.cluster.local
, which corresponds to the sysdig-agent service you created when you deployed the agent.
The bridge defines a liveness probe on http://:8182/health
.
The bridge defines cpu requests/limits. The default values should work for most environments, but you may want to either increase requests and/or limits based on your cluster size and the amount of k8s audit activity.
The brige exposes a prometheus-compatible metrics server on http://:25000/metrics
. The following bridge metrics are defined:
swb_poller_log_fetch_error
: The number of times the bridge had an error fetching a set of stackdriver logsswb_poller_log_entry_in
: The number of log entries receivedswb_poller_audit_event_out
: The number of audit events successfully passed along to the agentswb_poller_audit_payload_extract_error
: The number of times the bridge had an error extracting the audit payload from a log entryswb_poller_audit_payload_convert_error
: The number of times the bridge had an error converting an audit payload to an audit eventswb_poller_audit_event_marshal_error
: The number of times the bridge had an error marshaling an audit event to a json stringswb_poller_audit_event_send_error
: The number of audit events that could not successfully be sent to the agent
The Makefile has binary
, image
, and test
targets. There are unit tests that test the converter, ensuring that log entries are converted to expected K8s Audit Events.
GKE Uses a K8s Audit Policy that emits a more limited set of information than the audit policy recommended by Sysdig. In particular, audit events for configmaps generally do not contain a requestObject
field that contains the object that's being created/modified. As a result, the following Rules from k8s_audit_rules.yaml will not trigger:
- Create/Modify Configmap With Private Credentials: The contents of configmaps are not included in audit logs, so the contents can not be examined for sensitive information.
For many K8s distributions, an audit event representing a pod exec includes the command and specific container as arguments to the requestURI, for example:
"requestURI":"/api/v1/namespaces/default/pods/nginx-deployment-7998647bdf-phvq7/exec?command=bash&container=nginx1&container=nginx1&stdin=true&stdout=true&tty=true
In GKE, the audit event is missing those request parameters. This doesn't affect any of the Falco rules in k8s_audit_rules.yaml, but does limit the information that can be returned in the outputs of rules.
If you would rather use google pub/sub to receive stackdriver logs and forward them to a webhook, check out https://github.com/codeonline-io/falco-gke-audit-bridge.