google / exposure-notifications-verification-server

We added a request-id for normal logs that are sent to stderr, but the logs to stdout is a simple text log.

The request-id makes it easy to find all stderr logs from a request but we don't know what's the HTTP response code for that request.

The easiest fix would be to also log HTTP response in stderr, with the benefit that the oncaller would only need to care about a single log.

Can you clarify what you mean by stderr vs stdout? AFAIK, we use the logger everywhere - it shouldn't matter the destination.

If you use the following log filter:

resource.type="cloud_run_revision"
log_name="projects/$PROJECT/logs/run.googleapis.com%2Fstdout"

You can see lines like this:

{
  "textPayload": "169.254.8.129 - - [16/Dec/2020:04:46:50 +0000] \"GET /health HTTP/1.1\" 200 16 \"\" \"GoogleStackdriverMonitoring-UptimeChecks(https://cloud.google.com/monitoring)\"",
  "insertId": "5fd9913a000552f2d003e90e",
  "resource": {
    "type": "cloud_run_revision",
    "labels": {
      "service_name": "apiserver",
      "project_id": "$PROJECT",
      "revision_name": "apiserver-00067-xoq",
      "configuration_name": "apiserver",
      "location": "us-central1"
    }
  },
  "timestamp": "2020-12-16T04:46:50.348914Z",
  "labels": {
    "instanceId": "00bf4bf02dc7c259d26c5bdcfdf87d22966d1b61a7707ae4ff8c61e405b18e928029666487aa649b0fea1adf013fd2dbcb0d3931c5ec02d8e17e6ff841ebfb6ca33a235a"
  },
  "logName": "projects/encv-prod/logs/run.googleapis.com%2Fstdout",
  "receiveTimestamp": "2020-12-16T04:46:50.426717175Z"
}

This is collected by Cloud Run from the server process' stdout. Whereas majority of our other logs (those with request-id) are collected from stderr.

I think these logs are from handlers.CombinedLoggingHandler:

exposure-notifications-verification-server/cmd/apiserver/main.go

Line 219 in 10c0852

return srv.ServeHTTPHandler(ctx, handlers.CombinedLoggingHandler(os.Stdout, r))

Initially I thought something like this would be helpful: https://pkg.go.dev/github.com/gorilla/handlers#CustomLoggingHandler but then I realize it looks like there's no way to add the ctx (with the request-id) as part of the handlers.LogFormatter.

@yegle or @mikehelmick do you know why we use the combined logging thing? I never really understood the use case.

to get HTTP logs for local dev primarily -

I would be fine to just inject this logger via env var for local dev

Cloud run will insert HTTP logs automatically

/assign

This is not fully resolved, we still want the response code log lines with the request-id information.

@yegle you mean the HTTP logs? Those logs are generated outside of our app

Oh I see. I think I was asking an XY problem. What I really want is to be able to corelate the log from our code with the request log.

I found https://cloud.google.com/run/docs/logging#correlate-logs is what I want. I'll send PR to implement it.

`request-id` is not in the HTTP response from stdout