How do I correlate request logs in Cloud Run?

Question

I am using self structured JSON payloads for logging from my Node.js/Express based Cloud Run service and I am unable to get logs from the same request to correlate using the trace method.

The documentation says:

Container logs are not automatically correlated to request logs unless you use a Stackdriver Logging client library. If you want this correlation without using a client library, use a structured JSON log line that contains a trace field with the content of the incoming X-Cloud-Trace-Context header. Then your logs will be correctly correlated to the request log.

I know that my structured JSON logs are working as the level/severity and message are being extracted and displayed as expected.

I am passing as the value of trace exactly what is passed by the X-Cloud-Trace-Context header which I get using req.get method provided by Express: req.get('X-Cloud-Trace-Context').

Here is the JSON that is being logged:

{
    "message": "Create Query",
    "level": "debug",
    "severity": "DEBUG",
    "trace": "40f...........................cc/131...............23;o=1"
}

Below is an example of how that log line is present in Stackdriver Logging.

I have also tried using the logging.googleapis.com/trace property as mentioned in the Special fields in structured payloads documentation. I'm fairly certain the value of the X-Cloud-Trace-Context header is not valid for this property but I'm not sure how to format the header value to match the value as documented on this page.

---

Given the above, my questions are:

• Which is the correct property name to use for trace?
• How do I correctly format the value of this property based on the value of the X-Cloud-Trace-Context header?

---

Here is an example of a complete log message as presented (with IDs removed) in Stackdriver Logging:

{
 insertId:  "..."  
 jsonPayload: {
  level:  "debug"   
  message:  "Create Query"   
  trace:  "40f...........................cc/131...............23;o=1"   
 }
 labels: {
  instanceId:  "0.........................................2"   
 }
 logName:  "projects/b.............0/logs/run.googleapis.com%2Fstdout"  
 receiveTimestamp:  "2019-08-16T18:05:58.816240093Z"  
 resource: {
  labels: {
   configuration_name:  "a..................ing"    
   location:  "..."    
   project_id:  "b.............0"    
   revision_name:  "a..................ing-01987"    
   service_name:  "a..................ing"    
  }
  type:  "cloud_run_revision"   
 }
 severity:  "DEBUG"  
 timestamp:  "2019-08-16T18:05:58.479527Z"  
}

Answer 1

Which is the correct property name to use for trace?

The property name required in your JSON string is logging.googleapis.com/trace. This is pulled out of the jsonpayload and into the trace property that you can see example usage of on "INFO" logs.

How do I correctly format the value of this property based on the value of the X-Cloud-Trace-Context header?

The format required is as follows: projects/[project]/traces/[trace] where [project] is your Google Cloud Project i.e. b.......0 and [trace] as from your example is 40f...........................cc. The base and query param needs to be omitted (this isn't particular well documented).

Below is a snippet of this performed in JS from the Google documentation found here: https://cloud.google.com/run/docs/logging

// Build structured log messages as an object.
const globalLogFields = {};

// Add log correlation to nest all log messages beneath request log in Log Viewer.
const traceHeader = req.header('X-Cloud-Trace-Context');
if (traceHeader && project) {
  const [trace] = traceHeader.split('/');
  globalLogFields[
    'logging.googleapis.com/trace'
  ] = `projects/${project}/traces/${trace}`;
}

Below is a Golang function I have had success with in terms of pulling the information out and formatting it correctly:

func extractTracePath(r *http.Request, app *application) string {
    s := r.Header.Get("X-Cloud-Trace-Context")
    traceURL, err := url.Parse(s)
    if err != nil {
        app.infoLog.Fatal("Invalid trace URL")
    }
    tracePath := traceURL.Path
    trace := strings.Split(tracePath, "/")[0]
    project := os.Getenv("GCLOUD_PROJECT")
    partialTracePath, err := url.Parse("projects/" + project + "/traces/")
    tracePath = partialTracePath.Path
    tracePath = path.Join(tracePath, trace)
    return tracePath
}