Making GCP Load-balancer HTTP logs integrate with Cloud Trace (in GCP Logs Explorer)?

Question

Cloud Trace and Cloud Logging integrate quite nicely in most cases, described in https://cloud.google.com/trace/docs/trace-log-integration

Unfortunately, this doesn't seem to include the HTTP request logs generated by a Load Balancer when request logging is enabled.

The LB logs show the traces icon, and are correctly associated with an overall trace in the Cloud Trace system, but the context menu 'show trace details' is greyed out for those log items.

A similar problem arose with my application level logging/tracing, and was solved by setting the traceSampled attribute on the LogEntry, but this can't work for LB logs, since I'm not in control of their generation.

In this instance I'm tracing 100% of requests since the service is M2M and fairly low volume, but in the general case it makes sense that the LB can't know if something is actually generating traces without being told.

I can't find any good references in the docs, but in theory a response header indicating it was sampled could be observed by the LB and cause it to issue the appropriate log.

Any ideas if such a feature exists, in this form or any other?

(Last-ditch workaround might be to use Logs Router to feed LB logs into a pubsub queue (and exclude them from normal logging sinks), and resubmit them to the normal sink(s) with fields appropriately set by some Cloud Function or other pubsub consumer, but that seems like a lot of work and complexity for this purpose)

Answer 1

There is currently a Feature Request created for this, you can follow the status in the following link 1.

As a workaround, you could implement target proxies along with your Load Balancer, according to the documentation for a Global external HTTP(S) load balancer:

The proxies set HTTP request/response headers as follows:

• Via: 1.1 google (requests and responses)
• X-Forwarded-Proto: [http | https] (requests only)
• X-Cloud-Trace-Context: <trace-id>/<span-id>;<trace-options> (requests only) Contains parameters for Cloud Trace.
• X-Forwarded-For: [<supplied-value>,]<client-ip>,<load-balancer-ip> (see X-Forwarded-For header) (requests only)

You can find the complete documentation about external HTTP(S) load balancers and target proxies here 2.

And finally, take a look at the documentation on how to use and configure target proxies here 3.