Stackdriver Log Agent - Log Level Irrelevant with Google Cloud Logging Driver for Docker

Question

TL,DR; Log levels are ignored when making a Stackdriver logging API call using using a CloudLoggingHandler from a Docker container using the Google Cloud Logging driver.

Detail; The recommended way to get logs from a Docker container running on Google's Compute Engine is to use the Stackdriver Logging Agent:

It is a best practice to run the Stackdriver Logging agent on all your VM instances. The agent runs under both Linux and Windows. To install the Stackdriver Logging agent, see Installing the Logging Agent.

The following steps were completed successfully:

• Ensure Compute Engine default service account has Editor and Logs Writer roles.
• Ensure the VM instance has Cloud API access scope for Stackdriver Logging API (Full)
• Install and start Stackdriver Logging Agent.

I then copied the example CloudLoggingHandler example from Google's Cloud Platform Python docs.

import logging
import google.cloud.logging
from google.cloud.logging.handlers import CloudLoggingHandler

client = google.cloud.logging.Client()
handler = CloudLoggingHandler(client)

cloud_logger = logging.getLogger('cloudLogger')
cloud_logger.setLevel(logging.INFO)
cloud_logger.addHandler(handler)

cloud_logger.error('bad news error')
cloud_logger.warning('bad news warning')
cloud_logger.info('bad news info')

The Docker container is started with the Google Cloud Logging Driver flag (--log-driver=gcplogs):

sudo docker run --log-driver=gcplogs --name=server gcr.io/my-project/server:latest

This works, however all logs, irrespective of level are only visible in Stackdriver when viewing 'Any log level'. Strangely, the message itself contains the level:

2018-08-22 22:34:42.176 BST
ERROR:bad news error

2018-08-22 22:34:42.176 BST
WARNING:bad news warning

2018-08-22 22:34:42.176 BST
WARNING:bad news info

This makes it impossible to filter by level in the Stackdriver UI:

In the screenshot below, all icons on the LHS of every log entry show the level as Any:

Answer 1

From what I can tell, the CloudLoggingHandler is a standalone handler that sends logs to the Global log level. To integrate with gcplogs driver properly, try using the ContainerEngineHandler