Logging on either GCP or local

Question

Suppose there is a system that is run on GCP, but as a backup, can be run locally.

When running on the cloud, stackdriver is pretty straightforward.

However, I need my system to push to stackdriver if on the cloud, and if not on the cloud, use the local python logger.

I also don't want to include any logic to do so, and this should be automatic.

1. When logging, log straight to Python/local logger.
2. If on GCP -> push these to stackdriver.

I can write logic that could implement this but that is bad practice. There surely is a direct way of getting this to work.

Example

import google.cloud.logging

client = google.cloud.logging.Client()
client.setup_logging()


import logging
cl = logging.getLogger()
file_handler = logging.FileHandler('file.log')
cl.addHandler(file_handler)
logging.info("INFO!")

This will basically log to python logger, and then 'always' upload to cloud logger. How can I have it so that I don't need to explicitly add import google.cloud.logging and basically if stackdriver is installed, it directly gets the logs? Is that even possible? If not can someone explain how this would be handled from a best practices perspective?

Attempt 1 [works]

Created /etc/google-fluentd/config.d/workflow_log.conf

<source>
    @type tail
    format none
    path /home/daudn/this_log.log
    pos_file /var/lib/google-fluentd/pos/this_log.pos
    read_from_head true
    tag workflow-log
</source>

Created /var/log/this_log.log

pos_file /var/lib/google-fluentd/pos/this_log.pos exists

import logging
cl = logging.getLogger()
file_handler = logging.FileHandler('/var/log/this_log.log')
file_handler.setFormatter(logging.Formatter("%(asctime)s;%(levelname)s;%(message)s"))
cl.addHandler(file_handler)

logging.info("info_log")
logging.error("error_log")

This works! Look for your logs for the specific VM and not global>python

Answer 1

Fortunately, this is a story that is handled. Stackdriver Logging is a very versatile framework for logging. However, there are a variety of logging APIs and Google's intent was not that you had to rewrite all your existing applications to leveraging the Stackdriver logging native APIs. Instead, you can use a logging API of your choice (including standard and defacto APIs) and these logging APIs will then map to Stackdriver. If executed outside a GCP environment or you simply wish to switch to an alternate log collector, your applications would not have to be re-coded or recompiled.

A list of the logging APIs available for different languages can be found at Setting Up Language Runtimes and this includes Setting Up Stackdriver Logging for Python.

For Python, at runtime, you have a configuration property (eg an Environment variable) that declares whether or not you wish to use Stackdriver. If set to true, then .. and only then ... would you execute the login that sets up the native Python logging for Stackdriver otherwise that logic would not be called and hence you would have no dependency on Stackdriver.

A possible piece of code might be:

if os.environ.get('USE_STACKDRIVER') == 'true':
  import google.cloud.logging
  client = google.cloud.logging.Client()
  client.setup_logging()