Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Can't see application log in Google Cloud Logs

How can I view log messages on Google Cloud?: https://console.cloud.google.com/logs

This is what I see in the terminal when I run dev_appserver.py (locally running):

INFO     2016-05-16 14:00:45,118 module.py:787] default: "GET /static/images/contact.png HTTP/1.1" 304 -
INFO     2016-05-16 14:00:45,128 module.py:787] default: "GET /static/images/email.png HTTP/1.1" 304 -
INFO     2016-05-16 14:00:45,136 module.py:787] default: "GET /static/images/phone.png HTTP/1.1" 304 -
INFO     2016-05-16 14:00:45,487 basehandler.py:19] entering basehandler.py
INFO     2016-05-16 14:00:45,516 module.py:787] default: "GET /static/images/logo-349x209.png HTTP/1.1" 304 -
INFO     2016-05-16 14:00:45,562 requesthandlers.py:26] entering requesthandlers.py
INFO     2016-05-16 14:00:45,563 app.py:28] entering app.py
INFO     2016-05-16 14:00:45,563 app.py:198] Using development database

Both application log messages and request logging is displayed.
However when I view the log of the same code deployed I can only see the requests being logged:

gcloud-log

The code I'm using to generate application log messages is something like:

import logging
logger = logging.getLogger("someLogger")

logger.info("entering app.py")

But I've also tried using logging.info(...) directly with the same results.

I've tried finding an answer to this in various resources but I've come up empty-handed, most refer to how to set log level when developing locally.
I'm guessing that I need to enable some setting in order to view application logs on Google Cloud Logs.

Resources that I've looked at:

  • https://cloud.google.com/logging/docs/view/logs_viewer
  • https://cloud.google.com/appengine/docs/python/logs/
  • How to change the logging level of dev_appserver
  • How do I write to the console in Google App Engine?
  • Google App Engine - Can not find my logging messages
  • https://docs.python.org/3/howto/logging.html
like image 901
Petrus K. Avatar asked May 16 '16 14:05

Petrus K.


People also ask

How do I enable data access logs in GCP?

To enable Data Access audit logs, do the following: In the Data Access audit logs configuration table, select one or more Google Cloud services from the Service column. In the Log Types tab, select the Data Access audit log types that you want to enable for your selected services. Click Save.

What are the 3 options Google offers for viewing and interacting with your logs?

Download logs: Download logs from your query results. Hide similar logs: Hide large amounts of similar logs from your query results. Trace data: View trace details and refine your query based on the trace. For more information, see View trace data.


2 Answers

App engine groups the logs by request. You need to expand the log using the triangle/pointer on the left of the request in the 'new' GAE log viewer.

Personally I prefer using the old GAE log viewer, but I am unsure how much longer it will be around:

https://appengine.google.com/logs?app_id=s~xxx

(This viewer shows request + logs and allows log expansion)

like image 197
ChrisC73 Avatar answered Sep 27 '22 18:09

ChrisC73


An easy way to integrate Google Cloud Platform logging into your Python code is to create a subclass from logging.StreamHandler. This way logging levels will also match those of Google Cloud Logging, enabling you to filter based on severity. This solution also works within Cloud Run containers.

Also you can just add this handler to any existing logger configuration, without needing to change current logging code.

import json
import logging
import os
import sys
from logging import StreamHandler

from flask import request


class GoogleCloudHandler(StreamHandler):
    def __init__(self):
        StreamHandler.__init__(self)

    def emit(self, record):
        msg = self.format(record)
        # Get project_id from Cloud Run environment
        project = os.environ.get('GOOGLE_CLOUD_PROJECT')

        # Build structured log messages as an object.
        global_log_fields = {}
        trace_header = request.headers.get('X-Cloud-Trace-Context')

        if trace_header and project:
            trace = trace_header.split('/')
            global_log_fields['logging.googleapis.com/trace'] = (
                f"projects/{project}/traces/{trace[0]}")

        # Complete a structured log entry.
        entry = dict(severity=record.levelname, message=msg)
        print(json.dumps(entry))
        sys.stdout.flush()

A way to configure and use the handler could be:

def get_logger():
    logger = logging.getLogger(__name__)

    if not logger.handlers:
        gcp_handler = GoogleCloudHandler()
        gcp_handler.setLevel(logging.DEBUG)

        gcp_formatter = logging.Formatter(
            '%(levelname)s %(asctime)s [%(filename)s:%(funcName)s:%(lineno)d] %(message)s')
        gcp_handler.setFormatter(gcp_formatter)
        logger.addHandler(gcp_handler)
    return logger
like image 36
DfreakD Avatar answered Sep 27 '22 19:09

DfreakD