How to log from a custom ai platform model

Question

I recently deployed a custom model to google cloud's ai-platform, and I am trying to debug some parts of my preprocessing logic. However, My print statements are not being logged to the stackdriver output. I have also tried using the logging client imported from google.cloud, to no avail. Here is my custom prediction file:

import os
import pickle

import numpy as np
from sklearn.datasets import load_iris
import tensorflow as tf

from google.cloud import logging

class MyPredictor(object):
  def __init__(self, model, preprocessor):
    self.logging_client = logging.Client()
    self._model = model
    self._preprocessor = preprocessor
    self._class_names = ["Snare", "Kicks", "ClosedHH", "ClosedHH",  "Clap", "Crash", "Perc"]

  def predict(self, instances, **kwargs):
    log_name = "Here I am"
    logger = self.logging_client.logger(log_name)
    text = 'Hello, world!'
    logger.log_text(text)
    print('Logged: {}'.format(text), kwargs.get("sr"))

    inputs = np.asarray(instances)

    outputs = self._model.predict(inputs)

    if kwargs.get('probabilities'):
      return outputs.tolist()
      #return "[]"
    else:
      return [self._class_names[index] for index in np.argmax(outputs.tolist(), axis=1)]

  @classmethod
  def from_path(cls, model_dir):
    model_path = os.path.join(model_dir, 'model.h5')
    model = tf.keras.models.load_model(model_path, custom_objects={"adam": tf.keras.optimizers.Adam, 
 "categorical_crossentropy":tf.keras.losses.categorical_crossentropy, "lr":0.01, "name": "Adam"})

    preprocessor_path = os.path.join(model_dir, 'preprocessor.pkl')
    with open(preprocessor_path, 'rb') as f:
      preprocessor = pickle.load(f)

    return cls(model, preprocessor)

I can't find anything online for why my logs are not showing up in stackdriver (neither print statements nor the logging library calls). Has anyone faced this issue?

Thanks, Nikita

NOTE: If you have enough rep to create tags please add the google-ai-platform tag to this post. I think it would really help people who are in my position. Thanks!

Answer 1

From Documentation:

If you want to enable online prediction logging, you must configure it when you create a model resource or when you create a model version resource, depending on which type of logging you want to enable. There are three types of logging, which you can enable independently:

Access logging, which logs information like timestamp and latency for each request to Stackdriver Logging.

You can enable access logging when you create a model resource.

Stream logging, which logs the stderr and stdout streams from your prediction nodes to Stackdriver Logging, and can be useful for debugging. This type of logging is in beta, and it is not supported by Compute Engine (N1) machine types.

You can enable stream logging when you create a model resource.

Request-response logging, which logs a sample of online prediction requests and responses to a BigQuery table. This type of logging is in beta.

You can enable request-response logging by creating a model version resource, then updating that version.

For your use case, please use the following template to log custom information into StackDriver:

Model

gcloud beta ai-platform models create {MODEL_NAME} \
 --regions {REGION} \
 --enable-logging \
 --enable-console-logging

Model version

gcloud beta ai-platform versions create {VERSION_NAME} \
    --model {MODEL_NAME} \
    --origin gs://{BUCKET}/{MODEL_DIR} \
    --python-version 3.7 \
    --runtime-version 1.15 \
    --package-uris gs://{BUCKET}/{PACKAGES_DIR}/custom-model-0.1.tar.gz \
    --prediction-class=custom_prediction.CustomModelPrediction \
    --service-account custom@project_id.iam.gserviceaccount.com

I tried this and worked fine:

• I did some modification to the constructor due to the @classmethod decorator.
• Create a service account and grant it "Stackdriver Debugger User" role, use it during model version creation
• Add google-cloud-logging library to your setup.py
• Consider extra cost of enabling StackDriver logging
• When using log_struct check the correct type is passed. (If using str, make sure you convert bytes to str in Python 3 using .decode('utf-8'))
• Define the project_id parameter during Stackdriver client creation logging.Client(), otherwise you will get:

ERROR:root:Prediction failed: 400 Name "projects//logs/my-custom-prediction-log" is missing the parent component. Expected the form projects/[PROJECT_ID]/logs/[ID]" 

Code below:

%%writefile cloud_logging.py

import os
import pickle
import numpy as np

from datetime import date
from google.cloud import logging

import tensorflow.keras as keras
LOG_NAME = 'my-custom-prediction-log'

class CustomModelPrediction(object):
    def __init__(self, model, processor, client):    
        self._model = model
        self._processor = processor
        self._client = client

    def _postprocess(self, predictions):
        labels = ['negative', 'positive']
        return [
            {
                "label":labels[int(np.round(prediction))],
                "score":float(np.round(prediction, 4))
            } for prediction in predictions]

    def predict(self, instances, **kwargs):
        logger = self._client.logger(LOG_NAME)
        logger.log_struct({'instances':instances})
        preprocessed_data = self._processor.transform(instances)
        predictions =  self._model.predict(preprocessed_data)
        labels = self._postprocess(predictions)
        return labels

    @classmethod
    def from_path(cls, model_dir):        
        client = logging.Client(project='project_id') # Change to your project
        model = keras.models.load_model(
          os.path.join(model_dir,'keras_saved_model.h5'))
        with open(os.path.join(model_dir, 'processor_state.pkl'), 'rb') as f:
            processor = pickle.load(f)    
        return cls(model, processor, client)

# Verify model locally

from cloud_logging import CustomModelPrediction
classifier = CustomModelPrediction.from_path('.')

requests = ["God I hate the north", "god I love this"]
response = classifier.predict(requests)
response

Then I check with the sample library:

python snippets.py my-custom-prediction-log list
Listing entries for logger my-custom-prediction-log:
* 2020-02-19T19:51:45.809767+00:00: {u'instances': [u'God I hate the north', u'god I love this']}
* 2020-02-19T19:57:18.615159+00:00: {u'instances': [u'God I hate the north', u'god I love this']}

To visualize the logs, in StackDriver > Logging > Select Global and your Log name, if you want to see Model logs you should be able to select Cloud ML Model version.

You can use my files here: model and pre-processor