How to send numpy array to sagemaker endpoint using lambda function

Question

How to invoke sagemaker endpoint with input data type numpy.ndarray. I have deployed a sagemaker model and trying to hit it using lambda function. But I am unable to figure out how to do it. I am getting server error.

One row of the Input data. The total data set has shape=(91,5,12). The below is only one row of Input data.

array([[[0.30440741, 0.30209799, 0.33520652, 0.41558442, 0.69096432,
         0.69611016, 0.25153326, 0.98333333, 0.82352941, 0.77187154,
         0.7664042 , 0.74468085],
        [0.30894981, 0.33151662, 0.22907725, 0.46753247, 0.69437367,
         0.70410559, 0.29259044, 0.9       , 0.80882353, 0.79401993,
         0.89501312, 0.86997636],
        [0.33511896, 0.34338939, 0.24065546, 0.48051948, 0.70384005,
         0.71058715, 0.31031288, 0.86666667, 0.89705882, 0.82724252,
         0.92650919, 0.89125296],
        [0.34617355, 0.36150251, 0.23726854, 0.54545455, 0.71368726,
         0.71703244, 0.30228356, 0.85      , 0.86764706, 0.86157254,
         0.97112861, 0.94089835],
        [0.36269508, 0.35923332, 0.40285461, 0.62337662, 0.73325475,
         0.7274392 , 0.26241391, 0.85      , 0.82352941, 0.89922481,
         0.9343832 , 0.90780142]]])

I am using the following code but unable to invoke the endpoint

import boto3
def lambda_handler(event, context):
    # The SageMaker runtime is what allows us to invoke the endpoint that we've created.
    runtime = boto3.Session().client('sagemaker-runtime')

    endpoint = 'sagemaker-tensorflow-2019-04-22-07-16-51-717'

    print('givendata ', event['body'])
    # data = numpy.array([numpy.array(xi) for xi in event['body']])
    data = event['body']
    print('numpy array ', data)

    # Now we use the SageMaker runtime to invoke our endpoint, sending the review we were given
    response = runtime.invoke_endpoint(EndpointName = endpoint,# The name of the endpoint we created
                                       ContentType = 'application/json',                 # The data format that is expected
                                       Body = data) # The actual review

    # The response is an HTTP response whose body contains the result of our inference
    result = response['Body'].read().decode('utf-8')

    print('response', result)

    # Round the result so that our web app only gets '1' or '0' as a response.
    result = round(float(result))

    return {
        'statusCode' : 200,
        'headers' : { 'Content-Type' : 'text/plain', 'Access-Control-Allow-Origin' : '*' },
        'body' : str(result)
    }

I am unable to figure out what should be written in place of ContentType. Because I am not aware of MIME type in case of numpy.ndarray.

Answer 1

Illustration of what I had and how I solved

from sagemaker.tensorflow import TensorFlowPredictor

predictor = TensorFlowPredictor('sagemaker-tensorflow-serving-date')
data = np.array(raw_data)
response = predictor.predict(data=data)
predictions = response['predictions']
print(predictions)

What I did to find the answer:

• Looked up predictions.py and content_types.py implementation in sagemaker python library to see what content types it used and what arguments it had.
• First I thought that application/x-npy content_type was used and thus tried using serialisation code from predictor.py and passing the application/x-npy as content_type to invoke_endpoint.
• After receiving 415 (unsupported media type), the issue was still the content_type. The following print statements helped me to reveal what content_type predictor actually uses (application/json) and thus I took the appropriate serialisation code from predictor.py

from sagemaker.tensorflow import TensorFlowPredictor

predictor = TensorFlowPredictor('sagemaker-tensorflow-serving-date')
data = np.array(raw_data)
response = predictor.predict(data=data)
print(predictor.content_type)
print(predictor.accept)
predictions = response['predictions']
print(predictions)

TL;DR

Solution for lambda:

import json
import boto3

ENDPOINT_NAME = 'sagemaker-tensorflow-serving-date'
config = botocore.config.Config(read_timeout=80)
runtime= boto3.client('runtime.sagemaker', config=config)
data = np.array(raw_data)
payload = json.dumps(data.tolist())
response = runtime.invoke_endpoint(EndpointName=ENDPOINT_NAME,
                                   ContentType='application/json',
                                   Body=payload)
result = json.loads(response['Body'].read().decode())
res = result['predictions']

Note: numpy is not included in lambda thus you would either include the numpy yourself or instead of data.tolist() operate with python list and json.dump that list (of lists). From your code it seems to me you have python list instead of numpy array, so simple json dump should work.

Answer 2

If you are training and hosting custom algorithm on SageMaker using TensorFlow, you can serialize/de-serialize the request and response format as JSON as in TensorFlow Serving Predict API.

import numpy
from sagemaker.predictor import json_serializer, json_deserializer

# define predictor
predictor = estimator.deploy(1, instance_type)

# format request
data = {'instances': numpy.asarray(np_array).astype(float).tolist()}

# set predictor request/response formats
predictor.accept = 'application/json'
predictor.content_type = 'application/json'

predictor.serializer = json_serializer
predictor.deserializer = json_deserializer

# run inference using SageMaker predict class
# https://github.com/aws/sagemaker-python-sdk/blob/master/src/sagemaker/predictor.py
predictor.predict(data)

You can refer the example notebook here to train and host custom TensorFlow container.