How to deploy pre-trained model using sagemaker?

Question

I have a pre-trained model that will translate text from English to Marathi. You can find it here...

git clone https://github.com/shantanuo/Word-Level-Eng-Mar-NMT.git

Clone and Run the notebook. I am looking for a way to deploy it so that users can use it as an API

The guidelines for deploying the model can be found here... https://gitlab.com/shantanuo/dlnotebooks/blob/master/sagemaker/01-Image-classification-transfer-learning-cifar10.ipynb

I will like to know the steps to follow in order to deploy the model. Is it possible to create an android app for this?

Answer 1

Great News! So your model has already been deployed when you created the endpoint. Make sure you DON'T run the sage.delete_endpoint(EndpointName=endpoint_name) at the end of the notebook!

Now you can call that endpoint via command line or an SDK like boto3.

To attach it to a public API endpoint I recommend leveraging API Gateway, Lambdas, and Sagemaker in something similar to this tutorial.

API Gateway will handle the hosting and security/tokens (if desired). After the http request hits API Gateway it needs to be caught by a designated lambda. The lambda's job is to verify the incoming data, call the Sagemaker endpoint, and return the response in the correct format.

Step 1: Build Lambda

To correctly deploy your lambda you will need to create a Serverless Framework Service.

1) First install Serverless Framework

2) Navigate to the directory where you want to store the API Gateway and Lambda files

3) In the command line run:

serverless create --template aws-python

4) Create a new file named lambdaGET.py to be deployed inside your lambda

lambdaGET.py

import os
import io
import boto3
import json
import csv

'''
endpoint_name here should be a string, the same as which was created
in your notebook on line:
           endpoint_name = job_name_prefix + '-ep-' + timestamp


'''
ENDPOINT_NAME = endpoint_name 

client = boto3.client('sagemaker-runtime')
# Or client = boto3.client('runtime.sagemaker') should also be acceptable

def lambda_handler(event, context):
    print("Received event: " + json.dumps(event, indent=2))


    event = json.loads(json.dumps(event))
    # I recommend you verify the data here although it is not critical

    '''
    I'm assuming your going to attach this to a GET in which case the
    payload will be passed inside the "queryStringParameters"
    '''

    payload = event["queryStringParameters"]
    print(payload)


    response = client.invoke_endpoint(
                                      EndpointName=ENDPOINT_NAME,
                                      Body=payload,
                                      ContentType='application/x-image',
                                      Accept='Accept'
                                     )

    result = json.loads(response['Body'].read())
    print(result)

    '''
    After the lambda has obtained the results in needs to correctly 
    format them to be passed across the API Gateway
    '''

   response = {
                "isBase64Encoded": False,
                "statusCode": 200,
                "headers": {},
                "body": json.dumps(result)
        }



    return response

Step 2: Build Serverless.yml

In this step you need to build the serverless file to deploy the lambda, API Gateway, and connect them together.

service: Word-Level-Eng-Mar-NMT

provider:
  name: aws
  runtime: python2.7
  timeout: 30
  stage: ${opt:stage, 'dev'}
  region: ${opt:region, 'us-east-1'}
  profile: ${opt:profile, 'default'}
  apiName : Word-Level-Eng-Mar-NMT-${self:provider.stage}
  environment:
    region: ${self:provider.region}
    stage: ${self:provider.stage}
  stackTags:
    Owner : shantanuo
    Project : Word-Level-Eng-Mar-NMT
    Service : Word-Level-Eng-Mar-NMT
    Team : shantanuo
  stackPolicy: # This policy allows updates to all resources
    - Effect: Allow
      Principal: "*"
      Action: "Update:*"
      Resource: "*"

  iamRoleStatements:

    -  Effect: "Allow"
       Action:
         - "sagemaker:InvokeEndpoint"
       Resource:
         - "*"
# Note: Having * for the resource is highly frowned upon, you should change
# this to your acutal account number and EndpointArn when you get the chance


functions:
  lambdaGET:
    handler: lambdaGET.main
    events:
      - http:
          method: GET
          path: /translate
          resp: json

Step 3: Deploy

1) In this step you will need to install Serverless Framework

2) Install AWS commmand line

3) Set up your AWS configure

4) Make sure your directories are setup correctly: (lambdaGET.py and servless.yml should be in the same folder)

```
-ServiceDirectory
 --- lambdaGET.py
 --- serverless.yml
```

5) Navigate to the ServiceDirectory folder and in the command line run:

sls deploy

Step 4: Test

Your API can now be invoked using browsers or programs such as Postman

The base URL for all your services API endpoint can be found in console inside API Gateway > Service (in your case 'Word-Level-Eng-Mar-NMT') > Dashboard

Almost there... Now that you have the base URL you need to add the extention we placed on our endpoint: /translate

Now you can place this entire URL in Postman and send the same payload you were using in the creation and testing conducted in your notebooks. In your case it will be the file test.jpg

TAAA DAAA!!

If your model was handling text or relatively small package size bits of information this would be the end of the story. Now because you are trying to pass an entire image it is possible that you will be over the size limit for API Gateway. In this case we will need to create an alternive plan that involves uploading the image to a public location (S3 bucket for example) and passing the URI via API. Then the lambda would have to retreive the image from the bucket and invoke the model. Still doable, just a little more complex.

I hope this helps.

Answer 2

The basic idea to deploy model to SageMaker is: 1) Containerize your model. 2) Publish your model to ECR repository and grant SageMaker necessary permissions. 3) Call CreateModel, CreateEndpointConfig and CreateEndpoint to deploy your model to SageMaker.

Per your notebook of training the model, you didn't use any SageMaker sdk to containerize your model automatically, so it is more complicated to start from scratch. You may consider use any of the following sample notebooks with keras to containerize your model first: https://github.com/awslabs/amazon-sagemaker-examples