AWS SageMaker hosting multiple models on the same machine (ML compute instance)

Question

I am able to host the models developed in SageMaker by using the deploy functionality. Currently, I see that the different models that I have developed needs to deployed on different ML compute instances.

Is there a way to deploy all models on the same instance, using separate instances seems to be very expensive option. If it is possible to deploy multiple models on the same instance, will that create different endpoints for the models?

Answer 1

SageMaker is designed to solve deployment problems in scale, where you want to have thousands of model invocations per seconds. For such use cases, you want to have multiple tasks of the same model on each instance, and often multiple instances for the same model behind a load balancer and an auto scaling group to allow to scale up and down as needed.

If you don’t need such scale and having even a single instance for a single model is not economic for the request per second that you need to handle, you can take the models that were trained in SageMaker and host them yourself behind some serving framework such as MXNet serving (https://github.com/awslabs/mxnet-model-server ) or TensorFlow serving (https://www.tensorflow.org/serving/ ).

Please also note that you have control over the instance type that you are using for the hosting, and you can choose a smaller instance for smaller loads. Here is a list of the various instance types that you can choose from: https://aws.amazon.com/sagemaker/pricing/instance-types/

Answer 2

I believe this is a new feature introduced in AWS sagemaker, please refer below links which exactly does the same.

Yes, now in AWS sagemaker you can deploy multiple models in same ML instance.

In Below Link, https://github.com/awslabs/amazon-sagemaker-examples/blob/master/advanced_functionality/

You can find examples,

• multi_model_bring_your_own
• multi_model_sklearn_home_value
• multi_model_xgboost_home_value

Another link which explains multi-model XGboost in details. https://aws.amazon.com/blogs/machine-learning/save-on-inference-costs-by-using-amazon-sagemaker-multi-model-endpoints/

Hope this helps anyone who is looking to solve this issue in future.