Where is the best place to run a Python script in AWS?

Question

I am fairly new using AWS and I need to run a batch process (daily ) and store the data in a MySQL database. It would take approximately 30 minutes for extraction and transformation. As a side note, I need to run pandas.

I was reading that lambda functions are limited to 5 minutes. http://docs.aws.amazon.com/lambda/latest/dg/limits.html

I was thinking of using an EC2 micro instance with Ubuntu or an Elastic Beanstalk instance. And Amazon RDS for a MySQL DB.

Am I on the right path? Where is the best place to run my python code in AWS?

Answer 1

If you need to run these operations once or twice a day, you may want to look into the new AWS Batch service, which will let you run batch jobs without having to worry about DevOps.

If you have enough jobs to keep up the computer busy for most of the day, I believe the best solution is to run a Docker based solution, which will allow you to more easily manage your image and be able to test on your local host ( and more easily move to another cloud if you ever have to). AWS ECS makes this as easy as Elastic beanstalk.

I have my front end running on Elastic beanstalk and my back end workers running on ECS. In my case, my python workers are running on an infinite loop checking for SQS messages so the server can communicate with them via SQS messages. But I also have CloudWatch rules ( as cron jobs ) that wake up and call Lambda functions which then post SQS messages for the workers to handle. I can then have three worker containers running on the same t2.small ECS instance. If one of the workers ever fails, ECS will recreate one.

To summarize, use python on Docker on AWS ECS.

Answer 2

I'm using about 2-3 Ubuntu EC2 instances just to run Python scripts (via cronjob) for different purposes and using RDS for PostgresDB, all of them work well so far. So I think you should give EC2 and RDS a try. Good luck!