Schedule to start an EC2 instance and run a python script within it

Question

I am trying to schedule my python script in AWS, however I don't want the instances to be running all the time. So, trying to automate the process of:

1. Start the EC2 instance on a specific time
2. Run the python script within it
3. Stop the EC2 instance once the job is completed.

I cannot run this script directly as a Lambda function because the script does some parallel processing which requires more RAM, so choosing a bigger AWS instance rather than writing it as a lambda function. Also, don't want this instance to be running all the time as it is expensive.

So far, I followed Automatic starting and stopping of AWS EC2 instances with Lambda and CloudWatch · matoski.com and created a Lambda function to start and stop the instance at specific time, however I couldn't find a way to run the python script once the instance is started.

Can anyone point me in the right direction?

Answer 1

MY application runs an instance @ 13:39 UST everyday and self shuts down after processing is complete. It uses below

1. A scheduled lambda function using cloud watch event rule

Cloud watch Event/rules config

2. The lambda trigger will start an instance (with hardcoded id)

import boto3
def lambda_handler(event, context):
    ec2 = boto3.client('ec2', region_name='ap-south-1')
    ec2.start_instances(InstanceIds=['i-xxxxxxx'])
    print('started your instances: ' + str('i-xxxxxx'))
    return

3. This triggers an instance which has a cron running to execute Python script

   @reboot python /home/Init.py

4. Once script completes, python job shuts down itself using below snippet

import boto.ec2
import boto.utils
import logging
logger=logging.getLogger()
def stop_ec2():
    conn = boto.ec2.connect_to_region("ap-south-1") # or your region
    # Get the current instance's id
    my_id = boto.utils.get_instance_metadata()['instance-id']
    logger.info(' stopping EC2 :'+str(my_id))
    conn.stop_instances(instance_ids=[my_id])