Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Is there a way to run aws glue crawler after job is finished?

For example I run ETL and new fields or columns may be added for target table. To detect table changes a crawler should be run but it has only manual or schedule run.

Can crawler be triggered after job is finished?

like image 388
Cherry Avatar asked Jan 11 '18 05:01

Cherry


People also ask

Why is my AWS Glue ETL job running for a long time?

Some common reasons why your AWS Glue jobs take a long time to complete are the following: Large datasets. Non-uniform distribution of data in the datasets. Uneven distribution of tasks across the executors.


1 Answers

import boto3
glue_client = boto3.client('glue', region_name='us-east-1')
glue_client.start_crawler(Name='name_of_crawler')

Copy this code snippet at the end of your code.

like image 157
Ashutosh Avatar answered Oct 21 '22 03:10

Ashutosh