AWS Glue ETL job from AWS Redshift to S3 fails

Question

I am trying out AWS Glue service to ETL some data from redshift to S3. Crawler runs successfully and creates the meta table in data catalog, however when I run the ETL job ( generated by AWS ) it fails after around 20 minutes saying "Resource unavailable".

I cannot see AWS glue logs or error logs created in Cloudwatch. When I try to view them it says "Log stream not found. The log stream jr_xxxxxxxxxx could not be found. Check if it was correctly created and retry."

I would appreciate it if you could provide any guidance to resolve this issue.

Answer 1

So basically, the job you add to Glue will either run if there's not too much traffic in the region your Glue is. If there are no resources available, you need to either manually re-add the job again or you can also bind yourself to events from CloudWatch via SNS.

Also, there are parameters you can pass to the job like maximunRetry and timeout.

If you have a Ressource not available, it won't trigger a retry because the job did not fail, it just didn't even started. But if you set the timeout to let's say 60 minutes, it will trigger an error after that time, decrement your retry pool and re-launch the job.

Answer 2

The closest thing I see to Glue documentation on this is here:

If you encounter errors in AWS Glue, use the following solutions to help you find the source of the problems and fix them. Note The AWS Glue GitHub repository contains additional troubleshooting guidance in AWS Glue Frequently Asked Questions. Error: Resource Unavailable If AWS Glue returns a resource unavailable message, you can view error messages or logs to help you learn more about the issue. The following tasks describe general methods for troubleshooting. • A custom DNS configuration without reverse lookup can cause AWS Glue to fail. Check your DNS configuration. If you are using Amazon Route 53 or Microsoft Active Directory, make sure that there are forward and reverse lookups. For more information, see Setting Up DNS in Your VPC (p. 23). • For any connections and development endpoints that you use, check that your cluster has not run out of elastic network interfaces.