Resource Unavailable Error in AWS Glue Job

Question

We are using AWS Glue service to convert json files to parquet. Intermittently Glue job is failing with Resource unavailable Error. Can anyone help us how to resolve the same?

Answer 1

I've also experienced the "Resource unavailable" exception a few times but it's on their side (service) and they are actively working to improve it.

I've opened a support ticket regarding this exception and they said:

Glue creates a new spark environment at the backend in order to run the jobs submitted depending on account ID, IAM role, subnetID and security groups. Running the Glue jobs under different IAM user may help to mitigate this issue. By doing this, Glue creates a new spark environment at the backend in order to run the jobs submitted by different IAM role or different accountID.

So if your account has several jobs running concurrently and the "Resource unavailable" error keeps appearing I would recommend you to create different IAM roles and splitting the jobs between them in order to mitigate this particular issue.