Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Can we see or edit 'job bookmark' info in AWS glue or where it get stored?

I created a glue job and enabled the job bookmark, want to see the metadata stored by glue to keep track of processed files.

like image 790
niraj kumar Avatar asked Jul 11 '18 06:07

niraj kumar


1 Answers

Unfortunately AWS Glue job bookmarking details are not available for customers. As far as I know it is based on the job name, source file name and transformationContext strings passed to DynamicFrame's methods like getSinkWithFormat(), applyMapping(), getCatalogSource().

Besides that Job.init() and Job.commit() must be called at the beginning and end of a script respectively to make bookmarking working.

like image 54
Yuriy Bondaruk Avatar answered Oct 11 '22 21:10

Yuriy Bondaruk