Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

How to I add a current timestamp (extra column) in the glue job so that the output data has an extra column

How to I add a current timestamp (extra column) in the glue job so that the output data has an extra column. In this case:

Schema Source Table: Col1, Col2

After Glue job.

Schema of Destination: Col1, Col2, Update_Date(Current Timestamp)

like image 818
Kishore Bharathy Avatar asked Nov 28 '22 22:11

Kishore Bharathy


1 Answers

We do the following and works great without converting toDF()

datasource0 = glueContext.create_dynamic_frame.from_catalog(...)

from datetime import datetime
def AddProcessedTime(r):
    r["jobProcessedDateTime"] = datetime.today() #timestamp of when we ran this.
    return r

mapped_dyF = Map.apply(frame = datasource0, f = AddProcessedTime)
like image 137
Sabilace Avatar answered Jan 03 '23 04:01

Sabilace