Spark window function with synthetic timestamp?

Question

Say I have a datafile with records where each record has a timestamp, like this:

foo,bar,blaz,timestamp1
foo,flibble,baz,timestamp2
bleh,foo,gnarly,timestamp3
...

and I want to process this using Spark, in a way that requires using the window() function. Is there any way to read these records, and get each one into the DStream so that the timestamp that will be used by the window() function is provided by my code explicitly (based on parsing the timestamp field in the input records in this case)?

Answer 1

No, the default Spark processing is based on the system time. And if you want to build the window using the event time. I suggest you use up "updateStateByKey" function to handle the logic inside the update function.