How to specify OVERWRITE to writeAsText in Apache Flink Streaming 0.10.0?

Question

I have a method in scala

counts.writeAsText(path_to_file)

It throws an exception when file already exists and suggests to specify File or directory already exists. Existing files and directories are not overwritten in NO_OVERWRITE mode. Use OVERWRITE mode to overwrite existing files and directories.. But I have not found a method in DataStream class that accepts org.apache.flink.core.fs.FileSystem.WriteMode. There is only a signature that accepts Long for milliseconds.

Answer 1

The Scala API is currently missing an overload with the respective WriteMode parameter. This is a bug and will be fixed.

For the meantime, you can always control the default overwrite behaviour of Flink via the fs.overwrite-files configuration parameter. If you set this parameter to fs.overwrite-files: true in your flink-conf.yaml, then writing to a file will always overwrite.

The status of the corresponding JIRA issue can be found here.

Answer 2

Just a note: this has now been fixed. You can call .writeAsText(filePath, org.apache.flink.core.fs.FileSystem.WriteMode.OVERWRITE)